QF-Test - The Manual

QF-Test - The Manual
Version 4.2.2
Gregor Schmid
Quality First Software GmbH1
c 1999-2019 Quality First Software GmbH
Copyright March 8, 2018
1 http://www.qfs.de
Preface
As the name indicates, Quality First Software GmbH is dedicated to quality assurance
for the software development process. Our contribution to this area is the product QFTest the manual of which you are currently reading.
QF-Test is a professional tool for automating functional tests for Java or web applications
with a graphical user interface. Depending on size and structure of a company the
ungrateful task of testing sometimes falls to a QA department or team, sometimes to
the developers and sometimes to the unlucky customer. Users of QF-Test are therefore
usually developers or testers with varying knowledge about software development and
testing in general and Java or web GUIs in particular.
This manual is the primary source of information for QF-Test. We have tried to explain
things in a way that is understandable for all users, independent of their technical knowledge, yet provide a complete and concise reference. In case of Java specific questions,
testers may fare best by contacting their developers who will surely be able to assist.
Initially QF-Test did only support Java Swing GUIs. With version 2.0 support for
Eclipse/SWT was added and web support with version 3.0. Parts of QF-Test and this
manual owe to this history and most things are explained from the perspective of
testing a Swing GUI. In most cases the concepts are universal apply similarly to all
GUIs. Where things differ, specific notes explain the particularities of a web or SWT
GUI.
How to use this manual
This manual is available in HTML and PDF versions. The HTML version is split across
multiple files for better navigation and to avoid excessive memory consumption of the
web browser. Due to extensive cross-linking, the HTML document is the preferred version for online viewing, while the PDF version is better suited for printing. Searching
should also be done in the PDF version since the splitting of the HTML files prevents
searches through the whole manual.
The
PDF
version
of
the
manual
is
located
at
ii
qftest-4.2.2/doc/manual_en.pdf, the entry page of the HTML version is at
qftest-4.2.2/doc/manual/en/manual.html.
A web-browser for the HTML manual can be started directly from QF-Test. The
Help→Manual... menu item will take you to the entry page of the manual and
Help→News... will bring up the section documenting the latest changes.
Context-sensitive help is also available for all kinds of tree-nodes, attributes and
configurable options by clicking with the right mouse button and selecting What’s this?
from the popup menu. This may not work if the system’s browser is not accessible from
QF-Test.
The manual consists of three parts which are kept in one document for technical reasons
(it simplifies cross-linking and index generation). They may be split in a later version.
These parts are
(2)
User manual
This part explains how to install and run QF-Test and how to work with its user
interface. It shows how to create and organize tests, then continues with more
advanced material. To avoid duplication of text, the user manual often refers to
the reference manual for detailed explanation. We recommend that you follow
these links.
Reference manual
(316)
This is a complete reference that covers all configurable options, all parts of a
test-suite , etc. When looking for specific information, this is the place to go. The
reference manual also serves as the source for context-sensitive help.
(671)
Technical reference
The part about technical details contains in-depth and background information
about miscellaneous topics as well as a comprehensive API reference for the
scripting interface. Beginners will rarely need to take a look at this part, but for the
advanced user and the technically interested it is a valuable resource.
A learning-by-doing tutorial is also available in HTML and PDF versions. The HTML
version, which is directly accessible from the Help→Tutorial (HTML)... menu item, is
located at qftest-4.2.2/doc/tutorial/en/tutorial.html. The PDF version
is to be found at qftest-4.2.2/doc/tutorial/tutorial_en.pdf and is also directly accessible through Help→Tutorial (PDF)... .
The following notations are used throughout the manual:
• Menu→Submenu represents a menu or menu item.
iii
• Modifier-Key stands for a keystroke, where the modifier is one (or a combination)
of Shift , Control or Alt .
• Monospaced font is used for names of directories and files, user input and program output.
• In order to transfer at least part of the convenience of cross-linking to the paper
(i)
version, references in the PDF version are underlined and show the target page
number in small braces.
iv
Contents
I
User manual
1 Installation and startup
1.1
1
2
System requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3
1.1.1
Prime precondition . . . . . . . . . . . . . . . . . . . . . . . . . .
3
1.1.2
Supported technology versions . . . . . . . . . . . . . . . . . . .
3
Windows Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5
1.2.1
Installing via the Windows setup file QF-Test-4.2.2.exe . . .
5
1.2.2
Unpacking the self-extracting archive QF-Test-4.2.2-sfx.exe 6
1.2.3
Completing the installation and configuring Java . . . . . . . . .
6
1.3
Linux/Unix Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7
1.4
macOS Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
8
1.5
The license file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
8
1.6
The configuration files . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
9
1.7
Starting QF-Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.8
Firewall Security Warning . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.2
2 The user interface
12
2.1
The test-suite . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.2
Basic editing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.2.1
Navigating the tree . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.2.2
Insertion marker . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.2.3
Moving nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.2.4
Transforming nodes . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.2.5
Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
CONTENTS
2.3
v
2.2.6
Packing and Unpacking . . . . . . . . . . . . . . . . . . . . . . . 18
2.2.7
Sorting Nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Advanced editing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.3.1
Searching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.3.2
Replacing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.3.3
Complex searches and replace operations . . . . . . . . . . . . . 25
2.3.4
Multiple views . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3 Quickstart your application
3.1
27
Quickstart Wizard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3.1.1
Setup sequence creation . . . . . . . . . . . . . . . . . . . . . . 28
3.1.2
Executing the setup sequence . . . . . . . . . . . . . . . . . . . 29
3.1.3
Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
3.2
Program output and the Clients menu . . . . . . . . . . . . . . . . . . . . 31
3.3
An application started indirectly from an already connected SUT . . . . . 32
4 Capture and replay
33
4.1
Recording sequences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
4.2
Running tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
4.3
Recording checks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
4.4
Recording components . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
4.5
Recording of HTTP Requests (GET/POST) . . . . . . . . . . . . . . . . . 38
5 Components
40
5.1
GUI component hierarchy . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
5.2
Components in QF-Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
5.3
QF-Test IDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
5.4
Component class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
5.4.1
Generic classes . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
5.4.2
Class hierarchy for web applications . . . . . . . . . . . . . . . . 45
5.4.3
Settings for class recording . . . . . . . . . . . . . . . . . . . . . 46
5.5
Component inspection . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
5.6
The importance of naming components . . . . . . . . . . . . . . . . . . . 47
CONTENTS
vi
5.7
Considerations for setting names . . . . . . . . . . . . . . . . . . . . . . 49
5.8
Updating Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
5.9
Troubleshooting component recognition problems . . . . . . . . . . . . . 52
5.10 Accessing hidden fields on a web page . . . . . . . . . . . . . . . . . . . 54
6 Sub-items of complex GUI components
55
6.1
Item nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
6.2
The QF-Test ID of an Item . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
6.3
Special QF-Test ID-syntax for direct element access . . . . . . . . . . . . 57
6.4
6.3.1
Accessing simple subcomponents . . . . . . . . . . . . . . . . . 57
6.3.2
Addressing multi-level sub-items . . . . . . . . . . . . . . . . . . 58
Activating item recording for web tables . . . . . . . . . . . . . . . . . . . 60
7 Variables
62
7.1
Variable lookup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
7.2
Defining variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
7.3
Variable example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
7.4
Fetching data from the GUI . . . . . . . . . . . . . . . . . . . . . . . . . . 66
7.5
External data and special groups . . . . . . . . . . . . . . . . . . . . . . 67
7.6
Expressions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
7.7
Immediate and lazy binding . . . . . . . . . . . . . . . . . . . . . . . . . 72
8 Problem analysis and debugging
8.1
8.2
74
The run-log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
8.1.1
Error states . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
8.1.2
Navigating the run-log tree . . . . . . . . . . . . . . . . . . . . . 77
8.1.3
Accepting values of failed checks as good . . . . . . . . . . . . . 78
8.1.4
Split run-logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
8.1.5
Run-log options . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
8.1.6
Creating a test-suite from the run-log . . . . . . . . . . . . . . . . 80
8.1.7
Merging run-logs . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
The debugger . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
8.2.1
Entering the debugger . . . . . . . . . . . . . . . . . . . . . . . . 81
CONTENTS
vii
8.2.2
The debugger window . . . . . . . . . . . . . . . . . . . . . . . . 82
8.2.3
Debugger commands . . . . . . . . . . . . . . . . . . . . . . . . 83
8.2.4
Manipulating breakpoints . . . . . . . . . . . . . . . . . . . . . . 84
9 Organizing the test-suite
85
9.1
Test-suite structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
9.2
Test-set and Test-case nodes . . . . . . . . . . . . . . . . . . . . . . . . . . 87
9.2.1
Test management with Test-set and Test-case nodes . . . . . . . . 87
9.2.2
Concept . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
9.2.3
Variables and special attributes . . . . . . . . . . . . . . . . . . . 88
9.3
Sequence and Test-step nodes . . . . . . . . . . . . . . . . . . . . . . . . . 89
9.4
Setup and Cleanup nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
9.5
Procedures and Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
9.6
9.7
9.5.1
Local Procedures and Packages . . . . . . . . . . . . . . . . . . . . 92
9.5.2
Relative Procedures . . . . . . . . . . . . . . . . . . . . . . . . . . 92
9.5.3
Inserting Procedure call nodes . . . . . . . . . . . . . . . . . . . . 93
9.5.4
Parameterizing nodes . . . . . . . . . . . . . . . . . . . . . . . . 93
9.5.5
Transforming a Sequence into a Procedure . . . . . . . . . . . . . . 93
Dependency nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
9.6.1
Concept . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
9.6.2
Usage of Dependencies . . . . . . . . . . . . . . . . . . . . . . . . 95
9.6.3
Dependency execution and Dependency stack . . . . . . . . . . . . 96
9.6.4
Characteristic variables . . . . . . . . . . . . . . . . . . . . . . . . . 100
9.6.5
Forced cleanup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
9.6.6
Rolling back Dependencies . . . . . . . . . . . . . . . . . . . . . . 103
9.6.7
Error escalation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
9.6.8
Error handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
9.6.9
Name spaces for Dependencies . . . . . . . . . . . . . . . . . . 106
Documenting test-suites . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
10 Projects
110
11 The standard library
112
CONTENTS
viii
12 Scripting
114
12.1 General . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
12.2 The run-context rc . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
12.2.1 Logging messages . . . . . . . . . . . . . . . . . . . . . . . . . . 117
12.2.2 Performing checks . . . . . . . . . . . . . . . . . . . . . . . . . . 117
12.2.3 Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
12.2.4 Accessing the SUT’s GUI components . . . . . . . . . . . . . . . 121
12.2.5 Calling Procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
12.2.6 Setting options . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
12.2.7 Override components . . . . . . . . . . . . . . . . . . . . . . . . 123
12.3 Fundamentals of the Jython integration . . . . . . . . . . . . . . . . . . . 124
12.3.1 Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
12.3.2 Post-mortem debugging of Jython scripts . . . . . . . . . . . . . 125
12.3.3 Boolean type . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
12.3.4 Character encodings . . . . . . . . . . . . . . . . . . . . . . . . . 126
12.3.5 Getting the name of a Java class . . . . . . . . . . . . . . . . . . 127
12.3.6 A complex example
. . . . . . . . . . . . . . . . . . . . . . . . . 128
12.4 Scripting with Groovy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
12.4.1 Groovy packages . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
12.5 Scripting with JavaScript . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
12.5.1 JavaScript imports . . . . . . . . . . . . . . . . . . . . . . . . . . 135
12.5.2 NPM modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
12.5.3 Print statements . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
12.5.4 Execution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
13 Unit Tests
138
13.1 Java Classes as the Source for the Unit Test . . . . . . . . . . . . . . . . 138
13.2 Basics of the Test Scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
13.2.1 Groovy Unit Tests
. . . . . . . . . . . . . . . . . . . . . . . . . . 141
13.2.2 Jython Unit Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
13.2.3 JavaScript Unit-Test . . . . . . . . . . . . . . . . . . . . . . . . . 142
13.3 Injections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
CONTENTS
ix
13.3.1 Component-Injections . . . . . . . . . . . . . . . . . . . . . . . . 143
13.3.2 WebDriver-Injections . . . . . . . . . . . . . . . . . . . . . . . . . 145
13.4 Unit Tests in Report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
14 Web testing
148
14.1 Web testing using QF-Driver connection mode . . . . . . . . . . . . . . . 148
14.2 Website Testing using WebDriver . . . . . . . . . . . . . . . . . . . . . . 148
14.2.1 WebDriver in general . . . . . . . . . . . . . . . . . . . . . . . . . 149
14.2.2 WebDriver with Microsoft Edge . . . . . . . . . . . . . . . . . . . 150
14.2.3 WebDriver with Safari . . . . . . . . . . . . . . . . . . . . . . . . 150
14.2.4 Known limitations of the WebDriver mode . . . . . . . . . . . . . 151
14.3 Web-Tests without visible Window . . . . . . . . . . . . . . . . . . . . . . 151
15 Testing PDF documents
152
15.1 PDF Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
15.1.1 PDF Client start
. . . . . . . . . . . . . . . . . . . . . . . . . . . 152
15.1.2 PDF Client window . . . . . . . . . . . . . . . . . . . . . . . . . . 152
15.2 PDF events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154
15.2.1 Open a PDF document . . . . . . . . . . . . . . . . . . . . . . . 154
15.2.2 Switch page . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154
15.3 Checks for PDF components . . . . . . . . . . . . . . . . . . . . . . . . . 154
15.3.1 Check text . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
15.3.2 Check image . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
15.3.3 ’Check Font’ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
15.3.4 ’Check Font size’ . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
15.4 PDF component types . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
15.5 PDF component recognition . . . . . . . . . . . . . . . . . . . . . . . . . 158
16 Data-driven testing
159
16.1 Data-driver examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
16.2 General use of Data drivers . . . . . . . . . . . . . . . . . . . . . . . . . . 163
16.3 Examples for Data drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
16.4 Advanced use . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
CONTENTS
x
17 Reports and test documentation
167
17.1 Reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
17.1.1 Report concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
17.1.2 Report contents
. . . . . . . . . . . . . . . . . . . . . . . . . . . 169
17.1.3 Creating reports . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
17.2 Testdoc documentation for Test-sets and Test-cases . . . . . . . . . . . . . 172
17.3 Pkgdoc documentation for Packages, Procedures and Dependencies . . . . . 173
18 Test execution
176
18.1 Test execution in batch mode . . . . . . . . . . . . . . . . . . . . . . . . . 176
18.1.1 Command line usage . . . . . . . . . . . . . . . . . . . . . . . . 177
18.1.2 Windows batch script . . . . . . . . . . . . . . . . . . . . . . . . 179
18.1.3 Groovy
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
18.2 Executing tests in daemon mode
. . . . . . . . . . . . . . . . . . . . . . 182
18.2.1 Launching the daemon . . . . . . . . . . . . . . . . . . . . . . . . 182
18.2.2 Controlling a daemon from QF-Test’s command line . . . . . . . 183
18.2.3 Controlling a daemon with the daemon API . . . . . . . . . . . . 184
18.3 Re-execution of nodes (Re-run) . . . . . . . . . . . . . . . . . . . . . . . 187
18.3.1 Triggering re-run from a run-log . . . . . . . . . . . . . . . . . . . 187
18.3.2 Re-running failing nodes immediately
19 Distributed test development
. . . . . . . . . . . . . . . 190
193
19.1 Referencing nodes in another test-suite . . . . . . . . . . . . . . . . . . . 193
19.2 Managing Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
19.3 Merging test-suites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196
19.3.1 Importing Components . . . . . . . . . . . . . . . . . . . . . . . . 196
19.3.2 Importing Procedures and Testcases . . . . . . . . . . . . . . . 196
19.4 Strategies for distributed development . . . . . . . . . . . . . . . . . . . . 196
19.5 Static validation of test-suites
. . . . . . . . . . . . . . . . . . . . . . . . 198
19.5.1 Avoiding invalid references . . . . . . . . . . . . . . . . . . . . . 199
19.5.2 Unused procedures . . . . . . . . . . . . . . . . . . . . . . . . . 201
20 Automated Creation of Basic Procedures
202
CONTENTS
xi
20.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202
20.2 How to use the Procedure Builder . . . . . . . . . . . . . . . . . . . . . . 202
20.3 Configuration of the Procedure Builder . . . . . . . . . . . . . . . . . . . 204
20.3.1 The Procedure Builder definition file . . . . . . . . . . . . . . . . 205
21 Interaction with Test Management Tools
207
21.1 HP ALM - Quality Center . . . . . . . . . . . . . . . . . . . . . . . . . . . 207
21.1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207
21.1.2 Step-by-step integration guide . . . . . . . . . . . . . . . . . . . . 209
21.1.3 Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219
21.2 Imbus TestBench . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
21.2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
21.2.2 Creating QF-Test template from interactions . . . . . . . . . . . . 221
21.2.3 Importing test-execution results . . . . . . . . . . . . . . . . . . . 221
21.3 QMetry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222
21.3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222
21.3.2 Sample Configuration . . . . . . . . . . . . . . . . . . . . . . . . 224
21.4 Klaros . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225
21.4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225
21.4.2 Importing QF-Test results into Klaros . . . . . . . . . . . . . . . . 225
21.5 SQS-TEST/Professional . . . . . . . . . . . . . . . . . . . . . . . . . . . 226
21.5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226
21.5.2 Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226
21.6 TestLink . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226
21.6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226
21.6.2 Generating template test-suites for QF-Test from test-cases . . . 227
21.6.3 Execution of test-cases . . . . . . . . . . . . . . . . . . . . . . . 229
21.6.4 Importing QF-Test results into TestLink . . . . . . . . . . . . . . . 229
22 Integration with Development Tools
231
22.1 Eclipse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
22.1.1 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
CONTENTS
xii
22.1.2 Configuration of the test nodes . . . . . . . . . . . . . . . . . . . 232
22.2 Ant . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235
22.3 Maven . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236
22.4 Jenkins/Hudson . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238
22.4.1 Install and start Jenkins . . . . . . . . . . . . . . . . . . . . . . . 238
22.4.2 Requirements for GUI tests . . . . . . . . . . . . . . . . . . . . . 239
22.4.3 Install QF-Test Plugin . . . . . . . . . . . . . . . . . . . . . . . . 240
22.4.4 Configure a job for QF-Test . . . . . . . . . . . . . . . . . . . . . 241
23 Keyword-driven testing with QF-Test
245
23.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245
23.2 Simple Keyword-driven testing with QF-Test . . . . . . . . . . . . . . . . 247
23.2.1 Business-related Procedures . . . . . . . . . . . . . . . . . . . . 248
23.2.2 Atomic component-oriented procedures . . . . . . . . . . . . . . 251
23.3 Keyword-driven testing using dynamic or generic components . . . . . . 252
23.4 Behavior-driven testing (BDT) . . . . . . . . . . . . . . . . . . . . . . . . 255
23.4.1 Behavior-Driven Testing (BDT) from technical perspective . . . . 256
23.4.2 Behavior-Driven Testing (BDT) from business perspective . . . . 258
23.5 Scenario files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259
23.6 Custom test-case description . . . . . . . . . . . . . . . . . . . . . . . . . 262
23.7 Adapting to your software . . . . . . . . . . . . . . . . . . . . . . . . . . . 263
24 Performing GUI-based load tests
265
24.1 Background and comparison with other techniques . . . . . . . . . . . . 265
24.2 Load testing with QF-Test . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
24.2.1 Provision of test-systems . . . . . . . . . . . . . . . . . . . . . . 270
24.2.2 Conception of the test-run . . . . . . . . . . . . . . . . . . . . . . 270
24.2.3 Preparing test-systems prior to the test-run . . . . . . . . . . . . 271
24.2.4 Test execution
. . . . . . . . . . . . . . . . . . . . . . . . . . . . 271
24.2.5 Evaluating results
. . . . . . . . . . . . . . . . . . . . . . . . . . 272
24.3 Hints on test execution . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272
24.3.1 Synchronization . . . . . . . . . . . . . . . . . . . . . . . . . . . 272
CONTENTS
xiii
24.3.2 Measuring end-to-end response times . . . . . . . . . . . . . . . 274
24.4 Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274
25 Executing Manual Tests in QF-Test
276
25.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276
25.2 Step-by-step Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
25.3 Structure of the Excel file . . . . . . . . . . . . . . . . . . . . . . . . . . . 278
25.4 The ManualTestRunner test-suite . . . . . . . . . . . . . . . . . . . . . . 279
25.5 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 280
II
Best Practices
281
26 Introduction
282
27 How to start a testing project
283
27.1 Infrastructure and testing environment
. . . . . . . . . . . . . . . . . . . 283
27.2 Location of files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285
27.2.1 Network installation . . . . . . . . . . . . . . . . . . . . . . . . . 286
27.3 Component Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . 286
28 Organizing test-suites
288
28.1 Organizing tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 288
28.2 Modularization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289
28.3 Parameterization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 290
28.4 Working in multiple test-suites . . . . . . . . . . . . . . . . . . . . . . . . 290
28.5 Roles and responsibilities . . . . . . . . . . . . . . . . . . . . . . . . . . . 292
28.6 Managing components at different levels . . . . . . . . . . . . . . . . . . 294
28.7 Reverse includes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294
29 Efficient working techniques
295
29.1 Using QF-Test projects . . . . . . . . . . . . . . . . . . . . . . . . . . . . 295
29.2 Creating test-suites from scratch . . . . . . . . . . . . . . . . . . . . . . . 295
29.3 The standard library qfs.qft . . . . . . . . . . . . . . . . . . . . . . . . . . 296
CONTENTS
xiv
29.4 Component storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 296
29.5 Extending test-suites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297
29.6 Working in the script editor . . . . . . . . . . . . . . . . . . . . . . . . . . 298
30 How to achieve robust component recognition
299
30.1 Using the default configuration . . . . . . . . . . . . . . . . . . . . . . . . 299
30.2 Using the ’Name overrides everything’ mode . . . . . . . . . . . . . . . . 301
30.3 Using regular expressions for working with dynamic window titles . . . . 303
30.4 Influencing the names by implementing a NameResolver . . . . . . . . . 304
30.5 Handling classes of components . . . . . . . . . . . . . . . . . . . . . . . 305
30.5.1 Handling class changes due to re-factoring by using ’Record
system-classes only’ . . . . . . . . . . . . . . . . . . . . . . . . . 306
30.6 Avoiding recording every component or using generic components
30.7
. . . 307
Switching component recognition settings dynamically . . . . . . . . . . 308
31 Hints on setting up test-systems
309
31.1 Using the task scheduler . . . . . . . . . . . . . . . . . . . . . . . . . . . 309
31.2 Remote access to Windows systems . . . . . . . . . . . . . . . . . . . . 310
31.3 Automated logon on Windows systems . . . . . . . . . . . . . . . . . . . 311
31.4 Test execution on Unix . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312
32 Test execution
313
32.1 Dependencies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313
32.2 Timeout vs. delay . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313
32.3 What to do if the run-log contains an error . . . . . . . . . . . . . . . . . 314
III
Reference manual
33 Options
315
316
33.1 General options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 318
33.1.1 Projects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 320
33.1.2 Editing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 321
33.1.3 Bookmarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325
CONTENTS
xv
33.1.4 External tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325
33.1.5 Backup files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 328
33.1.6 Library path . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331
33.1.7 License . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 332
33.2 Recording options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333
33.2.1 Events to record . . . . . . . . . . . . . . . . . . . . . . . . . . . 334
33.2.2 Events to pack . . . . . . . . . . . . . . . . . . . . . . . . . . . . 336
33.2.3 Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 340
33.2.4 Recording sub-items . . . . . . . . . . . . . . . . . . . . . . . . . 346
33.2.5 Recording procedures . . . . . . . . . . . . . . . . . . . . . . . . 348
33.3 Replay options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 349
33.3.1 Client options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 352
33.3.2 Terminal options . . . . . . . . . . . . . . . . . . . . . . . . . . . 356
33.3.3 Event handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . 359
33.3.4 Component recognition . . . . . . . . . . . . . . . . . . . . . . . 364
33.3.5 Delays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 368
33.3.6 Timeouts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 370
33.3.7 Backward compatibility . . . . . . . . . . . . . . . . . . . . . . . . 374
33.4 Web options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 375
33.4.1 HTTP Requests . . . . . . . . . . . . . . . . . . . . . . . . . . . 379
33.4.2 Backward compatibility . . . . . . . . . . . . . . . . . . . . . . . . 381
33.5 Debugger options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 381
33.6 Run-log options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 383
33.6.1 General run-log options . . . . . . . . . . . . . . . . . . . . . . . 384
33.6.2 Options determining run-log content . . . . . . . . . . . . . . . . 389
33.6.3 Options for mapping between directories with test-suites . . . . . 392
33.7 Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 393
34 Elements of a test-suite
395
34.1 The test-suite and its structure . . . . . . . . . . . . . . . . . . . . . . . . 395
34.1.1 Test-suite . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 395
34.2 Test and Sequence nodes . . . . . . . . . . . . . . . . . . . . . . . . . . 398
CONTENTS
xvi
34.2.1 Test-case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 398
34.2.2 Test-set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 405
34.2.3 Test call . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 411
34.2.4 Sequence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 415
34.2.5 Test-step . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 417
34.2.6 Sequence with time limit . . . . . . . . . . . . . . . . . . . . . . . . 422
34.2.7 Extras . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 425
34.3 Dependencies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 426
34.3.1 Dependency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 427
34.3.2 Dependency reference . . . . . . . . . . . . . . . . . . . . . . . . . 430
34.3.3 Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 433
34.3.4 Cleanup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 435
34.3.5 Error handler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 438
34.4 Data driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 440
34.4.1 Data driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 440
34.4.2 Data table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 443
34.4.3 Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 446
34.4.4 Excel data file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 452
34.4.5 CSV data file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 456
34.4.6 Data loop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 460
34.5 Procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 462
34.5.1 Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 463
34.5.2 Procedure call . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 466
34.5.3 Return . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 469
34.5.4 Package . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 470
34.5.5 Procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 472
34.6 Control structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 473
34.6.1 Loop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 473
34.6.2 While . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 476
34.6.3 Break . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 479
34.6.4 If . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 481
CONTENTS
xvii
34.6.5 Elseif . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 484
34.6.6 Else . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 488
34.6.7 Try . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 490
34.6.8 Catch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 493
34.6.9 Finally . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 496
34.6.10 Throw . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 499
34.6.11 Rethrow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 500
34.6.12 Server script . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 502
34.6.13 SUT script . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 504
34.7 Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 507
34.7.1 Start Java SUT client . . . . . . . . . . . . . . . . . . . . . . . . . . 507
34.7.2 Start SUT client . . . . . . . . . . . . . . . . . . . . . . . . . . . . 511
34.7.3 Start process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 514
34.7.4 Start web engine . . . . . . . . . . . . . . . . . . . . . . . . . . . . 517
34.7.5 Open browser window . . . . . . . . . . . . . . . . . . . . . . . . . 521
34.7.6 Execute shell command . . . . . . . . . . . . . . . . . . . . . . . . 524
34.7.7 Start PDF client . . . . . . . . . . . . . . . . . . . . . . . . . . . . 526
34.7.8 Wait for client to connect . . . . . . . . . . . . . . . . . . . . . . . . 529
34.7.9 Stop client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 532
34.7.10 Wait for process to terminate . . . . . . . . . . . . . . . . . . . . . . 534
34.8 Events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 537
34.8.1 Mouse event . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 538
34.8.2 Key event . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 541
34.8.3 Text input . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 545
34.8.4 Window event . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 548
34.8.5 Component event . . . . . . . . . . . . . . . . . . . . . . . . . . . . 550
34.8.6 Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 553
34.8.7 File selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 557
34.9 Checks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 560
34.9.1 Check text . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 561
34.9.2 Boolean check . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 566
CONTENTS
xviii
34.9.3 Check items . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 572
34.9.4 Check selectable items . . . . . . . . . . . . . . . . . . . . . . . . . 576
34.9.5 Check image . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 581
34.9.6 Check geometry . . . . . . . . . . . . . . . . . . . . . . . . . . . . 586
34.10 Queries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 590
34.10.1 Fetch text . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 591
34.10.2 Fetch index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 594
34.10.3 Fetch geometry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 597
34.11 Miscellaneous . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 601
34.11.1 Set variable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 601
34.11.2 Wait for component to appear . . . . . . . . . . . . . . . . . . . . . 604
34.11.3 Wait for document to load . . . . . . . . . . . . . . . . . . . . . . . 608
34.11.4 Wait for download to finish . . . . . . . . . . . . . . . . . . . . . . . 612
34.11.5 Load resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 615
34.11.6 Load properties
. . . . . . . . . . . . . . . . . . . . . . . . . . . . 618
34.11.7 Unit test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 621
34.12 HTTP Requests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 626
34.12.1 Server HTTP request . . . . . . . . . . . . . . . . . . . . . . . . . . 626
34.12.2 Browser HTTP request . . . . . . . . . . . . . . . . . . . . . . . . . 631
34.13 Windows, Components and Items . . . . . . . . . . . . . . . . . . . . . . 634
34.13.1 Window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 634
34.13.2 Web page . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 640
34.13.3 Component . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 644
34.13.4 Item . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 650
34.13.5 Window group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 653
34.13.6 Component group . . . . . . . . . . . . . . . . . . . . . . . . . . . 654
34.13.7 Windows and components . . . . . . . . . . . . . . . . . . . . . . . 655
34.14 Deprecated nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 656
34.14.1 Test
35 Exceptions
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 656
662
CONTENTS
IV
xix
Technical reference
670
36 Command line arguments and exit codes
671
36.1 Call syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 671
36.2 Command line arguments . . . . . . . . . . . . . . . . . . . . . . . . . . 676
36.2.1 Arguments for the starter script . . . . . . . . . . . . . . . . . . . 676
36.2.2 Arguments for the Java VM . . . . . . . . . . . . . . . . . . . . . 677
36.2.3 Arguments for QF-Test . . . . . . . . . . . . . . . . . . . . . . . . 677
36.2.4 Placeholders in the filename parameter for run-log and report . . 694
36.3 Exit codes for QF-Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . 695
37 GUI engines
697
38 Running an application from QF-Test
699
38.1 Various methods to start the SUT . . . . . . . . . . . . . . . . . . . . . . 699
38.1.1 A standalone script or executable file . . . . . . . . . . . . . . . . 700
38.1.2 An application launched through Java WebStart . . . . . . . . . . 701
38.1.3 An applet in a web browser . . . . . . . . . . . . . . . . . . . . . 702
38.1.4 An application started with java -jar <archive> . . . . . . . 704
38.1.5 An application started with java -classpath <classpath>
<class> . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 706
38.1.6 A web application in a browser . . . . . . . . . . . . . . . . . . . 707
38.1.7 Opening a PDF Document . . . . . . . . . . . . . . . . . . . . . . 709
39 JDK and SWT instrumentation
710
39.1 JDK instrumentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 710
39.2 JDK instrumentation background
. . . . . . . . . . . . . . . . . . . . . . 713
39.2.1 Modifications to the JDK . . . . . . . . . . . . . . . . . . . . . . . 713
39.2.2 Effects of the instrumentation . . . . . . . . . . . . . . . . . . . . 714
39.3 SWT instrumentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 714
39.3.1 Preparation for manual SWT instrumentation . . . . . . . . . . . 715
39.3.2 Manual SWT instrumentation for eclipse based applications . . . 715
39.3.3 Manual instrumentation for standalone SWT applications . . . . 716
CONTENTS
xx
40 Technical details about miscellaneous issues
717
40.1 Drag&Drop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 717
40.2 Component recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . 718
40.3 Timing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 720
40.4 Regular expressions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 720
40.5 Line breaks under Unix and Windows . . . . . . . . . . . . . . . . . . . . 722
40.6 Quoting and escaping special characters . . . . . . . . . . . . . . . . . . 722
40.7 Include file resolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 723
41 Scripting (Jython, Groovy and JavaScript)
726
41.1 Module load-path . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 726
41.2 The plugin directory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 727
41.3 The package cache (Jython) . . . . . . . . . . . . . . . . . . . . . . . . . 727
41.4 Initialization (Jython) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 728
41.5 Namespace environment for script execution (Jython) . . . . . . . . . . . 728
41.6 Run-context API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 729
41.7 The qf module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 746
41.8 Image API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 749
41.8.1 The ImageWrapper class
. . . . . . . . . . . . . . . . . . . . . 749
41.9 Exception handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 752
41.10 Debugging scripts (Jython) . . . . . . . . . . . . . . . . . . . . . . . . . . 752
42 Special features for web applications
754
42.1 Pseudo DOM API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 754
42.1.1 The abstract Node class . . . . . . . . . . . . . . . . . . . . . . . 754
42.1.2 The DocumentNode class
. . . . . . . . . . . . . . . . . . . . . 761
42.1.3 The FrameNode class . . . . . . . . . . . . . . . . . . . . . . . . 764
42.1.4 The DomNode class . . . . . . . . . . . . . . . . . . . . . . . . . 764
42.1.5 The DialogNode class . . . . . . . . . . . . . . . . . . . . . . . 766
42.2 WebDriver SUT API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 766
42.2.1 The WebDriverConnection class . . . . . . . . . . . . . . . . 767
42.3 Special support for various AJAX toolkits . . . . . . . . . . . . . . . . . . 769
CONTENTS
xxi
42.3.1 AJAX resolver concepts . . . . . . . . . . . . . . . . . . . . . . . 770
42.3.2 Supported AJAX toolkits . . . . . . . . . . . . . . . . . . . . . . . 771
42.3.3 Setting unique IDs . . . . . . . . . . . . . . . . . . . . . . . . . . 777
42.3.4 Using older versions of AJAX resolvers
. . . . . . . . . . . . . . 779
42.4 Customizing QF-Test for your AJAX toolkit with CustomWebResolver . . 783
42.4.1 Determining specifics of your web page . . . . . . . . . . . . . . 784
42.4.2 Example for WebCarConfigurator . . . . . . . . . . . . . . . . . . 786
43 Extension APIs
800
43.1 The resolvers module . . . . . . . . . . . . . . . . . . . . . . . . . . . 800
43.1.1 Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 800
43.1.2 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 802
43.1.3 addResolver . . . . . . . . . . . . . . . . . . . . . . . . . . . . 804
43.1.4 removeResolver . . . . . . . . . . . . . . . . . . . . . . . . . . 806
43.1.5 listNames . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 807
43.1.6 The NameResolver Interface . . . . . . . . . . . . . . . . . . . 807
43.1.7 The ClassNameResolver Interface . . . . . . . . . . . . . . . 809
43.1.8 The GenericClassNameResolver Interface . . . . . . . . . . 810
43.1.9 The FeatureResolver Interface . . . . . . . . . . . . . . . . . 811
43.1.10 The ExtraFeatureResolver Interface . . . . . . . . . . . . . 812
43.1.11 The ItemNameResolver Interface . . . . . . . . . . . . . . . . 816
43.1.12 The ItemValueResolver Interface . . . . . . . . . . . . . . . 817
43.1.13 The TreeTableResolver Interface . . . . . . . . . . . . . . . . 818
43.1.14 The InterestingParentResolver Interface . . . . . . . . . 820
43.1.15 The TooltipResolver Interface . . . . . . . . . . . . . . . . . 821
43.1.16 The IdResolver interface . . . . . . . . . . . . . . . . . . . . . 821
43.1.17 The EnabledResolver Interface . . . . . . . . . . . . . . . . . 823
43.1.18 The VisibilityResolver Interface
. . . . . . . . . . . . . . 824
43.1.19 The MainTextResolver Interface . . . . . . . . . . . . . . . . 825
43.1.20 The BusyPaneResolver Interfaces . . . . . . . . . . . . . . . 826
43.1.21 The GlassPaneResolver Interfaces . . . . . . . . . . . . . . . 826
43.1.22 The EventSynchronizer Interface . . . . . . . . . . . . . . . 827
CONTENTS
xxii
43.1.23 The BusyApplicationDetector Interface . . . . . . . . . . . 828
43.1.24 Matcher . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 828
43.2 The webResolvers module . . . . . . . . . . . . . . . . . . . . . . . . . 830
43.2.1 General Information . . . . . . . . . . . . . . . . . . . . . . . . . 830
43.2.2 addResolver . . . . . . . . . . . . . . . . . . . . . . . . . . . . 831
43.2.3 removeResolver . . . . . . . . . . . . . . . . . . . . . . . . . . 832
43.2.4 listNames . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 833
43.2.5 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 833
43.3 The ResolverRegistry . . . . . . . . . . . . . . . . . . . . . . . . . . 837
43.4
Implementing custom item types with the ItemResolver interface
. . 844
43.4.1 ItemResolver concepts . . . . . . . . . . . . . . . . . . . . . . 845
43.4.2 The ItemResolver interface . . . . . . . . . . . . . . . . . . . . 846
43.4.3 The class SubItemIndex
. . . . . . . . . . . . . . . . . . . . . 850
43.4.4 The ItemRegistry . . . . . . . . . . . . . . . . . . . . . . . . . 851
43.4.5 Default item representations . . . . . . . . . . . . . . . . . . . . . 854
43.5
Implementing custom checks with the Checker interface . . . . . . . . 856
43.5.1 The Checker interface . . . . . . . . . . . . . . . . . . . . . . . 856
43.5.2 The class Pair . . . . . . . . . . . . . . . . . . . . . . . . . . . . 858
43.5.3
The CheckType interface and its implementation
DefaultCheckType . . . . . . . . . . . . . . . . . . . . . . . . 859
43.5.4 The class CheckDataType . . . . . . . . . . . . . . . . . . . . . 859
43.5.5 The class CheckData and its subclasses . . . . . . . . . . . . . 860
43.5.6 The CheckerRegistry . . . . . . . . . . . . . . . . . . . . . . 862
43.5.7 Custom checker example . . . . . . . . . . . . . . . . . . . . . . 863
43.6
Working with the Eclipse Graphical Editing Framework (GEF) . . . . . . 865
43.6.1 Recording GEF items . . . . . . . . . . . . . . . . . . . . . . . . 865
43.6.2 Implementing a GEF ItemNameResolver2 . . . . . . . . . . . . . 867
43.6.3 Implementing a GEF ItemValueResolver2 . . . . . . . . . . . . . 869
43.7 Test-run listeners . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 869
43.7.1 The TestRunListener interface . . . . . . . . . . . . . . . . . 870
43.7.2 The class TestRunEvent
. . . . . . . . . . . . . . . . . . . . . 871
43.7.3 The class TestSuiteNode . . . . . . . . . . . . . . . . . . . . . 872
CONTENTS
xxiii
43.8 ResetListener . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 873
43.9 DOM processors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 874
43.9.1 The DOMProcessor interface . . . . . . . . . . . . . . . . . . . . 875
43.9.2 The DOMProcessorRegistry . . . . . . . . . . . . . . . . . . . 876
43.9.3 Error handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 877
43.10 Image API extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 878
43.10.1 The ImageRep class . . . . . . . . . . . . . . . . . . . . . . . . . 878
43.10.2 The ImageComparator interface . . . . . . . . . . . . . . . . . 881
44 Daemon mode
882
44.1 Daemon concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 882
44.2 Daemon API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 883
44.2.1 The DaemonLocator . . . . . . . . . . . . . . . . . . . . . . . . 884
44.2.2 The Daemon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 884
44.2.3 The TestRunDaemon . . . . . . . . . . . . . . . . . . . . . . . . 886
44.2.4 The DaemonRunContext . . . . . . . . . . . . . . . . . . . . . . 889
44.2.5 The DaemonTestRunListener . . . . . . . . . . . . . . . . . . 896
44.3 Daemon security considerations . . . . . . . . . . . . . . . . . . . . . . . 896
44.3.1 Creating your own keystore . . . . . . . . . . . . . . . . . . . . . 897
44.3.2 Specifying the keystore . . . . . . . . . . . . . . . . . . . . . . . 898
44.3.3 Specifying the keystore on the client side . . . . . . . . . . . . . 898
45 The Procedure Builder definition file
899
45.1 Placeholders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 899
45.1.1 Fallback values for placeholders . . . . . . . . . . . . . . . . . . 901
45.2 Conditions for Package and Procedure Definition . . . . . . . . . . . . . 902
45.3 Interpretation of the Component Hierarchy . . . . . . . . . . . . . . . . . 902
45.4 Details about the @CONDITION tag . . . . . . . . . . . . . . . . . . . . 903
46 The ManualStepDialog
905
46.1 The ManualStepDialog API . . . . . . . . . . . . . . . . . . . . . . . . . . 905
47 Details about transforming nodes
907
CONTENTS
xxiv
47.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 907
47.2 Transformation with type changes . . . . . . . . . . . . . . . . . . . . . . 907
47.3 Additional transformations below the Extras node . . . . . . . . . . . . . . 908
47.3.1 Transformations without side-effects . . . . . . . . . . . . . . . . 908
47.3.2 Transformations with side-effects . . . . . . . . . . . . . . . . . . 908
48 Details about the algorithm for image comparison
910
48.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 910
48.2 Description of algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . 911
48.2.1 Classic image check . . . . . . . . . . . . . . . . . . . . . . . . . 911
48.2.2 Pixel-based identity check . . . . . . . . . . . . . . . . . . . . . . 912
48.2.3 Pixel-based similarity check . . . . . . . . . . . . . . . . . . . . . 913
48.2.4 Block-based identity check . . . . . . . . . . . . . . . . . . . . . 915
48.2.5 Block-based similarity check
. . . . . . . . . . . . . . . . . . . . 916
48.2.6 Histogram check . . . . . . . . . . . . . . . . . . . . . . . . . . . 918
48.2.7 Analysis with Discrete Cosine Transformation . . . . . . . . . . . 919
48.2.8 Block-based analysis with Discrete Cosine Transformation . . . . 921
48.2.9 Bilinear Filter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 923
48.3 Description of special functions . . . . . . . . . . . . . . . . . . . . . . . 924
48.3.1 Image-in-image search . . . . . . . . . . . . . . . . . . . . . . . 924
49 Result lists
927
49.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 927
49.2 Specific list actions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 929
49.2.1 All types of lists . . . . . . . . . . . . . . . . . . . . . . . . . . . . 929
49.2.2 Replacing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 929
49.2.3 Error list . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 930
49.3 Exporting and loading results
50 Generic classes
. . . . . . . . . . . . . . . . . . . . . . . . 930
931
50.1 Overview generic classes . . . . . . . . . . . . . . . . . . . . . . . . . . . 931
50.2 Mapping of complex components . . . . . . . . . . . . . . . . . . . . . . 936
50.2.1 Mapping of Comboboxes . . . . . . . . . . . . . . . . . . . . . . 936
CONTENTS
xxv
50.2.2 Mapping of Lists . . . . . . . . . . . . . . . . . . . . . . . . . . . 937
50.2.3 Mapping of TabPanels and Accordions . . . . . . . . . . . . . . . 938
50.2.4 Mapping of Tables and TreeTables . . . . . . . . . . . . . . . . . 938
50.2.5 Mapping of Trees . . . . . . . . . . . . . . . . . . . . . . . . . . . 939
50.3 Properties of generic classes . . . . . . . . . . . . . . . . . . . . . . . . . 940
50.3.1 Accordion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 940
50.3.2 BusyPane . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 940
50.3.3 Button . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 940
50.3.4 Calendar . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 941
50.3.5 ColorPicker . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 941
50.3.6 CheckBox . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 942
50.3.7 Closer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 942
50.3.8 Divider . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 943
50.3.9 ComboBox . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 943
50.3.10 Expander . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 944
50.3.11 FileChooser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 944
50.3.12 Graphics
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 944
50.3.13 Icon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 945
50.3.14 Indicator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 945
50.3.15 Item . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 946
50.3.16 Label . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 946
50.3.17 Link . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 947
50.3.18 List . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 947
50.3.19 LoadingComponent . . . . . . . . . . . . . . . . . . . . . . . . . 948
50.3.20 Maximizer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 948
50.3.21 Menu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 949
50.3.22 MenuItem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 949
50.3.23 Minimizer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 949
50.3.24 Popup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 950
50.3.25 RadioButton . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 950
50.3.26 Restore . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 951
CONTENTS
xxvi
50.3.27 Panel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 951
50.3.28 ProgressBar
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 952
50.3.29 ScrollBar . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 953
50.3.30 Separator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 953
50.3.31 Sizer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 953
50.3.32 Slider . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 954
50.3.33 Spacer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 954
50.3.34 Spinner . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 955
50.3.35 SplitPanel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 955
50.3.36 Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 955
50.3.37 TableCell . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 956
50.3.38 TableFooter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 957
50.3.39 TableHeader . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 957
50.3.40 TableHeaderCell . . . . . . . . . . . . . . . . . . . . . . . . . . . 957
50.3.41 TabPanel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 958
50.3.42 TableRow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 958
50.3.43 Text . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 959
50.3.44 TextArea . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 959
50.3.45 TextField . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 959
50.3.46 Thumb . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 960
50.3.47 ToggleButton . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 960
50.3.48 ToolBar . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 961
50.3.49 ToolBarItem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 961
50.3.50 ToolTip
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 961
50.3.51 Tree . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 962
50.3.52 TreeNode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 962
50.3.53 TreeTable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 963
50.3.54 Window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 963
51 Doctags
965
51.1 Doctags for reporting and documentation . . . . . . . . . . . . . . . . . . 965
51.1.1 @noreport Doctag . . . . . . . . . . . . . . . . . . . . . . . . . . 966
CONTENTS
xxvii
51.2 Doctags for test execution . . . . . . . . . . . . . . . . . . . . . . . . . . 966
51.3 Doctags for Editing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 966
A FAQ - Frequently Asked Questions
968
B Release notes
977
B.1
B.2
QF-Test version 4.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 977
B.1.1
Version 4.2.2 - March 9, 2018 . . . . . . . . . . . . . . . . . . . . 977
B.1.2
Version 4.2.1 - February 26, 2018
B.1.3
Highlights of version 4.2 . . . . . . . . . . . . . . . . . . . . . . . 979
B.1.4
Changes that can affect test execution . . . . . . . . . . . . . . . 979
B.1.5
Software that is no longer supported . . . . . . . . . . . . . . . . 980
B.1.6
Version 4.2.0 - January 10, 2018 . . . . . . . . . . . . . . . . . . 981
. . . . . . . . . . . . . . . . . 978
QF-Test version 4.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 984
B.2.1
Version 4.1.6 - October 27, 2017 . . . . . . . . . . . . . . . . . . 984
B.2.2
Version 4.1.5 - August 9, 2017 . . . . . . . . . . . . . . . . . . . 985
B.2.3
Version 4.1.4 - June 1, 2017 . . . . . . . . . . . . . . . . . . . . . 986
B.2.4
Version 4.1.3 - March 29, 2017 . . . . . . . . . . . . . . . . . . . 988
B.2.5
Version 4.1.2 - January 13, 2017 . . . . . . . . . . . . . . . . . . 990
B.2.6
Version 4.1.1 - November 23, 2016 . . . . . . . . . . . . . . . . . 991
B.2.7
Highlights of version 4.1 . . . . . . . . . . . . . . . . . . . . . . . 993
B.2.8
Changes that can affect test execution . . . . . . . . . . . . . . . 994
B.2.9
Software that is no longer supported . . . . . . . . . . . . . . . . 995
B.2.10 Version 4.1.0 - October 6, 2016 . . . . . . . . . . . . . . . . . . . 996
B.2.11 Version 4.1.0-rc2 - September 2, 2016 . . . . . . . . . . . . . . . 997
B.2.12 Version 4.1.0-rc1 - July 29, 2016 . . . . . . . . . . . . . . . . . . 998
B.3
QF-Test version 4.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1001
B.3.1
Version 4.0.11 - April 05, 2016 . . . . . . . . . . . . . . . . . . . 1001
B.3.2
Version 4.0.10 - January 28, 2016 . . . . . . . . . . . . . . . . . 1002
B.3.3
Version 4.0.9 - December 17, 2015 . . . . . . . . . . . . . . . . . 1003
B.3.4
Version 4.0.8 - November 4, 2015
B.3.5
Version 4.0.7 - August 27, 2015 . . . . . . . . . . . . . . . . . . . 1005
. . . . . . . . . . . . . . . . . 1003
CONTENTS
xxviii
B.3.6
Version 4.0.6 - July 16, 2015 . . . . . . . . . . . . . . . . . . . . 1005
B.3.7
Version 4.0.5 - May 12, 2015 . . . . . . . . . . . . . . . . . . . . 1006
B.3.8
Version 4.0.4 - March 5, 2015 . . . . . . . . . . . . . . . . . . . . 1008
B.3.9
Version 4.0.3 - January 29, 2015 . . . . . . . . . . . . . . . . . . 1009
B.3.10 Version 4.0.2-p2 . . . . . . . . . . . . . . . . . . . . . . . . . . . 1012
B.3.11 Version 4.0.2-p1 . . . . . . . . . . . . . . . . . . . . . . . . . . . 1012
B.3.12 Version 4.0.2 - October 22, 2014 . . . . . . . . . . . . . . . . . . 1012
B.3.13 Version 4.0.1 - August 28, 2014 . . . . . . . . . . . . . . . . . . . 1015
B.3.14 Main new features in version 4 . . . . . . . . . . . . . . . . . . . 1016
B.3.15 Version 4.0.0 - July 31, 2014 . . . . . . . . . . . . . . . . . . . . 1018
B.3.16 Version 4.0.0-rc1 - July 4, 2014 . . . . . . . . . . . . . . . . . . . 1020
B.3.17 Version 4.0.0-beta3 - May 13, 2014 . . . . . . . . . . . . . . . . . 1022
B.3.18 Version 4.0.0-beta2 - April 11, 2014 . . . . . . . . . . . . . . . . 1023
B.3.19 Version 4.0.0-beta1 - March 26, 2014 . . . . . . . . . . . . . . . 1023
B.4
QF-Test version 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1028
B.4.1
Version 3.5.7 - July 22, 2014 . . . . . . . . . . . . . . . . . . . . 1028
B.4.2
Version 3.5.6 - April 29, 2014 . . . . . . . . . . . . . . . . . . . . 1029
B.4.3
Version 3.5.5 - February 25, 2014
B.4.4
Version 3.5.4 - December 11, 2013 . . . . . . . . . . . . . . . . . 1031
B.4.5
Version 3.5.3 - September 17, 2013 . . . . . . . . . . . . . . . . 1032
B.4.6
Version 3.5.2 - July 2, 2013 . . . . . . . . . . . . . . . . . . . . . 1032
B.4.7
Version 3.5.1 - May 15, 2013 . . . . . . . . . . . . . . . . . . . . 1034
B.4.8
Main new features in version 3.5 . . . . . . . . . . . . . . . . . . 1035
B.4.9
Version 3.5.0 - March 20, 2013 . . . . . . . . . . . . . . . . . . . 1036
. . . . . . . . . . . . . . . . . 1030
B.4.10 Version 3.5M3 - January 8, 2013 . . . . . . . . . . . . . . . . . . 1039
B.4.11 Version 3.5M2 - November 16, 2012 . . . . . . . . . . . . . . . . 1040
B.4.12 Version 3.5M1 - October 29, 2012 . . . . . . . . . . . . . . . . . 1041
B.4.13 Version 3.4.14 - March 20, 2013 . . . . . . . . . . . . . . . . . . 1045
B.4.14 Version 3.4.13 - January 8, 2013 . . . . . . . . . . . . . . . . . . 1046
B.4.15 Version 3.4.12 - November 16, 2012 . . . . . . . . . . . . . . . . 1046
B.4.16 Version 3.4.11 - October 4, 2012 . . . . . . . . . . . . . . . . . . 1047
CONTENTS
xxix
B.4.17 Version 3.4.10 - August 7, 2012 . . . . . . . . . . . . . . . . . . . 1048
B.4.18 Version 3.4.9 - June 29, 2012 . . . . . . . . . . . . . . . . . . . . 1048
B.4.19 Version 3.4.8 - June 5, 2012 . . . . . . . . . . . . . . . . . . . . . 1049
B.4.20 Version 3.4.7 - April 24, 2012 . . . . . . . . . . . . . . . . . . . . 1050
B.4.21 Version 3.4.6 - March 27, 2012 . . . . . . . . . . . . . . . . . . . 1050
B.4.22 Version 3.4.5 - March 12, 2012 . . . . . . . . . . . . . . . . . . . 1051
B.4.23 Version 3.4.4 - January 30, 2012 . . . . . . . . . . . . . . . . . . 1052
B.4.24 Version 3.4.3 - October 27, 2011 . . . . . . . . . . . . . . . . . . 1053
B.4.25 Version 3.4.2 - September 15, 2011 . . . . . . . . . . . . . . . . 1054
B.4.26 Version 3.4.1 - July 15, 2011 . . . . . . . . . . . . . . . . . . . . 1055
B.4.27 Main new features in version 3.4 . . . . . . . . . . . . . . . . . . 1057
B.4.28 Version 3.4.0 - May 26, 2011 . . . . . . . . . . . . . . . . . . . . 1058
B.4.29 Version 3.4M2 - March 10, 2011 . . . . . . . . . . . . . . . . . . 1060
B.4.30 Version 3.4M1 - February 3, 2011
. . . . . . . . . . . . . . . . . 1062
B.4.31 Version 3.3.0 - November 30, 2010 . . . . . . . . . . . . . . . . . 1064
B.4.32 Version 3.2.2 - July 15, 2010 . . . . . . . . . . . . . . . . . . . . 1067
B.4.33 Version 3.2.1 - May 11, 2010 . . . . . . . . . . . . . . . . . . . . 1067
B.4.34 New features in QF-Test version 3.2 . . . . . . . . . . . . . . . . 1068
B.4.35 Version 3.2.0 - March 25, 2010 . . . . . . . . . . . . . . . . . . . 1069
B.4.36 Version 3.1.4 - February 9, 2010 . . . . . . . . . . . . . . . . . . 1070
B.4.37 Version 3.1.3 - December 1, 2009 . . . . . . . . . . . . . . . . . 1072
B.4.38 Version 3.1.2 - October 15, 2009 . . . . . . . . . . . . . . . . . . 1073
B.4.39 Version 3.1.1 - July 28, 2009 . . . . . . . . . . . . . . . . . . . . 1075
B.4.40 Version 3.1.0 - May 19, 2009 . . . . . . . . . . . . . . . . . . . . 1077
B.4.41 Version 3.0.2 - March 27, 2009 . . . . . . . . . . . . . . . . . . . 1079
B.4.42 Version 3.0.1 - January 23, 2009 . . . . . . . . . . . . . . . . . . 1080
B.4.43 New features in QF-Test version 3 - December 5, 2008 . . . . . . 1081
C Keyboard shortcuts
1083
C.1
Navigation and editing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1083
C.2
Record and replay functions . . . . . . . . . . . . . . . . . . . . . . . . . 1086
C.3
Keyboard helper . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1087
CONTENTS
xxx
D Glossary
1089
E Third party software
1090
xxxi
List of Figures
2.1
Structure of a test-suite . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.2
Insertion marker . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.3
Example table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.4
The simple search dialog . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.5
The advanced search dialog . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.6
Result list for ’Locate references’ . . . . . . . . . . . . . . . . . . . . . . . 22
2.7
Incremental search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.8
The replace dialog . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.9
The replace query dialog . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.1
Quickstart Wizard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3.2
Startup sequence created by the Quickstart Wizard . . . . . . . . . . . . 29
3.3
GUI technology information . . . . . . . . . . . . . . . . . . . . . . . . . . 32
4.1
Disabled and enabled Record button . . . . . . . . . . . . . . . . . . . . 33
5.1
Components of a GUI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
5.2
Component hierarchy of a Swing SUT . . . . . . . . . . . . . . . . . . . . 43
5.3
Pseudo class hierarchy for web elements . . . . . . . . . . . . . . . . . . 46
5.4
Update components dialog . . . . . . . . . . . . . . . . . . . . . . . . . . 51
6.1
An Item for a table cell . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
6.2
Extra Feature attributes for x-path or css-selector component recognition. 60
7.1
Direct and fallback bindings . . . . . . . . . . . . . . . . . . . . . . . . . 63
7.2
System variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
LIST OF FIGURES
xxxii
7.3
Variable example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
7.4
Variable bindings example . . . . . . . . . . . . . . . . . . . . . . . . . . 66
8.1
A simple test and its run-log . . . . . . . . . . . . . . . . . . . . . . . . . 75
8.2
Error states in a run-log . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
9.1
Test-suite structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
9.2
Test structure with simple Setup and Cleanup . . . . . . . . . . . . . . . . 90
9.3
Test execution with simple Setup and Cleanup . . . . . . . . . . . . . . . . 90
9.4
Packages and Procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
9.5
Dependency stack A-B-C . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
9.6
Good practice Setup node . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
9.7
Dependency stack A-B-D-E . . . . . . . . . . . . . . . . . . . . . . . . . 100
9.8
Dependency with Characteristic variables . . . . . . . . . . . . . . . . . . . . 102
9.9
Exception in forced cleanup sequence of C causes B to clean up . . . . 104
9.10 Typical Cleanup node . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
9.11 Example Test-set for name spaces . . . . . . . . . . . . . . . . . . . . . . 106
9.12 Dependency handling for test-case ’Data entry by User A’ . . . . . . . . 107
9.13 Dependency handling for test-case ’Offer processing by User C’ . . . . . 107
9.14 Dependency handling for test-case ’Check offer 1 in DMS’ . . . . . . . . 108
9.15 Dependency handling for test-case ’Data entry by User B’ . . . . . . . . 108
9.16 Dependency handling for test-case ’Offer processing by User D’ . . . . . 109
9.17 Dependency handling for test-case ’Check offer 2 in DMS’ . . . . . . . . 109
10.1 The project view . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
11.1 Standard library qfs.qft . . . . . . . . . . . . . . . . . . . . . . . . . . 113
12.1 Detail view of a Server script with help window for rc methods
. . . . . . 115
12.2 Overview of the types of variables in QF-Test . . . . . . . . . . . . . . . . 118
13.1 Unit Test node with Java classes . . . . . . . . . . . . . . . . . . . . . . . 139
13.2 Example Unit Test node with Injections . . . . . . . . . . . . . . . . . . . 144
13.3 Example Unit Test node with Injections . . . . . . . . . . . . . . . . . . . 146
LIST OF FIGURES
xxxiii
13.4 Unit Test Report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
15.1 PDF Client main window with PDF document . . . . . . . . . . . . . . . . 153
15.2 Check text ’default’ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
15.3 Check text ’Text positioned’ . . . . . . . . . . . . . . . . . . . . . . . . . . 155
15.4 Check Image ’default’ recording of a Text object . . . . . . . . . . . . . . 155
15.5 Check Image ’default’ recording of an Image object . . . . . . . . . . . . 156
15.6 Check Image ’unscaled’ recording of an Image object . . . . . . . . . . . 156
15.7 Check Image ’scaled’ recording of an Image object . . . . . . . . . . . . 156
16.1 A simple data-driven test . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
16.2 Data table example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
16.3 Run-log of a data-driven test . . . . . . . . . . . . . . . . . . . . . . . . . 161
16.4 Data-driven test with nested loops . . . . . . . . . . . . . . . . . . . . . . 161
16.5 Second data table example . . . . . . . . . . . . . . . . . . . . . . . . . . 162
16.6 Run-log of a data-driven test with nested loops . . . . . . . . . . . . . . . 163
17.1 Example report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
18.1 Dialog to re-run test-cases . . . . . . . . . . . . . . . . . . . . . . . . . . 189
19.1 Result of analyzing references . . . . . . . . . . . . . . . . . . . . . . . . 200
20.1 Recorded procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204
20.2 The Procedure Builder definition file . . . . . . . . . . . . . . . . . . . . . 205
21.1 Integration with ALM - QualityCenter . . . . . . . . . . . . . . . . . . . . 208
21.2 QF-Test VAPI-XP-TEST test-case in HP ALM - QualityCenter . . . . . . 209
21.3 In Test plan create new Test-set . . . . . . . . . . . . . . . . . . . . . . . 210
21.4 Create new test of type VAPI-XP-TEST . . . . . . . . . . . . . . . . . . . 211
21.5 HP VAPI-XP Wizard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212
21.6 Test details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
21.7 Copy template content to script text area . . . . . . . . . . . . . . . . . . 214
21.8 New test-set in Test lab section . . . . . . . . . . . . . . . . . . . . . . . 215
21.9 Add test to execution grid . . . . . . . . . . . . . . . . . . . . . . . . . . . 216
LIST OF FIGURES
xxxiv
21.10 Run the test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
21.11 Test result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218
21.12 Uploaded run-log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219
21.13 Script debug run . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220
21.14 QF-Test run-log in QMetry . . . . . . . . . . . . . . . . . . . . . . . . . . 224
22.1 Eclipse plugin configuration - tab ’Main’ . . . . . . . . . . . . . . . . . . . 232
22.2 Eclipse plugin configuration - Tab ’Settings’ . . . . . . . . . . . . . . . . . 233
22.3 Eclipse plugin configuration - Tab ’Initial Settings’ . . . . . . . . . . . . . 234
22.4 Jenkins after start-up. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239
22.5 Install QF-Test Plugin. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240
22.6 Set custom workspace. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241
22.7 Add build step to run QF-Test . . . . . . . . . . . . . . . . . . . . . . . . 242
22.8 Configure build step advanced options. . . . . . . . . . . . . . . . . . . . 242
22.9 Configure post build steps. . . . . . . . . . . . . . . . . . . . . . . . . . . 243
23.1 Excel file business-related keywords
. . . . . . . . . . . . . . . . . . . . 248
23.2 Test-suite business-related keywords . . . . . . . . . . . . . . . . . . . . 249
23.3 Procedure fillDialog . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251
23.4 Excel file of generic components . . . . . . . . . . . . . . . . . . . . . . . 253
23.5 Test-suite for generic components . . . . . . . . . . . . . . . . . . . . . . 254
23.6 Test-suite Behavior-driven testing technical . . . . . . . . . . . . . . . . . 257
23.7 Test-suite Behavior-driven testing from business perspective . . . . . . . 259
23.8 Excel file as scenario file . . . . . . . . . . . . . . . . . . . . . . . . . . . 260
23.9 Test-suite scenario file . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261
24.1 Load testing scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266
24.2 Overview load testing project . . . . . . . . . . . . . . . . . . . . . . . . . 268
24.3 Sample test-suite daemonController_twoPhases.qft . . . . . . . . . . . . 269
24.4 Call of rc.syncThreads in demo test-suite . . . . . . . . . . . . . . . . . . 273
25.1 Example for a ManualStepDialog . . . . . . . . . . . . . . . . . . . . . . 277
28.1 Structure of multiple test-suites . . . . . . . . . . . . . . . . . . . . . . . 291
LIST OF FIGURES
xxxv
28.2 Including test-suites of level 1 . . . . . . . . . . . . . . . . . . . . . . . . 292
28.3 Structure of different test-suites with roles
. . . . . . . . . . . . . . . . . 293
30.1 Default configuration for component recording . . . . . . . . . . . . . . . 300
30.2 Default configuration for component recognition . . . . . . . . . . . . . . 301
30.3 ’Name overrides everything’ configuration for component recording . . . 302
30.4 ’Name overrides everything’ configuration for component recognition . . 303
30.5 Using a regular expression in the Feature attribute . . . . . . . . . . . . . 304
30.6 Option to configure recording of system classes only . . . . . . . . . . . 306
33.1 Options tree . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317
33.2 General options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 318
33.3 Projects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 320
33.4 Editing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 322
33.5 Bookmarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325
33.6 External tools options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 326
33.7 Backup file options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 329
33.8 Library path option . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331
33.9 License options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 332
33.10 Recording options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333
33.11 Options for events to record . . . . . . . . . . . . . . . . . . . . . . . . . 334
33.12 Options for events to pack . . . . . . . . . . . . . . . . . . . . . . . . . . 336
33.13 Dragging to a sub-menu . . . . . . . . . . . . . . . . . . . . . . . . . . . 337
33.14 Options for recording components . . . . . . . . . . . . . . . . . . . . . . 340
33.15 Popup menu for recording components . . . . . . . . . . . . . . . . . . . 341
33.16 Options for recording sub-items . . . . . . . . . . . . . . . . . . . . . . . 347
33.17 Procedure Builder options . . . . . . . . . . . . . . . . . . . . . . . . . . 348
33.18 Replay options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 349
33.19 Client options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353
33.20 Terminal options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357
33.21 Event handling options . . . . . . . . . . . . . . . . . . . . . . . . . . . . 360
33.22 Component recognition options . . . . . . . . . . . . . . . . . . . . . . . 365
LIST OF FIGURES
xxxvi
33.23 Delay options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 368
33.24 Timeout options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 371
33.25 Options for replay backward compatibility . . . . . . . . . . . . . . . . . . 374
33.26 Web options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 376
33.27 Options for HTTP Requests . . . . . . . . . . . . . . . . . . . . . . . . . 379
33.28 Options for web backward compatibility . . . . . . . . . . . . . . . . . . . 381
33.29 Debugger options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 382
33.30 General run-log options . . . . . . . . . . . . . . . . . . . . . . . . . . . . 384
33.31 Options determining run-log content . . . . . . . . . . . . . . . . . . . . . 389
33.32 Options for mapping between directories with test-suites . . . . . . . . . 392
33.33 Variable options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 393
34.1 Test-suite attributes
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 396
34.2 Test-case attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 400
34.3 Test-set attributes
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 407
34.4 Test call Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 412
34.5 Sequence attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 416
34.6 Test-step attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 418
34.7 Sequence with time limit attributes . . . . . . . . . . . . . . . . . . . . . . . 423
34.8 Extras attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 426
34.9 Dependency attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 428
34.10 Dependency reference attributes . . . . . . . . . . . . . . . . . . . . . . . . 431
34.11 Setup attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 434
34.12 Cleanup attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 436
34.13 Error handler attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 438
34.14 Data driver attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 441
34.15 Data table attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 444
34.16 Database attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 448
34.17 Excel data file attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 453
34.18 CSV data file attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 457
34.19 Data loop attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 460
34.20 Procedure Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 464
LIST OF FIGURES
xxxvii
34.21 Procedure call Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 467
34.22 Return Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 469
34.23 Package Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 471
34.24 Procedures Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 472
34.25 Loop attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 474
34.26 While attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 477
34.27 Break attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 480
34.28 If attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 482
34.29 Elseif attributes
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 485
34.30 Else attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 488
34.31 Try attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 491
34.32 Catch attributes
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 494
34.33 Finally attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 497
34.34 Throw attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 499
34.35 Rethrow attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 501
34.36 Server script attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 502
34.37 SUT script attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 505
34.38 Start Java SUT client attributes . . . . . . . . . . . . . . . . . . . . . . . . . 508
34.39 Start SUT client attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . 512
34.40 Start process attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 515
34.41 Start web engine attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . 518
34.42 Open browser window attributes . . . . . . . . . . . . . . . . . . . . . . . . 522
34.43 Execute shell command attributes . . . . . . . . . . . . . . . . . . . . . . . 524
34.44 Start PDF client attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . 527
34.45 Wait for client to connect attributes . . . . . . . . . . . . . . . . . . . . . . 530
34.46 Stop client attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 533
34.47 Wait for process to terminate attributes . . . . . . . . . . . . . . . . . . . . . 535
34.48 Mouse event attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 538
34.49 Key event attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 542
34.50 Text input attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 546
34.51 Window event attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 549
LIST OF FIGURES
xxxviii
34.52 Component event attributes . . . . . . . . . . . . . . . . . . . . . . . . . . 551
34.53 Selection attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 556
34.54 File selection attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 558
34.55 Check text attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 562
34.56 Boolean check attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 567
34.57 Check items attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 573
34.58 Check selectable items attributes . . . . . . . . . . . . . . . . . . . . . . . . 577
34.59 Check image attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 582
34.60 Check geometry attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . 587
34.61 Fetch text attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 592
34.62 Fetch index attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 595
34.63 Fetch geometry attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . 598
34.64 Set variable attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 602
34.65 Wait for component to appear attributes . . . . . . . . . . . . . . . . . . . . 605
34.66 Wait for document to load attributes . . . . . . . . . . . . . . . . . . . . . . 609
34.67 Wait for download to finish attributes . . . . . . . . . . . . . . . . . . . . . 613
34.68 Load resources attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 616
34.69 Load properties attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . 619
34.70 Unit test server attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . 622
34.71 Unit test client attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 623
34.72 Server HTTP request Attribute . . . . . . . . . . . . . . . . . . . . . . . . . 628
34.73 Browser HTTP request Attribute . . . . . . . . . . . . . . . . . . . . . . . . 632
34.74 Window attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 635
34.75 Web page attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 640
34.76 Component attributes
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 645
34.77 Item attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 651
34.78 Window group attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 653
34.79 Component group attributes . . . . . . . . . . . . . . . . . . . . . . . . . . 654
34.80 Windows and components attributes . . . . . . . . . . . . . . . . . . . . . . 656
34.81 Test attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 658
38.1 Starting the SUT from a script or executable . . . . . . . . . . . . . . . . 700
LIST OF FIGURES
xxxix
38.2 Starting the SUT through Java WebStart . . . . . . . . . . . . . . . . . . 701
38.3 Starting the SUT as an applet in a web browser . . . . . . . . . . . . . . 702
38.4 Starting the SUT from a jar archive . . . . . . . . . . . . . . . . . . . . . 705
38.5 Starting the SUT via the main class . . . . . . . . . . . . . . . . . . . . . 706
38.6 Starting a web-based SUT in a browser . . . . . . . . . . . . . . . . . . . 708
38.7 Opening a PDF Document . . . . . . . . . . . . . . . . . . . . . . . . . . 709
39.1 Dialog for JDK instrumentation . . . . . . . . . . . . . . . . . . . . . . . . 711
42.1 HTML code inspection in Firefox . . . . . . . . . . . . . . . . . . . . . . . 785
42.2 Reduction of complexity for WebCarConfigurator demo . . . . . . . . . . 786
42.3 WebCarConfigurator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 787
42.4 Simplification due to simple class mapping . . . . . . . . . . . . . . . . . 788
42.5 Recording of ’-5%’ button in WebCarConfigurator . . . . . . . . . . . . . 789
42.6 Recording with genericClasses in WebCarConfigurator . . . . . . . . . . 790
42.7 Simplification due to advanced class mapping . . . . . . . . . . . . . . . 791
42.8 Recording of SPAN textfields . . . . . . . . . . . . . . . . . . . . . . . . . 792
42.9 Recording TextFields in WebCarConfigurator . . . . . . . . . . . . . . . . 793
42.10 Simplification for complex components . . . . . . . . . . . . . . . . . . . 794
42.11 Recording of table in WebCarConfigurator . . . . . . . . . . . . . . . . . 795
42.12 Recording of resolved table item in WebCarConfigurator . . . . . . . . . 797
42.13 Simplification of WebCarConfigurator demo . . . . . . . . . . . . . . . . 799
48.1 Original image . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 911
48.2 Classic image check . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 912
48.3 Pixel-based identity check . . . . . . . . . . . . . . . . . . . . . . . . . . 913
48.4 Pixel-based similarity check . . . . . . . . . . . . . . . . . . . . . . . . . 914
48.5 Block-based identity check . . . . . . . . . . . . . . . . . . . . . . . . . . 915
48.6 Block-based similarity check . . . . . . . . . . . . . . . . . . . . . . . . . 917
48.7 Histogram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 918
48.8 Analysis with Discrete Cosine Transformation . . . . . . . . . . . . . . . 920
48.9 Block-based analysis with Discrete Cosine Transformation . . . . . . . . 922
48.10 Bilinear Filter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 923
LIST OF FIGURES
xl
48.11 Image-in-image search: Expected image . . . . . . . . . . . . . . . . . . 925
48.12 Image-in-image search: Got image . . . . . . . . . . . . . . . . . . . . . 925
49.1 Sample result list for ’Locate references’ . . . . . . . . . . . . . . . . . . 928
A.1
Set browser maximum memory . . . . . . . . . . . . . . . . . . . . . . . 975
C.1
Keyboard helper . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1088
xli
List of Tables
1.1
Supported technology versions . . . . . . . . . . . . . . . . . . . . . . .
4
4.1
Test result counter in the status line . . . . . . . . . . . . . . . . . . . . . 36
6.1
Separators and index formats for element access . . . . . . . . . . . . . 57
7.1
Definitions in the special group qftest
9.1
Relative procedure calls . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
. . . . . . . . . . . . . . . . . . 71
15.1 Supported PDF objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
15.2 Color code for PDF objects . . . . . . . . . . . . . . . . . . . . . . . . . . 158
18.1 Choices for handling the run-log of a re-run . . . . . . . . . . . . . . . . . 188
23.1 Test-case using business-related keywords . . . . . . . . . . . . . . . . . 246
23.2 Test-case using atomic keywords . . . . . . . . . . . . . . . . . . . . . . 246
23.3 Test-case with Behavior-Driven Testing from a technical perspective . . . 246
23.4 Test-case with Behavior-Driven Testing from a business perspective . . . 247
23.5 Structure of SimpleKeywords.qft . . . . . . . . . . . . . . . . . . . . . . . 250
23.6 Structure of Keywords_With_Generics.qft . . . . . . . . . . . . . . . . . . 262
23.7 Necessary adaptions to your SUT . . . . . . . . . . . . . . . . . . . . . . 264
24.1 Content of load testing directory . . . . . . . . . . . . . . . . . . . . . . . 267
25.1 Description of the Excel file for the definition of manual tests . . . . . . . 278
25.2 Description of the Excel file with the results of manual tests . . . . . . . . 279
25.3 Description of the global variables in the ManualTestRunner test-suite . . 279
LIST OF TABLES
xlii
25.4 States of manual test-execution . . . . . . . . . . . . . . . . . . . . . . . 280
29.1 List of variables with auto-completion. . . . . . . . . . . . . . . . . . . . . 298
34.1 Placeholders for the Name for separate run-log attribute . . . . . . . . . . . 402
34.2 Placeholders for the Name for separate run-log attribute . . . . . . . . . . . 409
34.3 Placeholders for the Name for separate run-log attribute . . . . . . . . . . . 413
34.4 Placeholders for the Name for separate run-log attribute . . . . . . . . . . . 420
34.5 Placeholders for the Name for separate run-log attribute . . . . . . . . . . . 442
34.6 Iteration range examples . . . . . . . . . . . . . . . . . . . . . . . . . . . 445
34.7 Iteration range examples . . . . . . . . . . . . . . . . . . . . . . . . . . . 449
34.8 Database drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 450
34.9 Database connection strings . . . . . . . . . . . . . . . . . . . . . . . . . 451
34.10 Iteration range examples . . . . . . . . . . . . . . . . . . . . . . . . . . . 454
34.11 Iteration range examples . . . . . . . . . . . . . . . . . . . . . . . . . . . 458
34.12 Iteration range examples . . . . . . . . . . . . . . . . . . . . . . . . . . . 461
34.13 Condition examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 478
34.14 Condition examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 483
34.15 Condition examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 486
34.16 Modifier values
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 540
34.17 Modifier values
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 544
34.18 Supported SWT widgets for a Selection event . . . . . . . . . . . . . . . . 554
34.19 Supported DOM nodes for a Selection event . . . . . . . . . . . . . . . . 555
34.20 Provided Check types of Check text . . . . . . . . . . . . . . . . . . . . . 564
34.21 Provided Check types of Boolean check . . . . . . . . . . . . . . . . . . . 569
34.22 Components supported by Fetch text . . . . . . . . . . . . . . . . . . . . . 591
34.23 Components supported by Fetch geometry . . . . . . . . . . . . . . . . . . 597
34.24 Possible regular expressions . . . . . . . . . . . . . . . . . . . . . . . . . 625
34.25 Injection types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 625
34.26 Extra features assigned by QF-Test . . . . . . . . . . . . . . . . . . . . . 638
34.27 Extra features assigned by QF-Test . . . . . . . . . . . . . . . . . . . . . 643
34.28 Extra features assigned by QF-Test . . . . . . . . . . . . . . . . . . . . . 648
LIST OF TABLES
xliii
34.29 Sub-items of complex Swing components . . . . . . . . . . . . . . . . . . 650
34.30 Placeholders for the Name for separate run-log attribute . . . . . . . . . . . 659
36.1 Samples -suitesfile <file> . . . . . . . . . . . . . . . . . . . . . . 691
36.2 Placeholders in filename parameters . . . . . . . . . . . . . . . . . . . . 695
36.3 Exit codes for QF-Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . 696
36.4 calldaemon exit codes for QF-Test . . . . . . . . . . . . . . . . . . . . . 696
42.1 Supported AJAX toolkits . . . . . . . . . . . . . . . . . . . . . . . . . . . 769
42.2 Angular resolver versions . . . . . . . . . . . . . . . . . . . . . . . . . . . 771
42.3 Ext JS resolver versions . . . . . . . . . . . . . . . . . . . . . . . . . . . 772
42.4 GWT resolver versions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 772
42.5 ICEfaces resolver versions . . . . . . . . . . . . . . . . . . . . . . . . . . 773
42.6 jQuery UI resolver versions . . . . . . . . . . . . . . . . . . . . . . . . . . 773
42.7 jQuery EasyUI resolver versions . . . . . . . . . . . . . . . . . . . . . . . 773
42.8 Kendo UI resolver versions . . . . . . . . . . . . . . . . . . . . . . . . . . 774
42.9 PrimeFaces resolver versions . . . . . . . . . . . . . . . . . . . . . . . . 774
42.10 Qooxdoo resolver versions . . . . . . . . . . . . . . . . . . . . . . . . . . 774
42.11 RAP resolver versions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 775
42.12 RichFaces resolver versions . . . . . . . . . . . . . . . . . . . . . . . . . 775
42.13 Smart GWT resolver versions . . . . . . . . . . . . . . . . . . . . . . . . 776
42.14 Vaadin resolver versions . . . . . . . . . . . . . . . . . . . . . . . . . . . 776
42.15 ZK resolver versions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 776
42.16 Older Ext JS resolver versions . . . . . . . . . . . . . . . . . . . . . . . . 780
42.17 Older GWT resolver versions . . . . . . . . . . . . . . . . . . . . . . . . . 780
42.18 Older ICEfaces resolver versions . . . . . . . . . . . . . . . . . . . . . . 781
42.19 Older jQuery UI resolver versions . . . . . . . . . . . . . . . . . . . . . . 781
42.20 Older PrimeFaces resolver versions . . . . . . . . . . . . . . . . . . . . . 781
42.21 Older Qooxdoo resolver versions . . . . . . . . . . . . . . . . . . . . . . 782
42.22 Older RAP resolver versions . . . . . . . . . . . . . . . . . . . . . . . . . 782
42.23 Older RichFaces resolver versions . . . . . . . . . . . . . . . . . . . . . . 783
42.24 Older Vaadin resolver versions . . . . . . . . . . . . . . . . . . . . . . . . 783
LIST OF TABLES
xliv
43.1 Internal item representations for JavaFX GUI elements . . . . . . . . . . 854
43.2 Internal item representations for Swing GUI elements . . . . . . . . . . . 855
43.3 Internal item representations for SWT GUI elements . . . . . . . . . . . . 855
43.4 Internal item representations for DOM nodes . . . . . . . . . . . . . . . . 855
44.1 The run state . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 890
45.1 Placeholders for component procedures . . . . . . . . . . . . . . . . . . 900
45.2 Additional placeholders for container procedures . . . . . . . . . . . . . . 901
45.3 Comment attributes for procedure creation . . . . . . . . . . . . . . . . . 902
45.4 Hierarchy placeholders . . . . . . . . . . . . . . . . . . . . . . . . . . . . 903
45.5 Samples for the @CONDITION tag . . . . . . . . . . . . . . . . . . . . . 904
50.1 Generic classes for input and output . . . . . . . . . . . . . . . . . . . . . 932
50.2 Generic classes for interactions . . . . . . . . . . . . . . . . . . . . . . . 933
50.3 Generic classes for container components . . . . . . . . . . . . . . . . . 934
50.4 Generic classes for data containers . . . . . . . . . . . . . . . . . . . . . 935
50.5 Generic classes for data . . . . . . . . . . . . . . . . . . . . . . . . . . . 935
50.6 Generic classes for specific components . . . . . . . . . . . . . . . . . . 936
50.7 Mapping of ComboBoxes . . . . . . . . . . . . . . . . . . . . . . . . . . . 937
50.8 Mapping of Lists . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 937
50.9 Mapping of TabPanels . . . . . . . . . . . . . . . . . . . . . . . . . . . . 938
50.10 Mapping of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 939
50.11 Mapping of trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 939
50.12 Checktypes for Accordion . . . . . . . . . . . . . . . . . . . . . . . . . . . 940
50.13 Special qfs:type values for Buttons . . . . . . . . . . . . . . . . . . . . . 941
50.14 Special qfs:type values for CheckBoxes . . . . . . . . . . . . . . . . . . . 942
50.15 Checktypes for Checkbox . . . . . . . . . . . . . . . . . . . . . . . . . . . 942
50.16 Special qfs:type values for Closer . . . . . . . . . . . . . . . . . . . . . . 943
50.17 Checktypes for ComboBox . . . . . . . . . . . . . . . . . . . . . . . . . . 943
50.18 Special qfs:type values for Expander . . . . . . . . . . . . . . . . . . . . 944
50.19 Special qfs:type values for Icon . . . . . . . . . . . . . . . . . . . . . . . 945
50.20 Special qfs:type values for Indicator . . . . . . . . . . . . . . . . . . . . . 945
LIST OF TABLES
xlv
50.21 Special qfs:type values for Item . . . . . . . . . . . . . . . . . . . . . . . 946
50.22 Checktypes for Item . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 946
50.23 Special qfs:type values for Labels . . . . . . . . . . . . . . . . . . . . . . 947
50.24 Special qfs:type values for Links . . . . . . . . . . . . . . . . . . . . . . . 947
50.25 Special qfs:type values for List . . . . . . . . . . . . . . . . . . . . . . . . 948
50.26 Checktypes for List . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 948
50.27 Special qfs:type values for Maximizer . . . . . . . . . . . . . . . . . . . . 949
50.28 Special qfs:type values for Menu . . . . . . . . . . . . . . . . . . . . . . . 949
50.29 Special qfs:type values for Minimizer . . . . . . . . . . . . . . . . . . . . 950
50.30 Special qfs:type values for Popup . . . . . . . . . . . . . . . . . . . . . . 950
50.31 Special qfs:type values for RadioButtons . . . . . . . . . . . . . . . . . . 951
50.32 Checktypes for RadioButton . . . . . . . . . . . . . . . . . . . . . . . . . 951
50.33 Special qfs:type values for Restore . . . . . . . . . . . . . . . . . . . . . 951
50.34 Special qfs:type values for Panel . . . . . . . . . . . . . . . . . . . . . . . 952
50.35 Checktypes for ProgressBar . . . . . . . . . . . . . . . . . . . . . . . . . 953
50.36 Special qfs:type values for Sizer . . . . . . . . . . . . . . . . . . . . . . . 954
50.37 Checktypes for Slider . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 954
50.38 Special qfs:type values for Spacer . . . . . . . . . . . . . . . . . . . . . . 954
50.39 Checktypes for Spinner . . . . . . . . . . . . . . . . . . . . . . . . . . . . 955
50.40 Checktypes for Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 956
50.41 Checktypes for TableCell . . . . . . . . . . . . . . . . . . . . . . . . . . . 956
50.42 Checktypes for TableHeader . . . . . . . . . . . . . . . . . . . . . . . . . 957
50.43 Checktypes for TableHeaderCell . . . . . . . . . . . . . . . . . . . . . . . 958
50.44 Checktypes for TabPanel . . . . . . . . . . . . . . . . . . . . . . . . . . . 958
50.45 Special qfs:type values for Text . . . . . . . . . . . . . . . . . . . . . . . . 959
50.46 Checktypes for TextArea . . . . . . . . . . . . . . . . . . . . . . . . . . . 959
50.47 Special qfs:type values for TextField . . . . . . . . . . . . . . . . . . . . . 960
50.48 Checktypes for TextField . . . . . . . . . . . . . . . . . . . . . . . . . . . 960
50.49 Checktypes for ToggleButton . . . . . . . . . . . . . . . . . . . . . . . . . 961
50.50 Checktypes for Tree . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 962
50.51 Checktypes for TreeNode . . . . . . . . . . . . . . . . . . . . . . . . . . . 963
LIST OF TABLES
xlvi
50.52 Special qfs:type values for Window . . . . . . . . . . . . . . . . . . . . . 964
51.1 Doctags for reporting and documentation . . . . . . . . . . . . . . . . . . 965
51.2 Doctags for test execution . . . . . . . . . . . . . . . . . . . . . . . . . . 966
51.3 Doctags for editing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 967
B.1
Highlights in QF-Test version 4.2 . . . . . . . . . . . . . . . . . . . . . . . 979
B.2
Highlights in QF-Test version 4.1 (part 1) . . . . . . . . . . . . . . . . . . 993
B.3
Highlights in QF-Test version 4.1 (part 2) . . . . . . . . . . . . . . . . . . 994
B.4
Highlights in QF-Test version 4 . . . . . . . . . . . . . . . . . . . . . . . . 1017
B.5
New features in QF-Test version 3.5 . . . . . . . . . . . . . . . . . . . . . 1036
B.6
New features in QF-Test version 3.4 . . . . . . . . . . . . . . . . . . . . . 1058
B.7
New features in QF-Test version 3.2 . . . . . . . . . . . . . . . . . . . . . 1069
B.8
New features in QF-Test version 3 . . . . . . . . . . . . . . . . . . . . . . 1082
C.1
Shortcuts for navigation and editing . . . . . . . . . . . . . . . . . . . . . 1086
C.2
Shortcuts for special record and replay functions . . . . . . . . . . . . . . 1087
Part I
User manual
Chapter 1
Installation and startup
The installation of QF-Test on the supported operating systems is explained in detail in
the subsequent sections. The following packages are available for download:
(5)
Windows (section 1.2 )
On Windows QF-Test is normally installed via the setup program
QF-Test-4.2.2.exe which requires administrator privileges. If you are lacking
the required permissions or prefer to keep all QF-Test files together in one place
you can unpack the self-extracting archive QF-Test-4.2.2-sfx.exe instead.
(7)
Linux / Unix (section 1.3 )
On Linux and other Unix
QF-Test-4.2.2.tar.gz.
systems
please
unpack
the
archive
(8)
macOS (section 1.4 )
The disk-image QF-Test-4.2.2.dmg is provided for the installation on macOS.
It is possible to have different versions of QF-Test installed in parallel. Existing configuration files will not be overwritten during setup.
(285)
In section 27.2
you can find best practices about the QF-Test installation.
1.1.
System requirements
1.1
1.1.1
3
System requirements
Prime precondition
QF-Test itself requires Java 8. A 64 bit Java Runtime Environment (JRE) is provided
with QF-Test, so Java does not need to be installed on your system unless required by
the SUT.
Note
If your environment and/or your system under test (SUT) requires to still use Java 7 or
even Java 6, that is still possible. The Java command for the SUT can be configured
separately when creating the setup sequence for your SUT. In case of Java 6, older
versions of Jython and Groovy are used automatically.
1.1.2
Supported technology versions
The following table summarizes the officially supported versions of operating systems
and required software for this QF-Test version 4.2.2. Support for additional systems and
versions may be available on request but is not owed by QFS. Another option to get
support for older software can be to use one of the older QF-Test versions that are still
available for download at https://www.qfs.de/en/qf-test/download.html.
1.1.
System requirements
Technology
4
Version restriction
Operating system
7, 8, 8.1, 10, Server 2008 R2,
2012, 2016
Windows
Linux
Unix
No special restrictions.
Swing and JavaFX testing possible on Solaris, HP-UX, AIX,...
Swing, JavaFX and Web testing
possible.
macOS
Java
8 for QF-Test; 6, 7, 8, 9 or 10 for
the SUT
JDK/JRE
Swing
JavaFX
SWT
Internet Explorer
Firefox
Chrome
Comment
8, 9 and 10
3.7 - 4.7
All platforms.
All platforms.
Windows and Linux GTK
only.
For Eclipse/SWT
3.5 - 3.6 simply download
https://archive.qfs.de/
pub/qftest/swt_legacy.zip and
extract the contents into the
swt directory of your QF-Test
installation.
Web Browser
QF-Driver connection mode
11
38 - 43
Current stable Chromium ver- Windows only.
sion (57), part of the QF-Test
distribution
WebDriver connection mode, requires Java 8
Edge
Chrome
As supported by the included
ChromeDriver, currently the latest three Chrome versions, i.e.
64, 65, and 66
As supported by the included
GeckoDriver, i.e. the latest Firefox ESR plus the latest three
versions, currently 52esr, 56, 57
and 58
Firefox
Headless Firefox starting at version 56 (recommended min. 57)
Safari
AJAX Toolkits
Detailed list of supported toolk(769)
its in section 42.3
Table 1.1: Supported technology versions
1.2.
Windows Installation
1.2
5
Windows Installation
On Windows QF-Test can be installed in two variants.
1.2.1
Installing via the Windows setup file QF-Test-4.2.2.exe
This setup requires administrator privileges and follows the Windows standard of separating read-only program files from writable configuration files. If an older QF-Test
version is detected it is also possible to skirt Windows standards and install QF-Test
and its system configuration together at the place of the old installation.
4.2+
Windows compliant installation
Program files are saved to C:\Program Files\QFS\QF-Test or whichever target
directory you choose. The system configuration with writable data is stored in
%PROGRAMDATA%\QFS\QF-Test, irrespective of the selected target directory.
Note
%PROGRAMDATA% usually refers to the directory C:\ProgramData but the name may
vary depending on the Windows system. By default is is hidden in Windows Explorer.
A simple way to navigate to this directory is to enter %PROGRAMDATA% into the address
bar of Windows Explorer. In a PowerShell window use cd $env:PROGRAMDATA, in a
cmd console window cd /d %PROGRAMDATA% to change to the respective drive and
directory.
4.2+
Installation together with an existing QF-Test version
If an older QF-Test installation is found and there is no system configuration in
%PROGRAMDATA%\QFS\QF-Test yet, you choose to follow the Windows compliant
installation using %PROGRAMDATA% or to stick with the existing structure and install
QF-Test there.
In the first case, after selecting the target directory for the QF-Test program files, the
system configuration files are copied - just this once - from the existing installation to
%PROGRAMDATA%\QFS\QF-Test.
When installing into the existing structure, QF-Test is installed into that directory and
shares the system configuration that is already present there.
In both cases the directory %PROGRAMDATA%\QFS\QF-Test\qftestpath is added to
the system PATH and the program files qftest.exe and qftestc.exe are copied there. This
allows to start QF-Test from anywhere.
Independent of the installation choice both old and new QF-Test can be run in parallel. In case of the Windows compliant installation with %PROGRAMDATA% the old and
new system configuration are independent. If the old structure is kept, all versions
share the same system configuration. In the medium to long term we advise to move
to %PROGRAMDATA% because the old structure requires changing access rights in the
program directory, which is questionable. However, while migrating tests from QF-Test
1.2.
Windows Installation
6
4.1 to 4.2 it may be convenient to keep both versions close together. The move to a
Windows compliant installation using %PROGRAMDATA% can also be made in the course
of a later installation.
1.2.2
Unpacking
the
QF-Test-4.2.2-sfx.exe
self-extracting
archive
If you don’t have administrator privileges or want to keep all QF-Test files together in a
single place, unpack the archive QF-Test-4.2.2-sfx.exe at a suitable place. To do
so, copy the file to the desired location and execute it there. If 7-Zip is installed on your
system you can also right-click the archive to open and extract it with 7-Zip. This will
create a directory named qftest at the target location which we will refer to as the root
directory of QF-Test and that will also hold QF-Test’s system configuration files.
After unpacking the files you can run the program minisetup-noadmin.exe in the
sub-directory qftest-4.2.2. It will create associations for the file extensions
belonging to QF-Test and optionally a startup menu entry and a desktop icon for
QF-Test. If you have administrator privileges you can run minisetup-admin.exe
instead which applies the same settings for all users and also adds the directory
%PROGRAMDATA%\QFS\QF-Test\qftestpath to the system PATH and copies the
program files qftest.exe and qftestc.exe there.
If you’d rather have a fully portable installation instead, you can create a folder named
userdir in the qftest directory which will then serve as the user-specific configuration directory in place of %APPDATA%\QFS\QF-Test so that really all files belonging to
QF-Test are kept together in one place and no changes are made to the system.
1.2.3
Completing the installation and configuring Java
As the last step each of the setup programs will offer to configure the Java program for
QF-Test which is done with the help of a small dialog in which you can
make your choices.
With a portable installation you can run the program
qftest\qftest-4.2.2\bin\qfconfig.exe to achieve the same.
4.2+
A 64 bit Java Runtime Environment is installed with QF-Test into its installation folder.
It is recommended to use it. On a 32 bit Windows machine you must provide a Java
8 JDK or JRE yourself. You can either copy a 32 bit JRE 8 into QF-Test’s installation
directory as ...\qftest-4.2.2\jre\win32 and also chose to use the QF-Test Java
or you can install it as the system JRE or in any other convenient place and select that
as the alternative Java program.
The dialog also lets you adjust the maximum amount of memory to be used by QF-Test
with a default of 512 MB.
1.3.
Linux/Unix Installation
7
The third value to be configured is the language for QF-Test. Normally the language is
determined by the system settings, but you can also choose to always use the English
or the German version.
The values above are stored in the file launcherwin.cfg in QF-Test’s system configuration directory from where they are read by the qftest.exe start program. You can
run the configuration program any time from the system menu to change these settings.
1.3
Linux/Unix Installation
First select a convenient directory that will contain this release of QF-Test as well as
future updates. Common choices are /opt or /usr/local. Make sure you have write
access to this directory and change to it. When upgrading to a new QF-Test version,
use the same directory again.
Unpack the .tar.gz archive with tar xfzv QF-Test-4.2.2.tar.gz. This will
create a directory named qftest, which we will refer to as the main or root directory of
QF-Test. On a Linux/Unix system this also serves as the system directory holding the
system configuration files of QF-Test.
After unpacking a QF-Test archive for the first time, QF-Test’s root directory will hold only
the version-specific subdirectory qftest-4.2.2. When upgrading, a new subdirectory
for the current version will be added.
To finish the installation, change to the specific directory for the current QF-Test version with cd qftest/qftest-4.2.2 and run one of the two setup scripts provided.
setup.sh is a plain Bourne shell script while setup.ksh is written for the Korn shell.
On Solaris you must run ./setup.ksh. On other Unix systems both scripts should
work equally well but the Bourne shell version ./setup.sh is preferred. Note that if
you need to use the Korn shell scripts and want to create you own link or start QF-Test
directly from its bin directory, you need to use the qftest.ksh start script instead of
the plain qftest script.
The setup script will create the directories log, jython groovy and javascript
under QF-Test’s root directory unless they already exist. Additionally it will offer to
create a symbolic link from the /usr/local/bin directory (or /usr/bin if there is
no /usr/local/bin) to the respective Bourne shell or Korn shell run script for the
qftest command. You need to have write permission to the /usr/local/bin directory for the link to be created.
On Linux QF-Test should normally use its own JRE. Alternatively the default java program for QF-Test can be defined now. Either way it can be overridden at execution
(677)
argument. The setup script searches PATH
time with the -java <executable>
and proposes to use the first java program it detects. If you want to use a different
program or if none was found, you can enter one. The script determines the JDK
1.4.
macOS Installation
8
version automatically.
Next setting to perform is the maximum amount of memory to be used by QF-Test. As
default 512 MB are taken. Alternatively QF-Test can be started with the -J-XmxZZZm
command line argument, where ZZZ defines the memory in MB.
Finally the language for QF-Test can be configured. By default the language depends on
the system settings, but you can also choose to always use the English or the German
version. Note that this setting will affect all QF-Test users. Alternatively you can run QFTest with the -J-Duser.language=XX option using en for English or de for German.
Those of the above settings that differ from the default are written to the file
launcher.cfg in QF-Test’s root directory.
This file is read by the qftest
launch-script and also evaluated during an update of QF-Test.
1.4
macOS Installation
To install QF-Test on a macOS System, simply mount the QF-Test-4.2.2.dmg disk
image and copy the QF-Test app to your Applications directory (or any other folder)
and start it from there.
Note
Unfortunately, Apple’s JavaRuntimeSupport contains a bug, which is present in all
versions of Java for macOS since OS X 10.6. and which might lead to a crash of
QF-Test during start. The bug has been fixed in OS X 10.11 and in Java for OS X
2015-001. You can avoid the crash by installing the Java SE 6 package posted at
https://support.apple.com/kb/DL1572.
Note
To configure custom program arguments like memory used by QF-Test or the language
there is a separate options-page in the QF-Test options (General->Startup). You can
configure the settings there and they will then be applied after restarting QF-Test.
1.5
The license file
QF-Test requires a license file to run, which you should have received from Quality First
Software GmbH.
4.0+
Since QF-Test 4.0 the preferred way to activate or update your QF-Test license is by
way of the menu Help→Update license... .
The traditional way as described below is also still valid.
Place the license file into the system directory of QF-Test. On Windows, depending on
(5)
the type of installation, this will be %PROGRAMDATA%\QFS\QF-Test (see section 1.2 )
or the root directory of your QF-Test installation as on Linux. Make sure the file is named
license with no extension. Some mail clients try to guess the file type and add an
1.6.
The configuration files
9
extension on their own. When upgrading to a new QF-Test version you can simply keep
the license file provided that it is valid for the new version.
Note
For a complete list of the directories relevant to QF-Test please open the info dialog via
the menu Help→Info and select the ”System info” tab.
If you need to upgrade your license, for example to increase the number of concurrent
QF-Test instances or when upgrading to a new version, you will receive a file called
license.new from Quality First Software GmbH which is typically not a valid license
in itself but must be combined with your current license. To do so, proceed as follows:
• Place the file license.new in the same directory as the current license. Make
sure that this directory and the file license are writable by you.
• Start QF-Test in interactive mode. QF-Test will detect the license update, verify its
validity and offer to upgrade your license file.
• If you agree, the current license will be renamed to license.old and the new,
combined license will be written to license. When you are satisfied that everything is OK, you can remove the files license.old and license.new.
• If QF-Test doesn’t seem to recognize the license upgrade, make sure that the
timestamp of the file license.new is newer than that of the file license. Also
make sure that no other instance of QF-Test is running on your computer.
In case you need to specify a special name or location for the license file or work with
(683)
more than one license, this can be achieved with help of the -license <file>
(671)
argument as described in chapter 36 .
1.6
Note
The configuration files
For a complete list of the directories relevant to QF-Test please open the info dialog via
the menu Help→Info and select the ”System info” tab.
QF-Test saves all of its window configuration and those global options that represent
personal preferences together in a file named config located in the QF-Test user configuration directory which also holds run-logs for tests run in interactive mode, profile
directories for web testing and temporary files for editing and running scripts.
4.2+
On Windows the user configuration directory defaults to %APPDATA%\QFS\QF-Test
for new installations. If that directory doesn’t exist and you already used a QF-Test
version older than 4.2 on the same system that created the directory .qftest in your
home directory for the user configuration, QF-Test will continue to use that .qftest
directory.
1.7.
Starting QF-Test
10
You can manually move the content of the directory .qftest to
%APPDATA%\QFS\QF-Test and delete .qftest afterwards. QF-Test since version
4.2.0 will then use this directory only. You should not move the files if you still want to
use a version older than 4.2.0!
On Linux the user configuration directory is always ∼/.qftest.
On macOS it is located at /Users/<username>/Library/Application
Support/de.qfs.apps.qftest.
The personal config file is not read when QF-Test is run in batch mode (see section
(10)
1.7 ). Irrespective of the system default you can always specify an explicit location for
(693)
the user configuration directory as a whole with the -userdir <directory>
(693)
command line argument and just for the user config file with -usercfg <file> .
System specific options that need to be shared between users are saved in a file called
qftest.cfg in the system configuration directory which also serves as the home for
the license file, script modules, Java plugins and other customization files. On
Windows the location of the system configuration directory depends on the installation
(5)
variant (c.f. section 1.2 ). It is either located in %PROGRAMDATA%\QFS\QF-Test or in
the root directory of QF-Test.
On Linux and macOS the default system configuration directory is the root directory of
QF-Test.
The location of the system config file can be changed with the command line argument
(691)
and that of the entire system directory with
and -systemcfg <file>
(691)
-systemdir <directory> .
1.7
Starting QF-Test
QF-Test can be run in two modes. In normal mode QF-Test is the editor for test-suites
and run-logs and the control center for running programs, capturing events and execut(676)
ing tests. When run with the -batch
argument, QF-Test goes into ”batch” mode.
Instead of opening an editor window, the test-suites given on the command line are
loaded and executed automatically without the need for supervision. The result of the
(695)
(74)
test is reflected in QF-Test’s exit code , optional run-logs (see section 8.1 ) and re(167)
ports (see chapter 17 ).
The setup script for Linux/Unix offers to create a symbolic link from /usr/local/bin
to the qftest start script in the qftest-4.2.2/bin directory under QF-Test’s root
directory. That way you can simply enter qftest at the shell prompt to launch the
application.
On Windows a menu shortcut is created as well as an optional desktop icon. You can
either launch QF-Test from one of these or by double-clicking a test-suite or a run-log,
since these files are associated with the QF-Test application. To run QF-Test from the
console type qftest.
1.8.
Firewall Security Warning
11
When run from the command line, QF-Test offers a wide range of arguments for customization, like selecting the Java VM to use. These are explained in detail in chapter
(671)
36 .
In case different versions of QF-Test are installed at the same time, a specific version
can be started by calling the qftest executable directly from the respective
qftest-X.Y.Z/bin directory.
Mac
In case QF-Test is not starting up anymore because of some incorrect settings under
Options->General->Startup the default startup settings need to be restored. This can
be done through running the following two commands from a macOS shell terminal.
defaults write de.qfs.qftest /de/qfs/qftest/ \
-dict-add JVMOptions/ ’{"Xmx"="-Xmx512M";"Xms"="-Xms16m";}’
defaults write de.qfs.qftest /de/qfs/qftest/ \
-dict-add JVMArguments/ ’{"args"="";}’
Example 1.1: Resetting startup settings to defaults under macOS
1.8
Firewall Security Warning
On startup of QF-Test and/or the System Under Test (SUT) via QF-Test you might get
a security warning from the Windows firewall asking whether to block Java or not. As
QF-Test communicates with the SUT by means of network protocols, this must not be
blocked by the local firewall in order to allow automated testing.
Chapter 2
The user interface
This chapter explains the structure of QF-Test’s main window. When you are done
reading, it might be a good idea to run QF-Test and try things out. In the Help menu
there is an entry labeled Tutorial that should bring up your web browser with a
hands-on, learning-by-doing tutorial.
Should this fail because QF-Test cannot
determine your system’s standard browser, the tutorial can be found in the directory
qftest-4.2.2/doc/tutorial, where a PDF version is also available.
2.1
The test-suite
Automating a GUI test basically requires two things: control structure and data. The
control structure defines what to do and when to do it. The data for a test consists of
information about the SUT’s GUI components, the events that will be triggered and the
expected results.
QF-Test combines all of these into one data structure, a tree hierarchy that we call a testsuite. The elements of the tree are called nodes. Nodes can contain child nodes (often
just called children) and are themselves contained in a parent node (or just parent). The
root node of the tree represents the test-suite as a whole.
There are more than 60 different kinds of nodes all of which are explained in detail in the
(395)
reference manual . Some nodes are used as containers for data while others control
the execution of a test. All of them have their own unique set of attributes.
The attributes of the currently selected node are displayed to the right of the tree in a
detail view which can be toggled on and off via the View→Details menu item.
2.1.
The test-suite
13
Figure 2.1: Structure of a test-suite
The image above shows an example for a simple test-suite. The attributes of the node
named ”Simple test” can be edited in the detail view to the right.
(395)
The basic structure of a test-suite and thus the child nodes of the Test-suite root node
(405)
(398)
(427)
is fixed. An arbitrary number of Test-set and Test-case with or without Dependency
(472)
(425)
(655)
nodes are followed by the Procedures , Extras and Windows and components nodes.
(470)
(463)
The Procedures node holds Packages and Procedures which are explained further in
(91)
section 9.5 . The Extras node is a kind of playground or clipboard where all kinds
of nodes can be added for experimentation or temporary storage. The windows and
(634)
(644)
components of the SUT’s user interface are represented as Window and Component
2.2.
Basic editing
14
nodes which are located below the Windows and components node.
To get detailed information about a node or one of its attributes, click on it with the
right mouse button and select What’s this? from the context menu. This will bring up a
browser displaying the corresponding section of the reference manual.
2.2
Basic editing
Editing a test-suite falls into two categories: operations like Cut/Copy/Paste on the tree’s
nodes and changing the attributes of a node. The latter
can be done either by editing the
Return , or by bringing up a dialog
fields in the detail view and selecting OK or pressing
for the selected node with Edit→Properties or Alt-Return and changing the values
there. If you change some values in the detail view and forget to press OK before
moving the selection to another node, QF-Test will pop up a dialog with the changed
values, asking you to either confirm your changes or discard them. This feature can be
(322)
turned off with the option Ask before implicitly accepting detail modifications .
Some non-obvious
key-bindings
may come in handy when editing multi-line text
attribute:
Ctrl-TAB and Shift-Ctrl-TAB move the focus out of the text field, while
Ctrl-Return is a shortcut to select the OK button.
An extremely useful feature is the Edit→Undo function (Ctrl-Z ) which will take back
any kind of change made to the test-suite, including recordings or use of the replace
dialog. Changes are undone step by step. If you find you went too far and undid more
than you wanted, you can use Edit→Redo (Ctrl-Y ) to undo the undone. The number
of steps that can taken back are limited only by available memory and can be configured
(323)
with the option Number of undo levels per suite (default 30).
2.2.1
Navigating the tree
Though the key-bindings for tree navigation are similar to those of most tree components, it won’t hurt to mention them here. Besides, QF-Test comes with a few nonstandard bindings that may come in handy.
The cursor keys are used for basic navigation. Up and Down are obvious. Right either expands a closed node or moves down one row while Left closes an open node
or moves to its parent.
QF-Test’s trees support a special variant of multi-selection. Multiple discontinuous regions can be selected, but only among siblings, i.e. children of the same node. If
multi-selection across the whole tree were allowed,
cutting
and pasting nodes would
become a real brain-teaser. Keys to try are Shift-Up and Shift-Down to extend the
2.2.
Basic editing
15
selection, Ctrl-Up and Ctrl-Down to move without affecting the selection and Space to toggle the selection of the
current
node.
Similarly,
mouse-clicks
with
Shift extend the
selection while clicks with Ctrl toggle the selection of the node being clicked on.
Special bindings include Alt-Right and Alt-Left which recursively expand or collapse a
node and all of its children. Alt-Down and Alt-Up can be used to move to the next or
previous sibling of a node, skipping the intermediate child nodes.
QF-Test keeps a history of recently visited nodes. Ctrl-Backspace will take you back to
the previously selected node. Also worthy of note are Ctrl-Right and Ctrl-Left which
will scroll the tree to the right or left if it doesn’t fit its frame.
2.2.2
Insertion marker
When inserting a new node or pasting in a copy of some other nodes, the insertion
marker shows the place where the nodes will end up.
Figure 2.2: Insertion marker
Nodes are always inserted after the selected node. If the selected node is expanded,
the new node is inserted as the first child of the selected node, otherwise it becomes
a sibling of same. This behavior will take a little to get used to, especially for long-time
users of the Windows explorer. However, there is no other way to insert a node at a
(15)
definite position. In the example shown in figure 2.2 above, a new node would be
(538)
inserted as the first child of the sequence called ”Events”, just before the Mouse event .
2.2.3
Moving nodes
Nodes can be copied and pasted or moved around within a
or to another
suite.
test-suite
The standard keyboard shortcuts for cut, copy and paste, Ctrl-X , Ctrl-C and Ctrl-V are
available as well as entries in the context menu.
2.0+
Alternatively, nodes can be moved using standard Drag&Drop operations. The default
2.2.
Basic editing
16
operation will move the selected node(s). If the CTRL key is held down during the drop,
the nodes are copied instead.
While dragging the mouse over the tree of a test-suite, the insertion marker shows
where the nodes will be dropped when the mouse button is released and whether the
operation is allowed. A green marker signals a legal operation, a red marker an illegal
one. Nothing will happen if the nodes are dropped on an illegal target position.
During the drag you can expand or collapse nodes by dragging the mouse cursor over
the expansion toggle and keeping it there for a moment. That way you can easily navigate to the desired target location without interrupting and restarting the drag.
The Drag&Drop operation can be aborted at any time by pressing Esc .
2.2.4
Transforming nodes
Some nodes can be transformed into different node types, which is a lot more convenient
than first creating the desired target node and then copying over the required attributes.
(415)
(417)
(502)
Examples of interchangeable nodes are Sequence
and Test-step
or Server script
(504)
and SUT script . The transformation of a node is possible only if its childnodes and
its current position in the tree are also valid for the desired target node. The potential
transformation targets can be selected from the entry Transform node into in the context menu. If the entry is not available there are no valid target nodes. In that case,
(425)
moving the node to the Extras node first may help.
You can find more details about the conversion mechanism under Details about
(907)
transforming nodes .
2.2.5
Tables
In various places QF-Test employs tables to view and edit a set of values, e.g. when
(62)
(560)
defining variables or for checks of multiple elements.
2.2.
Basic editing
17
Figure 2.3: Example table
The buttons above the tables have the following keyboard shortcuts and effects:
Shift-Insert Insert a new row.
Shift-Return , Alt-Return Edit a row. Opens a dialog with fields for every cell of the selected row.
Shift-Delete Delete the selected row.
Shift-Ctrl-Up Move the selected row up by one.
Shift-Ctrl-Down Move the selected row down by one.
Some tables also offer the ability to add and remove columns and edit the
column title.
For these, the following additional buttons are available:
Insert a new column.
Delete the selected column.
Edit the title of the selected column.
To enter a value directly into the selected cell just start typing. This way you overwrite
the
current value of the cell. To edit the current value, either double click the cellor press
Escape .
F2 . To finish editing press Return , to cancel and restore the old value press If you try to enter an invalid value the cell’s border will turn red and you can’t accept the
value.
Multi-selection of table rows is supported via mouse-clicks with Shift/Ctrl and
2.2.
Basic editing
18
Shift/Ctrl-Up/Down . Cut copy and paste of the selected rows is done with Ctrl-X/C/V .
Pasting is restricted to tables with a similar column structure.
In the table’s context menu additional actions might be available, e.g. show line numbers, locate component, etc.
A mouse click in a column header will activate sorting of table rows. A double-click in a
column header will resize the column to fit the largest value in the column or opens the
editor for the header text (data table).
2.2.6
Packing and Unpacking
During test development it is often necessary to move several nodes into a new parent
node. A typical situation could be the re-factoring of procedures to re-organize them in
packages or to wrap a workflow into a Try/Catch block.
For such requirements QF-Test allows the user to pack nodes into others. This can be
achieved by selecting the nodes to pack, right-clicking and selecting Pack nodes and
the desired parent node.
QF-Test also allows the user to unpack such nodes and remove their parent. This can
be used to remove unnecessary packages or test-sets from the structure or to dispense
with sequences or Try/Catch blocks that are no longer required. For unpacking right-click
the node to unpack and select Unpack nodes .
Note
The packing and unpacking actions are only shown in the menu if the desired target
structure is legal.
2.2.7
Sorting Nodes
QF-Test allows sorting nodes. This can be achieved by clicking at a node with the
right mouse button and selecting Sort child nodes . Alternatively you can also select
multiple nodes, perform a right mouse click and then choose Sort nodes , which will
sort the current selected nodes.
To guarantee a better overview the sorting algorithm puts ciphers prior to capital letter
and those prior to small letters. Sorting doesn’t modify the base structure of QF-Test
nodes. It also follows the rule to keep Package nodes always prior to Dependency nodes
and those always prior to Procedure nodes.
Note
The base structure of a test-suite will not be altered during sorting. You can sort testcases or procedures but the Procedures node will always stay prior to the Windows and
components node.
2.3.
Advanced editing
2.3
19
Advanced editing
This section explains how to use the more advanced editing techniques such as
search/replace and multiple views on the same test-suite.
2.3.1
Searching
QF-Test provides two kinds of search operations, a general search through all nodes
and attributes of a test-suite or run-log and in incremental search through the contents
of a text area, including script terminals or program output.
General search
Though search and replace operations in QF-Test have much in common, there are
significant differences, especially in the scope of the operation. Searching normally
starts at the selected node and traverses the whole tree depth-first to the end. After
asking for confirmation the search continues from the root of the tree to the original
start of the search so each node is traversed exactly once. This is not unlike search
operations in common text processors and should be intuitive to use.
By default QF-Test shows the search dialog in ’simple’ mode, which allows searching for
any appearance of a given text.
Figure 2.4: The simple search dialog
For a more specific search QF-Test allows limiting the search to specific attributes, node
types or specific states of nodes. Therefore you have to switch to the ’advanced’ mode
2.3.
Advanced editing
20
by clicking the ’Switch mode’ button in the toolbar of the search dialog.
Figure 2.5: The advanced search dialog
By default QF-Test will search all attributes in all kinds of nodes for the requested string.
Use the ”In attribute” option to limit the search to a specific attribute and/or the ”Node
type” option to limit it to nodes of a specific kind. The other options should be self(720)
explanatory. Regular expressions are explained in section 40.4 . If ”Match whole
attribute” is selected, a search for the word ”tree”, for example, will not match an attribute
value of ”treeNode”. Activating options in ”Only nodes with the following states” limits the
search to nodes that have all of the activated states, e.g. a green mark and a breakpoint.
The option ”Scope of operation” tells QF-Test where to search for the given expression,
2.3.
Advanced editing
21
below the selected node(s), in the current test-suite or in all currently opened suites.
Note
(540)
To search for values of boolean attributes like Replay as ”hard” event , use ”true” or
”false” (no quotes). If you want to search for an empty value you have to check ”Match
whole attribute”.
If the search is successful, the resultant node is selected and a message in the status
line displays the name of the attribute that contains the value.
3.4+
As already mentioned the searching process usually starts from the currently selected
node. In case you want to select other nodes during your search process you can
continue the previous search by using the ”Search continue” button.
Once you have closed the search dialog you can still continue the search pressing F3 .
You can even trigger the same search from a new node pressing Ctrl-G .
A very useful feature is the ability to quickly locate all Procedure call nodes that call a
given Procedure or all event nodes that refer to a given Component node, etc. Simply
select the entry Locate references... from the context menu of a node that can be
called or referred to. This will show a new frame showing all available references of it.
You can reach the node in the test-suite via a double click at the row in the list.
2.3.
Advanced editing
22
Figure 2.6: Result list for ’Locate references’
3.1+
It is also possible to get a list of all found nodes via pressing the ”Show result list” button
in the search dialog. From this dialog you can then reach any single node in your testsuite.
Incremental text search
3.1+
In addition to searching the tree, components containing text like terminal areas or respective attributes in the details view can be searched independently by use of QF-Test’s
incremental search feature. This feature can be invoked
either by selecting Search...
from the component’s context menu or by pressing Ctrl-F when the component is selected and owns the keyboard focus. Then the incremental search popup dialog appears
at the upper right corner of the respective component. The figure below shows an incremental search for the terminal with highlighted search hits.
2.3.
Advanced editing
23
Figure 2.7: Incremental search
The search can be limited to a certain part of the contained text by selecting the region
to be searched
and invoking Search in selection... from the component’s context menu
or pressing Ctrl-Shift-F .
Beside this, the way the incremental search works as well as the available options
should be self-explanatory.
2.3.2
Replacing
Once you understand how the scope of the replace operation differs from searching,
the replace dialog should be just as intuitive to use as the search dialog. When the
replace operation is in progress, you have a choice of replacing one match at a time
or all matches at once. To avoid unexpected results when selecting the latter, there
needs to be a way to limit the nodes that will possibly be affected. To that end, replace
operations can be limited to the currently selected nodes and their direct or indirect child
2.3.
Advanced editing
24
nodes. For a replace operation that covers the whole tree, either select the root node or
choose the respective ”Scope of the operation” option in the dialog.
Figure 2.8: The replace dialog
The options are identical to the ones for searching. When ”Match whole attribute” is
turned off, multiple replacements within one attribute are possible. When replacing ”a”
with ”b” for example, ”banana” would change to ”bbnbnb”. Be sure to read section
(720)
40.4 about how to use regular expressions for replacing.
2.3.
Advanced editing
25
If the search is successful, the resultant node is selected and a confirmation dialog is
brought up that shows the target attribute and its value before and after the change.
Figure 2.9: The replace query dialog
It offers the following choices:
• When Replace is selected, the attribute’s value is changed and the search carries
on, showing the query again for the next match.
• Replace all means change this value and all the rest of the matches in one go
without asking again.
• Skip leaves this attribute unchanged. The search continues and the query dialog
is shown again in case of another match.
• Obviously Cancel ends the replace operation.
If you know what to expect you can skip the query entirely by selecting Replace all in the
replace dialog. After the attributes have been changed, the number of affected nodes is
shown in a message dialog.
3.1+
After performing the actual replacement QF-Test will show a list of all touched nodes.
You can also open a list of all nodes, which will be touched before the actual replacement
pressing the ”Show result list” button in the replace dialog.
Note
Whether values are replaced one by one or all at once also affects the way the undo
function will take these changes back. All changes of a Replace all operation are taken
back in one step, while single changes are undone one at a time.
2.3.3
3.3+
Complex searches and replace operations
Sometimes a simple search is not enough. Imagine, for example, that you want to set a
2.3.
Advanced editing
26
Timeout of 3000 milliseconds for all text checks on a certain component. You know the
component’s QF-Test ID, but if you search for that QF-Test ID you will also find events
and other kinds of checks referencing it. And if you search for the node text ’Check text’
you will find all Check text nodes, not just those for the given component.
Instead of providing several combinable levels of search criteria QF-Test offers complete
flexibility through its marks. First perform a search for your first criterion, e.g. the node
text and select
’Show result list’. In the resulting dialog select all entries in the table by
pressing Ctrl-A , press ’Set mark’ to assign the blue mark to all result nodes and close
the dialog. You can now perform a second search or a replacement with the scope
limited to nodes with a given mark. In our example you would perform a replacement
of the empty string with ’3000’ on all Timeout attributes with the search scope set the all
nodes with the blue mark in the whole tree.
2.3.4
Multiple views
It is possible to open multiple views that show different parts of the same tree structure
simultaneously. This can be useful when managing large test-suites or to compare the
attributes of different nodes.
Additional views are opened via the View→New window... menu item. The current
node will be the root node for the new view. Additional views are similar to the primary
views, but with a limited range of menus.
Chapter 3
Quickstart your application
This chapter provides instructions on how to quickly set up your application as the SUT
(System Under Test).
In order to make you application recognized by QF-Test as SUT it basically needs to be
started out of QF-Test. There are a number of special process nodes available within
the Insert→Process nodes to perform this task but the straight forward way is to use
the Quickstart Wizard as described below. For those with an aversion to wizard dialogs,
(699)
the manual way is explained at section 38.1 .
A precondition for testing Java based SUTs is that QF-Test can hook into the GUI toolkit:
Swing
For Swing/JavaFX or combined Swing/JavaFX and SWT applications QF-Test hooks
into the Java’s JVM Tool Interface. Normally QF-Test can do this directly. Only for
some non-standard JDKs it may be necessary to instrument those first. See JDK
(710)
instrumentation for details if necessary.
JavaFX
For JavaFX applications and respective combinations the connection works exclusively
(354)
via the QF-Test agent. Please ensure the option Connect via QF-Test agent is activated.
SWT
For Eclipse/SWT based applications, an instrumentation of the SWT library is necessary. The Quickstart Wizard, which is described below, will support you in the task to
add the necessary step to the setup sequence. Background information can be found
(714)
in section 39.3 .
Web
Web application testing does not require instrumentation but there are some constraints
(148)
to consider that are explained in chapter 14 .
3.1.
Quickstart Wizard
3.1
28
Quickstart Wizard
2.0+
3.1.1
Setup sequence creation
With the Quickstart Wizard QF-Test offers a convenient utility for creating a startup sequence for your application.
You can open the Quickstart Wizard via the Extras→Quickstart Wizard... menu item.
Please follow the steps which should be self explanatory.
Figure 3.1: Quickstart Wizard
As result the Wizard delivers a startup sequence under the ”Extras”, as shown in the
following figure:
3.1.
Quickstart Wizard
29
Figure 3.2: Startup sequence created by the Quickstart Wizard
The created setup sequence varies depending on the specific type of the application.
But all of them follow the same standard. At the beginning you will find a Set variable
node which specifies the name of the SUT client for QF-Test. This node is followed by a
Wait for client to connect node that checks whether its necessary to start the application.
The subsequent If evaluates the result of its predecessor and contains the start steps for
your application. The actual launch takes place in the start node which is specific to the
type of application. That node is followed by an other Wait for client to connect node which
ensures that QF-Test connects to your application during the startup procedure. (Details
(699)
about the different start node types and attributes can be found in section 38.1 .)
Swing
JavaFX
For Swing or JavaFX based applications the setup sequence typically is straight forward
except for Applet testing in a browser or Java WebStart applications which uses some
(112)
special procedures from the standard library .
SWT
For SWT based applications an additional procedure call node for SWT
(714)
instrumentation is added.
Web
The standard startup sequence for web includes some additional nodes for setting variables, initializing browser cache and cookie settings and possibly install an AJAX toolkit
(148)
resolver. See chapter 14 for further information about starting a web based SUT.
3.1.2
Executing the setup sequence
The setup sequence can be executed directly after creation via selecting the green setup
sequence node in the tree and pressing ”Replay” toolbar button
.
When executing the setup sequence your application should start up and the ”Start
recording” button
in the QF-Test toolbar should become activated which indicates
that QF-Test is properly connected to the SUT.
Now you are able to record and replay your first test sequences as described in chapter
(33)
4 . There is also a learning by doing tutorial available from the QF-Test help menu
which guides you through all features of QF-Test.
In case you are facing an error message or the red ”Start recording” button stays inactive, please proceed with the following paragraph.
3.1.
Quickstart Wizard
3.1.3
30
Troubleshooting
If your application (or the browser window in case of web testing) doesn’t come up at
all:
• The error dialog QF-Test typically displays should provide a first indication.
• Please look for error messages in the terminal window. If there is no terminal
window visible in the bottom area of QF-Test, it can be activated through the menu
item View→Terminal→Show . Additional information about program output can
(31)
be found in section 3.2 .
• Be sure to double-check the attribute values in the setup sequence nodes are
correct. Possibly a typo has crept in somewhere. Details about the different start
(699)
node types and attributes can be found in section 38.1 .
(702)
Swing
• If you are testing an Applet (possibly on Windows 8) please see section 38.1.3
for additional information.
Web
• As browser development cycles i.e. those of Firefox tend to shorten, be sure the
installed browser is supported by the QF-Test version you are using. The terminal
(977)
output should show a respective error message. See release notes in appendix
B for latest supported browser versions. Possibly you need to update QF-Test to
(148)
for further
a later version or temporarily use another browser. See chapter 14
information.
If the SUT gets visible but QF-Test is not able to connect to the client
(666)
(ClientNotConnectedException ):
• Please double-check the terminal output content (see also above) for possible
error messages.
• In case the case the red record button in the toolbar gets activated after the error
(529)
message occurred, the timeout value in the Wait for client to connect node needs
to be increased.
(355)
Swing
• Please ensure the option Connect without JDK instrumentation (Swing) is acti(710)
vated. If you are using a non-standard JDK please check out the section 39.1
chapter.
Swing
• If you are testing an Applet, please see section 38.1.3
There are a number of special cases to consider.
JavaFX
• For JavaFX applications please ensure the option Connect via QF-Test agent
activated.
(702)
for additional information.
(354)
is
3.2.
SWT
Program output and the Clients menu
31
• For an Eclipse/SWT application first make sure that you specified the correct ap(74)
plication directory. You may want to take a look at the run-log (see section 8.1 )
to see if any warnings or errors were logged during execution of the Procedure
qfs.swt.instrument.setup.
• Check the run-log in general for possible additional error indications (see section
(74)
8.1 ).
After possibly having adapted your test-suite or settings retry executing your setup sequence. If you are not getting any further you might want to consider trying a sample
test-suite from the tutorial or you contact our support.
3.2
Program output and the Clients menu
The standard output and error streams of all processes started by QF-Test are captured
and stored in the run-log under the node that represents the respective starter node. In
this QF-Test does not distinguish between SUT clients and arbitrary processes or shell
(514)
(524)
scripts started with a Start process or Execute shell command node.
The main window contains a shared terminal view that shows the output of all processes
started by a test that was run from this window. The View→Terminal sub-menu holds
items to configure whether this terminal is visible, whether the tree or the terminal should
use the are in the lower left corner, whether long lines are wrapped and whether it is
automatically scrolled to the end when new output arrives. Other items let you clear the
terminal or save its contents to a file. The maximum amount of text that the terminal
(357)
holds is configurable in the option Maximum size of shared terminal (kB) .
In addition to the shared terminal, for each active or recently terminated process there is
an individual terminal window that shows its output. These individual terminal windows
can be opened from the Clients menu. The shared terminal’s intention is to provide
visual feedback whenever new output arrives, while the individual terminals are better
suited for actually studying that output.
Active processes can also be stopped with the help of the Clients menu, either individually in the respective sub-menu or all at once with Clients→Stop all .
The number of terminated clients that are kept in the Clients menu is set with the option
(353)
Number of terminated clients in menu . If your processes generate lots of output and
you are low on memory you may want to reduce that number.
Note
The Clients menu also serves well in case you are not sure which specific QF-Test
product you need to purchase. The GUI technologies used by your applications are
shown in ’[ ]’ next to the active client name. The example below shows two clients using
Java swing and web which suggests to buy a QF-Test/swing+web license.
3.3.
An application started indirectly from an already connected SUT
32
Figure 3.3: GUI technology information
3.3
An application started indirectly from an already
connected SUT
If a second Java VM is started from an already connected SUT, QF-Test will recognize
this as an indirect connection attempt from a child process of the first SUT and automatically assign an artificial client name to this new SUT. The name is created by appending
’:2’ to the client name of the parent SUT, signifying that this is the second process for
this client. Yet another Java VM started by either of these SUTs would get ’:3’ appended
to the original client name unless the second process was already terminated so the ’:2’
was again free for use.
To summarize, the sequence for connecting to an indirectly started SUT typically consists of an event node that triggers something like a button click or menu selection,
(529)
causing the SUT to launch the second SUT, followed by a Wait for client to connect
node for the ’:2’ extended client name.
Chapter 4
Capture and replay
Once the SUT is up and running under QF-Test, the next step is to record sequences of
events and play them back.
4.1
Recording sequences
In order to record a sequence of events in the SUT, the SUT must have been run from
(27)
QF-Test (see chapter 3 ) and the connection between QF-Test and the SUT must be
established. A visual indicator of this is the color of the record button which turns red
when it is enabled.
Figure 4.1: Disabled and enabled Record button
To record a sequence, simply start recording be pressing the record button
or selecting Record→Start . Then switch to the SUT, execute a few commands, switch back
to QF-Test and stop the recording with the stop button
or Record→Stop . The
recorded events will be added to the test-suite, either directly at the position indicated
(15)
(415)
(425)
by the insertion marker or as a new Sequence under the Extras node, depending
(333)
on the setting of the Insert recording at current selection option. Pause the recording
with Record→Pause or the pause button
if you need to execute some steps in the
SUT that should not be recorded and you don’t want to stop and restart the recording.
Recording mode can be started and stopped directly in the SUT by use of the Hotkey
4.2.
Running tests
(333)
for recording
34
. Default key is F11 .
(644)
Any Components referred to by the newly recorded events are added automatically to
(655)
the Windows and components node if they are not there already.
There are many options that influence the way QF-Test records events and how it treats
(333)
the components of the GUI. All of these are explained in detail in section 33.2 of the
reference manual. Once you are familiar with QF-Test you should take the time to skim
through it.
Here’s some general advice for recording:
• Record short sequences at a time.
• After recording, take a look at the sequence, try to understand what you got and
whether it represents the actions you took.
• Edit the sequence to remove unnecessary events, especially those at the beginning and end caused by switching windows. QF-Test has excellent filters that
should catch nearly all of these, but some might remain and have to be removed
manually.
• Finally, try out the new sequence to see whether it replays OK. Then you can
cut/copy/paste as needed to integrate it into larger parts.
Mac
For SUT’s running on macOS QF-Test disables use of the screen menu bar and activates normal menu bar behavior like on other platforms. This is due to the fact that
QF-Test cannot fully access the screen menu bar which prevents proper capture/replay
of menu actions. In case the typical Mac screen menu bar behavior is necessary for any
reason, this can be forced by adding the line qfs.apple.noScreenMenuBar=false
to the file qfconnect.properties that is located in QF-Test’s root directory. After
restarting the SUT the screen menu bar is supposed to work as normal on Mac.
4.2
Running tests
To run some tests, select the node or nodes to execute and press Return or the play
button
or select Run→Start . QF-Test will mark each node with a small arrow as it
is executed and also show progress messages in the status bar. This can slow down
(350)
execution a little and can be turned off with the options Mark nodes during replay and
(350)
Show replay messages in status line .
When the test is finished, the result is shown in the status bar. If things are fine you
should see ”No errors”, otherwise the number of warnings, errors and exceptions is
4.2.
Running tests
35
shown. Additionally, a message dialog is shown in case of errors or exceptions to make
sure you don’t miss these.
As for recording there are many options that influence the replay of tests. Some of these
are only for convenience while others have a major impact on the outcome of the tests.
(349)
Be sure to read section 33.3 some time to familiarize yourself with these.
To abort execution before the test is finished, press the stop button
or select
Run→Stop . You can also suspend execution temporarily via the pause button
or
(74)
by selecting Run→Pause . This will also enable the debugger (see chapter 8 ). To
continue, press pause again.
While a test is run at full speed it can be tricky to stop or interrupt it, especially when the
mouse cursor is actually moved across the screen or the SUT’s windows are raised on
(349)
(the default is Alt-F12 ).
every event. To regain control, press the (Don’t) Panic key
This will pause all running tests immediately. To continue, press the same combination
again.
While building a test-suite you will often want to execute some sequences to get the
SUT to a point where you can continue recording. Sometimes you may want to skip
certain nodes at this stage because they don’t get you where you want, but you don’t
want to delete them or move them to some other place. In that case use the
Edit→Toggle disabled state menu item to disable the node(s). When you want to use
them again later you can re-enable them.
The current error state during replay as well as the final result is shown in the status
line at the bottom of the QF-Test main window. The visibility of this status line can be
controlled via View→Show status line .
(405)
(398)
(87)
In case Test-set
or Test-case
nodes (section 9.2 describes their usage) are executed the status line also contains relevant result counters from the following list.
4.3.
Recording checks
Counter Icon
36
Description
Total number of test-cases. This counter value starts with a ’>’ symbol in case there
are skipped test-sets.
Number of test-cases with exceptions.
Number of test-cases with errors.
(402)
Number of test-cases with expected errors. Expected to fail if...
marks a test-case
expected to fail.
Number of successful test-cases.
(402)
Number of skipped test-cases. A test-case is skipped when its (optional) Condition
fails. This counter value starts with a ’>’ symbol in case there are skipped test-sets.
(409)
Number of skipped test-sets. A test-set is skipped when its (optional) Condition
fails.
Number of not implemented test-cases. A test-case is not implemented when it
doesn’t contain nodes that were executed during the test-run.
Number of executed test-cases.
Percent test-cases passed.
Table 4.1: Test result counter in the status line
The final test result counts also appear in the report which can be created for any test
(167)
run. Reports are discussed in chapter 17 .
Note
(67)
The counter values above can also be accessed as variables during the test-run.
(869)
A TestRunListener
can help to keep track of counter values and trigger dependent
actions.
4.3
Recording checks
Though it can be quite entertaining to record sequences and watch the magic dance
of the SUT as they are played back, the task at hand is to find out whether the SUT
(560)
actually works as expected. This is where checks come into play. The most common
(561)
check, the Check text node, reads the text displayed by a component, e.g. a text field,
and compares it to a given value. If the values differ, an error is signaled.
(560)
How checks work is explained in detail in the Reference manual . There is a range
(581)
available from simple text check to advance Check image
and even custom check
(856)
types can be implemented. Here we are going to concentrate on the most convenient
way to create checks, which is to record them.
While recording, the SUT is in record mode, which means that all events are collected and sent to QF-Test. With the help of the check button
or by selecting
Record→Check you can bring it into check mode, recognizable through the different
4.4.
Recording components
37
mouse cursor. In this mode, recording events is suspended. Instead, the mouse cursor
is tracked and the component under it is highlighted. When you click on the component,
a check for the component is recorded using the value that is currently displayed. To get
back to record mode, select the check button or menu item again.
(560)
There are different kinds of checks
that can be performed. Which kinds of checks
are applicable depends on the selected component. Some components don’t display
(561)
any text, so a Check text node doesn’t make sense for, say, a scroll bar. Clicking on
a component with the right mouse button while in check mode brings up a menu of
applicable checks for this component. Select one of the items to create the respective
check node. Clicking with the left mouse button always records the default check, which
is the topmost one in the popup menu.
If you hold down the Shift or Ctrl key while clicking with the right mouse button, the
check menu will stay open after making a selection. That way, you can easily record
multiple kinds of checks for the same component.
Checks integrate well with events and you’ll soon develop a recording style a’la click,
click, type, click, check, click, click, check... Having to switch back and forth between
QF-Test and the SUT every time you want to create a check can be a real pain. That
(334)
is where the Hotkey for checks option comes into play. It defines a key
which toggles
the SUT between record mode and check mode. The default value is F12 , but if this
key has some defined meaning for your application you can change it to whatever you
like. To record a sequence of interspersed events and checks, simply start recording,
switch to the
SUT and record the sequence. Whenever you want to add some checks,
just press F12 (orwhatever you defined), record the checks, then switch back to record
mode by pressing F12 again and continue. This way you can work with the SUT for the
whole sequence and need to switch back to QF-Test only to stop the recording.
One word of warning should be repeated: Don’t let this convenience tempt you into
recording extremely long sequences. When something changes that causes such a
sequence to fail you will be hard put to find out what went wrong and how to cope.
4.4
Recording components
As already described component information is automatically stored when recording
events or checks. Nevertheless there are situations where capturing of just components
proves useful.
To activate component recording mode you simply need to press the record components
or select Record→Record components from the menu. Then switch to the
button
SUT window where you will notice a special behavior in which the component below the
mouse cursor is highlighted.
4.5.
Recording of HTTP Requests (GET/POST)
38
Clicking with the left mouse button on a component will record the single component
while pushing the right button instead will pop up a menu with choices to record the
nested components as well or all components in the window. Multiple components can
be captured in this way. Then switch back to QF-Test and release the record components button
or deactivate Record→Record components . Now the recorded
(644)
component information is stored in shape of respective Component nodes under the
(655)
Windows and components node.
Component recording
mode can be alternatively controlled by a configurable hotkey.
(340)
Default binding is F11 , the option that configures it is Hotkey for components .
Pressing F11 (default setting) in the SUT window starts component recording. Further
(340)
details can be found in the documentation for the option Hotkey for components .
Note
Only one test-suite at a time can receive the recorded components. If more than one
test-suite is open and each is shown in an individual window, i.e. workbench mode
is deactivated, either the test-suite
in which the recording is stopped (toolbar button
or menu) or - when using F11 , the test-suite that can be selected via the menu item
Record→Suite is receiver for recording will receive the components.
The component recording feature can also be used to quickly locate a component independent of whether it has been recorded before. When creating event or checks by
(539)
hand or changing the target component, the QF-Test component ID attribute needs to
be specified. When a component is recorded, its QF-Test ID is saved in the clipboard
and can be pasted directly into the QF-Test component
ID field with
Ctrl-V . You can
also jump directly to the Component node with Shift-Ctrl-Backspace or by choosing the
Edit→Select next node menu item or clicking the respective toolbar button.
The popup menu which appears when clicking with the right mouse button on a component also contains an entry Show methods which allows of some kind of component
(47)
inspection (see section 5.5 ).
Components play a central role in the structure of a test-suite which is explained further
(40)
in chapter 5 .
4.5
Recording of HTTP Requests (GET/POST)
Web
In order to record a (GET/POST) request sent by the SUT, the SUT must have been
(27)
launched from QF-Test (see chapter 3 ) and the connection between QF-Test and the
SUT must be established.
While recording, the SUT is in record mode, which means that all events are collected
and sent to QF-Test. With the help of the HTTP-request button
or by selecting
Record→Record HTTP Requests you can bring it into request recording mode. In
4.5.
Recording of HTTP Requests (GET/POST)
39
(33)
contrast to Recording sequences all GET/POST-request send by the web browser are
saved as http-request nodes in this special recording mode. To get back to record mode,
select the HTTP-request button or menu item again.
(375)
In section Web options the ability to change the type of the recorded request is de(631)
scribed. By default Browser HTTP request is recorded. This Request type is likely used
to automate large web form inputs, the use of separate input nodes will be avoided.
The form data will be submitted within the browser, so that the response will be shown
as well. At this point the test execution could be continued. In opposition to this the
(626)
Server HTTP request will be directly submitted through QF-Test without the need of a
running browser. The response is also only available in QF-Test and a eventually running browser will not be affected.
All attributes of an accordingly recorded request node as well as the parametrization of
(626)
requests are explained in detail in the HTTP Requests section of the reference part
of this manual.
Chapter 5
Components
Though
they
often
go
unnoticed,
at
least
until
the
first
(662)
(644)
ComponentNotFoundException occurs, the Component nodes are the heart of
a test-suite. Everything else revolves around them. Explaining this requires a little
side-tracking:
5.1
GUI component hierarchy
The GUI of an application consists of one or more windows which hold a number of
components. The components are nested in a hierarchical structure. Components that
hold other components are called containers. As QF-Test itself is a complex application,
its main window should serve well as an example:
5.1.
GUI component hierarchy
41
Figure 5.1: Components of a GUI
The window contains a menu bar which holds the menus for QF-Test. Below that is
the toolbar with its toolbar buttons. The main area employs a split pane to separate
the tree view from the details. The tree view consists of a label (”Test-suite”) and the
tree itself. The detail view contains a complex hierarchy of various components like
text fields, buttons, a table, etc. Actually there are many more components that are not
obvious. The tree, for example, is nested in a scroll pane which will show scroll bars if
the tree grows beyond the visible area. Also, various kinds of panes mainly serve as
containers and background for other components, like the region that contains the ”OK”
and ”Cancel” buttons in the detail view.
5.2. Components in QF-Test
42
SWT
In SWT the main GUI components are called Control, Widget or Item. Unless explicitly stated otherwise the term ”component”, as used in this manual, also applies to
these and not only to AWT/Swing/JavaFX Components.
JavaFX
The same is valid for JavaFX components called Nodes that build up the component
hierarchy denominated as Scene graph.
Web
The internal representation of an HTML page is based on the Document Object Model
(DOM) as defined by the W3C, a tree structure consisting of nodes. The root node,
a Document can contain Frame nodes with further Document nodes and/or a root
Element with a tree structure of further Element nodes. Though an HTML page with
its DOM is quite different from a Swing, JavaFX or SWT interface, the abstractions QFTest uses work just as well and the general term ”component” also applies to DOM
nodes.
Actions by the end-user of an application are transformed into events by the Java VM.
Every event has a target component. For a mouse click this is the component under the
mouse cursor, for a key press it is the component that has the keyboard focus. When
an event is recorded by QF-Test, the component information is recorded as well, so that
the event can later be replayed for the same component.
This may sound trivial and obvious, but component recognition is actually the most
complex part of QF-Test. The reason for this is the necessity to allow for change. QFTest is a tool designed for regression testing, so when a new version of the SUT is
released, tests should continue to run, ideally unchanged. So when the GUI of the SUT
changes, QF-Test needs to adapt. If, for example, the ”OK” and ”Cancel” buttons were
moved from the bottom of the detail view to its top, QF-Test would still be able to replay
events for these buttons correctly. The extent to which QF-Test is able to adapt varies
and depends on the willingness of developers to plan ahead and assist a little bit in
(47)
making the SUT well-suited to automated testing. But more on that later (section 5.6
(49)
and section 5.7 ).
5.2
Components in QF-Test
(634)
(644)
and Component
nodes
The recorded components are transformed into Window
which form a hierarchy that represents the actual structure of the GUI. These nodes
(655)
are located under the Windows and components node. The following image shows part
of the Components representing QF-Test’s main window.
5.3. QF-Test IDs
43
Figure 5.2: Component hierarchy of a Swing SUT
Web
(640)
Instead of a Window, the root Document of a web page is represented as a Web page
node. Nested Documents inside Frames are represented as Component nodes.
Every time a sequence is recorded, nodes are generated for components that are not
yet represented. When the sequence is discarded later on, the Components remain,
hence Component nodes have a tendency to proliferate. The popup menu (right button
click) for Window and Component nodes has two items, Mark unused components...
and Remove unused components , which will mark or remove those Component nodes
that are no longer being referred to. Be careful though if you are referencing Components
across test-suite boundaries or use variable values in QF-Test component ID attributes
as these are not taken into account unless the test-suites belong to the same project or
(396)
the Dependencies (reverse includes) attribute of the Test-suite root node is set correctly .
Note
4.0+
Besides this way of representing components as nodes it is also possible to address
components as multi-level sub-items with an XPath-like syntax called QPath as ex(58)
plained in section 6.3.2
The attributes of Components and the algorithm for component recognition are explained
(718)
in detail in section 40.2 . Here we will concentrate on the association between Component nodes and the rest of the test-suite.
5.3
QF-Test IDs
Every node of the test suite has a QF-Test ID attribute which is secondary for most kinds
(645)
of nodes. For Component nodes however, the QF-Test ID
has an important function.
It is the unique identifier for the Component node by which events, checks and other
5.4.
Component class
44
nodes that have a target component refer to it. Such nodes have a QF-Test component ID
attribute which is set to the Component’s QF-Test ID. This level of indirection is important.
If the GUI of the SUT changes in a way that QF-Test cannot adapt to automatically, only
the Component nodes for the unrecognized components need to be updated to reflect
the change and the test will run again.
It is essential to understand that the Component’s QF-Test ID is an artificial concept for
QF-Test’s internal use and should not be confused with the Name attribute, which serves
for identifying components in the SUT and is explained in detail in the following section.
The actual value of the QF-Test ID is completely irrelevant, except for the requirement
to be unique, and it bears no relation whatever to the actual component in the GUI of
the SUT. However, the QF-Test ID of the Component is shown in the tree of the test-suite,
for Component nodes as well as for events and other nodes that refer to a Component.
For this reason, Components should have expressive QF-Test IDs that allude to the actual
GUI component.
When creating a Component node, QF-Test has to assign a QF-Test ID automatically. It
does its best to create an expressive value from the information available. The option
(345)
Prepend parent QF-Test ID to component QF-Test ID controls part of this process. If
the generated QF-Test ID doesn’t suit you, you can change it. QF-Test will warn you if
you try to assign a QF-Test ID that is not unique and if you have already recorded events
that refer to the Component, it will change their QF-Test component ID attribute to reflect
the change. Note that this will not cover references with a variable QF-Test component ID
attribute.
Note
A common mistake is changing the QF-Test component ID attribute of an event instead
of the QF-Test ID itself. This will break the association between the event and the Com(667)
ponent, leading to an UnresolvedComponentIdException . Therefore you should
not do this unless you want to change the actual target component of the event.
Experienced testers with a well-structured concept for automated testing will find the
(37)
component recording feature described in section 4.4 useful. It can be used to record
the component hierarchy first in order to get an overview over the structure of the GUI
and to assign QF-Test IDs that suit you. Then you can continue to record the sequences
and build the test-suite around the components.
5.4
Component class
The class of a component is a very important attribute as it describes the type of the
recorded component. Once QF-Test records a button, it will only look for a button on
replay, not for a table or a tree. Thus the component class conveniently serves to partition the components of a GUI. This improves performance and reliability of component
recognition, but also helps you associate the component information recorded by QF-
5.4.
Component class
45
Test with the actual component in the GUI.
Besides its role in component identification, the class of a component is also important
for registering various kinds of resolvers that can have great influence on the way QF(807)
Test handles components. Resolvers are explained in detail in section 43.1.6 .
4.0+
5.4.1
Generic classes
Each toolkit defines its own system-specific classes for components like Buttons or Tables. In case of Buttons, that definition could be javax.swing.JButton for Java Swing
or org.eclipse.swt.widgets.Button for Java SWT or javafx.scene.control.ButtonBase For
JavaFX or INPUT:SUBMIT for web applications. In order to allow your tests to run independently of the actually utilized technology QF-Test unifies those classes via so-called
generic classes, e.g. all buttons are simply called Button now. This approach provides
a certain degree of independence from the dedicated technical classes and will allow
you to create tests without taking care about the specific technology. You can find a
(931)
detailed description of generic classes at chapter 50 . In addition to generic classes
(646)
with the state ”Ignore”. In
QF-Test records system-specific classes as Extra features
case of component recognition problems due to too many similar components these can
be activated to have a stricter component recognition at the expense of flexibility.
Another reason for generic classes is that dedicated technical classes could get
changed during development, e.g. due to introduction of a new base framework or
even another technology. In such cases QF-Test needs to be quite flexible in order to
recognize a proper class. Here the concept of generic classes allows you to be able to
cope with those changes and for the most part to re-use existing tests. You can find
(46)
more details at section 5.4.3 .
5.4.2
Class hierarchy for web applications
Web
For Swing, FX and SWT QF-Test works with the actual Java GUI classes whereas a
pseudo class hierarchy is used for web applications as follows:
5.4.
Component class
46
Figure 5.3: Pseudo class hierarchy for web elements
As shown, ”NODE” is at the root of the pseudo class hierarchy. It matches any kind of element in the DOM. Derived from ”NODE” are ”DOCUMENT”, ”FRAME”, ”DOM_NODE”
and ”DIALOG”, the types of nodes implementing the pseudo DOM API explained in
(754)
section 42.1 . ”DOM_NODE” is further sub-classed according to the tag name of the
node, e.g. ”H1”, ”A” or ”INPUT” where some tags have an additional subclass like ”INPUT:TEXT”.
5.4.3
Settings for class recording
QF-Test can record classes of component in various ways, therefore it organizes component classes in various categories. Those categories are called as the specific class,
the technology-specific system class, the generic class and the dedicated type of the
(646)
generic class. Each category is recorded at Extra features .
(342)
The option Record generic class names for components is checked by default. Using
this option allows you to record generic classes in order to share and re-use your tests
when testing a different technology with just minor changes to the existing tests.
In case you work with one Java engine only and you prefer to work with the ”real” Java
classes, you could also work without the generic class recording. But in this case you
(342)
should consider to check the option Record system class only . This option makes
QF-Test to record the technology-specific system class instead of the derived class. If
you switch off this option you will get the derived class which enables you to make a
very well targeted recognition but could cause maintenance efforts in case of changes
coming from refactoring.
5.5.
Web
Component inspection
47
In web applications QF-Test records classes as described in the previous chapter
(45)
section 5.4.2 . In case you have to work with a supported AJAX toolkit (see section
(769)
42.3 ), QF-Test records generic classes as well. You shouldn’t modify the default
options for this technology.
5.5
Component inspection
3.1+
Depending on its class a component has a set of (public) methods and fields which
(504)
can be used in an SUT script
once you have a reference to the object (see section
(121)
12.2.4 ). Select the entry Show the component’s methods... from the context menu
(655)
of a node under the Windows and components branch to display the methods and fields
of the corresponding class or right click on a component in the SUT while you are in
(37)
component recording mode (see section 4.4 ).
Web
The methods and fields displayed for (HTML) elements in a browser cannot be used
directly with an object returned by rc.getComponent(). These are at JavaScript
(754)
level and require a wrapping of the method calls into evalJS (cf. section 42.1 ).
5.6
The importance of naming components
Test automation can be improved tremendously if the developers of the SUT have either planned ahead or are willing to help by defining names for at least some of the
components of the SUT. Such names have two effects: They make it easier for QF-Test
to locate components even after significant changes were made to the SUT and they
are highly visible in the test-suite because they serve as the basis for the QF-Test IDs
QF-Test assigns to components. The latter should not be underestimated, especially for
components without inherent features like text fields. Nodes that insert text into components called ”textName”, ”textAddress” or ”textAccount” are far more readable and
maintainable than similar nodes for ”text”, ”text2” or ”text3”. Indeed, coordinated naming
of components is one of the most deciding factors for the efficiency of test automation
and the return of investment on QF-Test. If development or management is reluctant to
spend the little effort required to set names, please try to have them read this chapter of
the manual.
Note
(646)
attribute of Component
Please note that recorded names are stored in the Name
nodes. Because they also serve as the basis for the QF-Test ID for the same node,
Name and QF-Test ID are often identical. But always keep in mind that the QF-Test ID is
used solely within QF-Test and that the Name is playing the critical part in identifying the
component in the SUT. If the name of a component changes, it is the Name attribute that
must be updated, there is no need to touch the QF-Test ID.
5.6.
The importance of naming components
48
The technique to use for setting names during development depends on the kind of
SUT:
Swing
All AWT and Swing components are derived from the AWT class Component, so its
method setName is the natural standard for Swing SUTs and some developers make
good use of it even without test automation in mind, which is a great help.
JavaFX
For JavaFx setId is the pendant of Swing’s setName method to set identifiers for
components (called ’Nodes’). Alternatively IDs can be set via the FXML attribute fx:id.
While the ID of a ’Node’ should be unique within the scene graph, this uniqueness is not
enforced. This is analogous to the ’ID’ attribute on an HTML element.
SWT
Unfortunately SWT has no inherent concept for naming components. An accepted standard convention is to use the method setData(String key, Object value) with
the String ”name” as the key and the designated name as the value. If present, QF-Test
will retrieve that data and use it as the name for the component. Obviously, with no default naming standard, very few SWT applications today have names in place, including
Eclipse itself.
Fortunately QF-Test can derive names for the major components of Eclipse/RCP based
applications from the underlying models with good results - provided that IDs were
specified for those models. See the Automatic component names for Eclipse/RCP
(344)
applications option for more details.
Web
The natural candidate for naming the DOM nodes of a web application is the ’ID’ attribute of a DOM node - not to be confused with the QF-Test ID attribute of QF-Test’s
Component nodes. Unfortunately the HTML standard does not enforce IDs to be unique.
Besides, ’ID’ attributes are a double-edged sword because they can play a major role
in the internal JavaScript operations of a web application. Thus there is a good chance
that ’ID’ attributes are defined, but they cannot be defined as freely as the names in a
Swing, JavaFX or SWT application. Worse, many DHTML and Ajax frameworks need
to generate ’ID’ attributes automatically, which can make them unsuited for naming.
(376)
The option Turn ’ID’ attribute into name where ”unique enough” determines whether
QF-Test uses ’ID’ attributes as names.
Web
In case you want to test a web application using a supported AJAX toolkit, please take
(777)
a look at section 42.3.3 for details about assigning IDs.
If developers have implemented some other consistent naming scheme not based on
the above methods, those names can still be made accessible to QF-Test by implement(807)
ing a NameResolver as described in section 43.1.6 .
The reason for the tremendous impact of names is the fact that they make component
recognition reliable over time. Obviously, locating a component that has a unique name
assigned is trivial. Without the help of a name, QF-Test uses lots of different kinds of
information to locate a component. The algorithm is fault-tolerant and configurable and
has been fine-tuned with excellent results. However, every other kind of information
besides the name is subject to change as the SUT evolves. At some time, when the
5.7.
Considerations for setting names
49
changes are significant or small changes have accumulated, component recognition
will fail and manual intervention will be required to update the test-suite.
Another aspect of names is that they make testing of multi-lingual applications independent of the current language because the name is internal to the application and does
not need to be translated.
5.7
Considerations for setting names
There is one critical requirement for names: They must not change over time, not from
one version of the SUT to another, not from one invocation of the SUT to the next and
not while the SUT executes, for example when a component is destroyed and later created anew. Once a name is set it must be persistent. Unfortunately there is no scheme
for setting names automatically that fulfills this requirement. Such schemes typically
create names based on the class of a component and an incrementing counter and
invariably fail because the result depends on the order of creation of the components.
Because names play such a central role in component identification, non-persistent
names, specifically automatically generated ones, can cause a lot of trouble. If development cannot be convinced to replace them with a consistent scheme or at least drop
them, such names can be suppressed with the help of a NameResolver as described
(807)
in section 43.1.6 .
QF-Test does not require ubiquitous use of names. In fact, over-generous use can
even be counter-productive because QF-Test also has a concept for components being
”interesting” or not. Components that are not considered interesting are abstracted away
so they can cause no problem if they change. Typical examples for such components
are panels used solely for layout. If a component has a non-trivial name QF-Test will
always consider it interesting, so naming trivial components can cause failures if they
are removed from the component hierarchy in a later version.
Global uniqueness of names is also not required. Each class of components has its
own namespace, so there is no conflict if a button and a text field have the same name.
Besides, only the names of components contained within the same window should be
unique because this gives the highest tolerance to change. If your component names
(365)
are unique on a per-window basis, set the options Name override mode (replay) and
(343)
Name override mode (record) to ”Override everything”. If names are not unique per
window but identically named components are at least located inside differently named
ancestors, ”Hierarchical resolution” is the next best choice for those options.
Two questions remain: Which components should have names assigned and which
names to use? As a rule of thumb, all components that a user directly interacts with
should have a name, for example buttons, menus, text fields, etc. Components that are
not created directly, but are automatically generated as children of complex components
5.8.
Updating Components
50
don’t need a name, for example the scroll bars of a JScrollPane, or the list of a
JComboBox. The component itself should have a name, however.
If components were not named in the first place and development is only willing to
spend as little effort as possible to assign names to help with test automation, a good
strategy is to assign names to windows, complex components like trees and tables, and
to panels that comprise a number of components representing a kind of form. As long as
the structure and geometry of the components within such forms is relatively consistent,
this will result in a good compromise for component recognition and useful QF-Test ID
attributes. Individual components causing trouble due to changing attributes can either
be named by development when identified or taken care of with a NameResolver.
Since QF-Test ”knows” the components for which setName is most useful, it comes with
a feature to locate and report these components. QF-Test even suggests names to assign, though these aren’t necessarily useful. This feature is similar to component record(340)
ing and is explained in the documentation for the option Hotkey for components .
Web
The suggested names for DOM nodes are currently not very useful.
5.8
Updating Components
Unavoidably the components of the SUT are going to change over time. If names are
used consistently this is not really a problem, since in that case QF-Test can cope with
just about any kind of change.
Without names however, changes tend to accumulate and may reach a point where
component recognition fails. To avoid that kind of problem, QF-Test’s representation
of the SUT’s components should be updated every now and then to reflect the current state of affairs. This can be done with the help of the Update component(s)
menu-item in the context menu that you get by right-clicking on any node under the
(655)
Windows and components node.
Note
This function can change a lot of information in your test-suite at once and it may be
difficult to tell whether everything went fine or whether some components have been
misidentified. To avoid problems, always create a backup file before updating multiple
components. Don’t update too many components at once, take things Window by Window. Make sure that the components you are trying to update are visible except for the
menu-items. After each step, make sure that your tests still run fine.
Provided that you are connected to the SUT, this function will bring up the following
dialog:
5.8.
Updating Components
51
Figure 5.4: Update components dialog
If you are connected to multiple SUT clients, you must choose one to update the components for.
(644)
Select whether you only want to update the selected Component
nodes as well.
node or all its child
You can choose to include components that are not currently visible in the SUT. This is
mostly useful for menu-items.
(645)
The QF-Test ID
for an updated node is left unchanged if ”Use QF-Test component
ID of original node” is selected. Otherwise, updated nodes will receive a QF-Test ID
generated by QF-Test. If the QF-Test ID of a node is changed, all nodes referring to that
node via their QF-Test component ID attribute will be updated accordingly. QF-Test also
checks for references to the component in all suites of the same project and in those
(396)
(395)
suites that are listed in the Dependencies (reverse includes) attribute of the Test-suite
node. Those suites are loaded automatically and indirect dependencies are resolved as
well.
Note
In this case, QF-Test will open modified test-suites automatically, so you can save the
changes or undo them.
After pressing ”OK”, QF-Test will try to locate the selected components in the SUT and
fetch current information for them. Components that are not found are skipped. The
Component nodes are then updated according to the current structure of the SUT’s GUI,
which may include moving nodes to different parents.
Note
For large component hierarchies this very complex operation can take a while, in extreme cases even a few minutes.
This function is especially useful when names have been set for the first time in the SUT.
If you have already generated substantial test-suites before convincing the developers
to add names, you can use this function to update your Components to include the new
names and update their QF-Test IDs accordingly. This will work best if you can get hold
5.9.
Troubleshooting component recognition problems
52
of an SUT version that is identical to the previous one except for the added names.
Note
Very important note: When updating whole windows or component hierarchies of significant size you may try to update components that are not currently visible or available. In
that case it is very important to avoid false-positive matches for those components. You
may want to temporarily adjust the bonus and penalty options for component recognition
(364)
described in section 33.3.4
to prevent this. Specifically, set the ’Feature penalty’ to
a value below the ’Minimum probability’, i.e. to 49 if you have not changed the default
settings. Don’t forget to restore the original value afterwards.
(365)
If you need to change the setting of the options Name override mode (replay)
and
(343)
Name override mode (record)
because, for example, component names turned out
not to be unique after all, change only the setting for the recording options before updating the components. When finished, change the replay option accordingly.
5.9
Troubleshooting component recognition problems
If your SUT has changed in a way that makes it impossible for QF-Test to locate a
(662)
component, your test will fail with a ComponentNotFoundException . This should
(667)
not be confused with an UnresolvedComponentIdException which is caused by
removing a Component node from the test-suite or changing the QF-Test component ID
attribute of an Event node to a non-existing QF-Test ID.
Note
There are two videos available that explain in detail how to deal with a
ComponentNotFoundException.
A simple case video can be found at
http://www.qfs.de/en/yt/cnfes40.html, a more complex case is discussed in the video
http://www.qfs.de/en/yt/cnfec40.html.
When you get a ComponentNotFoundException, rerun the test with QF-Test’s debugger activated so that the test gets suspended and you can look at the node that
caused the problem. Here it pays if your QF-Test ID attributes are expressive because
you need to understand which component the test tried to access. If you cannot figure
out what this node is supposed to do, try to deactivate it and rerun the test to see if it
runs through now. It could be a stray event that was not filtered during recording. In
general your tests should only contain the minimum of nodes required to achieve the
desired effect.
If the node needs to be retained, take a look at the SUT to see if the target component is
currently visible. If not, you need to modify your test to take that situation into account.
If the component is visible, ensure that it was already showing at the time of replay by
checking the screenshot in the run-log and try to re-execute the failed node by singlestepping. If execution now works you have a timing problem that you need to handle by
(368)
either modifying the options for default delays (section 33.3.5 ) or with the help of a
(604)
Wait for component to appear node or a Check node with a Timeout. As a last resort you
5.9.
Troubleshooting component recognition problems
53
can work with a fixed delay.
If the component is visible and replay fails consistently, the cause is indeed a change
in the component or one of its parent components. The next step is identifying what
changed and where. To do so, re-record a click on the component, then look at the old
(655)
and new Component node in the hierarchy under Windows and components .
Note
You can jump directly from the Event node to the corresponding Component node by
pressing Ctrl-W or right-clicking and selecting Locate component . You can jump back
via Ctrl-Backspace or Edit→Select previous node . A clever trick is to mark the Component nodes to compare by setting breakpoints on them to make them easier to spot.
The crucial point is where the hierarchy for those two components branches. If they are
located in different Window nodes, the difference is in the Window itself. Otherwise the
old and new Component have a common ancestor just above the branching point and the
crucial difference is in the respective nodes directly below that branch. When you have
located those nodes, examine their attributes top-to-bottom and look for differences.
Note
You can open a second QF-Test window via View→New window... so as to place the
detail views of the nodes to compare side to side.
The only differences that will always cause recognition failures are Class name and Name.
Differences in Feature, structure or geometry attributes can usually be compensated
unless they accumulate.
A change in the Class name attribute can be caused by refactoring done by development,
in which case you need to update your Class name attribute(s) to reflect the change(s).
Another possible cause is obfuscation, a technique for making the names of the application classes illegible for protection against prying eyes. This poses a problem because
the class names can then change with each version. You can prevent both refactoring
(342)
and obfuscation problems by activating the option Record system class only .
If the Name has changed things get more difficult. If the change is apparently intentional, e.g. a typo was fixed, you can update the Name attribute accordingly. More likely
the cause is some automatically generated name that may change again anytime. As
explained in the previous section, your options in this case are discussing things with development or suppressing such names with the help of a NameResolver as described
(807)
in section 43.1.6 .
Changes to the Feature attribute are common for Window nodes, where the Feature represents the window title. When combined with a significant change in geometry such
a change can cause recognition to break. This can be fixed by updating the Feature to
match the new title or, preferably, by turning it into a regular expression that matches all
variants.
Depending on the kind and amount of changes to accommodate there are two ways to
deal with the situation:
5.10.
Accessing hidden fields on a web page
54
• Update the attributes of the old node and remove the newly recorded ones.
• Keep the new node and remove the old one, first taking care that events referring
to the old node are updated to the new QF-Test ID. There’s a clever trick to achieve
that: Edit the QF-Test ID of the old Component node to match the new one. QF-Test
will complain about the duplicate QF-Test ID, which you can ignore, and offer to
update all references, which you should accept. Then you can safely remove the
old node.
Note
Automatic updates for references from other test-suites require that the suites belong to
(396)
the same project or the correct setting the Dependencies (reverse includes) attribute of
the Test-suite root node.
5.10
Accessing hidden fields on a web page
Web
Hidden fields are not captured by default and therefore not stored under the
(655)
Windows and components node.
In case you frequently need to access hidden fields you can deactivate the Take visibility
(377)
of DOM nodes into account option.
Another way to get hidden fields recorded is the following:
(37)
• Activate the record components mode, navigate the mouse cursor to the parent
item containing the hidden field (most likely a FORM element).
• Press right mouse button and select Component and children from the popup
menu.
(37)
• Deactivate the record components
(19)
mode.
(655)
• A search within Windows and components
your destination component quickly.
for e.g. ’HIDDEN’ should get you to
To access a hidden field’s attributes (e.g. the ’value’ attribute) you can create a simple
(504)
SUT script
as shown below. Details on scripting in general, the used methods and
(114)
(729)
(754)
parameters can be found in Scripting , Run-context API
and Pseudo DOM API
respectively.
node = rc.getComponent(’id of the hidden component’, hidden=1)
node.getAttribute(’value’)
Example 5.1: Accessing the value attribute of a hidden field
Chapter 6
Sub-items of complex GUI
components
Some components of a rich GUI, trees or tables most notably, are rather complex and
can contain an arbitrary number of elements like tree-nodes or table-cells. Depending
on the kind of application, these elements may themselves be GUI components (Nodes
in JavaFX, DOM nodes in a web application) or just a graphical representation of some
data (Swing renderer concept, SWT Tree and Table items). This is a purely technical
distinction and because it makes sense, from the tester’s point of view, to treat these
elements uniformly as separate entities and possible targets for events. QF-Test rep(650)
resents such sub-items in one of two ways, either as an Item node or with a special
syntax.
6.1
Item nodes
An Item is defined by two parts, the component it belongs to and an index into this
component. The component is identified by the parent node of the Item node. The index
can be either numeric or textual. Numeric indexes start with 0. In a List, for example,
the element with index 1 is the second element from the top. Plain numeric indexes are
not very useful for tree-nodes, because only the visible nodes are counted. Expanding
or collapsing a tree node changes the numeric indexes of all nodes below it.
A textual index refers to the element by the text that is shown in the GUI. A list-item
shown as ”item1” in a List component would be recorded with at textual index of
”item1”. The textual representation is more flexible but can cause problems if the names
of the elements in a component are not unique. A textual index can also be given as
(720)
a regular expression (see section 40.4 ). In this case, the first element found that
matches the regular expression will be used.
6.1. Item nodes
56
(347)
The option Sub-item format
controls the format for recording indexes.
Almost all kinds of Items have only one index. The exception is the table-cell in a Table
component. Since tables are two-dimensional structures, two indexes are required to
(651)
refer to a table-cell. The first one, the Primary index , defines the column of the table
(651)
and the Secondary index the row.
Figure 6.1: An Item for a table cell
Tree-nodes are also special. The hierarchic structure of a tree doesn’t map well to a
linear one. One reason is the dependence of the numeric index on the current expansion state of the nodes. Another reason is that many kinds of trees have nodes with
non-unique names. However, when the direct and indirect parent-nodes are taken into
account to work with paths instead of names, uniqueness is often restored.
QF-Test uses a special syntax to represent tree nodes. An index that starts with a ’/’
character is considered to be a path index. Imagine a tree that shows a file-system hierarchy. The file ”/tmp/file1” could be represented flat as just ”file1”, possibly colliding with
another ”file1” in some other directory, or as the full, unique path ”/tmp/file1”. QF-Test
also supports numeric indexes with this syntax. As a numeric index ”/2/3” designates
the fourth child node of the third child of the root. A mixed-format index like ”/tmp/2” to
get the third child of the node named ”tmp” is currently not possible.
Note
This special syntax makes ’/’ a special character for elements of tree components, so
(722)
it needs to be escaped when a literal ’/’ character is required. See section 40.6
for
details about escaping and quoting special characters.
6.2.
The QF-Test ID of an Item
6.2
57
The QF-Test ID of an Item
(645)
(43)
Everything that was said about the QF-Test ID attribute of Components in section 5.3
(651)
also applies to an Item’s QF-Test ID . It has to be unique and can be referred to by
events and similar kinds of nodes that require a component target.
When assigning an Item node’s QF-Test ID automatically, QF-Test creates it by concatenating the containing Component’s QF-Test ID and the index (or indexes) of the element.
These kinds of QF-Test IDs are expressive and typically easy to recognize. However,
they are also the source of a common misunderstanding: If you want to change the
index of a recorded Item to point to some other element, you must not change the QFTest component ID attribute of the node that refers to it (unless you use the special syntax
described in the following section). Change the Primary index of the Item node instead.
6.3
6.3.1
Special QF-Test ID-syntax for direct element access
Accessing simple subcomponents
There are alternative ways to target an event on an element of a complex component
that does not require creation of an Item node. By using a special syntax you can specify
the index of the element directly in the QF-Test component ID attribute of the respective
node. Instead of the QF-Test ID of an Item node, specify the QF-Test ID of the containing
Component and follow it with a special separator character and the index of the element.
The character that separates the QF-Test ID and the index designates the format for the
index:
Separator
@
&
%
Index format
Textual index
Numeric index
Regular expression
Table 6.1: Separators and index formats for element access
To address a table-cell with Primary index and Secondary index simply append another
separator character and the second index. That way, the Primary index and Secondary
index can be of different format.
3.3+
Example: If your test-suite contains a Component node for a Table component with the
QF-Test ID mytable and you want to simulate a click on table-cell in the 6th row of
(538)
the column with the title ”Name”, create a Mouse event
node and set its
(539)
QF-Test component ID attribute to mytable@Name&5.
6.3.
Special QF-Test ID-syntax for direct element access
58
(844)
Almost all internal ItemResolver implementations (cf. section 43.4 ) allow of negative indices to start counting from the end. For example, mytable&-1&-1 addresses
the last cell in the rightmost table column.
(650)
Whether it is preferable to use an Item
node or the special syntax depends on the
situation. You can use either method as you see fit and mix them as you like. As a rule
of thumb, Item nodes are better suited to components with a few more or less constant
elements, the columns of a Table for example, or the tabs of a TabbedPane. The special syntax is more flexible and generally preferable for indexes with variable values and
(347)
determines
for elements with varying or editable values. The option Sub-item type
whether Item nodes are created or the special syntax is used when recording. With the
default value ”Intelligent” QF-Test will follow the rules outlined above.
Note
The special meaning of the separator characters implies that literal ’@’, ’&’ or ’%’ char(722)
acters in element indexes need to be escaped. See section 40.6
for details about
escaping and quoting special characters.
6.3.2
Addressing multi-level sub-items
4.0+
In more complex GUI applications it is possible for items like a table cell to contain
arbitrary components. Typical examples are CheckBoxes or TextFields, but it is even
possible to nest tables. Such inner components can be addressed using a special multilevel item syntax.
Addressing components via QPath
Any QF-Test component ID used in an event or check node, regardless of whether it
already contains sub-item syntax or not, can be affixed with one or more special subitem indexes of the form @:ClassName<idx>, where <idx> is optional. This instructs
QF-Test to first resolve the target component (and possibly item) for the part of the QFTest component ID before the ’@:’ and therein search for visible components of class
ClassName. If <idx> is specified, it is treated as a 0-based index into the list of visible
candidate components. No <idx> is equivalent to <0>.
Addressing via XPath and/or CSS-Selectors
Web
4.1+
XPath’s and also CSS-Selectors are relative powerful, standardized features to describe components into a webbrowser. (The official specification can be found here:
https://www.w3.org/TR/xpath/ and https://www.w3.org/TR/css3-selectors/).
Since QF-Test 4.1.0 QF-Test is also official supporting componentrecognition via xPaths
6.3.
Special QF-Test ID-syntax for direct element access
59
and CSS-Selectors in webtests. In order to allow the faster migration of existing webtests of under tools towards QF-Test.
The
Internet
is
already
providing
numerous
tutorials
on
how
web-components
can
be
described
by
using
CSS-Selectors
(e.g.
https://www.w3schools.com/cssref/css_selectors.asp)
as
well
on
how
web-components
can
be
described
by
using
xpaths
(e.g.
https://www.w3schools.com/xml/xpath_syntax.asp). In this respect the basics of this
componentrecognition technics didn’t get included into this manual.
Usage in QF-Test ID
Assumed that QF-Test should recognize a webcomponent via it’s X-Path ”$(xpath)” or
it’s css-selector ”$(css)”, the recognition can happen by using multiple ways. The most
often most easiest/fastest way is to specify the X-Path or rather the css-selector in the
”QF-Test component ID”-attribute of an arbitrarily Event-Node. Hereby the following
syntax get’s used:
genericHtml@:xpath=${quoteitem:$(xpath)}
genericHtml@:css=${quoteitem:$(css)}
or equivalent:
genericDocument@:xpath=${quoteitem:$(xpath)}
genericDocument@:css=${quoteitem:$(css)}
This Syntax can get nested in any order. For example with:
genericDocument@:xpath=${quoteitem:$(xpath)}@:css=${quoteitem:$(css)}
one can advise QF-Test to first search a component via X-Path and then to search for a
subcomponent via the css-selector.
Note
Please note that out of obvious reasons the @:xpath/@:css-syntax assumes that the
used XPath/CSS-Syntax returns a single component. The usage of a XPath-syntax
that does not return a single component, but for example an integer (Example:
”count(.//input[@id!=’Google’])”) or a boolean (Example: ”nilled($in-xml//child[1])”), may
lead to unexpected behaviour.
Usage in scripts
The rc-Modul contains some function, that equally allows addressing web-components
via xpath or css-selectors respectively.
6.4.
com
res
res
res
res
Activating item recording for web tables
=
=
=
=
=
60
rc.getComponent("genericHtml") # or rc.getComponent("genericDocument")
com.getByXPath(rc.lookup("xpath"))
# find subcomponent via xpath
com.getByCSS(rc.lookup("css"))
# find subcomponent via css
com.getAllByXPath(rc.lookup("xpath")) # find all subcomponent via xpath
com.getAllByCSS(rc.lookup("css"))
# find all subcomponent via css
Example 6.1: Find Components via xpath / css-selectors in scripts
In order to use an XPath that does not return component(s) please use the evalJSfunction:
node = rc.getComponent(’genericDocument’)
print node.evalJS("""document.evaluate("count(.//input[@id=’Google’])",
document, null, 0, null).numberValue;""")
Example 6.2: Example usage for an XPath-Selector that does not return component(s)
Usage in component nodes
Within component-nodes QF-Test can equally get advised to recognize components via
X-Path or css-selectors. Therefor one is specifying an additional recognition-feature
below ”Extra features” that is looking like:
State
Regexp
Negate
Name
Value
Must match
No
No
qfs:item
@:xpath=${quoteitem:$(xpath)} or @:css=${quoteitem:$(css)}
Figure 6.2: Extra Feature attributes for x-path or css-selector component recognition.
6.4
Activating item recording for web tables
Web
Within web pages, tables are used both for ’real’ visible tables and for invisible layout
purpose. Also nested tables are possible. QF-Test typically cannot know where a real
table starts, and therefore item based recording is not possible without further information.
Hence this information can be provided by the user. By use of the procedure
qfs.web.table.recording.activateItemRecording from the standard
(112)
library item recording can be activate for a given web table.
6.4.
Activating item recording for web tables
61
In case you don’t know which QF-Test component ID to provide to the procedure, simply
record a click to an arbitrary table cell. Jump to the respective component node by
pressing Ctrl-W or right-clicking and selecting Locate component and look for the next
node with the class name Table upwards in the component hierarchy. That should be
the one to use.
Note
(769)
When using one of the AJAX toolkit resolvers you typically don’t have to do any separate activation. AJAX tables are handled by the resolver.
Chapter 7
Variables
Variables are the primary means to add flexibility to a test-suite. Though they are used
(463)
mainly as parameters for Procedures , they are also useful in many other cases.
Variable syntax can be used in almost every attribute of the nodes of a suite, except for
boolean values (check boxes). There are three versions of variable references:
• $(varname) expands to the value of some previously defined variable.
(67)
• ${group:name} accesses external data from a resource bundle or property
file. The groups system and qftest are always defined and have a special
(67)
meaning (see section 7.5 ).
(71)
• $[expression] evaluates some mathematical expressions .
7.1
Variable lookup
To understand the reasons for why and how variables are defined in multiple places, you
first have to learn about how the values of variables are determined.
Each set of variable definitions, called bindings, is placed on one of two stacks of bindings. One stack is used for actual - or direct - definitions and one for fallback bindings or default values. When a variable’s value is requested via $(...), QF-Test
first searches the stack of direct bindings from top to bottom, then the stack of fallbacks, also top down. The first value found is used. If there is no binding at all for a
(664)
name, an UnboundVariableException is thrown unless you use the special syntax ${default:varname:defaultvalue} to provide a default value for this case as
(67)
described in section 7.5 .
7.2.
Defining variables
63
Topmost bindings
(highest precedence)
...
Bottommost bindings
(lowest precedence)
Primary stack(Direct bindings)
Topmost bindings
(highest precedence)
...
Bottommost bindings
(lowest precedence)
Secondary stack(Default values)
Figure 7.1: Direct and fallback bindings
This
mechanism
supports
recursive
or
self-referencing
variable
definitions.
For example,
setting a variable named classpath
to the value some/path/archive.jar:$(classpath) will extend a
binding for classpath with lower precedence.
If no such binding exists, a
(664)
RecursiveVariableException is thrown.
7.2
Defining variables
Variables are defined in various places with different orders of precedence. You define a
variable by adding a row to the appropriate table with the name and value of the variable
in the respective columns.
Figure 7.2: System variables
7.2.
Defining variables
64
(393)
Four sets of variable definitions are accessible from the global options
:
System variables
The most basic set of bindings, commonly used for system-specific definitions
like path names, JDK version, operating system stuff etc. These are placed at the
bottom of the fallback stack so they have the least possible precedence. The
system bindings are saved in the system configuration file together with the other
QF-Test system options.
Suite variables
These are general definitions for a test-suite and saved as part of the suite. The
bindings of the suite from which a test run is started, are placed at the bottom of
the direct bindings stack. When a procedure from a different suite is called, or a
different suite’s window or component is accessed, the bindings of that suite are
temporarily placed on top of the fallback stack.
Variables from the command line
You can define variables on the command line when starting QF-Test with the
(693)
-variable <name>=<value>
argument which can be used multiple times.
These bindings are placed above the suite bindings on the direct bindings stack,
so they override the bindings of the suite. For example, a variable count=1 can
(475)
be defined for the suite and used as $(count) for the Number of iterations of a
(473)
Loop
node for quick trial runs.
Then you can use qftest -batch
-variable count=100 ... for the actual testing. The command line bindings
are accessible mainly for your information, but you can also edit them for
experimentation.
Global variables
Placed above the command line variables on the direct call stack, these bindings
are a means to pass values between unrelated parts of a test run. Values read
(591)
(594)
from the SUT by a Fetch text or a Fetch index node are bound here, as well as
(601)
definitions from a Set variable node. The globals are not cleared in between test
runs as a convenience and you can edit them for the same reason. To simulate a
real test run you should clear them via the Run→Clear global variables menu
(10)
first. When run in batch mode (see section 1.7 ) QF-Test clears the global
(691)
variables before the execution of each test given with the -test <n>|<ID>
command line argument.
The rest of the variable definitions are part of the test-suite itself:
7.3.
Variable example
65
• Each sequence type node defines a set of bindings that are placed on top of the
direct bindings stack when the test run enters the node and removed when the
node is exited.
(466)
• A Procedure call’s bindings are placed on top of the direct bindings stack before
(463)
executing the Procedure . The bindings defined by a Procedure are placed on top
of the fallback stack during execution of the Procedure. Think of them as default
values for the expected parameters. Once the procedure returns, both sets of
bindings are removed.
7.3
Variable example
Consider the following example:
Figure 7.3: Variable example
The Sequence ”Login” contains a Procedure call of the Procedure ”login” which expects
two parameters: user and password.
The Procedure’s default bindings are
user=username and password=pwd. The Procedure call overrides these with
user=myname and password=mypassword.
The ”login” Procedure itself holds Procedure calls of Procedures in another test-suite called
”lib.qft” to write the user and password to some GUI components. We assume that
the Procedures in this library have most parameters in common, so they don’t define
default values themselves but in the suite bindings of ”lib.qft” where they are set to
user=libuser and password=libpwd. This is useful for creating and editing the
procedures in the library since they can all be executed and tried out separately without
having to define Procedure calls.
The following diagram shows the State of the binding stacks during the execution of the
Procedure ”lib.qft#setUser”:
7.4.
Fetching data from the GUI
66
Procedure
call
”lib.qft#setUser”
Procedure call ”login”
user=myname
password=mypassword
Procedure ”lib.qft#setUser”
Test-suite ”lib.qft”
user=libuser
password=libpwd
Procedure ”Login”
user=user
password=pwd
Sequence ”Login”
Global bindings
Command line bindings
Suite bindings
Primary stack(Direct bindings)
System bindings
Secondary stack(Default values)
Figure 7.4: Variable bindings example
The important thing to note here is that the Procedure call to ”lib.qft#setUser” inside the
Procedure ”login” does not need to define the user parameter again, the parameter is
”passed through”. As a rule of thumb, when calling one Procedure from another, define
a parameter value in the Procedure call if and only if the value has not been explicitly
defined yet or you want to pass a different value.
7.4
Fetching data from the GUI
Often it is necessary to fetch some kind of data from the SUT’s GUI to use it as test
input.
QF-Test offers a special set of query nodes
Insert→Miscellaneous :
(591)
• Fetch text
for this kind of task, available via
to fetch a component or item text,
(594)
• Fetch index
(590)
to fetch the index of an components item,
(597)
• Fetch geometry
to fetch a component or items geometry.
The retrieved values are assigned to local or global variables that can be declared in the
fetch node.
Instead of inserting fetch nodes by hand they can easily be created by first recording a
(16)
mouse click node the the respective component and then use the transform operation
to convert the same to the fetch node you need.
7.5.
External data and special groups
7.5
67
External data and special groups
(615)
(618)
External data is made available by the Load resources
and Load properties
nodes.
These assign a group name to a set of definitions from a resource bundle or properties
file. To access the value of the definition for name, use the syntax ${group:name}.
(10)
When run in batch mode (see section 1.7 ) QF-Test clears the resources and prop(691)
erties before the execution of each test given with the -test <n>|<ID> command
line argument. In normal mode QF-Test keeps them around to ease building a suite, but
for a true trial run you should clear them via the Run→Clear resources and properties
menu first.
Some special group names are predefined and always available:
system
The group system gives access to the system properties of the Java VM
(for
programmers:
java.lang.System.getProperties()),
e.g.
${system:user.home}
for
the
user’s
home
directory
or
${system:java.class.path} for the CLASSPATH with which QF-Test was
started. Which names are defined in the group system depends on the utilized
JDK.
The group always refers to the VM QF-Test was started with, because variable
expansion takes place there.
env
On operating systems which support environment variables like PATH,
CLASSPATH or JAVA_HOME (practically all systems QF-Test runs on), these
environment variables can be accessed with the help of the group env.
default
You can specify a default value for a variable with the group default. The
syntax is ${default:varname:defaultvalue}. This is extremely useful for
things like generic components or in almost every place where there is a
reasonable default for a variable because the default value is then tightly coupled
with the use of the variable and doesn’t have to be specified at Sequence or
test-suite level. Of course you should only use this syntax if the variable lookup in
question is more or less unique. If you are using the same variable with the same
default in different places it is preferable to use normal syntax and explicitly set
the default, so that the default for all values can be changed in a single place.
3.4+
id
7.5.
3.1+
External data and special groups
68
The group id can be used to reference QF-Test component IDs. Values in this
group simply expand to themselves, i.e. ”${id:whatever}” expands to ”whatever”.
Though QF-Test component IDs can be referenced without the help of this group,
its use increases the readability of tests. Most notably however, QF-Test
component ID references in this group will be updated automatically in case the
referenced target component gets moved or its QF-Test ID changed.
quoteitem
4.0+
Via the quoteitem group you can conveniently escape special characters like
’@’, ’&’ and ’%’ in the name of a textual sub-item index to prevent it from being
treated as several items, e.g.
”${quoteitem:user@host.org}” will result in
”user\@host.org”.
quoteregex, quoteregexp
4.0+
The group quoteregex with its alias quoteregexp can be used to escape
characters with special meaning in regular expressions. This is often useful when
building regular expressions dynamically or when referencing subitems with
special characters in their name by a regular expression index, e.g.
”componentid%${quoteregex:foo(baa)}.*” allows you to address the first
occurrence of items beginning with ’foo(baa)’.
qftest
The special group named qftest provides miscellaneous values that may be
useful during a test run. The following tables list the values currently defined.
Name
32 or 32bit
64 or 64bit
batch
client.exitcode.<name>
client.output.<name>
clients
Meaning
true if QF-Test is running in a 32bit Java VM - which is not
the same as running on a 32bit Operating System - false
otherwise.
true if QF-Test is running in a 64bit Java VM, false otherwise.
true if QF-Test is running in batch mode, false for interactive mode.
(508)
The exit-code of the last process started with the Client
attribute set to <name>. In case the process is still alive
the result is the empty string.
(508)
The output of the last process started with the Client
attribute set to <name>. The maximum size for buffered
output is defined by the option Maximum size of client
(354)
terminal (kB) .
A list of the names of all active process clients, separated
by a newline.
7.5.
External data and special groups
clients.all
count.exceptions
count.errors
count.warnings
count.testcases
count.testcases.exception
count.testcases.error
count.testcases.expectedtofail
count.testcases.ok
count.testcases.ok.percentage
count.testcases.skipped
count.testcases.notimplemented
count.testcases.run
count.testsets.skipped
dir.groovy
dir.javascript
dir.jython
dir.log
dir.plugin
dir.root
dir.runlog
dir.system
dir.user
dir.version
license
systemcfg
usercfg
executable
isInRerun
isInRerunFromLog
69
A list of the names of all process clients, separated by a
newline. This includes live clients as well as the recent
dead clients similar to those listed in the ”Clients” menu.
Number of exceptions in the current test-run.
Number of errors in the current test-run.
Number of warnings in the current test-run.
Total number of total test-cases (run and skipped) in the
current test-run.
Number of test-cases with exceptions in the current testrun.
Number of test-cases with errors in the current test-run.
Number of test-cases expected to fail in the current testrun.
Number of successful test-cases in the current test-run.
Percentage of successful test-cases in the current testrun.
Number of skipped test-cases in the current test-run.
Number of not implemented test-cases in the current testrun.
Number of run test-cases in the current test-run.
Number of skipped test-sets in the current test-run.
Directory of Groovy
Directory of JavaScript
Directory of Jython
Log directory of QF-Test
Plugin directory of QF-Test
Root directory of QF-Test
Run-log directory of QF-Test
System-specific configuration directory of QF-Test.
User-specific configuration directory of QF-Test
Version-specific directory of QF-Test
The path to the license file
The path to the system configuration file
The path to the user specific configuration file
The qftest executable matching the currently running
QF-Test version, including the full path to its bin directory
and with .exe appended on Windows. Useful if you need
to run QF-Test from QF-Test for example to call a daemon
or create reports.
”true”, if current execution is in re-run mode, ”false” other(190)
wise, see section 18.3.2 .
”true”, if test-run has been re-started from run-log, ”false”
(187)
otherwise, see section 18.3.1 .
7.5.
External data and special groups
java
linux
macos
os.fullversion
os.mainversion
os.name
os.version
reruncounter
return
runid
screen.height
screen.width
skipnode
suite.dir
suite.file
suite.path
testcase.name
testcase.id
testcase.qname
testcase.reportname
70
Standard Java program (javaw under Windows, java
under Linux/Unix) or the explicit Java argument if QF-Test
(677)
is started with -java <executable>
”true” under Linux, ”false” otherwise
”true” under macOS, ”false” otherwise
The whole version of the operating system
The main version of the operating system, e.g. ”10” for
Windows 10
The name of the operating system
The version of the operating system. In some cases that’s
not the whole one then you should use os.fullversion
instead.
Number of current re-run attempt, default is 0, for details
(190)
see section 18.3.2 .
(463)
The most recent value returned from a Procedure
(469)
through a Return
node.
(168)
The runid of the current test-run. See section 17.1
for
further information about the runid.
Screen height in pixels
Screen width in pixels
This magic value is not for the casual user. It causes
QF-Test to skip execution of the current node. Its primary use is as the value for a variable defined in the
(546)
(545)
Text
attribute of a Text input
node which also has
(547)
its Clear target component first
attribute set. An empty
value would clear the field whereas $_{qftest:skipnode}
leaves the field unchanged. But skipnode is also applicable for fine-grained execution control by placing a variable in the comment of a node and selectively passing
$_{qftest:skipnode} to that variable. Please note that you
almost always want to use lazy syntax ’$_’ with this variable. Otherwise its expansion as the parameter in a Procedure call node would cause skipping the whole call.
Directory of the current suite
File name of the current suite without directory
File name of the current suite including directory
The name of the current Test-case, empty if no Test-case is
currently being executed.
The QF-Test ID of the current Test-case, empty if no Testcase is currently being executed.
The qualified name of the current Test-case, including the
names of its parent Test-sets. Empty if no Test-case is currently being executed.
The expanded report name of the current Test-case, empty
if no Test-case is currently being executed.
7.6.
Expressions
71
testcase.splitlogname
testset.name
testset.id
testset.qname
testset.reportname
testset.splitlogname
teststep.name
teststep.qname
teststep.reportname
thread
threads
version
version.build
windows
The qualified name of the current Test-case converted to
a filename, including the names of its parent Test-sets as
directories. Empty if no Test-case is currently being executed.
The name of the current Test-set, empty if no Test-set is
currently being executed.
The QF-Test ID of the current Test-set, empty if no Test-set
is currently being executed.
The qualified name of the current Test-set, including the
names of its parent Test-sets. Empty if no Test-set is currently being executed.
The expanded report name of the current Test-set, empty
if no Test-set is currently being executed.
The qualified name of the current Test-set converted to a
filename, including the names of its parent Test-sets as directories. Empty if no Test-set is currently being executed.
The name of the current Test-step, empty if no Test-step is
currently being executed.
The qualified name of the current Test-step, including the
names of its parent Test-steps, but not including Test-cases
or Test-sets. Empty if no Test-step is currently being executed.
The expanded report name of the current Test-step, empty
if no Test-step is currently being executed.
The index of the current thread.
Always 0 except if QF-Test is started with the argument
(693)
-threads <number> .
The number of parallel threads.
Always 1 except if QF-Test is started with the argument
(693)
-threads <number> .
QF-Test version
QF-Test build number
”true” under Windows, ”false” otherwise
Table 7.1: Definitions in the special group qftest
7.6
Expressions
The $[...] syntax for mathematical expressions supports operations like +, -, *, /
and % (modulo) as well as braces. It comes in handy when calculating coordinates, e.g.
in a loop.
7.7.
Immediate and lazy binding
72
In fact these expressions are far more powerful, since they are evaluated by the Jython
interpreter. You can use arbitrary constructs that are legal syntax for the Jython method
(114)
eval. See chapter 12 for details about Jython scripting.
Note
Accessing QF-Test variables in an expression follows the same rules as in Jython
(118)
scripts (see section 12.2.3 ). You can use the standard QF-Test syntax $(...) and
${...:...} for numeric or Boolean values. String values should be accessed with
rc.lookup(...).
7.7
Immediate and lazy binding
3.0+
There is a very subtle issue in using QF-Test variables that requires further explanation:
When a new set of variable bindings is pushed on one of the variable stacks, there are
two possibilities for handling variable references in the value of a binding, for example
when the variable named ’x’ is bound to the value ’$(y)’. The value ’$(y)’ can be stored
literally, in which case it will be expanded some time in the future when ’$(x)’ is referenced somewhere, or it can be expanded immediately, so that the value of the variable
’y’ is bound instead. The first approach is called ’lazy’ or ’late binding’, the second
approach ’immediate binding’.
The difference, of course, is the time and thus the context in which a variable is expanded. In most cases there is no difference at all, but there are situations where it is
essential to use either lazy or immediate binding. Consider the following two examples:
A utility test-suite contains a procedure for starting the SUT with different JDK versions.
The variable ’jdk’ is passed as a parameter to this procedure. For ease of use, the author
of the test-suite defines some additional useful variables at test-suite level, for example a
variable for the java executable named ’javabin’ with the value ’/opt/java/$(jdk)/bin/java’.
At the time ’javabin’ is bound in the test-suite variables, ’jdk’ may be undefined, so
immediate binding would cause an exception. But even if ’jdk’ were bound to some
value, immediate binding would not have the desired effect, because the java executable
is supposed to be the one from the JDK defined later by passing the parameter ’jdk’ to
a procedure. Thus lazy binding is the method of choice here.
Imagine another utility test-suite with a procedure to copy a file. Two parameters called
’source’ and ’dest’ specify the source file and destination directory. The caller of the
procedure wants to copy a file called ’data.csv’ from the same directory as the calling test-suite to some other place. The natural idea is to bind the variable ’source’ to
the value ’${qftest:suite.dir}/data.csv’ in the procedure call. With immediate binding,
’${qftest:suite.dir}’ will indeed expand to the directory in which the calling suite resides.
However, if lazy binding were used, the actual expansion would take place inside the
procedure. In that case, ’${qftest:suite.dir}’ would expand to the directory of the utility
suite, which most likely is not what the caller intended.
7.7.
Immediate and lazy binding
73
In versions of QF-Test up to and including 2.2 all variable expansion was lazy. As the
examples above show, both variants are sometimes necessary. Since immediate binding is more intuitive it is now the default. This can be changed with the option When
(393)
binding variables, expand values immediately . The option Fall back to lazy binding
(393)
if immediate binding fails
complements this and helps to ease migration of old testsuites to the use of Immediate Binding. The warnings issued in this context help locating
the few spots where you should use explicit lazy binding as described below. Except for
very rare cases where lazy binding is required but immediate binding also works so that
the fallback is not triggered, all tests should work out of the box.
In the few cases where it makes a difference whether a variable is expanded immediately
or lazily, the expansion of choice can be selected individually, independent of the setting
of the above option, by using an alternative variable syntax. For immediate binding use
’$!’ instead of just ’$’. Lazy binding is selected with ’$_’. For example, to define a
variable at test-suite level that specifies a file located in this test-suite’s directory, use
’$!{qftest:suite.dir}/somefile’. If immediate binding is the default and you require lazy
binding as in the ’jdk’ example above, use ’$_(jdk)’.
Note
With lazy binding the order of variable or parameter definitions in a node or a data driver
did not matter because nothing was expanded during the binding stage. With immediate
bindings, variables are expanded top-to-bottom or, in a data driver, left-to-right. This
means that if you define x=1 and y=$(x) it will work, with y being set to 1, if x is defined
first. If y is defined first the definition will either fail or trigger the lazy definition fallback
described above.
Chapter 8
Problem analysis and debugging
The whole point of creating automated tests is to uncover problems in the SUT. Therefore we can justifiably expect the tests to fail occasionally.
After the execution of a test has finished, a message will appear in the status line that
will hopefully say ”No errors”. If something went wrong, the numbers of warnings, errors
and exceptions that occurred is shown. Additionally an error dialog may pop up. In that
case you will need to find out what went wrong.
For some problems the cause may be obvious, but very often it is not. First and foremost in this case is the need to determine whether the test failed due to a bug in the
SUT or whether the SUT behaved correctly but the logic of the tests was wrong. The
dilemma here is that any potential problem in the SUT must be duly reported as soon as
possible, yet every false bug report is a waste of time and will cause resentment among
the developers. Therefore, each problem needs to be thoroughly analyzed and every
alleged bug in the SUT should ideally be reproducible before a report is submitted.
QF-Test supports testers in this crucial task in two ways. A detailed log is created
for each test-run that holds all the relevant information for post mortem error analysis,
including screenshots taken at the time that an error occurred. The integrated test
debugger helps with analyzing and understanding the flow of control and information
during a test-run.
8.1
The run-log
During test replay QF-Test creates a run-log that records everything that is going on.
The run-logs for recent tests are accessible from the Run menu, the current or most
recent run-log can also be opened by typing Ctrl-L or the respective
button in the
(79)
toolbar. See section 8.1.5 for information about options influencing run-log creation.
8.1.
The run-log
75
The structure of a run-log is similar to that of a test-suite, but there are subtle differences.
(433)
(435)
Nodes are added to the run-log when they are executed. Setup and Cleanup nodes,
for example, are typically executed more than once, in which case multiple copies will
be recored in the run-log as shown below:
Test-suite
Run-log
Figure 8.1: A simple test and its run-log
A run-log is the essential resource for determining what went wrong in a test, where
it went wrong and maybe even get an idea about why it went wrong. Therefore the
emphasis is on completeness of information rather than readability and a run-log should
not be confused with a report or summary. Report generation is covered in chapter
(167)
17 .
In addition to the nodes copied from the test-suite, a run-log contains failure information,
optional annotations, information about variable expansion, various kinds of messages
and, not to forget, timestamps and durations.
Note
The difference between the values of ”Duration” and ”Real time spent” are explicit delays
introduced in nodes via the ’Delay before/after’ attribute or user interrupts.
The information gathered from a long test-run accumulates and can eat up enormous
amounts of memory, so QF-Test normally creates a special form of compact run-log in
which only the information relevant for report generation and for error diagnosis is re(391)
tained. This functionality is configurable through the option Create compact run-log
(679)
and the command line argument -compact . This should not be mixed up with the
option to save compressed run-logs as *.qrz files which simply uses plain gzip compression to reduce the size of the log file, but doesn’t have any impact on memory
usage. Another alternative to save memory is to create split run-logs as described in
(78)
section 8.1.4 . Compact and split run-logs can be combined for maximum efficiency.
8.1.
The run-log
8.1.1
76
Error states
There are three kinds of failures differing in the level of severity:
Warnings
Warnings indicate problems that are typically not serious, but might lead to
trouble in the future and may need looking at. For example, QF-Test issues a
warning, if the best match for a component barely meets the requirements and
differs in some significant way.
Errors
Errors are considered to be serious defects that require closer inspection. They
indicate that the SUT does not fulfill some requirement. A typical cause for an
(561)
error is a mismatch in a Check text node.
Exceptions
Exceptions are the most serious kinds of errors. An exception is thrown when a
situation occurs in which QF-Test cannot simply continue with the execution of
the test. Most exceptions indicate problems with the logic of the test, though they
(662)
can just as well be caused by the SUT. A ComponentNotFoundException ,
for example, is thrown when no component in the SUT matches the intended
(662)
target for an event. A list of all possible exceptions is available in chapter 35 .
Each node of a run-log has an associated state which can be one of normal, warning,
error or exception. This state is visually represented by a frame around the node’s icon
which is orange for warning, red for error and bold red for exception.
Figure 8.2: Error states in a run-log
8.1.
The run-log
77
As shown in the (somewhat reduced) screenshot above, error states propagate from
bottom to top. The exception state takes precedence over the error state, which in turn
overrides warning. The most severe kind of error that propagates to the top of the tree
determines the final result of a test and QF-Test’s exit code when run in batch mode
(695)
(see section 36.3 ).
(415)
If necessary, the propagation of errors can be restricted for all kinds of Sequence
(416)
nodes with the help of the Maximum error level
attribute. This can be useful for sequences which are known to contain errors that should not be taken into account just
(490)
(493)
and Catch
nodes. The
yet. Exceptions can be handled with the help of the Try
(495)
Maximum error level attribute of the Catch node determines the state to propagate for
a caught exception.
8.1.2
Navigating the run-log tree
All of the basic editing methods for a run-log are similar to those for a test-suite. One
significant difference is that can neither add or remove any nodes nor edit the attributes
of the nodes copied from the test-suite. You can add annotations though, for example
to document the reason for an error if it is known.
The first question to answer when looking at a run-log is ”What happened”?
Button, or Edit→Find next error , Ctrl-N for short, moves the selection to the
The
next place at which an error actually occurred.
Respectively,
or Edit→Find previous error (Ctrl-P ) moves backwards.
(386)
(386)
The options Find next error function locates and Skip suppressed errors determine
the kinds of errors to look for and whether to ignore those that didn’t propagate up to
the root node. There’s a menu item shortcut Edit→Skip suppressed errors to quickly
toggle the latter option.
The next question might be ”Where did this happen?”.
Though a run-log is similar in many ways to a test-suite,
the connection isn’t always
obvious. The function Edit→Find node in test-suite (Ctrl-T ) will take you to the exact
node in the test-suite that is represented by the selected node in the run-log, always
provided that the test-suite can be located and hasn’t changed in a way that prevents
this. If the run-log is loaded from a file, the corresponding test-suite may not be located
at the same place as when the test was executed. If the test-suite cannot be found, a
dialog will pop up that lets you select a different file. In case you select a wrong file or
some other test-suite is found instead of the one the run-log was created from, you may
end up at some totally different node, or none at all. In that case you can use the menu
item Edit→Locate corresponding test-suite to explicitly change the test-suite.
8.1.
The run-log
78
If you want to set the link between the file path of the executed test-suite and the file
path where you develop the test-suite permanently you can do so in the options menu
(392)
for the log-file as explained in Directory map for test-suites .
8.1.3
Accepting values of failed checks as good
A noteworthy feature of QF-Test’s run-log is the ability to quickly accept the actual values of a failed Check node as good. When QF-Test logs a failed Check it includes the
complete current state of the Check node’s target component in the SUT. This is much
more useful than the failure message alone, which, for example, might just tell you that
a table column has 10 rows instead of the expected 9, but not what its contents are.
If you are analyzing a failed Check and see that the value in the SUT was actually correct
and the expected value stored in the test-suite wrong, you can press Ctrl-U or select
Update check node with current data from the context menu to accept the data from
the failed Check as the new correct value for the Check node.
(561)
Warning: QF-Test currently doesn’t preserve regular expressions in Check text
(572)
Check items nodes, they will simply get overwritten.
8.1.4
or
Split run-logs
3.0+
Run-logs for long-running tests can get very large and consume an enormous amount
of memory, even more so in case many screenshots are kept. Compact run-logs help,
but not enough to make tests that run for days on end possible without turning off the
run-log entirely. The best way to overcome the memory problem are split run-logs.
For split run-logs, whenever a certain part of a test has finished, QF-Test takes the runlog for that part, removes it from the main run-log, saves it to a separate file and replaces
it with a single node that references that file. The partial logs are complete run-logs in
themselves and can be viewed and archived independently, though normally they are
accessed through the main run-log. When navigating the main run-log or when creating reports, QF-Test transparently loads the partial run-logs from the separate files as
required and removes them from memory when no longer needed. This makes it possible to navigate huge run-logs while still retaining a relatively small memory footprint.
Of course operations like searching or report creation that need to traverse the whole
run-log become slower, but jumping from error to error remains quite fast and loading
the main run-log is sped up drastically.
There are two ways for storing a main run-log and its partial logs: All combined together
in a single ZIP file with the extension .qzp or with the partial logs in a separate directory.
The latter is named after the main run-log with the extension .qrl or .qrz removed
and the suffix _logs appended. Inside a .qzp ZIP file the same layout is used so that
8.1.
The run-log
79
it is possible to zip or unzip files manually without breaking the internal references in the
run-log. This compatibility is also the reason why by default partial logs inside the ZIP
are are stored compressed with the extension .qrz. This is slightly less efficient than
storing uncompressed .qrl files, but that way a .qzp run-log can be unzipped without
its overall size exploding.
To make use of split run-logs you can explicitly define points at which a run-log is bro(442)
ken and split into parts. This is done via the Name for separate run-log
attribute of
(440)
(398)
(405)
(411)
(417)
Data driver , Test-case , Test-set , Test call or Test-step nodes. When used with
a Data driver, the logs for each iteration are saved separately, in the other cases the node
with the Name for separate run-log attribute is split off. Otherwise partial run-logs are split
off automatically when they reach a certain size. This functionality can be configured
(387)
via the option Minimum size for automatic splitting (kB) .
Split run-logs are also very handy for tracking the progress of a test in batch mode. In
that context it is extremely useful that the file names for the partial logs can be created
using the same placeholders as when specifying the name of the main run-log on the
command line. In particular the error state of the partial log can be made part of its
(442)
filename. Please see the documentation for the attribute Name for separate run-log for
details.
8.1.5
Run-log options
There are several options that influence the creation of run-logs and their content. These
(383)
are explained in detailed in section 33.6 . Following is a short summary of the most
important options:
(385)
• Unless Automatically save run-logs
is turned off, recent run-logs are saved in
the directory .qftest in the user’s home directory to save memory and to make
them persistent between sessions.
(385)
• Number of run-logs in menu
sets the number of run-logs to keep around after
a test is run. Unless you turn off automatically saved run-logs there is no need to
reduce this number.
(389)
• Log level for the SUT determines which kinds of messages are logged. Logging
all messages uses up lots of memory.
(391)
• If Create compact run-log is set, only nodes containing relevant error or report
information are retained. This is a very useful setting in most cases.
(387)
• By turning Create split run-logs
off, split run-logs can be disabled. Normally
the Name for separate run-log attributes of various nodes determine if and where a
run-log is split.
8.1.
The run-log
80
(391)
• With Don’t create run-log creation of run-logs can be suppressed entirely. This
is dangerous, since no information will be available in case of errors. Since split
run-logs have become available this option is no longer required to make longrunning demos or stress tests possible, but it is retained for backwards compatibility.
(392)
to map the directory for test-suites to be
• Specify Directory map for test-suites
executed on one computer with the developing directory on another computer.
Then you can continue using ’Find node in test-suite’ and ’Locate corresponding
test-suite’ from the run-log in case the directory structure differs on the two computers.
8.1.6
Creating a test-suite from the run-log
3.3+
If several people are involved in the test development process, it might be useful to
generate a test-suite from the run-log directly. The generated test-suite could be used
to reproduce a test-run on-the-fly without having the entire structure of test-suites.
You can create a test-suite from the run-log via performing a right mouse click at any
node in the run-log and selecting Create test-suite from the context menu.
QF-Test creates a new file containing all executed steps of the respective tests under
(425)
Extras as well as the used components.
Note
QF-Test only adds the executed steps to the new test-suite. Variables will be expanded
immediately, so you can only see their value in the new file. Organizational nodes like
procedures or control structures will not become created.
You have to set a couple of options in order to get this feature properly working (Under
Run-log -> Content):
(391)
• Create compact run-log
(389)
• Log variable expansion
needs to be switched off.
needs to be switched on.
(389)
• Log parent nodes of components
needs to be switched on.
If you have access to all test-suites, you can use also use information from them for
creating the new one. Therefore select Create test-suite from existing structure from
the context menu. In contrast to the approach described above, it is not required to
(389)
switch on the option Log parent nodes of components .
8.2.
The debugger
8.1.7
81
Merging run-logs
4.1+
During test development you might face the requirement, that you have a run-log with
the test-results for your test-cycle. But in several cases you might need to re-run one
test-case which was failing due to subtle reasons during the previous test-run. Once
that re-run has taken place you would like to update your test-report to have that new
execution in that test-report instead the previous one. For this purpose it’s possible to
merge run-logs via command line.
A typical merge command looks like this:
qftest -batch -mergelogs -mergelogs.mode=replace
-merglogs.masterlog full_log.qzp
-mergelogs.resultlog newresult_log.qzp rerun.qzp
Example 8.1: Sample call of merging run-logs for update
The command above takes the result of the re-run from the run-log
newresult_log.qzp, searches for the test-case in the run-log full_log.qzp and
store the updated run-log to newresult_log.qzp. If you set the parameter
mergelogs.mode to merge the new test-cases will be added to the existing structure
and the original test-cases will remain in the run-log.
Another case might be to add run-logs of several test-runs into one large run-log in
order to get a more readable report. This kind of merging is also implemented and can
be achieved by another command line call like this:
qftest -batch -mergelogs -mergelogs.mode=append
-mergelogs.resultlog newresult_log.qzp run1.qzp run2.qzp
Example 8.2: Sample call of merging run-logs for adding
The call above takes the run-logs run1.qzp and run2.qzp and creates a run-log
newresult_log.qzp which contains the results from both run-logs.
8.2
The debugger
As in any development environment, at some point the need will arise to debug problems
introduced into a test-suite which cannot readily be solved by a straight-forward analysis
of the elements and structure of the suite. To this end, QF-Test includes an intuitive
debugger. Those of you familiar with debugging programs in Java or other programming
languages will find this debugger similar in function and usefulness.
8.2.
The debugger
8.2.1
82
Entering the debugger
The QF-Test debugger can be started directly by selecting a node (or some nodes) to
execute and pressing the step-in
or step-over
buttons, or by using the menu
operations Debugger→Step in and Debugger→Step over or the keyboard shortcuts
F7 and F8 .
(83)
See section 8.2.3
for a detailed explanation of these operations.
If you are running tests on your test-suite and use the play button to start execution (see
(34)
section 4.2 ), the debugger will normally not be entered. However, the debugger will
be activated automatically when any one of the following occur:
(84)
• A user-defined breakpoint is reached (see section 8.2.4
points).
on turning on/off break
• Execution is interrupted manually by pressing the pause button or F9 or selecting
the Run→Pause menu item.
• A caught or uncaught exception is thrown, an error happens or a warning is logged
and the respective option to break under that condition is set (see option Automatic
(382)
breaks ).
When the debugger suspends execution of the test, the node about to be executed will
be shown with a colored frame around its icon that reflects the cause for the break. If the
debugger is stopped due to manual intervention, a user breakpoint or when stepping,
the frame will be black. When stopping due to a warning, error or exception the frame
will be orange, red or thick red respectively, exactly like the error indicators in the run-log.
Note
When the debugger is entered due to a warning, error or exception it will move execution
back to the beginning of the node that caused it, giving you a chance to fix the cause
of the problem and re-execute that node. If this is undesirable or impossible you can
(83)
simply skip the node (see section 8.2.3 ).
8.2.2
The debugger window
The debugger can be run either from within the normal test-suite view, or by opening
a dedicated debugger window by selecting Debugger→Show debugger window once
the debugger has been entered.
You can also cause the debugger window to open automatically whenever the debugger
(382)
is entered by setting the option Always open debugger window in the global options
dialog or under the Debugger→Options menu. If you open or close the debugger
window explicitly, this is considered a ”manual override” and this option will be ignored
for the rest of the test-run.
8.2.
The debugger
83
The debugger window is similar to a normal test-suite window. You can select nodes
and edit their attributes, but you cannot delete or insert nodes, there are no file operations and no recorder. For the more complex operations you can quickly jump from
the
debugger window to the same node in the respective test-suite window by pressing
Ctrl-T or selecting Find node in test-suite from the Edit menu or the context popup
menu.
The lower half of the debugger window shows the nodes that are binding variables on
(62)
the primary and fallback variable stacks (see chapter 7 ). For the primary stack all
nodes are shown, even if they are not binding any variables. This is useful because it
serves as a kind of stack-trace for the current point of execution of the test-run. You can
double-click on any node to quickly navigate to the node in its associated test-suite.
A single click on a node brings up its variable bindings in the right half of the variable
display. There the variable values can be edited, new variables can be added or existing
ones removed. These changes immediately affect the current test-run, but are of a
temporary nature. They are not propagated to the nodes in which the variables were
bound in the first place.
8.2.3
Debugger commands
Most of the debugger commands are similar to those of any other debugger. However,
some additional commands are included that deal with special situations.
Step-wise debugging of a test-suite is available through three operations:
(F7 , Debugger→Step in ) executes the currently selected
• The step-in button
node and will set the execution mark to the next deepest node, regardless of how
deep that node may lie in the node structure. This operation is useful, for example,
to step into and debug a Procedure or Sequence.
(F8 , Debugger→Step over ) executes the currently se• The step-over button
lected node as well as any child nodes that lie under it and then sets the execution
mark to the next node at the same level. This is helpful for being able to execute
an entire Procedure or Sequence without stepping through each step individually.
• The step-out button
(Ctrl-F7 , Debugger→Step out ) executes the currently
selected node as well as any other nodes at the same level (including any child
nodes of these nodes) and then sets the execution mark to the next node at the
next higher level. This type of operation is useful when, for example, you are
debugging a Procedure or Sequence and don’t want to step through the rest of the
nodes in it. By simply using step-out, you can execute the rest of the nodes and
return.
8.2.
The debugger
84
The skip functions expand the QF-Test debugger in a powerful way which is not typically
possible for a debugger in a standard programming environment. In short, they allow
you to jump over one or more nodes without having to execute those nodes at all.
• The skip-over button
(Shift-F9 , Debugger→Skip over ) jumps over the current
node without executing it, moving the execution mark to the next node.
(Ctrl-F9 , Debugger→Skip out ) ends the execution of the
• The skip-out button
current Procedure or Sequence and jumps to the next node at the next higher level.
Even more powerful is the ability to continue the test-run at any arbitrary node, even
in a completely different test-suite. QF-Test will keep as much of the current execution
context as possible, including variable bindings. The closer the new target location is to
the current point of execution, the more information can be salvaged.
You can switch execution to a different node by pressing Ctrl-, or by selecting the menu
item Run→Continue execution from here or the respective item in the context menu.
When you do so, execution will not continue immediately, only the next node to be
executed will change. You can continue the test as usual by single-stepping or resuming
the test-run.
The following additional commands are available:
( Debugger→Rethrow exception ) is only active
• The rethrow-exception button
when the debugger was entered due to an exception. It lets you rethrow the exception to be handled by the test-suite just as if the debugger had never caught it
in the first place.
• The locate-current-node button
( Debugger→Locate current node ) quickly
moves the selection in the tree-view to the node that is about to be executed. It is
a useful shortcut in case you get lost while moving around the test-suite.
8.2.4
Manipulating breakpoints
Setting a breakpoint on a node will tell the debugger to suspend execution just before it
enters that node. Breakpoints are displayed in the tree-view by prepending ”(B)” to the
name of a node.
Breakpoints can be set or removed individually with Ctrl-F8 or with the
Debugger→Breakpoint on/off menu item. After finishing a debugging session you can
use Debugger→Clear all breakpoints to remove any breakpoints that might have been
left hanging around. This command will remove all breakpoints from all test-suites.
8.2.
Note
The debugger
Breakpoints are transient and will not be saved with the test-suite.
85
Chapter 9
Organizing the test-suite
Creating useful, reliable tests requires more than just recording sequences and playing
them back. You can fill a test-suite with lots of sequences in a short time, but you are
bound to lose track of what you’ve got sooner or later if you do not organize your tests
in some logical structure. QF-Test provides you with a number of structure elements to
achieve this.
Before you start recording sequences and put them into a structure make sure that you
• You have a good idea of what you are testing.
• You are testing the right thing.
• Your tests are reliable and repeatable.
• Your tests are easy to maintain.
• The results of your tests are conclusive.
The essential prerequisite of getting the components right has been discussed in
(40)
chapter 5 . Here we are going to concentrate on structuring the actual test-sets,
test-cases, test-steps, sequences, events, checks, etc.
9.1.
9.1
Test-suite structure
87
Test-suite structure
Figure 9.1: Test-suite structure
QF-Test provides structure elements on different levels:
The QF-Test files for saving the tests and components in the file directory. These can
be bundled in projects.
The Test-suite has a set structure starting with the testing section that can hold any
(405)
(398)
number of Test-set nodes, which in turn can have any number of Test-case nodes or
more Test-sets.
(463)
(463)
Next comes the Procedures section, where you can place any number of Procedure
(470)
nodes. QF-Test provides Package nodes as structure element in this section. A package node can hold any number of procedure nodes or more package nodes.
(425)
After that you will find the Extras where you place any type of node and try out tests
before moving them to the testing section.
(655)
The last section, Windows and components
by the tests.
, is reserved for the components referenced
QF-Test provides a number of structure elements for the tests themselves like Test-case
9.2. Test-set and Test-case nodes
88
and Test-set nodes as well as Setup and Cleanup nodes for setting up the preconditions
for the tests, cleaning up after the test and error handling.
9.2
9.2.1
Test-set and Test-case nodes
Test management with Test-set and Test-case nodes
2.0+
(405)
(398)
The Test-set and Test-case nodes provide a small-scale, pragmatic form of test management right inside QF-Test. Their main feature is the smart dependency management
(94)
described in Dependency nodes that allows Test-cases to be implemented completely in(427)
dependent from each other. With properly written Dependencies , cleanup of the SUT
for previously executed tests is handled automatically along with the setup for the next
test and all error handling.
9.2.2
Concept
Conceptually a Test-case node represents a single elementary test case. As such it
is the main link between test planning, execution and result analysis. With the help of
(427)
Dependencies , Test-cases can be isolated from each other so that they can be run in any
arbitrary order. QF-Test automatically takes care of the necessary test setup. Cleanup
is also automatic and will be performed only when necessary in order to minimize overhead in the transition from one test to the next. This enables things like running subsets
of functional test-suites as build tests or retesting only failed Test-cases.
Test-sets basically are bundles of Test-cases that belong together and typically have similar
requirements for setup and cleanup. Test-sets can be nested. The whole structure of Test(470)
(463)
sets and Test-cases is very similar to Package and Procedure nodes. The Test-suite root
node can be considered a special kind of Test-set.
Test-suite, Test-set and Test-case nodes can be called from other places using a Test call
node. That way, tests that run only a subset of other tests can easily be created and
managed. Test call nodes are allowed everywhere, but should not be executed from
within a Test-case node because that would break the atomicity of a Test-case from the
report’s point of view. A warning is issued if Test-case execution is nested.
9.2. Test-set and Test-case nodes
9.2.3
89
Variables and special attributes
Default value
As both Test-sets and Test-case can be called via a Test call node they each have a set
(463)
of default parameters similar to those of a Procedure . These will be bound on the
secondary variable stack and can be overridden in the Test call node.
Variable definitions
A Test-case has an additional set of variable bindings. These are direct bindings for the
primary variable stack that will be defined during the execution of the Test-case and
cannot be overridden via a Test call node or the command line parameter
(693)
-variable <name>=<value> .
Primary and secondary variable stack are
(62)
described in section 7.1 .
Characteristic variables
The list of Characteristic variables is a set of names of variables that are part of the characteristics of the test for data-driven testing. Each execution of the Test-case with a
different set of values for these variables is considered a separate test case. The expanded values of these variables are shown in the run-log and report for improved error
analysis.
Condition
(482)
(481)
of an If
Another useful attribute is the Condition which is similar to the Condition
node. If the Condition is not empty, the test will only be executed if the expression
evaluates to true. Otherwise the test will be reported as skipped.
Expected to fail if...
Sometimes a Test-case is expected to fail for a certain period of time e.g. when it is created prior to the implementation of the respective feature or before a bug-fix is available
in the SUT. The Expected to fail if... attribute allows marking such Test-cases so they are
counted separately and don’t influence the percentage error statistics.
9.3. Sequence and Test-step nodes
9.3
90
Sequence and Test-step nodes
(415)
(417)
and Test-step
nodes which
The primary building block of a test are the Sequence
execute their child nodes one by one in the order as they appear. They are used to
structure the child nodes of a Test-case.
(415)
(417)
(417)
The difference between Sequence and Test-step nodes is that Test-step
(415)
show up in the report whereas Sequences will not.
9.4
nodes will
Setup and Cleanup nodes
Since it is in the nature of testing that tests may fail from time to time it is crucial to have
structure elements that will help you set up a defined initial state for a test. Setup and
Cleanup nodes are for simple cases and are inserted as child nodes of Test-case nodes.
However, in most cases Dependency nodes, that contain Setup and Cleanup nodes, will
prove far more efficient.
Test-case nodes with well designed Setup and Cleanup nodes have the following properties
important to successful testing:
• The Test-case can be executed independently of previous test cases that may have
failed.
• Test-case nodes can be added at any position in Test-suite and Test-set nodes without
influencing other Test-cases
• You can work on a Test-case or just run it without having to execute previous Testcases to get the SUT into the state required by your test.
• You can execute any number of Test-case nodes in case you do not want to run the
whole Test-set or Test-suite.
In the simplest case exactly the same initial condition is required by all the Test-case
nodes of a Test-set. This can be implemented via the following structure:
9.4. Setup and Cleanup nodes
91
Figure 9.2: Test structure with simple Setup and Cleanup
In the run-log you can see that for each Test-case node first the Setup node and then the
Cleanup node is run:
Figure 9.3: Test execution with simple Setup and Cleanup
In this simple example the cleanup is done in any case, even if the next test could be
executed with the state the previous test left the SUT in. QF-Test provides a more comprehensive structure for setting up the SUT and handling cleanup much more efficiently,
(94)
and even including error handling. This is explained in chapter section 9.6 in detail.
9.5. Procedures and Packages
9.5
92
Procedures and Packages
In a way, writing good tests is a little like programming. After mastering the initial steps,
tests and source code alike tend to proliferate. Things work fine until some building block
that was taken for granted changes. Without a proper structure, programs as well as
tests tend to collapse back upon themselves at this point as the effort of adapting them
to the new situation is greater than the one needed for recreating them from scratch.
The key to avoiding this kind of problem is reuse or avoidance of redundancy. Generating redundancy is one of the main dangers of relying too much on recording alone.
To give an example, imagine you are recording various sequences to interact with the
components in a dialog. To keep these sequences independent of each other, you start
each one by opening the dialog and finish it by closing the dialog again. This is good
thinking, but it creates redundancy because multiple copies of the events needed to
open and close the dialog are contained in these sequences. Imagine what happens if
the SUT changes in a way that invalidates these sequences. Let’s say a little confirmation window is suddenly shown before the dialog is actually closed. Now you need to go
through the whole suite, locate all of the sequences that close the dialog and change
them accommodate the confirmation window. Pure horror.
To stress the analogy again, this kind of programming style is called Spaghetti Programming and it leads to the same kind of maintenance problems. These can be avoided
by collecting the identical pieces in one place and referring to them wherever they are
needed. Then the modifications required to adapt to a change like the one described
above are restricted to this place only.
Figure 9.4: Packages and Procedures
QF-Test comes with a set of nodes that help to achieve this kind of modularization,
(463)
(466)
(470)
namely the Procedure , Procedure call and Package nodes. A Procedure is similar to
(415)
(464)
a Sequence except that its Name attribute is a handle by which a Procedure call node
can refer to it. When a Procedure call is executed, the Procedure it refers to is looked up
and execution continues there. Once the last child node of the Procedure has finished,
the Procedure call has completed as well.
Packages are just a way to give even more structure to Procedures. A hierarchy of Pack(472)
ages and Procedures, rooted at the special Procedures
node, is used to group sets of
9.5. Procedures and Packages
93
Procedures with a common context together and to separate them from other Procedures
used in different areas.
A Procedure that always does exactly the same, no matter where it is called from, is
only marginally useful. To expand on the above example, let’s say we want to extend
the Procedure that opens the dialog to also set some initial values in some of its fields.
Of course we don’t want to have these initial values hard-coded in the Procedure node,
but want to specify them when we call the Procedure to get different values in different
contexts. To that end, parameters can be defined for the Procedure. When the Procedure
call is executed, it specifies the actual values for these parameters during this run. How
(62)
all of this works is explained in Variables . Also please take a look at the detailed
(463)
(466)
explanation for the Procedure and Procedure call nodes for a better understanding of
how these complement each other.
A test-suite library with a set of commonly useful Procedures is provided with QF-Test
under the name qfs.qft. An entire chapter of the Tutorial is devoted to this library and
(193)
section 19.1 explains how to include it in your test-suites.
9.5.1
Local Procedures and Packages
3.1+
If you work with several test-suite libraries you might face a situation, where you define
reusable test-steps or sequences, which you only want to use within a dedicated testsuite. If you want to create such local Procedures, you can put a ’_’ as first sign of the
procedure’s name. This marks a Procedure as test-suite local.
A call of a local Procedure can only be inserted within the test-suite, where it is defined.
You can use the same concept for local Packages.
3.1+
9.5.2
Relative Procedures
If you call Procedures from other Procedures, it could be convenient not to specify the full
procedure name all the time.
So called ’relative’ procedure calls can only be added to a Package, which has the Border
(471)
for relative calls (see Border for relative calls ) attribute specified. The structure of that
call follows the concept below:
Level
Procedures of the same level
Procedures one level higher
Procedures one level deeper
Call
.Name of Procedure
..Name of Procedure
.Name of Package.Name of Procedure
Table 9.1: Relative procedure calls
9.5. Procedures and Packages
94
As you can see each dot stands for one level. So calling a Procedure two levels higher
requires three dots (Current level also requires a dot.)
9.5.3
Inserting Procedure call nodes
As you should organize your tests in separate test steps, which are ideally the same like
QF-Test’s procedures, QF-Test offers several ways to insert those Procedure call nodes:
1. Via the menu Insert→Procedures→ProcedureCall
2. Via right mouse click and selecting Insert node→Procedures→ProcedureCall
3. Copy a Procedure node and insert it at the location of the Procedure call using the
normal Copy/Paste actions
4. Via Drag&Drop operation, i.e. dragging the Procedure node to its target node
5. Via the keyboard shortcut Ctrl-A This approach is also valid for inserting Dependency reference nodes, except the keyboard
shortcut.
9.5.4
Parameterizing nodes
3.1+
You can create parameters for a Procedure, Dependency or Test-case automatically via the
menu Operations→Parameterize node .
The parameter details dialog allows you to define for which actions you want to create
parameters, e.g. only text-inputs or check nodes.
9.5.5
Transforming a Sequence into a Procedure
3.0+
This transformation is very useful for developing procedures immediately after recording!
Under Extras you can convert a recorded Sequence node into a Procedure and move that
to the Procedures node.
3.1+
If you transform a Sequence under Test-cases QF-Test automatically creates a Procedure
node and inserts a Procedure call to the previous location of the transformed node.
9.6. Dependency nodes
9.6
9.6.1
95
Dependency nodes
Concept
Dependencies are a powerful and optimized concept for handling pre- and
post-conditions. They are indispensable when running tests in the QF-Test Daemon
(882)
mode mode. They basically work the following way:
1. Set up a list of all dependencies required for the test-case.
2. Compare the list of dependencies needed for the current test-case with the list of
dependencies of the test-case executed last.
3. Execute all Cleanup nodes of the dependencies no longer part of the current
(100)
dependencies list plus the ones where the values of Characteristic variables
changed, including the Cleanup nodes of the Dependencies based on them.
4. Execute all Setup nodes of the current dependencies list.
(427)
Test-cases as well as other dependencies can make use of Dependency nodes placed
(463)
(430)
in the Procedures
section via Dependency reference
nodes. Therefore, Setup and
Cleanup nodes placed in a Dependency node can be used by various test-cases - in
contrast to those placed directly in Test-case or Test-set nodes.
In order to understand the concept of Dependency nodes it might be helpful to have a
look at how a manual tester would proceed: He would do the setup for the first test case
and then run it. In case of errors he may want to run special error cleanup routines. After
that he would first check the requirements of the second test case. Only then would he
do any cleanup. And he would only clean up as much as is necessary. Next he would
check that the SUT still meets all preconditions required by the next test case and if not
execute the necessary steps. In case the previous test case failed badly he might need
to clean up the SUT completely before being able to set up the initial condition for the
second test case.
This is exactly what you can implement using QF-Test Dependencies.
Dependencies give an answer to the disadvantages of the classical Setup and Cleanup
(89)
nodes where Setup nodes can only be nested by nesting test-sets and where Cleanup
nodes will be executed in any case, both of which is not very efficient. Moreover, Dependency nodes provide structure elements for handling errors and exceptions.
Quite a number of the provided sample test-suites make use of Dependencies, e.g.:
• The test-suite in the directory doc/tutorial named dependencies.qft. You
will find a detailed description in the tutorial in chapter 16.
9.6. Dependency nodes
96
• The test-suite in demo/carconfig named carconfig_de.qft, showing a realistic example.
• The SWT demo test-suite named swt_addressbook.qft, with an example for
SWT users
• The test-suite in demo/eclipse named eclipse.qft, containing nested Dependencies.
• The data driver demo datadriver.qft in doc/tutorial also uses Dependencies.
Single-stepping through these suites in the debugger, looking at the variable bindings
and examining the run-logs should help you to familiarize yourself with this feature.
Please take care to store modified test-suites in a project-related folder.
9.6.2
Usage of Dependencies
You can define Dependencies in two places:
• You can implement Dependency nodes at the beginning of Test-suite, Test-set or Testcase nodes. Additionally to their own Dependency a Test-case or Test-set may inherit
the Dependency of its parent node.
• Dependencies used by a number of Test-cases or used as a basis for other Depen(463)
node and be placed in the
dencies may be implemented just like a Procedure
(472)
(470)
Procedures section, e.g. in a Package node named ”Dependencies”. The fully
(463)
qualified name has the same structure as that of a Procedure. Just like Procedures
Dependencies can be referred to by other nodes, in this case via Dependency reference nodes.
One Dependency should deal with one precondition. Then you can reduce the test overhead generated by cleanup activities. In case a Dependency itself relies on preconditions
these should be implemented in separate Dependency nodes. Dependencies can either
be inherited from a parent node or referred to explicitly via Dependency reference nodes.
The implementation of the actual pre- and post-conditions is done in the Setup and
Cleanup nodes of the Dependency.
In case a Test-set or Test-case node has a Dependency node as well as Setup and Cleanup
nodes the Dependency will be executed first. Setup and Cleanup nodes have no influence
on the dependency stack.
9.6. Dependency nodes
97
9.6.3 Dependency execution and Dependency stack
The execution of a Dependency has three phases:
1. Generate a list of required Dependencies and check with list of previously executed
Dependency nodes
2. Execute Cleanup nodes if required
3. Execute Setup nodes of all required Dependency nodes
The examples used in this chapter all refer to tests with the following preconditions and
cleanup activities:
Dependency A:
Setup: start application if necessary
Cleanup: stop application
Dependency B:
Setup: log in user if necessary
Cleanup: log off user
Dependency C:
Setup: load application module 1 if necessary
Cleanup: close application module 1
Dependency D:
Setup: load application module 2 if necessary
Cleanup: close application module 2
Dependency E:
Setup: open dialog in module 2 if necessary
Cleanup: close dialog
Dependency C depends on B, B in turn on A.
Dependency E depends on D, D on B therefore also on A.
Before executing a Test-case node QF-Test checks whether it has a Dependency node of
its own and/or inherits one from its parent nodes. In that case QF-Test checks whether
9.6. Dependency nodes
98
the Dependency node itself relies on other dependencies. Based on this analysis QFTest generates a list of the dependencies required. This is done in step 1 of the example
below.
Next, QF-Test checks if previous tests have already executed dependencies. If so, QFTest checks if it has to execute any Cleanup nodes. After that QF-Test goes through
all the setup nodes, starting with the most basic ones. The name of each Dependency
executed is noted down in a list called dependency stack. See step 2 of below example.
Example: Test-case 1
Test of application module 1. First test-case to be executed.
1. step
Analyze the dependencies: the dependencies A-B-C have to be executed.
2. step
Compare the dependencies to be executed with the dependency stack: In
this example the dependency stack is still empty as the test-case is the first
one to be executed.
3. step
Execute the Setup nodes, starting with A (start application), then B (login user
(user name: Standard)), and last C (load application module 1).
The dependency stack now reads A-B-C.
4. step
Execute the Test-case.
In the run-log you can see exactly what QF-Test did:
9.6. Dependency nodes
99
Figure 9.5: Dependency stack A-B-C
After executing the test-case the application remains in the condition the last test-case
left it in. Only after analyzing the dependencies of the next test-case Cleanup nodes
might be run and the respective Dependency be deleted from the dependency stack.
When Cleanup nodes need to be run they are executed in reverse order to the Setup
nodes. After maybe clearing up dependencies no longer needed the Setup nodes of
all required Dependencies are executed. Just like a manual tester will check that all
requirements for the next test-case are fulfilled QF-Test will do the same. A manual
tester may not be conscious of checking the basic requirements. However, if he notices
that the last test-case left the application in a very bad state like a deadlock, he will
probably kill the process if nothing else helped and start it again. To this end QF-Test
explicitly runs all Setup nodes. These should be implemented in a way that they first
check if the application is already in the required state and just in case not run the
whole Setup node.
9.6. Dependency nodes
100
Figure 9.6: Good practice Setup node
Setups nodes should first check if the required condition already exists before actually
executing the node. Cleanup nodes should first check if the requested cleanup action
(e.g. closing a dialog) has already been performed. Also they should be programmed
in such a way that they are in grade of clearing up error states of the application (e.g.
error messages) so that a failed test-case will not affect the following ones.
Example: test-case 2
Test a dialog in application module 2
1. step:
Analyze the dependencies: the dependencies A-B-D-E have to be executed.
2. step:
Compare the list of dependencies to be executed with the dependency stack:
Dependency C is not required for test-case 2. Therefore Cleanup node of Dependency C is executed (close application module 1).
3. step:
Execute the Setup nodes, starting with A (check the application is already
started and skip the rest of the setup), then B (check the user is already
logged in, and skip the rest of the setup), then D (check if application module
2 is loaded and as it is not execute the complete Cleanup node), then E (same
as D).
4. step:
Execute the Test-case.
You can see in the run-log that the cleanup was done:
9.6. Dependency nodes
101
Figure 9.7: Dependency stack A-B-D-E
9.6.4 Characteristic variables
Values of certain variables may determine whether a dependency has to be cleared up
and the setup re-executed, like the user name for dependency B ’Login’. These variables
are called Characteristic variables. The values of the Characteristic variables are always
taken into account when comparing dependency stacks. Two Dependencies on the stack
are only considered identical if the values of all Characteristic variables from the previous
and the current run are equivalent. Consequently it is also possible for a Dependency
to directly or indirectly refer to the same base Dependency with different values for its
Characteristic variables. In that case the base Dependency will appear multiple times in the
linearized dependency stack.
Furthermore, QF-Test stores the values of the Characteristic variables during execution
of the Setup of a Dependency. When the Dependency is rolled back, i.e. its Cleanup node
9.6. Dependency nodes
102
is executed, QF-Test will ensure that these variables are bound to the same value as
during execution of the Setup. This ensures that a completely unrelated Test-case with
conflicting variable definitions can be executed without interfering with the execution
of the Cleanup nodes during Dependency rollback. Consider for example the commonly
used ”client” variable for the name of an SUT client. If a set of tests for one SUT has
been run and the next test will need a different SUT with a different name, the ”client”
variable will be changed. However, the Cleanup node for the previous SUT must still refer
to the old value of ”client”, otherwise it wouldn’t be able to terminate the SUT client. This
is taken care of automatically as long as ”client” was added to the list of Characteristic
variables.
Example: Test-case 3:
Test of the same dialog for the user Administrator.
1. Step:
Analyze the dependencies: same list of dependencies A-B-D-E as in Test-case
2. However, the Characteristic variables of Dependency B has a different value,
i.e. ’Administrator’.
2. Step:
Compare the dependency list with the dependency stack: The required Dependency B differs from the Dependency B saved on the dependency stack
because of the values of the Characteristic variables ’username’, which is ’Standard’ on the dependency stack. This means that the dependency stack will be
rolled back including Dependency B, starting with the Cleanup for Dependency E
(close dialog), then Cleanup for Dependency D (stop module 2), then Cleanup
for Dependency B (log off user - the variable ’username’ then has the value
’Standard’ saved via the Characteristic variables on the dependency stack).
3. Step:
Execute the Setup nodes, starting with A (check the application is already
started and skip the rest of the setup), then B (log in user ’Administrator’),
then D (load module 2), then E (open dialog).
4. Step:
Execute the Test-case.
In the run-log you can see the values of the Characteristic variables behind the respective
Dependency:
9.6. Dependency nodes
103
Figure 9.8: Dependency with Characteristic variables
Other examples for Characteristic variables are JDK versions when the SUT needs to
be tested for various JDK versions or the browser name with web applications. In our
example these would be specified as Characteristic variables for Dependency A.
9.6.5 Forced cleanup
In some use cases it may be necessary to execute the Cleanup node of a Dependency
after each Test-case. Then you should set the attribute Forced cleanup.
9.6. Dependency nodes
104
If Forced cleanup is activated for a Dependency node on the list of dependencies the
Cleanup node of this and maybe of subsequent Dependencies will be executed.
Example:
In this example the test logic requires module 2 to be stopped after test
execution. The attribute Forced cleanup is activated for Dependency D.
In our example the Cleanup nodes of Dependencies E (close dialog) and D (stop
modul) would be executed after each Test-case.
9.6.6
Rolling back Dependencies
QF-Test rolls back Dependencies depending on the needs of the Test-cases.
If you want to clear the list of dependencies explicitly there are two ways to do it:
• The menu item Run→Roll back dependencies rolls back the list of dependencies
’cleanly’ executing all the Cleanup nodes in reverse order to the setup activities.
• The menu item Run→Reset dependencies just deletes the list of dependencies
without executing any nodes.
When a Test-case does not use Dependencies the list of dependencies remains untouched,
i.e. no Cleanup nodes are executed.
9.6.7
Error escalation
Another thing that is just grand about Dependencies is the convenient way that errors
can be escalated without any additional effort. Let’s again consider the example from
the previous section after the first dependency stack has been initialized to A-B-C (Application started, user logged in, module one loaded) and the Setups have been run.
Now what happens if the SUT has a really bad fault, like going into a deadlock and not
reacting to user input any longer?
When a Cleanup node fails during rollback of the dependencies stack, QF-Test will roll
back an additional Dependency and another one if that fails again and so on until the
stack has been cleared. Similarly, if one of the Setups fails, an additional Dependency is
rolled back and the execution of the Setups started from scratch.
Example:
In the example Test-case 1 above the SUT would for example get deadlocked. In
9.6. Dependency nodes
105
Test-case 1 an exception would be thrown, Test-case 1 would be stopped and
execution passed on to Test-case 2.
1. Step:
Analyze the dependencies: the list of dependencies A-B-D-E has to be executed. (Application started, user logged in, module 2 loaded, dialog opened)
2. Step:
Comparing the dependency list with the dependency stack set up by Test-case
1 A-B-C (application started, user logged in, module 1 loaded) results in the
execution of the Cleanup node of Dependency C (stop module 1). Of course,
the exception is thrown again. Now QF-Test runs the Cleanup node for the
next Dependency B (log off user). This will fail again so that now the basic
Dependency A will be rolled back, which successfully stops the application.
3. Step:
Execute the Setup nodes, starting with A (start the application), then B (log in
user), then D (load module 2), then E (open dialog).
4. Step:
Test-case 2 will be executed despite the deadlock in Test-case 1.
Figure 9.9: Exception in forced cleanup sequence of C causes B to clean up
For this to work it is very important to write Cleanup sequences in a way that ensures
that either the desired state is reached or that an exception is thrown and that there
is a more basic dependency with a more encompassing Cleanup. For example, if the
Cleanup node for the SUT Dependency just tries to cleanly shut down the SUT through
its File->Exit menu without exception handling and further safeguards, an exception in
that sequence will prevent the SUT from being terminated and possibly interfere with
9.6. Dependency nodes
106
all subsequent tests. Instead, the shutdown should be wrapped in a Try/Catch with a
Finally node that checks that the SUT is really dead and if not, kills the process as a last
resort.
Figure 9.10: Typical Cleanup node
With good error handling in place, Test-cases will rarely interfere with each other even in
case of really bad errors. This helps avoid losing a whole night’s worth of test-runs just
because of a single error.
9.6.8
Error handling
Besides supporting automatic escalation of errors a Dependency can also act as an error
or exception handler for the tests that depend on it. Catch nodes, which can be placed
at the end of a Dependency, are used to catch and handle exceptions thrown during a
test. Exceptions thus caught will still be reported as exceptions in the run-log and the
report, but they will not interfere with subsequent tests or even abort the whole test-run.
An Error handler node is another special node that may be added to a Dependency after
the Cleanup and before the Catch nodes. It will be executed whenever the result of
a Test-case is ”error”. In case of an exception, the Error handler node is not executed
automatically because that might only cause more problems and even interfere with
the exception handling, depending on the kind of exception. To do similar things for
errors and exception, implement the actual handler as a Procedure and call it from the
Error handler and the Catch node. Error handlers are useful for capturing and saving
miscellaneous states that are not automatically provided by QF-Test. For example, you
may want to create copies of temporary files created during execution of your SUT that
may hold information pertaining to the error.
9.6. Dependency nodes
107
Only the topmost Error handler that is found on the dependency stack is executed, i.e.
if in a dependency stack of [A,B,C,D] both A and C have Error handlers, only C’s Error
handler is run. Otherwise it would be difficult to modify the error handling of the more
basic Dependency A in the more specialized Dependency C. To reuse A’s error handling
code in C, implement it as a Procedure.
9.6.9
Note
Name spaces for Dependencies
You might be interested in reading this section in case you want to run several SUTs
at the same time where you do not want the Dependency node for a test on one of the
SUTs to trigger cleanup actions for another SUT. Otherwise feel free to skip it.
A typical use case would be the test of whole process chains over several applications.
Consider the following situation: Sales representatives enter data for offers via a web
application into a database at headquarters. There, the offers will be completed, printed
and posted. A copy of each printed offer will be saved in a document management
system (DMS).
Figure 9.11: Example Test-set for name spaces
9.6. Dependency nodes
108
In above example two sales representatives (UserA and UserB) enter offers and two
different persons (UserC and UserD) process the offers at headquarters. Then the
offers will be checked in the document management system. Since you do not want the
dependencies of the test-cases to interfere with one another you need to add a suitable
(431)
name in the Dependency namespace attribute of each Dependency reference node.
After running the test-set you can see in the run-log that a dependencies stack was set
up in the name space ’data entry’ for the first test-case:
Figure 9.12: Dependency handling for test-case ’Data entry by User A’
A dependencies stack is set up in the name space ’database’ for the second test-case.
The dependencies stack in the name space ’data entry’ remains unheeded. Looking
at the applications, this means the database is started whereas the application for data
entry is left as it is.
Figure 9.13: Dependency handling for test-case ’Offer processing by User C’
A dependencies stack is set up in the name space ’DMS’ for the third test-case. The
dependencies stacks in the name spaces ’data entry’ and ’database’ remain unheeded.
Looking at the applications, this means the document management system is started
whereas the other two applications are left as they are.
9.6. Dependency nodes
109
Figure 9.14: Dependency handling for test-case ’Check offer 1 in DMS’
In test-case number four the required dependencies are checked against the ones on
the dependencies stack in the name space ’data entry’ of the first test-case. The dependencies stacks in the other two name spaces remain unheeded. Looking at the
applications, this means User A is logged off, User B is logged into the data entry application and the other two applications are left as they are.
Figure 9.15: Dependency handling for test-case ’Data entry by User B’
In test-case number five the required dependencies are checked against the ones on
the dependencies stack in the name space ’database’ of the second test-case. The
dependencies stacks in the other two name spaces remain unheeded. Looking at the
applications, this means User C is logged off, User D is logged into the database application and again the other two applications are left as they are.
9.7.
Documenting test-suites
110
Figure 9.16: Dependency handling for test-case ’Offer processing by User D’
In the last test-case the required dependencies are checked against the ones on the
dependencies stack in the name space ’DMS’ of the third test-case. The dependencies
stacks in the other two name spaces remain unheeded. Looking at the applications, this
means no clean up action has to be done on the DMS. The other two applications are
left as they are, anyway.
Figure 9.17: Dependency handling for test-case ’Check offer 2 in DMS’
9.7
Documenting test-suites
Like with any programming-related task it is important for successful test-automation
to properly document your efforts. Otherwise there is a good chance (some might say
a certainty) that you will lose the overview over what you have done so far and start
re-implementing things or miss out tests that should have been automated. Proper
documentation will be invaluable when working through a run-log, trying to understand
the cause of a failed test. It will also greatly improve the readability of test reports.
(411)
(405)
(398)
(470)
Based on the Comment
attributes of Test-set , Test-case , Package
and
(463)
Procedure nodes, QF-Test can create a set of comprehensive HTML documents that
will make all required information readily available. The various kinds of documents
(167)
and the methods to create them are explained in detail in chapter 17 .
Chapter 10
Projects
3.5+
Projects provide a better overview, improve navigation between test-suites and expand
the scope for search and replace operations. Also, QF-Test automatically manages
dependencies resulting from includes or absolute references between test-suites that
(193)
belong to the same project (see section 19.1 ). Many other features have already
been implemented or are under development.
Technically a QF-Test project is a set of test-suites located in one or more directories
with a common root. There is a 1:1 relation between the project and its directory and
the name of the directory automatically becomes the name of the project.
To create a new project, select the menu item File→New project... and choose the
directory. QF-Test then creates a file called qftest.qpj in that directory which identifies it as a project. All test-suites located below that directory, except those specified in
(321)
the option Project files and directories to exclude automatically belong to this project.
(320)
Please see section 33.1.1 for options affecting projects, including the exclusion list.
A sub-project is sub-directory of a project which is itself a project. Test-suites within a
sub-project also belong to all outer projects containing the sub-project. The project of a
test-suite is the innermost sub-project it belongs to. Automatic dependency resolution
always covers the whole outermost project of a suite including all sub-projects.
Projects
112
Figure 10.1: The project view
The project view with one or more projects can be turned on or off via the menu item
View→Show projects . The project tree shows the hierarchy of directories and testsuites starting from the project root, possibly limited by the filter at the top of the tree
which matches
on test-suite names. Double clicking a test-suite opens it, as does press
ing the Return key. You can select several files or directories to be opened in one go,
including all test-suites located below the selected directories.
The hierarchy is refreshed automatically at intervals defined in the option Project refresh
(320)
interval (s) . You can refresh
a directory including its complete hierarchy at any time
by selecting it and pressing F5 . For a more thorough rescan that does not rely on
modification times but may take significantly longer for large projects, press Shift-F5 instead.
To switch
keyboard
focus back and forth between the test-suite and the project view,
press F6 . Via Shift-F6 you can navigate to the node representing the current testsuite in the project tree. If necessary, project view and project are automatically shown.
Chapter 11
The standard library
The standard library qfs.qft, a test-suite that is part of the QF-Test distribution, contains many useful procedures for a diverse set of tasks.
The standard library
114
Figure 11.1: Standard library qfs.qft
Among others there are procedures for accessing and checking components (AWT,
Swing, JavaFX, SWT, Web) in a generic manner, file system and database access,
logging messages or screenshots to the run-log and report and for performing cleanup.
A complete description of all packages and procedures including parameters and return
values is provided in the library’s HTML documentation, also accessible from the QFTest Help menu. The latest version is also available online.
qfs.qft is included by default in every newly created test-suite. As its directory is on
(331)
(396)
the library path , specifying just qfs.qft in the Include files of the Test-suite node is
sufficient.
Note
All procedures referring to an SUT use the generic variable $(client) as an implicit
parameter. You must make sure that this variable is set correctly either globally or locally
or specified as an explicit parameter in the procedure call.
Chapter 12
Scripting
One of QF-Test’s benefits is that complex tests can be created without writing a single
line of code. However, there are limits to what can be achieved with a GUI alone. When
testing a program which writes to a database, for example, one might want to verify that
the actual values written to the database are correct; or one might want to read values
from a database or a file and use these to drive a test. All this and more is possible with
the help of powerful scripting languages like Jython, Groovy or JavaScript.
4.2+
While Jython is supported since the beginning of QF-Test, Groovy has found its way
into QF-Test a bit later (QF-Test version 3). This language might be more convenient
than Jython for those who are familiar with Java. Version 4.2 enabled JavaScript which
might be more suitable for web developers. It’s mainly a matter of individual preference
whether to utilize Jython, Groovy or JavaScript scripting inside QF-Test.
In this chapter the basics of the scripting features available in all supported languages
are explained. Most of the examples can be applied exactly or with few changes in
other script languages. Methods calls which vary in syntax are exemplified in the affected languages. Particularities of the script languages are described in the sections
(124)
(131)
Fundamentals of the Jython integration , Scripting with Groovy
and Scripting with
(134)
JavaScript .
3.0+
(502)
(504)
The scripting language to use for a given Server script
or SUT script
node is de(503)
termined by its Script language attribute, so you can mix all three languages within a
test-suite. The default language to use for newly created script nodes can be set via the
(319)
option Default script language for script nodes .
12.1
General
The approach to scripting in QF-Test is inverse from that of other GUI test tools. Instead
of driving the whole test from a script, QF-Test embeds scripts into the test-suite. This
12.1.
General
116
(502)
is achieved with the two nodes Server script
Both nodes have a Script
(502)
(504)
and SUT script
.
attribute for the actual code.
Figure 12.1: Detail view of a Server script with help window for rc methods
3.0+
The internal script editor has some useful features to ease the typing of code. Reserved
keywords, built-in functions, standard types, literals and comments are highlighted.
In
dentation is handled automatically inside of code blocks. With TAB and Shift-TAB respectively several selected lines can be indented manually.
However, the probably most useful feature - at least for the QF-Test newbie - might be
the input assistance for many built-in methods. Type,
rc. and maybe
for example,
some initial letters of a method name. Then press Ctrl-Space to open a pop-up window displaying the appropriate methods and descriptions of QF-Test’s run-context (cf.
(726)
chapter 41 ). Select one of the methods and confirm with Return toinsert it into the
script code. To get a list of all objects equipped with help, just press Ctrl-Space with
the mouse cursor positioned after white space.
Server scripts are useful for tasks like calculating the values of variables or reading and
parsing data from a file and using it to drive a test. SUT scripts on the other hand give
full access to the components of the SUT and to every Java API that the SUT exposes.
An SUT script might be used to retrieve or check values in the SUT to which QF-Test
(505)
doesn’t have access. The SUT script node has a Client
attribute which requires the
name of the SUT client to run in.
12.2.
The run-context rc
117
Server scripts are run in script interpreters for the different script languages embedded
in QF-Test itself, while SUT scripts are run in a script interpreter embedded in the SUT.
These interpreters are independent of each other and do not share any state. However,
QF-Test uses the RMI connection between itself and the SUT for seamless integration
of SUT scripts into the execution of a test.
Through the menu items Extras→Jython terminal... or Extras→Groovy terminal... etc.
you can open a window with an interactive command prompt for the language interpreter
embedded in QF-Test. You can use this terminal to experiment with Jython scripts, get
a feeling for the language, but also to try
sophisticated
stuff like setting up
out some
database connections. The keystrokes Ctrl-Up and Ctrl-Down let you cycle through
previous input and you can also edit any other line or mark a region in the terminal and
simply press Return to send it to the interpreter. In that case QF-Test will filter the ’»>’
and ’...’ prompts from previous interpreter output.
Similar terminals are available for each SUT client. The respective menu items are
located below the Clients menu.
Note
When working in a SUT script terminal, there’s one thing you need to be aware of: The
commands issued to the interpreter are not executed on the event dispatch thread, contrary to commands executed via SUT script nodes. This may not mean anything to you
and most of the time it doesn’t cause any problems, but it may deadlock your application
if you access any Swing or SWT components or invoke their methods. To avoid that,
QF-Test provides the global method runAWT (and runSWT respectively) that executes
arbitrary code on the event dispatch thread. For example, to get the number of visible
nodes in a JTree component named tree, use runAWT(”tree.getRowCount()”)
(or runAWT { tree.getRowCount() } in Groovy) to be on the safe side.
12.2
The run-context rc
When executing Server scripts and SUT scripts, QF-Test provides a special environment
in which a variable named rc is bound. This variable represents the run-context which
encapsulates the current state of the execution of the test. It provides an interface
(729)
(fully documented in section 41.6 ) for accessing QF-Test variables, for calling QFTest procedures and can be used to add messages to the run-log. To SUT scripts it also
provides access to the actual Java components of the SUT’s GUI.
For those cases where no run-context is available, i.e. Resolvers, TestRunListeners,
code executing in a background thread etc. QF-Test also provides a module called qf
(746)
with useful generic methods for logging and other things. Please see section 41.7 for
details.
Working
examples
are
also
doc/tutorial/demo-script.qft.
provided
in
the
test-suite
12.2.
The run-context rc
12.2.1
118
Logging messages
One thing the run-context can be used for is to add arbitrary messages to the run-log
that QF-Test generates for each test-run. These messages may also be flagged as
warnings or errors.
rc.logMessage("This is a plain message")
rc.logWarning("This is a warning")
rc.logError("This is an error")
Example 12.1: Logging messages from scripts
When working with compact run-logs (which is strongly encouraged, see the option
(391)
Create compact run-log ), nodes which most likely will not be needed for error analysis
may be deleted from the run-log to preserve memory. This does not apply to error
messages (rc.logError). They are kept, along with about 100 nodes preceding the
error. Warnings (rc.logWarning) are also kept, however, without preceding nodes.
Normal messages (rc.logMessage) may be subject to deletion. If you really need to
make sure that a message will definitely be kept in the run-log you can enforce this by
specifying the optional second parameter dontcompactify, e.g.
rc.logMessage("This message will not be removed", dontcompactify=true)
# or simply
rc.logMessage("This message will not be removed", 1)
Example 12.2: Logging messages that will not get removed in compact run-logs
12.2.2
Performing checks
Most of the time logging messages is tied to evaluating some condition. In that case, it
is often desirable to get a result in the HTML or XML report equivalent to that of a Check
node. The methods rc.check and rc.checkEqual will do just that:
var = 0
rc.check(var == 0, "Value of var is 0")
rc.checkEqual(’${system:user.language}’, ’en’, "English locale required",
rc.EXCEPTION)
Example 12.3: Performing checks
The optional last argument changes the error level in case of failure. Possible values
are rc.EXCEPTION, rc.ERROR, rc.OK or rc.WARNING.
12.2.
The run-context rc
12.2.3
119
Variables
QF-Test has different kinds of variables. On the one hand you find variables belonging
to the QF-Test environment and on the other variables of the script languages. Variables
of the script languages are separated in server and SUT side variables of the specific
script interpreter. The following graphic clarifies theses differences:
Figure 12.2: Overview of the types of variables in QF-Test
To share the different kinds of variables between QF-Test and the script interpreters
provides the rc object which has several methods for the purpose. The methods are
explained in the next section.
Accessing variables
Using QF-Test variables in scripts is not difficult. You can use the run-context’s lookup
(729)
method (see section 41.6 for API reference) whenever you want to access a QF-Test
value as a string.
12.2.
The run-context rc
120
# access a simple variable
text = rc.lookup("someText")
# access a property or resource
version = rc.lookup("qftest", "version")
Example 12.4: Using rc.lookup to access string variables
Setting variables
To make the results of a script available during further test execution, values can be
(601)
stored in global or local variables. The effect is identical to that of a Set variable node.
The corresponding methods in the run-context are rc.setGlobal and rc.setLocal.
# Test if the file /tmp/somefile exists
from java.io import File
rc.setGlobal("fileExists", File("/tmp/somefile").exists())
Example 12.5: Using rc.setGlobal
After executing the above example $(fileExists) will expand to true if the file
/tmp/somefile exists and to false if it doesn’t.
To clear a variable, set it to None, to clear all global variables use rc.clearGlobals()
from a Server script.
Global variables
Sometimes it is helpful to have a variable available in several scripting nodes of the same
language. If the value of the variable is not a simple string or integer, it is normally not
sufficient to use rc.setGlobal(...) to store it in a global QF-Test variable because
the value will be converted to a string in the process. Instead, such a variable should be
declared global as shown in the following example.
global globalVar
globalVar = 10000
Example 12.6: Global Jython variable
The globalVar is now accessible within all further scripting nodes of the same type
(Server scripts or SUT scripts of the same client). For changing the value of globalVar
within another script, the global declaration is necessary again. Otherwise, a new
12.2.
The run-context rc
121
local variable is created instead of accessing the existing global. Use the del statement
to remove a global Jython variable:
global globalVar
del globalVar
Example 12.7: Delete a global Jython variable
In Groovy and JavaScript the global variables declaration is even easier than in Jython.
All variables that are not declared locally are assumed to be global.
myGlobal = ’global’
Example 12.8: Defining a global variable in Groovy or JavaScript
assert myGlobal == ’global’
def globals = binding.variables
assert globals[’myGlobal’] == ’global’
globals.remove(’myGlobal’)
assert globals.find { it == ’myGlobal’ } == null
Example 12.9: Usage and deletion of a global Groovy variable
Exchanging variables between interpreters
Sometimes one would like to use variable values that have been defined in one interpreter in a different interpreter. For example, an SUT script might have been used to
create a list of items displayed in a table. Later we want to iterate over that list in a
Server script.
To simplify such tasks, the run-context provides a symmetrical set of methods to access or set global variables in a different interpreter. For SUT scripts these methods
are named toServer and fromServer. The corresponding Server script methods are
toSUT and fromSUT.
The following example illustrates how an SUT script can set a global variable in the QFTest Jython interpreter:
12.2.
The run-context rc
122
cellValues = []
table = rc.lookup("idOfTable")
for i in range(table.getRowCount()):
cellValues.append(table.getValueAt(i, 0))
rc.toServer(tableCells=cellValues)
Example 12.10: Setting a server variable from an SUT script
After the above script is run, the global variable named ”tableCells” in the QF-Test Jython
interpreter will hold the array of cell values.
Note
The cell values in the above example are not necessarily strings. They could be
numbers, date values, anything. Unfortunately Jython’s pickle mechanism isn’t smart
enough to transport instances of Java classes (not even realizable ones), so the whole
exchange mechanism is limited to primitive types like strings and numbers, along with
Jython objects and structures like arrays and dictionaries.
12.2.4
Accessing the SUT’s GUI components
For SUT scripts the run-context provides an additional method that is extremely useful. Calling rc.getComponent(”componentId”) will retrieve the information of the
(644)
(645)
Component node in the test-suite with the QF-Test ID ”componentId” and pass that
to QF-Test’s component recognition mechanism. The whole process is basically the
same as when simulating an event, including the possible exceptions if the component
cannot be found.
If the component is located, it will be passed to Jython, not as some abstract data but as
the actual Java object. All methods exposed by the Java API for the component’s class
can now be invoked to retrieve information or achieve effects which are not possible
(47)
through the GUI alone. To get a list of a component’s method see section 5.5 .
# get the custom password field
field = rc.getComponent("tfPassword")
# read its encrypted value
passwd = field.getCryptedText()
rc.setGlobal("passwd", passwd)
# get the table component
table = rc.getComponent("tabAddresses")
# get the number of rows
rows = table.getRowCount()
rc.setGlobal("tableRows", rows)
Example 12.11: Accessing components with rc.getComponent
You can also access sub-items this way. If the componentId parameter references
12.2.
The run-context rc
123
an item, the result of the getComponent call is a pair, the component and the item’s
index. The index can be used to retrieve the actual value. The following example shows
how to get the value of a table cell. Note the convenient way Jython supports sequence
unpacking during assignment.
# first get the table and index
table, (row,column) = rc.getComponent("tableAddresses@Name@Greg")
# then get the value of the table cell
cell = table.getValueAt(row, column)
Example 12.12: Accessing sub-items with rc.getComponent
12.2.5
Calling Procedures
(463)
The run-context can also be used to call back into QF-Test and execute a Procedure
node.
rc.callProcedure("text.clearField",
{"component": "nameField", "message" : "nameField cleared"})
Example 12.13: Simple procedure call in Jython
(470)
In the example above the Procedure named ”clearField” in the Package
named ”text”
will be called. The parameter named ”component” is set to the value ”nameField” and
the parameter named ”message” is set to the value ”nameField cleared”.
The same example with Groovy syntax:
rc.callProcedure("text.clearField",
["component" : "nameField", "message" : "nameField cleared"])
Example 12.14: Simple procedure call in Groovy
And in JavaScript:
rc.callProcedure("text.clearField",
{"component" : "nameField", "message" : "nameField cleared"})
Example 12.15: Simple procedure call in JavaScript
(469)
The value returned by the Procedure through a Return
of the rc.callProcedure call.
node is returned as the result
12.2.
Note
The run-context rc
124
Great care must be taken when using rc.callProcedure(...) in SUT script nodes.
Only short-running Procedures should be called that won’t trigger overly complex ac(664)
tions in the SUT. Otherwise, a DeadlockTimeoutException might be caused. For
data-driven tests where for some reason the data must be determined in the SUT, use
rc.toServer(...) to transfer the values to QF-Test interpreter, then drive the test
from a Server script node where these restrictions do not apply.
12.2.6
Setting options
3.1+
(316)
can also be set at runtime via
Many of the options described in chapter 33
rc.setOption. Constants for option names are predefined in the class Options. It
is automatically available for all script languages.
A real-life example where this might be useful is if you want to replay an event on a
disabled component, so you need to temporarily disable QF-Test’s check for the enabled/disabled state:
rc.setOption(Options.OPT_PLAY_THROW_DISABLED_EXCEPTION, false)
Example 12.16: Example for setOption
After replaying this special event, the original value read from the configuration file or
set in the option dialog can be restored by unsetting the option as the following example
shows:
rc.unsetOption(Options.OPT_PLAY_THROW_DISABLED_EXCEPTION)
Example 12.17: Example for unsetOption
Note
Be sure to set QF-Test options in a Server script node and SUT options in an SUT script
node, otherwise the setting will have no effect. The option documentation in chapter
(316)
33 shows which one to use.
12.2.7
Override components
You might face a situation where you want to work with a component which you have
to search before working with it. Sometimes recording all required components can
be exhaustive or might be too complicated. For such cases you can use the method
rc.overrideElement to set the found component (either by generic components or
via scripting) to a QF-Test component. Now you can work with the assigned component
and use all available QF-Test nodes.
12.3.
Fundamentals of the Jython integration
125
Let’s imagine that we have a panel and we want to work with the first textfield, but
because of changing textfields we cannot rely on the standard way of the recognition.
Now we can implement a script, which looks for the first textfield and assigns that
textfield to the PriorityAwtSwingComponent from the standard library qfs.qft.
Once we have executed that script we can work with any QF-Test nodes using the
PriorityAwtSwingComponent, which actually performs all actions on the found
textfield.
panel = rc.getComponent("myPanel")
for component in panel.getComponents():
if qf.isInstance(component, "javax.swing.JTextField"):
rc.overrideElement("PriorityAwtSwingComponent", component)
break
Example 12.18: Using rc.overrideElement
This concept is very useful if you know an algorithm to determine the target component
of your test-steps.
You can find such priority-components for all engines in the standard library qfs.qft.
You can also find an illustrative example in the provided demo test-suite
carconfig_en.qft, located in the directory demo/carconfig in your QF-Test
installation.
12.3
Fundamentals of the Jython integration
Python is an excellent, object oriented scripting language written in C by Guido van
Rossum. A wealth of information including an excellent Python tutorial is available at
http://www.python.org. Python is a standard language that has been around for years
with extensive freely accessible documentation. Therefore, this manual only explains
how Jython is integrated into QF-Test, not the language itself. Python is a very natural
language. Its greatest strength is the readability of Python scripts, so you should have
no problems following the examples.
Jython (formerly called JPython) is a Java implementation of the language Python. It
has the same syntax as Python and almost the same set of features. The object systems of Java and Jython are very similar and Jython can be integrated seamlessly into
applications like QF-Test. This makes it an invaluable tool for Java scripting. Jython has
its own web page at http://www.jython.org. There is also an extensive tutorial available
which may help you get started with this scripting language.
QF-Test uses Jython version 2.7 which supports a large majority of the standard Python
library.
12.3.
Note
Fundamentals of the Jython integration
126
In Jython QF-Test variables with the syntax $(var) or ${group:name} are expanded
before the execution of the script.
This can lead to unexpected behavior.
rc.lookup(...), which will be evaluated during execution of the script, is the
(118)
preferred method in this case (see section 12.2.3 for details).
12.3.1
Modules
Modules for Jython in QF-Test are just like standard Python modules. You can import the
modules into QF-Test scripts and call their methods, which simplifies the development
of complex scripts and increases maintainability since modules are available across
test-suites.
Modules intended to be shared between test-suites should be placed in the directory jython under QF-Test’s root directory. Modules written specifically for one testsuite can also be placed in the test-suite’s directory. The version-specific directory
qftest-4.2.2/jython/Lib is reserved for modules provided by Quality First Software GmbH. Jython modules must have the file extension .py.
The following Jython module defines a procedure sorting an array of numbers.
def insertionSort(alist):
for index in range(1,len(alist)):
currentvalue = alist[index]
position = index
while position>0 and alist[position-1]>currentvalue:
alist[position]=alist[position-1]
position = position-1
alist[position]=currentvalue
Example 12.19: The Jython module pysort.py
The procedure defined in above module is beeing called in the following Jython script:
import pysort
alist = [54,26,93,17,77,31,44,55,20]
pysort.insertionSort(alist)
print(alist)
Example 12.20: Jython script using a module
12.3.2
Post-mortem debugging of Jython scripts
Python comes with a simple line-oriented debugger called pdb. Among its useful features is the ability for post-mortem debugging, i.e. analyzing why a script failed with an
12.3.
Fundamentals of the Jython integration
127
exception. In Python you can simply import the pdb package and run pdb.pm() after
an exception. This will put you in a debugger environment where you can examine the
variable bindings in effect at the time of failure and also navigate up to the call stack
to examine the variables there. It is somewhat similar to analyzing a core dump of a C
application.
Though Jython comes with pdb, the debugger doesn’t work very well inside QF-Test for
various reasons. But at least post-mortem debugging of Jython scripts is supported from
(124)
(502)
the Jython terminals (see section 12.3 ). After a Server script node fails, open QF(504)
node open the respective SUT Jython
Test’s Jython terminal, for a failed SUT script
terminal, then just execute debug(). This should have a similar effect as pdb.pm()
described above. For further information about the Python debugger please see the
documentation for pdb at https://docs.python.org/2/library/pdb.html.
12.3.3
Boolean type
Jython now has a real boolean type with values True and False whereas in older
versions integer values 0 and 1 served as boolean values. This can cause problems if
boolean results from calls like file.exists() are assigned to a QF-Test variable, e.g.
(482)
”fileExists” and later checked in a Condition attribute in the form $(fileExists) ==
1. Such conditions generally be written as just $(fileExists) which works well with
all Jython versions.
12.3.4
Character encodings
All Java strings are sequences of 16-bit characters. Python’s original strings are made
of 8-bit characters. Later, unicode strings with 16-bit characters were added. Jython
literal strings like ”abc” are 8-bit, prepending ’u’ for u”abc” turns them into unicode
strings.
In Jython 2.2, Java strings were converted to 8-bit Python strings based on the default
encoding of the Java VM, typically ISO-8859-1 (also known as latin-1) in western countries. In Jython 2.5, every Java string is now interpreted as a unicode Jython string.
This results in a lot more implicit conversion between 8-bit and unicode strings, for example when concatenating a Java string - now converted to unicode - and a literal string
like rc.lookup(”path”) + ”/file”. Most of the time this works well, but if the
literal string contains characters outside the 7-bit ASCII character-set, things start to get
messy. The default encoding for 8-bit Jython characters can be specified in the option
(319)
Default character encoding for Jython
with a default of latin-1 for maximum backwards compatibility. On the upside it is now possible to have default encodings other
than latin-1 and to specify literal strings of characters in international character sets.
One thing to watch out for is existing code of the form
12.3.
Fundamentals of the Jython integration
128
import types
if type(somevar) == types.StringType:
...
The type types.StringType is the 8-bit string. It does not match unicode strings. To
test whether some variable is a Jython string, regardless of whether it’s 8-bit or
unicode, change that to
import types
if type(somevar) in types.StringTypes:
...
One new requirement - coming from newer Python versions - is that Python module
files containing characters outside the 7-bit ASCII character must specify the character
encoding to be used in a comment line close to the top of the file, e.g.
# coding:
latin-1
Please see http://www.python.org/peps/pep-0263.html for details.
12.3.5
Getting the name of a Java class
This simple operation is surprisingly difficult in Jython. Given a Java object you would
expect to simply write obj.getClass().getName(). For some objects this works
fine, for others it fails with a cryptic message. This can be very frustrating. Things go
wrong whenever there is another getName method defined by the class, which is the
case for AWT Component, so getting the class name this way fails for all AWT/Swing
component classes.
In Jython 2.2.1 the accepted workaround was to use the Python idiom
obj.__class__.__name__. This no longer works in Jython 2.5 because it no
longer returns the fully qualified class name, only the last part.
Instead of
java.lang.String you now get just String. The only solution that reliably works
for version 2.5 is:
from java.lang import Class
Class.getName(obj.getClass())
This also works for 2.2, but it is not nice, so we initiated a new convenience module
with utility methods called qf that gets imported automatically. As a result you can now
simply write
12.3.
Fundamentals of the Jython integration
129
qf.getClassName(obj).
12.3.6
A complex example
We are going to close this section with a complex example, combining features from
Jython and QF-Test to execute a data-driven test. For the example we assume that a
simple table with the three columns ”Name”, ”Age” and ”Address” should be filled with
values read from a file. The file is assumed to be in ”comma-separated-values” format
with ”|” as the separator character, one line per table-row, e.g.:
John Smith|45|Some street, some town
Julia Black|35|Another street, same town
The example verifies the SUT’s functionality in creating new table rows. It calls a QFTest procedure that takes the three parameters, ”name”, ”age”, and ”address”, creates
a new table-row and fills it with these values. Then the Jython script is used to read and
parse the data from the file, iterate over the data-sets and call back to QF-Test for each
table-row to be created. The name of the file to read is passed in a QF-Test variable
named ”filename”. After filling the table, the script compares the state of the actual table
component with the data read from the file to make sure everything is OK.
12.3.
Fundamentals of the Jython integration
130
import string
data = []
# read the data from the file
fd = open(rc.lookup("filename"), "r")
line = fd.readline()
while line:
# remove whitespace
line = string.strip(line)
# split the line into separate fields
# and add them to the data array
if len(line) > 0:
data.append(string.split(line, "|"))
line = fd.readline()
# now iterate over the rows
for row in data:
# call a qftest procedure to create
# one new table row
rc.callProcedure("table.createRow",
{"name": row[0], "age": row[1],
"address": row[2]})
# verify that the table-rows have been filled correctly
table = rc.getComponent("tabAddresses")
# check the number of rows
rc.check(table.getRowCount() == len(data), "Row count")
if table.getRowCount() == len(data):
# check each row
for i in range(len(data)):
rc.check(table.getValueAt(i, 0)) == data[i][0],
"Name in row " + str(i))
rc.check(table.getValueAt(i, 1)) == data[i][1],
"Age in row " + str(i))
rc.check(table.getValueAt(i, 2)) == data[i][2],
"Address in row " + str(i))
Example 12.21: Executing a data-driven test
Of course, the example above serves only as an illustration. It is too complex to be
edited comfortably in QF-Test and too much is hard-coded, so it is not easily reusable.
For real use, the code to read and parse the file should be parameterized and moved to
a module, as should the code that verifies the table.
This is done in the following Jython script with the methods loadTable to read
the data from the file and verifyTable to verify the results.
It is saved
in a module named csvtable.py.
An example module is provided in
qftest-4.2.2/doc/tutorial/csvtable.py. Following is a simplified version:
12.3.
Fundamentals of the Jython integration
131
import string
def loadTable(file, separator="|"):
data = []
fd = open(file, "r")
line = fd.readline()
while line:
line = string.strip(line)
if len(line) > 0:
data.append(string.split(line,separator))
line = fd.readline()
return data
def verifyTable(rc, table, data):
ret = 1
# check the number of rows
if table.getRowCount() != len(data):
if rc:
rc.logError("Row count mismatch")
return 0
# check each row
for i in range(len(data)):
row = data[i]
# check the number of columns
if table.getModel().getColumnCount() != len(row):
if rc:
rc.logError("Column count mismatch " +
"in row " + str(i))
ret = 0
else:
# check each cell
for j in range(len(row)):
val = table.getModel().getValueAt(i, j)
if str(val) != row[j]:
if rc:
rc.logError("Mismatch in row " +
str(i) + " column " +
str(j))
ret = 0
return ret
Example 12.22: Writing a module
The code above should look familiar. It is an improved version of parts of example
(129)
12.21 . With that module in place, the code that has to be written in QF-Test is reduced
to:
12.4.
Scripting with Groovy
132
import csvtable
# load the data
data = csvtable.loadTable(rc.lookup("filename"))
# now iterate over the rows
for row in data:
# call a qftest procedure to create
# one new table row
rc.callProcedure("table.createRow",
{"name": row[0], "age": row[1],
"address": row[2]})
# verify that the table-rows have been filled correctly
table = rc.getComponent("tabAddresses")
csvtable.verifyTable(rc, table, data)
Example 12.23: Calling methods in a module
For more complex import of data QF-Test can be extended with existing Python modules. For example, at http://python-dsv.sourceforge.net/ an excellent module for very
flexible CSV import is freely available.
12.4
Scripting with Groovy
Groovy is another established scripting language for the Java Platform. It was invented
by James Strachan and Bob McWhirter in 2003. All you need for doing Groovy is a
Java Runtime Environment (JRE) and the groovy-all.jar file. This library contains
a compiler to create Java class files and provides the runtime when using that classes
in the Java Virtual Machine (JVM). You may think of Groovy as being Java with an
additional .jar file. In contrast to Java, Groovy is a dynamic language, meaning that
the behavior of an object is determined at runtime. Groovy also allows to load classes
from sources without creating class files. Finally, it is easy to embed Groovy scripts into
Java applications like QF-Test.
The Groovy syntax is similar to Java, maybe more expressive and easier to read. When
coming from Java you can embrace the Groovy style step by step. Of course we cannot
explain all aspects of the Groovy language here. For in-depth information, please take
a look at the Groovy home page at http://groovy-lang.org/ or read the excellent book
”Groovy in Action” by Dierk Koenig and others. Perhaps the following tips may help a
Java programmer getting started with Groovy.
• The semicolon is optional as long as a line contains only one statement.
• Parentheses are sometimes optional, e. g. println ’hello qfs’ means the
same as println(’hello qfs’).
12.4.
Scripting with Groovy
• Use for (i in 0..<len) { ...
len; i++) { ... }.
133
} instead of for (int i = 0; i <
• The following imports are made by default: java.lang.*, java.util.*,
java.io.*, java.net.*, groovy.lang.*, groovy.util.*,
java.math.BigInteger, java.math.BigDecimal.
• Everything is an object, even integers like ’1’ or booleans like ’true’.
• Instead of using getter and setter methods like obj.getXxx(), you can simply
write obj.xxx to access a property.
• The operator == checks for equality, not identity, so you can write if (somevar
== ”somestring”) instead of if (somevar.equals(”somestring”)).
The method is() checks for identity.
• Variables have a dynamic type when being defined with the def keyword. Using
def x = 1 allows for example to assign a String value to the variable x later in
the script.
• Arrays are defined differently from Java, e. g. int[] a = [1, 2, 3] or def
a = [1, 2, 3] as int[]. With def a = [1, 2, 3] you define a List in
Groovy.
• Groovy extends the Java library by defining a set of extra methods for many
classes. Thus, you can for example apply an isInteger() method to any
String object in a Groovy script. That’s what is called GDK (according to the
JDK in Java). To get a list of those methods for an arbitrary object obj, you can
simply invoke obj.class.metaClass.metaMethods.name or use the
following example:
import groovy.inspect.Inspector
def s = ’abc’
def inspector = new Inspector(s)
def mm = inspector.getMetaMethods().toList().sort() {
it[Inspector.MEMBER_NAME_IDX] }
for (m in mm) {
println(m[Inspector.MEMBER_TYPE_IDX] + ’ ’ +
m[Inspector.MEMBER_NAME_IDX] +
’(’ + m[Inspector.MEMBER_PARAMS_IDX] + ’)’)
}
Example 12.24: GDK methods for a String object
• Inner classes are not supported, in most cases you can use Closures instead. A
Closure is an object which represents a piece of code. It can take parameters
12.4.
Scripting with Groovy
134
and return a value. Like a block, a Closure is defined with curly braces { ...
}. Blocks only exist in context with a class, an interface, static or object initializers, method bodies, if, else, synchronized, for, while, switch, try,
catch, and finally. Every other occurrence of {...} is a Closure. As an
example let’s take a look at the eachFileMatch GDK method of the File class.
It takes two parameters, a filter (e. g. a Pattern) and a Closure. That Closure
takes itself a parameter, a File object for the current file.
def
def
def
new
dir = rc.lookup(’qftest’, ’suite.dir’)
pattern = ~/.*\.qft/
files = []
File(dir).eachFileMatch(pattern) { file ->
files.add(file.name)
}
files.each {
// A single Closure argument can also be referred with "it"
rc.logMessage(it)
}
Example 12.25: Closures
• Working with Lists and Maps is simpler than in Java.
def myList = [1, 2, 3]
assert myList.size() == 3
assert myList[0] == 1
myList.add(4)
def myMap = [a:1, b:2, c:3]
assert myMap[’a’] == 1
myMap.each {
this.println it.value
}
Example 12.26: Working with lists and maps
12.4.1
Groovy packages
Just like Java classes, Groovy source files (.groovy) can be organized in packages.
Those intended to be shared between test-suites should be placed in the directory
groovy under QF-Test’s root directory. Others that are written specifically for one testsuite can also be placed in the directory of the test-suite. The version-specific directory
qftest-4.2.2/groovy is reserved for Groovy files provided by Quality First Software
GmbH.
12.5.
Scripting with JavaScript
135
package my
class MyModule
{
public static int add(int a, int b)
{
return a + b
}
}
Example 12.27: MyModule.groovy
The file MyModule.groovy could be saved in a sub directory my below the suite directory. Then you can use the add method from MyModule as follows:
import my.MyModule as MyLib
assert MyLib.add(2, 3) == 5
Example 12.28: Using MyModule
This code also shows another groovy feature: Type aliasing. By using import and as
in combination you can reference a class by a name of your choice.
12.5
Scripting with JavaScript
JavaScript has become the most widely used programming language in the web area
and is one of the most popular script languages. QF-Test supports scripting with ECMAScript, which provides a common standard for the variety of different implementations of JavaScript.
QF-Test must run with at least Java 8 to use JavaScript.
It is possible to write code for the ECMAScript 6 standard. QF-Test automatically transpiles the code to the EcmaScript 5 standard before the execution.
Special features of JavaScript as compared to other programming languages:
• There are two different null values: undefined and null. A variable is
undefined when it has no value. null is an intended null value that has to be
assigned.
• The == operator checks for equality instead of identity. So you can use if
(somevar == ”somestring”) to check for equality. To check for identity use
the === operator.
12.5.
Scripting with JavaScript
136
• Variables declared with the let keyword are dynamically typed. E.g. let x = 1
makes it possible to assign String to x. Constants can be declared with const.
12.5.1
JavaScript imports
The following example shows how functionality can be transfered in a module. The
module must be placed in the javascript directory inside the QF-Test root directory.
The module can look like this:
var fibonacci = function(n) {
return n < 1 ? 0
: n <= 2 ? 1
: fibonacci(n - 1) + fibonacci(n - 2);
}
function sumDigits(number) {
var str = number.toString();
var sum = 0;
for (var i = 0; i < str.length; i++) {
sum += parseInt(str.charAt(i), 10);
}
return sum;
}
// Module exports (Node.js style)
exports.fibonacci = fibonacci;
exports.sumDigits = sumDigits;
Example 12.29: The moremath.js module
The moremath.js module defines the two function: fibonacci and sumDigits.
Each function has to be exported to . This can be achieved via Node.js like function
exports.
The following code can now be used inside the script node to take advantage of the
moremath.js modules functions:
moremath = require(’moremath’);
console.log(moremath.fibonacci(13));
console.log(moremath.sumDigits(123));
Example 12.30: Usage of the moremath.js module
There are multiple ways to import modules. Modules provided by QF-Test can be imported using the import function.
12.5.
Scripting with JavaScript
137
import {Autowin} from ’autowin’;
Autowin.doClickHard(0, 0, true);
Example 12.31: Using the autowin module
Java classes can also be imported using the import function.
import {File} from ’java.io’;
Example 12.32: Importing Java classes
It is also possible to use the ”require” function for importing npm modules, which are
explained in the following section.
12.5.2
NPM modules
npm is a package manager for JavaScript with over 350.000 packages. The available
packages are listed here https://www.npmjs.com/. The packages can be used in QFTest scripts. They need to be installed in the javascript folder of the QF-Test root directory.
npm install underscore
This line installs the npm underscore package from the os command line.
There are a few npm modules that are incompatible with the ECMAScript standard as
they were written for Node.js.
_ = require(’underscore’);
func = function(num){ return num % 2 == 0; }
let evens = _.filter([1, 2, 3, 4, 5, 6], func);
console.log(evens);
Example 12.33: Usage of the ’underscore’ package
12.5.3
Print statements
Besides console.log() there is another method implemented in QF-Test to show
output on the terminal. Note that this print is not defined in ECMAScript and was
added for convenience in QF-Test.
12.5.
Scripting with JavaScript
138
print([1,2,3,4]);
Example 12.34: Printing an array
12.5.4
Execution
JavaScript scripts are not executed inside the browser but in a specific engine on the
server or SUT side. This engine is called Oracle Nashorn Engine and comes with JDK
8. It allows the execution of EcmaScript directly in the JVM.
Chapter 13
Unit Tests
With Unit Tests, i.e. component tests, you can check the functional units. They explicitly
test the functionality of single components. For this reason they are much less complex
compared to integration and system tests.
(621)
The Unit test node executes Unit Tests via the JUnit framework. The results are available in the run-log as well as in the report.
The tests can be started from two possible sources: Java classes containing the JUnit
test-cases or Unit Tests scripted directly in QF-Test. The parameters of the node vary
with the use case.
13.1
Java Classes as the Source for the Unit Test
It is possible to execute Unit Tests from Jar or Class files. It is also possible to execute
Unit Tests that are available from the SUT’s runtime. QF-Test executes the tests of the
(621)
test classes specified in the respective attribute of the Unit test node. In the report they
will be displayed as test-steps within a test-case. The following example demonstrates
(621)
the usage of a Unit test node with Java test classes.
13.1.
Java Classes as the Source for the Unit Test
Figure 13.1: Unit Test node with Java classes
140
13.2.
Basics of the Test Scripts
141
package de.qfs.test;
import org.junit.Assert;
import org.junit.Test;
public class StringTest {
@Test
public void testubstring() {
String s = new String("Long text");
s = s.substring(5, 9;
assert("text".equals(s));
}
@Test
public void testReplace() {
String s = new String("example");
s = s.replace(’e’, ’i’);
Assert.assertEquals("ixampli", s);
}
}
Example 13.1: Java Unit test
(624)
The class de.qfs.test.StringTest must exist in the unittests.jar specified in the Classpath
attribute. The path is determined relative to the path of the directory of the current suite.
In this example the jar file is in the same directory as the suite.
JUnit test classes are Java classes where the methods have the @Test annotation. The
(624)
Unit test node executes all classes specified in the Test classes table. Thus a Unit test
node can execute several test classes.
13.2
Basics of the Test Scripts
The second option to execute Unit Tests is to script the Unit Test directly in the Unit test
node. You can use any of the Script languages QF-Test offers. The most appropriate
one is Groovy because it supports the Java annotations. The JUnit framework is used
to execute the scripts.
13.2.
Basics of the Test Scripts
13.2.1
142
Groovy Unit Tests
@BeforeClass
static void onbefore(){
println("Set Up")
}
@Test(expected=IndexOutOfBoundsException.class)
void indexOutOfBoundsAccess() {
def numbers = [1,2,3,4]
numbers.get(4)
}
@Test
void noFailure() {
assert true
}
Example 13.2: Unit Test Script with Groovy
In Groovy the required JUnit 4 classes are automatically imported at run-time. Just like
in Java all tests with the @Test annotation will be executed. You can ignore expected
exceptions using the expected parameter of the @Test annotation. The methods with
the @BeforeClass annotation will be executed before the test methods will be run.
13.2.2
Jython Unit Tests
def setUp(self):
print "Set Up"
def testMathCeil(self):
import math
self.assertEqual(2, math.ceil(1.01))
self.assertEqual(1, math.ceil(0.5))
self.assertEqual(0, math.ceil(-0.5))
self.assertEqual(-1, math.ceil(-1.1))
def testMultiplication(self):
self.assertAlmostEqual(0.3, 0.1 * 3)
Example 13.3: Unit Test script with Jython
Because Jython does not support Java annotations, the tests run as JUnit 3 tests. All
methods beginning with the keyword test are considered to be a test and executed
as QF-Test checks. The methods must have the self parameter because they are
automatically enclosed in a class. The setUp method is executed at the beginning of
each test.
13.3.
Injections
13.2.3
143
JavaScript Unit-Test
setUp(){
print("Set up");
}
tearDown() {
print("Tear Down");
}
testUpperCase(){
let s = "text";
assertEquals("TEXT", s.toUpperCase());
}
testOk() {
assertTrue(true);
}
Example 13.4: Unit Test Script with JavaScript
Also JavaScript does not support Java annotations, the tests are executed as JUnit-3
(141)
Tests (cf. section 13.2.2 ). Just like in Jython all functions beginning with the keyword
tests are executed as QF-Test checks.
13.3
Injections
It is possible to use the Unit Test node for the so called Live Tests. In this case the Unit
Tests are executed in the running SUT. Using ’Injections’ the Unit Tests inject QF-Test
objects like WebDriver, components and variables into the Unit Tests or scripts directly.
13.3.
Injections
13.3.1
144
Component-Injections
import static org.junit.Assert.*;
import javax.swing.JComponent;
import org.junit.Test;
public class ComponentTest
{
/** The component to test in this unit test */
static JComponent component;
/** Expected value */
static String accessibleName;
@Test
public void accessibleNameIsCorrect()
{
/** component and accessible name are injected at run-time */
final String currentName =
component.getAccessibleContext().getAccessibleName();
assertEquals(accessibleName,currentName);
}
}
Example 13.5: Java Unit-Test
13.3.
Injections
145
Figure 13.2: Example Unit Test node with Injections
The example shows the injection of two QF-Test objects: component and variable.
The parameter ”Field” corresponds to the name of the field static JComponent
component; in the Java class. The java field must be static.
13.3.
Injections
13.3.2
146
WebDriver-Injections
import static org.junit.Assert.*;
import org.junit.Test;
import org.openqa.selenium.WebDriver;
public class WebdriverTest
{
/** The driver of the window currently opened by QF-Test. */
static WebDriver driver;
@Test
public void urlIsCorrectedLoaded()
{
// driver is injected at run-time
final String currentUrl = driver.getCurrentUrl();
assertEquals("http://www.example.com", currentUrl);
}
}
Example 13.6: Java Unit Test with WebDriver Injections
13.3.
Injections
147
Figure 13.3: Example Unit Test node with Injections
This example shows how to inject a WebDriver object into a Java class. When no value
for the WebDriver driver is specified QF-Test determines the value via the given
client.
13.4.
13.4
Unit Tests in Report
148
Unit Tests in Report
The greatest benefit from using the Unit test node is that the results are displayed nicely
formatted in the HTML report. All Unit Test classes executed via this node are considered QF-Test test-cases. Unit test nodes should not be run separately. In order to see
them correctly displayed in the HTML report, run them as part of a Test-case. Each test
method is handled like a QF-Test check, e.g. a failed check does not abort the tests
execution.
Figure 13.4: Unit Test Report
Chapter 14
Web testing
QF-Test supports several browsers and is able to control them so that users can record
manual interactions with the browser and replay those actions again. To gain full control
over the browser QF-Test uses two different approaches which will be explained in the
following sections.
Note
Both approaches are necessary and they will continue to coexist and be fully maintained. Most browsers can be accessed by only one of the two, the only exceptions
are Chrome on Windows and Firefox up to version 43 on Windows and Linux. QFTest attempts to do the right thing by default but you can take control via the attribute
(519)
(517)
Browser connection mode of the Start web engine node.
14.1
Web testing using QF-Driver connection mode
This approach integrates the locally installed browser into a wrapper-window. This approach is also called embedding. QF-Test natively embeds the browser into its own
window, thus gaining access to its automation-interfaces. By using these interfaces QFTest can listen for events from the browser and is also able to inject events the other
way into the browser.
Native embedding of browsers into a separate window is not working satisfactorily for
newer browsers and in case of Microsoft Edge this approach cannot work at all. For
this reason an alternative mechanism was implemented in QF-Test 4.1. Explanations
on this mechanism can be found in the following section.
14.2
4.1+
Website Testing using WebDriver
Given that the existing approach using embedding is not maintained anymore by some
14.2.
Website Testing using WebDriver
150
browser vendors or is not supported at all, a new mechanism was implemented for
QF-Test 4.1 to support future browsers and browser versions. This mechanism uses
Selenium WebDriver as a bridge between the browser and QF-Test.
Note
The WebDriver connection mode is not yet on par with QF-Driver mode in terms of
(151)
performance and feature completeness (see section 14.2.4 ). When recording, events
may get lost. For the time being we recommend to primarily use QF-Driver mode for
(3)
recording. Please see the system requirements in section 1.1 for detailed information
about which browsers are supported for each connection mode and operating system.
14.2.1
WebDriver in general
WebDriver is slowly evolving into a W3C-standard for interacting with web browsers.
(http://www.w3.org/TR/webdriver). WebDriver is a remote control interface that enables
introspection and control of browsers, based on a platform and language-neutral wire
protocol.
The various browser vendors have agreed on this quasi-standard, so that the WebDriver
integration is partly implemented directly by the vendors themselves. Partially the integration is based on plugins, some vendors already include the integration in the default
setup of their browsers.
QF-Test uses the WebDriver interfaces to interact with the browser. Since the WebDriver
approach is only partially fitting the concepts of QF-Test, its web engine was extended
so that most of QF-Test’s functionality is also available via WebDriver, including the
added benefits like synchronization, abstraction of components etc.
Moreover the QF-Test approach can be combined with already existing Selenium
scripts. When using WebDriver mode QF-Test is able to directly execute those in an
(504)
(766)
SUT script node (see section 42.2 ).
Note
If you use the WebDriver mode, you are no longer limited to Firefox if you want to
(519)
specify a browser installation folder using the attribute Directory of browser installation
(517)
of the Start web engine node. If no matching browser can be localized in the specified
directory, an exception will be thrown. If no directory is specified, QF-Test will try to start
a default browser of the given browser type.
Note
QF-Test needs to deep inspect the browser content in order to enable the familiar testing
features (e.g. event and check recording, feature based component recognition, web
and custom resolvers) in WebDriver mode. This might provoke in some browsers a
warning based on mixed content display or an untrusted certificate. If this warning or
error message only appears while running the web site in testing mode, you can safely
ignore it.
14.2.
Website Testing using WebDriver
14.2.2
151
WebDriver with Microsoft Edge
Because Microsoft Edge comes without pre-installed WebDriver support, the WebDriver
integration needs to be installed manually once.
Please navigate your web browser to the Microsoft WebDriver website1 and download
and install the WebDriver version matching your operating system.
In addition, a loopback exemption needs to be defined for the Edge browser. Windows
10 uses an isolation technology that may prevent requests being sent from Edge to
QF-Test. Although this does not always happen, we recommend enabling the loopback
before testing with Microsoft Edge.
To enable loopback for Edge, open a command shell with administration rights and
execute:
CheckNetIsolation LoopbackExempt -a
-n="Microsoft.MicrosoftEdge_8wekyb3d8bbwe"
To disable loopback for Edge, open a command shell with administration rights and
execute:
CheckNetIsolation LoopbackExempt -d
-n="Microsoft.MicrosoftEdge_8wekyb3d8bbwe"
Details about for enabling loopback for Windows apps can be found in the Windows Dev
Center2 .
After successful installation you can also use Microsoft Edge as browser in your tests.
14.2.3
WebDriver with Safari
There are two options to use Safari for web testing: By means of the SafariDriver bundled with Safari 10+ on macOS 10.11+, or using a Safari extension. Both methods
require a manual user setup.
To activate the Apple SafariDriver, open the Safari Preferences and select ”Show
Develop menu in menu bar”. In this menu, activate the ”Allow Remote Automation”
option.
After that, open a Terminal window and execute once the command
/usr/bin/safaridriver -p 0 to authorize the driver.
Note
Due to special security restrictions imposed by the Apple SafariDriver, the interaction
between QF-Test and Safari is limited in the following ways:
1
2
https://developer.microsoft.com/en-us/microsoft-edge/tools/webdriver
https://msdn.microsoft.com/en-us/library/windows/apps/Hh780593.aspx
14.3.
Web-Tests without visible Window
152
• Tests can only be replayed but not recorded
• No hard events are possible
• Only one browser instance is allowed
For a limited time, it will be possible to control Safari using the Selenium
SafariDriver extension. To activate this, select QF-Test in Finder and show the
package contents using the context menu.
Navigate to the folder
Contents/Resources/qftest/qftest-4.2.2/webdriver/mac64 and copy the
file SafariDriver.safariextz to a local directory. Then open this file to install the
extension. Please uninstall any older version of the extension before.
14.2.4
Known limitations of the WebDriver mode
The WebDriver connection mode is under active development. Due to this, some features known from QF-Driver connection mode are not yet available, mostly due to restrictions of the WebDriver specification.
• No support of file downloads and HTTP authentication.
• It is not possible to record or replay HTTP requests directly.
• wd.getComponent(WebElement) does not work currently on elements in inner
frames.
• Events triggering a page load are sometimes not recorded.
• Event-Synchronization is in some cases delayed.
14.3
4.2+
Web-Tests without visible Window
Using the WebDriver-connection-mode it is possible to drive Firefox and Chrome in the
so called ”headless”-mode. In this mode, the browser is started in the background,
but without any visible window on the screen. All interactions with the webpage are
executed inside this ”invisible” window.
To execute an existing web test using the ”headless”-mode, simply change the type
(517)
step from chrome to headless-chrome or from
of browser in the Start web engine
firefox to headless-firefox. Please note that the ”headless”-mode is only supported since Chrome 60 and Firefox 57.
Chapter 15
Testing PDF documents
4.2+
From version 4.2 onwards QF-Test offers the possibility to test PDF documents similarly
to GUIs, i.e. QF-Test analyzes the structure of a PDF document and recognizes single
components, which can be tested individually.
(33)
(154)
Using Capture and replay QF-Test can directly record and replay Events as well as
(154)
Checks .
15.1
PDF Client
QF-Test loads the PDF document to be tested into a viewer, which QF-Test starts as a
client process.
15.1.1
PDF Client start
The Quickstart Wizard allows to create a setup sequence to start the PDF client. Please
(27)
choose ”PDF document” as Type of Application. (c.f. chapter 3 ). This allows to start
(526)
the viewer. The node Start PDF client is used as start node.
15.1.2
PDF Client window
The left side of the window of the PDF Client displays a column with an overview of all
pages of the PDF document.
The right part of the window shows the currently selected page.
The following screenshot shows the PDF Client with a demo PDF document.
15.1.
PDF Client
Figure 15.1: PDF Client main window with PDF document
154
15.2.
PDF events
15.2
155
PDF events
To change the opened document during test execution or the shown page you may use
(553)
a Selection step. These actions can be recorded directly in the recording mode. The
following events can be replayed:
15.2.1
Open a PDF document
You can load another PDF document during the test execution. To do so you have to
(556)
(553)
set the Detail attribute of the Selection step to open: .
Now you can set the path to the PDF document. If a relative path is specified it is
resolved relatively to the directory containing the current test-suite.
open:C:\Users\qfs\meinPDFDokument.pdf
Example 15.1: Loading a PDF document
If the document cannot be found or opened a TestException
15.2.2
(662)
will be thrown. >
Switch page
(556)
To switch to a specific page the Detail
goto:.
(553)
attribute of the Selection
step can be set to
(527)
Just like the Page of PDF Document attribute, the page can either be set as an integer
to set the page number or a string in quotation marks for the page name.
goto:3 or goto:"Introduction"
Example 15.2: Open a specific page
If the desired page cannot be determined, a PageNotFoundException will be thrown.
15.3
Checks for PDF components
(157)
The following checks exist for PDF components (c.f. section 15.4
be directly recorded via the Check recording mode.
). These checks can
15.3.
Checks for PDF components
156
15.3.1 Check text
(561)
For a description of the Check text node please refer to Check text . There are two
check types available for PDF text components: ’default’ and ’Text positioned’.
PDF documents do not necessarily contain line breaks, and spaces. The spaces between words and rows result from the coordinates of the single letters. The check type
’default’ checks the text as it is represented in the PDF document - without line breaks
and spaces when the text object does not contain any. From the coordinates of the single letters QF-Test calculates where there should be line breaks and spaces. The check
type ’Text positioned’ checks this processed text.
Figure 15.2: Check text ’default’
Figure 15.3: Check text ’Text positioned’
15.3.2 Check image
(581)
For a description of the Check image node please refer to Check image
’default’ is provided for all PDF object types.
. The check type
The check type ’default’ checks the object as it is displayed on the PDF page, with
scaling and with overlapping objects or parts of objects.
Figure 15.4: Check Image ’default’ recording of a Text object
15.3.
Checks for PDF components
157
Figure 15.5: Check Image ’default’ recording of an Image object
In a PDF document there may also be real embedded images. The images can be
scaled for display on the PDF page. Moreover, other objects may overlap the image on
the displayed page. For these Images the following check types are also available:
The check type ’unscaled’ checks the original unscaled image embedded into the file.
Figure 15.6: Check Image ’unscaled’ recording of an Image object
The check type ’scaled’ checks the image displayed on the PDF page without overlapping objects, however, taking into account scaling. This allows to check partly invisible
images.
Figure 15.7: Check Image ’scaled’ recording of an Image object
15.4.
PDF component types
15.3.3
158
’Check Font’
The Check text node with the check type ’text_font’ allows to check the font of a text
object.
The letters within one PDF text object may have various fonts. ’Check Font’ checks the
font which is used predominantly.
15.3.4
’Check Font size’
The Check text node with the check type ’text_fontsize’ allows to check the fontsize of a
text object.
The letters within one PDF text object may have various font sizes. ’Check Font size’
checks the font size which is used predominantly.
15.4
PDF component types
QF-Test recognizes the following object types:
Text
PDF object type
QF-Test component type
Text or Label
Image
Graphics
Shader and Vectors
Graphics
Main stage
MainPanel
Comment
Collection of letters, which have
a font and a font size.
Collection of pixels. May also
have the form of a letter.
Collection of vectors, which either represent geometrical figures or maybe also letters.
The basic page for all objects.
Table 15.1: Supported PDF objects
QF-Test highlights all recognized PDF objects with a colored border line if this feature
is enabled via the menu View -> Show components or the keyboard shortcut CTRL-T.
This feature must be disabled during capture and replay, otherwise image checks will
show the colored border lines.
The following color code applies to the object types:
15.5.
PDF component recognition
159
Color
Red
Blue
Green
Cyan
PDF object type
Text
Image
Shader and Vectors
Main stage
Table 15.2: Color code for PDF objects
15.5
PDF component recognition
QF-Test represents PDF objects as Swing components, which can be accessed via the
(114)
Swing API by SUI scripts, for example (c.f. chapter 12 ).
The basic data QF-Test needs to identify the PDF object on the page are the same as
with all QF-Test components: class, geometry and structure information (index). For text
components QF-Test also saves the predominantly used font and predominantly used
font size in the Extra features table. For image objects QF-Test also records the image
hash and saves it in the Extra features table, for shader objects the shader type.
Moreover, QF-Test checks for every text object whether according to its features it might
be a label. If so, the text object is given the class ’Label’. Via the standard algorithm
for ’qfs:label’ in the Extra features table QF-Test will assign the label component to other
components where appropriate.
As the standard algorithm for the recognition of labels is based on assumptions and
probabilities it may happen that labels are not recognized or the falsely identified. In this
(800)
case you may want to use resolvers (section 43.1 ) to improve recognition. Resolvers
can also be used to improve the assignment of label components to the respective field.
Chapter 16
Data-driven testing
Data-driven testing is a very important aspect of test automation. In short, the goal is
to run a given test or set of tests multiple times with different sets of input data and
expected results. QF-Test has various means to store data or load external data for
use in data-driven tests. The most convenient is based on a Data driver node which
sets up an environment for iterating over the sets of data combined with one or more
Data binder nodes to bind the variables for test execution. Note that there is no Data
binder node as such. The name serves as a generic term for the specific nodes like a
Data table or a CSV data file. This is best explained through some examples. A demo
test-suite with simple and advanced examples named datadriver.qft is provided in
the directory doc/tutorial below QF-Test’s root directory. Please take care to store
modified test-suites in a project-related folder.
16.1
Data-driver examples
Figure 16.1: A simple data-driven test
16.1.
Data-driver examples
161
The image above shows a Test-set with a Data driver node that contains a single Data
binder in the form of a Data table node. The contents of the Data table are as follows:
Figure 16.2: Data table example
When the Test-set is executed, it will iterate over the rows of the Data table shown above.
For each of the three iterations the values of the respective row in the table are bound
to the variable named in the matching column header. Thus, during the first iteration the
variable named ”Model” is bound to ”Rolo”, ”Variant” to ”None” and ”Price” to ”19000”.
During the second iteration, ”Model” is set to ”I5” and to ”Minigolf” during the third and
last iteration. For each iteration, all Test-case child nodes of the Test-set are run.
The following image shows a run-log for the above Test-set.
16.1.
Data-driver examples
Figure 16.3: Run-log of a data-driven test
The next example shows that data-driven testing is not limited to a single loop:
Figure 16.4: Data-driven test with nested loops
The Data driver now contains a second Data table node with the following contents:
162
16.1.
Data-driver examples
163
Figure 16.5: Second data table example
The Test-set will now go through a total of six iterations because for each of the three
iterations of the outer loop ”cars”, both iterations of the inner loop ”accessories” will be
run as shown in the following run-log.
16.2.
General use of Data drivers
164
Figure 16.6: Run-log of a data-driven test with nested loops
Note
The extremely useful dynamic names of the loop nodes in the run-logs are obtained by
setting the attribute Name for loop pass in the run-log to the value ”car Model: $(Model)”in
the first and to ”car Model: $(Model), accessory Name: $(Accessory)” in the second
example. As you can see, that name is expanded individually for each iteration, so you
can make use of the variables bound for that iteration.
16.2
General use of Data drivers
As seen in the example above the Data driver node must be placed in a Test-set node,
between the optional Dependency and Setup nodes. When the Test-set is executed it will
check for Data driver and run it. The contents of the Data driver node are not limited to
Data binders. Like a normal Sequence the Data driver node can hold any executable node
to be able to perform any setup that may be required to retrieve the data. Thus it is
also possible to share Data binders by putting them inside a Procedure and calling the
Procedure from inside the Data driver.
Conceptually, a Data binder represents a loop where a different set of variables is bound
for each iteration. A Data binder must be registered with a name in the Data driver context
(479)
node with
of a Test-set. This ensures that the loop can be interrupted by a Break
the same name. Once the Test-set has run the Data driver node, it will iterate over the
registered data loops and perform the tests.
In case of nested loops the Data binder that was registered first represents the outermost
loop. Its variables are bound first and have lesser precedence than the variables from
the inner loop(s).
16.3.
16.3
Examples for Data drivers
165
Examples for Data drivers
We provide a couple of examples for reading CSV or Excel files in the test-suite
doc/tutorial/datadriver.qft.
16.4
Advanced use
(443)
Besides the Data table node there are various other means for binding data in a data
(452)
(456)
(446)
(460)
driver. The Excel data file , CSV data file , Database and Data loop nodes are all
(440)
explained in detail in section 34.4 .
It is also possible to bind data by calling the Procedures qfs.databinder.bindList
or qfs.databinder.bindSets in the standard library qfs.qft. These take as parameters strings with lists or sets of values to split and iterate over. Please see tutorial
chapter 8 for information about the standard library.
And finally, data can be bound directly from Jython (and analogous from Groovy and
JavaScript) with the help of the databinder module, which offers the following methods:
bindList(Object rc, String loopname, String varname, Object
values, String separator=None, String counter=None, String
intervals=None)
Create and register a databinder that binds a list of values to a variable.
Parameters
rc
The current run-context.
loopname
The name under which to bind the data, equivalent to the
Name attribute of a Data binder node.
varname
The name of the variable to bind to.
values
The values to bind. Either a sequence type or a string to
split.
separator
Optional separator character to split the values at in case
they’re a string. Default is whitespace.
counter
An optional variable name for the iteration counter.
intervals
Optional ranges of indices, separated by comma, e.g.
”0,2-3”.
16.4.
Advanced use
166
bindSets(Object rc, String loopname, Object varnames, Object
values, String separator=None, String counter=None, String
intervals=None)
Create and register a databinder that binds a list of value-set to a set of variables.
Parameters
rc
The current run-context.
loopname
The name under which to bind the data, equivalent to the
Name attribute of a Data binder node.
varnames
The names of the variables to bind to. Either a sequence
type or a string to split.
values
The value-sets to bind. Either a sequence of sequences each inner sequence being one set of data to bind - or a
string to split.
separator
Optional separator character to split the varnames and the
values of a value-set at in case they’re a string. Default is
whitespace. Value-sets are separated by line-breaks.
counter
An optional variable name for the iteration counter.
intervals
Optional ranges of indices, separated by comma, e.g.
”0,2-3”.
bindDict(Object rc, String loopname, dictionary dict, String
counter=None, String intervals=None)
Create and register a databinder that binds data from a dictionary. The keys of the
dictionary are the names of the variables and the values are sequences of values to be
bound.
Parameters
rc
The current run-context.
loopname
The name under which to bind the data, equivalent to the
Name attribute of a Data binder node.
dict
The dictionary to bind.
counter
An optional variable name for the iteration counter.
intervals
Optional ranges of indices, separated by comma, e.g.
”0,2-3”.
Some examples:
16.4.
Advanced use
import databinder
# Three iterations with the values "spam", "bacon" and "eggs"
# bound to the variable named "ingredient"
databinder.bindList(rc, "meal", "ingredient", ["spam", "bacon", "eggs"])
# Same with string values
databinder.bindList(rc, "meal", "ingredient", "spam bacon eggs")
# Same with string values and special separator
databinder.bindList(rc, "meal", "ingredient", "spam|bacon|eggs", "|")
# Two iterations, the first with item="apple" and number="5",
# the second with item="orange" and number="3"
databinder.bindSets(rc, "fruit", ["item", "number"],
[["apple",5], ["orange",3]])
# Same with string values, note the linebreak
databinder.bindSets(rc, "fruit", "item number", """apple 5
orange 3""")
# Same as before with the data stored in a dict
databinder.bindDict(rc, "fruit",
{"item": ["apple", "orange"],
"number": [5,3]})
Example 16.1: Examples for use of the databinder module
167
Chapter 17
Reports and test documentation
Besides test-suites and run-logs QF-Test can create a number of additional documents.
Most important of these is the report, which summarizes the overall results of a testrun along with an overview over the test-suites executed and their individual results.
The report is easy to read and understand without further knowledge about QF-Test
and thus complements the run-log which is geared towards error analysis and requires
some QF-Test experience to fully understand.
Following is an example of a report summary:
17.1.
Reports
169
Figure 17.1: Example report
The other documents are more static in nature, describing the content of test-suites instead of test-run results. The testdoc document gives an overview over the structure
(405)
(398)
of Test-set
and Test-case
nodes in a given set of test-suites. It is intended for the
test architect or QA project manager and documents the current state of test devel(470)
(463)
opment. The pkgdoc documentation is similar, but focuses on Package , Procedure
(427)
and Dependency nodes instead. The result is a library reference comparable to Java’s
JavaDoc. The target audience for pgkdoc are test implementers requiring information
about already existing procedures and their parameters.
17.1
Reports
Before we can start explaining how to create reports, some basic terms and concepts
need to be defined.
17.1.
Reports
17.1.1
170
Report concepts
A report represents the results of one or more test-runs. A test-run comprises the
execution of either a single test-suite or multiple test-suites, typically executed together
in one batch-run. A test-run is identified by a runid. It is possible to execute a test-run
in several steps by assigning the same runid to the resulting run-logs.
A report is identified by a report name. For a report that covers a single test-run, the
report name is usually the same as the runid. For reports summarizing the results of
several test-runs a distinct report name can be specified.
Reports can be created in multiple variants: XML, HTML and JUnit. Most users will
probably use the HTML variant which can be viewed in a browser, printed and archived.
The XML variant can serve as the basis for collecting the results of a test-run for further
processing, for example to collect test-results in a database or to create customized
HTML reports. We suggest that you always create both HTML and XML reports unless
you have a good reason to do otherwise. JUnit reports base on the JUnit XML format
as created by Apache Ant by use of its JUnitReport task. This format is not as pretty
and detailed as the first two report variants QF-Test offers but it is directly understood
by many continuous integration tools and may prove useful for a quick integration with
those.
A report consists of one summary document, plus one document per run-log. These
files are collected together with complementary files like icons, stylesheets and screenshot images in a directory. At the file level, this directory represents the report.
The layout of the files inside the report directory depends on some command line options
explained below. Basically there are two ways to lay out the files: Based on the file
structure of the original test-suites or based on the file structure of the run-logs.
17.1.2
Report contents
In advance to the overall test result, a report as shown above starts with a summary
containing informational system data and a legend describing the meaning of counter
(34)
icons used in the report (see Running tests ).
Note
The difference between ”Time spent in tests” and ”Elapsed time” are explicit delays
introduced in nodes via the ’Delay before/after’ attribute or user interrupts.
The contents of a report are based on the original structure of the executed test-suites.
(405)
(398)
(411)
The main structure is created from Test-set
and Test-case
nodes. The Comment
attributes of the root node as well as the Test-set and Test-case nodes share the doctags
(172)
with testdoc documents as explained in section 17.2 . In addition to those doctags
the ’@title’ doctag can be specified in the comment of the root node to set a title for the
report document created for the respective test-suite.
17.1.
Reports
171
(687)
If -report.teststeps is specified in batch mode (true by default) or the respective
option is active in the interactive dialog, Test-cases can be further broken down into steps
(417)
nodes. In addition to explicitly wrapping steps into a Testwith the help of Test-step
step, any node can be turned into a test-step by specifying the doctag ’@teststep’ in its
Comment, followed by an optional name for the step. For Test-step nodes the ’@author’,
’@version’ and ’@since’ doctags are also applicable. The names, comments and tag
values of the various nodes can contain variables that will be expanded at execution
time so that the expanded value is shown in the report. This is especially useful for
test-steps within a procedure.
If listing of test-steps is active, Setup, Cleanup and Dependency nodes are also listed
and checks, screenshots and messages, including warnings, errors and exceptions are
properly integrated into the nested steps. If the test-suites are set up properly the resulting report can serve as a very readable summary of what was going on during the
execution of a test.
Whether warnings and checks are listed is determined by the command line arguments
(688)
(686)
-report.warnings and -report.checks or the respective interactive options.
Warnings from component recognition are never listed because they are too technical
and could easily flood the report. For checks one must distinguish between checks that
represent an actual verification step and those that are used solely for control flow, for
example to check whether a checkbox is already selected and click it only in case it is
not. By default QF-Test lists those Check nodes in the report that have the default result
settings, i.e. the Error level of message is ’Error’, no exception is thrown and no result
variable bound. All others are treated as helpers for control flow and not listed in the
report. For cases where this default treatment is not appropriate, you can force a Check
into the report via the doctag ’@report’ in its Comment attribute or prevent its listing
via ’@noreport’. Of course failed checks are treated as warnings, errors or exceptions
(depending on their Error level of message) and cannot be excluded from the report if
messages at the respective level are shown.
Additional messages, checks and screenshots can be added to the report by scripts via
the methods rc.logMessage, rc.logImage and rc.check and its variants, which
have an optional report parameter. For details, please see the run-context API docu(729)
mentation in section 41.6 .
17.1.3
Creating reports
There are three ways to create reports:
• Interactively from a run-log through the menu item File→Create report... .
• In batch mode as the result of a test-run.
17.1.
Reports
172
• In batch mode by transforming already existing run-logs.
The interactive variant is easy to use. Just select the target directory for the report and
whether you want the XML and/or the HTML variant.
For report creation in batch mode there are a number of command line options which
(676)
are listed and explained in section 36.2 . Let’s look at the variant of creating reports
as the result of a test-run first:
The command line syntax for plain test-execution in batch mode is qftest -batch
<test-suite> [<test-suite>...]
(686)
To create a combined XML and HTML report, use -report <directory> .
To create only one version or to separate the XML, HTML variants, use
(688)
(686)
-report.xml <directory>
and/or -report.html <directory> .
For
(687)
JUnit reports -report.junit <directory> works respectively.
(688)
The runid of a test-run is specified with -runid <ID> , the name of the report with
(687)
-report.name <name> . If the report name is unspecified it will default to the runid.
To lay out the files in the report directory according to the file structure of the test-suites,
(689)
use -sourcedir <directory> . To use the file structure of the run-log as the
(688)
basis, use -runlogdir <directory> .
The following is a typical example of a command line for a batch run making use of the
(694)
placeholders explained in section 36.2.4 :
qftest -batch -runid +M+d -runlog logs/+i -report report_+i
-sourcedir . suite1.qft subdir/suite2.qft
Example 17.1: Creating a report as the result of a test-run
Creating a report as a separate step by transforming a set of run-logs is similar in many
respects. The run-logs to transform have to be specified instead of the test-suites to execute and the -runid <ID> and -sourcedir <directory> command line options
have no effect. The following is an example for how to create a weekly summary report
based on the assumption that you have collected all run-logs below the directory named
logdir, possibly in subdirectorys thereof:
qftest -batch -genreport -report report_+M+d
-report.name week_of_+y+M+d logdir
Example 17.2: Creating a weekly summary report
17.2.
Testdoc documentation for Test-sets and Test-cases
17.2
173
Testdoc documentation for Test-sets and Test-cases
The type of test documents called testdoc provide overview and detailed information
(405)
(398)
over the Test-set
and Test-case
nodes of one or more test-suites. When Test-cases
(417)
contain Test-steps
those steps will be included in the testdoc. By default QF-Test
(411)
nodes during testdoc creation.
By setting the option
ignores Test call
(692)
-testdoc.followcalls =true the real targets Test-case, Test-set or the whole
test-suite are processed as if they were part of the original test-suite.
This documentation is a valuable tool for QA project managers to keep track of the
current state of test development. Similar to reports, testdoc documents are laid out as
directories with one summary file and one detailed file per test-suite.
A testdoc document for a single suite can be created interactively from a test-suite by
selecting Create testdoc documentation... from the File menu. This is very useful
during test development to quickly check whether all tests are properly documented.
For actual use as a reference it is preferable to create complete sets of documents
spanning multiple test-suites for a whole project. This can be done by running QF-Test
(681)
in batch mode with the -gendoc command line argument. In its simplest form, a call
to create testdoc documentation for a whole directory tree would look as follows:
qftest -batch -gendoc -testdoc test_documentation
directory/with/test-suites
Example 17.3: Creating testdoc documentation
(671)
Please see chapter 36
arguments.
for detailed information about the available command line
(411)
attributes of
To get optimal results you can use HTML markup in the Comment
Test-set and Test-case nodes and also make use of doctags. A doctag is a keyword
beginning with ’@’, sometimes followed by a name and always by a description. This is
a proven concept in JavaDoc, the standard documentation format for Java programs
(see
http://www.oracle.com/technetwork/java/javase/documentation/index137868.html#tag).
Note
All doctags must appear after the main description. Description after the doctags will be
ignored, as well doctags inside the descripiton are not allowed.
The following doctags are supported for Test-set and Test-case nodes:
@deprecated
If a Test-set or Test-case is no longer to be used, this description should explain
17.3.
Pkgdoc documentation for Packages, Procedures and Dependencies
174
when and why the node was deprecated and especially which replacement
should be used.
@condition
Non-formal explanation of the condition under which the node is executed or
skipped.
@param
Description for a parameter. Following are the name of the parameter and its
description.
@charvar
Description for a characteristic variable. Following are the name of the variable
and its description.
@author
Author of the Test-set or Test-case.
@version
Version of the Test-case or Test-case.
@since
The version since which this Test-set or Test-case has been available.
In addition to the doctags described above, the doctag ’@title’ in the comment of the root
node can be used to specify a title for the testdoc document created for the respective
test-suite.
17.3
Pkgdoc documentation for Packages, Procedures
and Dependencies
The concepts of and methods for creation of pkgdoc documents are nearly identical to
testdoc, so this section is brief. Instead of Test-set and Test-case nodes, pkgdoc docu(470)
(463)
(427)
and Dependency
nodes. They are intended for
ments cover Package , Procedure
the test developer to keep track of the procedures available for use in implementing
tests.
17.3.
Pkgdoc documentation for Packages, Procedures and Dependencies
175
Please refer to the standard library qfs.qft as a good example how a pkgdoc may
look like.
A pkgdoc document can also either be created interactively using
(671)
File→Create HMTL/XML pkgdoc... or in batch mode. Again, please see chapter 36
for detailed information about the available command line arguments.
As the following example shows, testdoc and pkgdoc can even be created together in a
single batch run:
qftest -batch -gendoc -testdoc tests -pkgdoc procedures
directory/with/test-suites
Example 17.4: Creating testdoc and pkgdoc documentation in a single run
Of course pkgdoc also supports HTML markup and doctags. The following doctags are
supported for Package, Procedure and Dependency nodes:
@deprecated
If a Procedure, Dependency or Package is no longer to be used, this description
should explain when and why the node was deprecated and especially which
replacement should be used.
@param (Procedure and Dependency only)
A parameter of a Procedure or Dependency.
parameter and its description.
Following are the name of the
@charvar (Dependency only)
Description for a characteristic variable of a Dependency. Following are the name
of the variable and its description.
@return (Procedure only)
The return value of the procedure.
@result (Procedure and Dependency only)
Can be used to document side-effects of the Procedure or Dependency like setting
a global variable.
@throws (Procedure only)
17.3.
Pkgdoc documentation for Packages, Procedures and Dependencies
176
Expected exception. Following are the name of the exception and a description of
its cause.
@catches (Dependency only)
An exception being caught by the Dependency. Following are the name of the
exception and a description of the handler.
@author
Author of the Package, Procedure or Dependency.
@version
Version of the Package, Procedure or Dependency.
@since
The version since which this Package, Procedure or Dependency is available.
In addition to the doctags described above, the doctag ’@title’ in the comment of the root
node can be used to specify a title for the pkgdoc document created for the respective
test-suite.
Chapter 18
Test execution
When talking about test execution, there are two aspects to be considered. On one
hand you need to run tests while they are developed to check them for proper operation.
(34)
This situation has already been described in section 4.2 . Basically all you have to do
to run a test interactively is invoking Run→Start from the main menu.
On the other hand you want to run your tests periodically to ensure the stability of the
system under test, for example in nightly regression tests. Instead of launching QF-Test,
loading the test-suite and running it from the graphical user interface, it is much more
convenient here to execute tests from the command line in batch mode. This kind of
running tests is explained in the first section of this chapter (Test execution in batch
(176)
mode ).
Sometimes, for instance when you want to run the test on a remote computer, a second
variant comes into play: the daemon mode. This type of test execution, which uses a
running QF-Test instance to execute tests, is the topic of the second section (Executing
(182)
tests in daemon mode ).
For integration of QF-Test with build tools like ant, maven or Hudson/Jenkins, please
(231)
refer to chapter 22 .
18.1
Test execution in batch mode
There are a lot of command line arguments when running QF-Test in batch mode; an
(671)
overview can be found in chapter 36 . Here we will present examples showing the
most important of them.
The examples are written for the Windows operating system, but you may easily adapt
them for the Linux platform. What is different is the path specification and also the
(694)
syntax for placeholders (section 36.2.4 ): On Linux you can use +X as well as %X.
On Windows there’s a separate console application qftestc.exe. In contrast to its
18.1.
Test execution in batch mode
178
GUI variant qftest.exe, it waits until the execution of QF-Test has terminated and
(502)
also displays print output from a Server script . You can use qftestc.exe in place of
qftest.exe wherever you’ll find it convenient.
18.1.1
Command line usage
Let’s start with the most simple QF-Test command to execute a test:
qftest -batch -run c:\mysuites\suiteA.qft
Example 18.1: Test execution from the command line
The argument -batch makes QF-Test start without a graphical user interface. The
second argument, -run, is the specifier for test execution. Finally, at the end of the
command line, you find the test-suite to be executed.
Note
The argument -run is optional, i. e. the test execution is defined as default for the batch
mode.
When running the above command, all top-level Test-case and Test-set nodes of
suiteA.qft will be executed one after another. After the test-run you will find a
run-log file in the current directory; it has the same name as the test-suite (except from
the extension, which can be .qrl, .qrz or .qzp). The run-log file shows the result of
the test-run.
By specifying -nolog you can suppress the creation of a run-log. Probably this only
makes sense, if you have extended your test by your own log output (written to a file).
Otherwise you’d have to check the result code of QF-Test, whereas 0 means that everything is alright. A positive value in contrast indicates that warnings, errors or exceptions
(695)
occurred during the test-run (see section 36.3 ). That’s why in most situations you’ll
probably prefer to create a run-log and save it at a fixed place in the file system. This
can be achieved with the parameter -runlog:
qftest -batch -compact -runlog c:\mylogs\+b c:\mysuites\suiteA.qft
Example 18.2: Test execution with run-log creation
A run-log file suiteA.qrz will now be created in the specified directory c:\mylogs.
The placeholder +b is responsible for its name being identical with that of the test-suite.
The additional switch -compact prevents the run-log from growing too large: Only the
nodes needed for a report and those immediately before an error or an exception are
kept in the run-log. Especially in case of very long test-runs this may help to reduce the
18.1.
Test execution in batch mode
179
amount of required memory. In the meantime however, QF-Test offers split run-logs for
(74)
the same reason. For more information about this topic see section 8.1 .
Note
Whether the file is indeed created as compressed run-log (to be distinguished from the
above ”compact”) with extension .qrz, depends on the system settings. To force the
creation of a particular format you can set the file extension explicitly. With -runlog
c:\mylogs\+b.qrl, for example, an uncompressed XML file will be produced.
Sometimes you may want to execute not the whole test-suite but only parts of it. By
using the parameter -test you can run a specific node of the test-suite:
qftest -batch -runlog c:\mylogs\+b -test "My test-case" c:\mysuites\suiteA.qft
Example 18.3: Executing a specified node
The parameter -test expects the QF-Test ID attribute of the node to follow or the qualified name of a Test-case or Test-set. If you want to execute several nodes, you can define
-test <ID> multiple times. Apart from the node’s QF-Test ID, -test accepts also the
numerical index of a top-level node. For example, -test 0 will run the first child of the
Test-suite node.
The run-log provides a rather technical view of the test-run; it is helpful mainly when
(74)
analyzing errors (cf. section 8.1 ). The report in contrast contains a summary of the
(167)
executed test-cases and errors (cf. chapter 17 ) in XML or HTML format. It is created
from the run-log either in a separate step after running the test or automatically with the
test-run:
qftest -batch -runlog c:\mylogs\+b
-report c:\mylogs\rep_+b_+y+M+d+h+m
c:\mysuites\suiteA.qft
Example 18.4: Creating a report
In this example the XML and HTML files are saved in a directory which name consists
of the test-suite and a timestamp like c:\mylogs\rep_suiteA_0806042152. When
replacing the argument -report with -report.xml or -report.html respectively,
only an XML or HTML report will be created.
Test-cases often uses variables to control the execution of the test. For example, you
(395)
may have defined the variable myvar in the Test-suite
node of the suite. You can
overwrite its default value when running the test-suite from the command line:
qftest -batch -variable myvar="Value from command line"
-runlog c:\mylogs\+b c:\mysuites\suiteA.qft
Example 18.5: Test execution with variables
18.1.
Test execution in batch mode
180
If needed, you can specify -variable <name>=<wert> multiple times to set values
for different variables.
18.1.2
Windows batch script
Running tests from the command line is fundamental for integrating QF-Test in test
(207)
management systems (see Interaction with Test Management Tools ). Otherwise, living without such a tool, you may find it convenient to embed the command for the test
execution into a script. A simple Windows batch script (qfbatch.bat) looks like this:
@echo off
setlocal
if "%1" == "" (
echo Usage: qfbatch Testsuite
goto end
) else (
set suite=%~f1
)
set logdir=c:\mylogs
pushd c:\programs\qftest\qftest-4.2.2\bin
@echo on
.\qftest -batch -compact -runlog %logdir%\+b %suite%
@echo off
if %errorlevel% equ 0 (
echo Test terminated successfully
goto end
)
if %errorlevel% equ 1 (
echo Test terminated with warnings
goto end
)
if %errorlevel% equ 2 (
echo Test terminated with errors
goto end
)
if %errorlevel% equ 3 (
echo Test terminated with exceptions
goto end
)
if %errorlevel% leq -1 (
echo Error %errorlevel%
goto end
)
:end
popd
Example 18.6: Batch script qfbatch.bat to execute a test-suite
18.1.
Test execution in batch mode
181
Now you can simply run that script with only the file name of the test-suite as parameter.
Everything else is done automatically: The test-suite will be executed, the run-log file
stored in logdir and finally the script will print out the state of the test-run (depending
on the QF-Test result code).
18.1.3
Groovy
3.0+
(114)
Since version QF-Test 3 the language Groovy is part of the release (cf. chapter 12 ).
It is meant mainly for scripting inside QF-Test (Server and SUT scripts), but it can, like
Jython, also be used outside of QF-Test. Groovy is probably well suited to create a little
test execution management system by yourself. By the way, Groovy simplifies working
with Ant, too: Instead of dealing with bulky XML files, which makes it hard to define
conditions, you can work with the Groovy AntBuilder. However, that’s out of scope
here, the following example doesn’t rely on Ant but only on the basic Groovy features:
18.1.
Test execution in batch mode
182
def suite = ’’
if (args.size() == 0) {
println ’Usage: groovy QfExec Testsuite’
return
}
else {
suite = args[0]
}
def qftestdir = ’c:\\programs\\qfs\\qftest\\qftest-4.2.2’
def qftest = qftestdir + ’\\bin\\qftest.exe’
def command = "$qftest -batch -compact -runlog c:\\mylogs\\+b \"$suite\""
def printStream = { stream ->
while (true) {
try {
stream.eachLine { println it }
} catch (IOException) {
break
}
}
}
println "Running command: $command"
def proc = command.execute()
new Thread().start() { printStream(proc.in) }
new Thread().start() { printStream(proc.err) }
proc.waitFor()
switch (proc.exitValue()) {
case ’0’: println ’Test terminated successfully’; break
case ’1’: println ’Test terminated with warnings’; break
case ’2’: println ’Test terminated with errors’; break
case ’3’: println ’Test terminated with exceptions’; break
default: println "Error ${proc.exitValue()}"
}
Example 18.7: Groovy script QfExec.groovy to execute a test-suite
If you have Groovy installed on your computer independently of QF-Test, you can run the
example test-suite simply via groovy QfExec c:\mysuites\suiteA.qft. Otherwise you can use the Groovy jar file from the QF-Test installation, preferably again with
help of a batch script:
18.2.
Executing tests in daemon mode
183
@echo off
setlocal
if "%1" == "" (
echo Usage: qfexec Testsuite
goto end
)
set qftestdir=c:\programs\qftest\qftest-4.2.2
set scriptfile=QfExec.groovy
java -cp %qftestdir%/lib/groovy-all.jar groovy.ui.GroovyMain %scriptfile% %*
:end
Example 18.8:
Batch script qfexec.bat to run a Groovy script (here:
QfExec.groovy)
Now execute the test-suite with qfexec c:\mysuites\suiteA.qft.
18.2
Executing tests in daemon mode
In daemon mode QF-Test listens to RMI connections and provides an interface for remote test execution. This is useful for simplifying test execution in a distributed load(265)
testing scenario (chapter 24 ), but also for integration with existing test-management
(207)
or test-execution tools (chapter 21 ).
Note
(309)
GUI tests require an active user session. Chapter Hints on setting up test-systems
contains useful tips and tricks to set-up the daemon process. In FAQ 14 you can find
technical details.
18.2.1
Launching the daemon
!!! Warning !!!
Anybody with access to the QF-Test daemon can start any program on the machine
running the daemon with the rights of the user account that the daemon is running
under, so access should be granted only to trusted users.
If you are not running the daemon in a secure environment where every user is trusted
or if you are creating your own library to connect to the QF-Test daemon, you definitely
(896)
should read section 44.3 about how to secure daemon communication with SSL.
To work with a daemon, you must first launch it on any computer in your network (of
course, this host can also be localhost):
18.2.
Executing tests in daemon mode
184
qftest -batch -daemon -daemonport 12345
Example 18.9: Launching a QF-Test daemon
Note
Important compatibility note:
3.5+
Starting with QF-Test version 3.5, SSL is used for daemon communication by default.
To interact with a QF-Test version older than 3.5 you must start the daemon with an
(682)
empty -keystore <keystore file> argument in the form:
qftest -batch -keystore= -daemon -daemonport 12345
Example 18.10: Launching a QF-Test daemon without SSL
If you omit the argument -daemonport, the daemon will listen on QF-Test’s standard
port 3543. You may check whether the daemon is running by means of the netstat
utility:
Windows
netstat -a -p tcp -n | findstr ”12345”
Linux
netstat -a --tcp --numeric-ports | grep 12345
If you want to launch a daemon on a remote host, you may use for instance ssh or VNC.
Your network administrator knows whether and how this works. To follow the examples
below, a local daemon will be sufficient.
18.2.2
Controlling a daemon from QF-Test’s command line
3.0+
The easiest way to get in touch with a daemon is running QF-Test from the command
line in the calldaemon mode. The following example checks if a daemon is listening at
the specified host and port:
qftestc -batch -calldaemon -daemonhost localhost -daemonport 12345 -ping
Example 18.11: Pinging a QF-Test daemon
In contrast to the netstat command from above -ping also works between different
computers (if you check the daemon on your local computer, you can omit the argument
-daemonhost).
What you actually want from a daemon is executing your test-case(s) and getting back
a run-log file. It sounds and indeed looks quite similar to what you have seen before
when running a test in batch mode:
18.2.
Executing tests in daemon mode
185
qftest -batch -calldaemon -daemonhost somehost -daemonport 12345
-runlog c:\mylogs\+b
-suitedir c:\mysuites
suiteA.qft#"My test-case"
Example 18.12: Running a test-case with the QF-Test daemon
Note
In contrast to the batch mode, a Test-case or a Test-set node is always referenced here
by its qualified name, for instance ”My Test-set.My Test-case” (just to remember: -test
<ID> may be used in batch mode). To execute the complete suite suiteA.qft, you
can simply omit the test-case or write suiteA.qft#..
If the daemon is running on a remote host, you have to specify it explicitly via
-daemonhost (default is -daemonhost localhost). Note that the parameter
-suitedir refers to the remote host (where the daemon is running) while -runlog
defines a local file.
3.4+
In case you cannot easily observe the test running on a remote host, you may find
it convenient to add the argument -verbose to get status output in the console (on
Windows, use qftestc to see the output).
A running daemon, no matter whether local or remote, can be terminated with the calldaemon command -terminate:
qftest -batch -calldaemon -daemonport 12345 -daemonhost localhost -terminate
Example 18.13: Terminating a QF-Test daemon
A complete list of the calldaemon parameters can be found in the chapter Command
(671)
line arguments and exit codes .
18.2.3
Controlling a daemon with the daemon API
Using the QF-Test command line to control a daemon was quite easy. On the other
hand, to get all capabilities of a daemon, you have to deal with the daemon API. In this
section we will concentrate on some basic examples, the whole interface is described in
(882)
chapter 44 .
To get started with the daemon API, insert a Server script node with the following code:
18.2.
Executing tests in daemon mode
186
from de.qfs.apps.qftest.daemon import DaemonRunContext
from de.qfs.apps.qftest.daemon import DaemonLocator
host = "localhost"
port = 12345
# Leading r means raw string to allow normal backslashes in the path string.
testcase = r"c:\mysuites\suiteA.qft#My test-case"
timeout = 60 * 1000
def calldaemon(host, port, testcase, timeout=0):
daemon = DaemonLocator.instance().locateDaemon(host, port)
trd = daemon.createTestRunDaemon()
context = trd.createContext()
context.runTest(testcase)
if not context.waitForRunState(DaemonRunContext.STATE_FINISHED, timeout):
# Run did not finish, terminate it
context.stopRun()
if not context.waitForRunState(DaemonRunContext.STATE_FINISHED, 5000):
# Context is deadlocked
raise UserException("No reply from daemon RunContext.")
rc.logError("Daemon call did not terminate and had to be stopped.")
result = context.getResult()
log = context.getRunLog()
rc.addDaemonLog(log)
context.release()
return result
result = calldaemon(host, port, testcase, timeout)
rc.logMessage("Result from daemon: %d" %result)
Example 18.14: Daemon API in a Server script
The script shows the basic mechanisms to control a daemon:
• First find a running daemon with locateDaemon.
• Provide an environment for test-runs by calling createTestRunDaemon.
• To run a test, you need a context object (createContext). The creation of that
object requires a QF-Test run-time license.
• Now the context enables you to start a test-run (runTest) and to query about its
current state. waitForRunState waits during the defined timeout (in milliseconds) until the specified state has occurred. In the example above, we wait for the
test to terminate within one minute.
• Finally, when the test-run has terminated, the context can query the test result with
(695)
the method getResult (cf. Exit codes for QF-Test ).
• Moreover, you can use the context to get the run-log of the daemon test-run. It
can be included in the local run-log by means of the rc method addDaemonLog.
18.2.
Executing tests in daemon mode
187
Note
To keep it small and simple, the example script does not contain any error handling.
However, particularly when working with a daemon, you should check every method
call.
Note
Driving a daemon from a Server script has the disadvantage of consuming an additional
QF-Test license to run the script node interactively or in batch mode. However, this
doesn’t apply nor for the above-mentioned calldaemon mode neither for the case when
controlling a daemon outside QF-Test (see below).
The usage of the daemon API is not restricted to Server scripts. Outside QF-Test a
daemon can be contacted by means of a Java program or, more easily, a Groovy script.
The following Groovy script works with several running daemons and may serve as a
starting point for load tests. Suppose we have started some daemons in our network,
each on a separate machine. We want to execute a test-case simultaneously by all of
the daemons and we want to save a run-log for every single test-run (daemon1.qrl,
..., daemonN.qrl). The test-suite containing the test-case to be executed may be
available for all daemon instances via the network drive z:).
import de.qfs.apps.qftest.daemon.DaemonLocator
import de.qfs.apps.qftest.daemon.DaemonRunContext
def testcase = "z:\\mysuites\\suiteA.qft#My test-case"
def logfile = "c:\\mylogs\\daemon"
def timeout = 120 * 1000
def daemons = DaemonLocator.instance().locateDaemons(10000)
def contexts = []
// Start tests
for (daemon in daemons) {
def trd = daemon.createTestRunDaemon()
trd.setGlobal(’machines’, daemons.size().toString())
def context = trd.createContext()
contexts << context
context.runTest(testcase)
}
// Wait for tests to terminate
for (i in 0..<contexts.size()) {
def context = contexts[i]
context.waitForRunState(DaemonRunContext.STATE_FINISHED, timeout)
byte[] runlog = context.getRunLog()
def fos = new FileOutputStream("$logfile${i + 1}.qrl")
fos.write(runlog)
fos.close()
context.release()
}
Example 18.15: Groovy daemon script CallDaemon.groovy
To run that Groovy script, you need the QF-Test libraries qftest.jar,
qfshared.jar, and qflib.jar as well as the Groovy library, which is also part of
18.3.
Re-execution of nodes (Re-run)
188
the QF-Test installation. The following batch script shows how it works:
@echo off
setlocal
set qftestdir=c:\programs\qftest\qftest-4.2.2
set qflibdir=%qftestdir%\qflib
set classpath=%qftestdir%\lib\groovy-all.jar
set classpath=%classpath%;%qflibdir%\qftest.jar;%qflibdir%\qfshared.jar;
%qflibdir%\qflib.jar
java -cp %classpath% groovy.ui.GroovyMain CallDaemon
Example 18.16: Batch script calldaemon.bat to run Calldaemon.groovy
(265)
To make the daemon example a load test (cf. chapter 24 ), you have to synchronize
the test-runs inside of ”My test-case” (e. g. after starting the SUT). This can be done by
means of the rc method syncThreads:
def machines = rc.getNum(’machines’)
rc.syncThreads(’startup’, 60000, -1, machines)
Example 18.17: Groovy Server script node to synchronize the test-runs
The variable machines denotes the number of hosts with a daemon running on them.
Best define it in the Test-suite node of the test-suite with a default value of 1. When
running the Groovy script, it will be overwritten with the correct value.
18.3
Re-execution of nodes (Re-run)
18.3.1
Triggering re-run from a run-log
4.1+
When a test-run has finished, the run-log or report is a good entry point for evaluating
the results. In case of errors you may face various challenges. You might want to reexecute the failed test-cases to investigate the reason for the error or because you want
to perform an official re-test of this failing situation after removing the error condition.
If the re-test results are to be shown in the test-report, you may want to replace the
previous results or append them to the existing ones. Or you might just want to repeat
the test-case with the previous variable settings and keep the new run-logs and reports
separately.
To that end QF-Test offers the capability to trigger re-execution from the run-log. You
can trigger a re-run via selecting the run-log node or any test-set node and choose
Re-run test-cases from the Edit menu or from the context menu. Alternatively you
18.3.
Re-execution of nodes (Re-run)
189
can select the nodes to re-run it the error-list and use the context menu entry
Re-run test-cases of selected nodes . The dialog then shown lets you select the
test-cases for the re-run and choose how to handle run-logs via the selection box Mode
for merging run-logs with the following options:
Choice
Replace test-cases
Merge run-logs
Append run-log
Keep run-logs separated
Meaning
Replace the test-cases from the original run-log
with the results from the re-run, i.e. the previous
results will get lost. The previous run-log will be
saved in a backup copy.
The new test-results will be merged into the existing structure.
The new test-results will be appended to the end
of the run-log. The test-set structure will be ignored.
The new test-results will be written to a new runlog, the original one remains unchanged.
Table 18.1: Choices for handling the run-log of a re-run
18.3.
Re-execution of nodes (Re-run)
190
Figure 18.1: Dialog to re-run test-cases
For each test-case the variable values are taken from the run-log of the original testrun. During re-execution the variable ${qftest:isInRerunFromLog} is set to true,
making it possible to distinguish between a normal test-run and a a re-run.
Note
Merging of run-logs makes use of names of test-cases and test-sets. Therefore, those
18.3.
Re-execution of nodes (Re-run)
191
names should be unique. In case of data-driven testing you should take care to keep
those names unique via the attributes Name for separate run-log or Characteristic variables.
18.3.2
Re-running failing nodes immediately
During your test-automation project you can sometimes face situations where some teststeps don’t provide reliable results, failing sometimes but not always. Most of the time
(604)
such cases depend on timing and can be stabilized using Wait for component to appear
nodes, or checks for conditions, delays, scripts or other control structures. As an alternative or additional approach QF-Test offers the capability to repeat such steps whenever
they fail.
This automated re-running in case of error can be applied to every executable node
using a certain doctag in the comment attribute. This doctag can look like this:
@rerun attempts=3;errorlevel>=ERROR;newerrorlevel=WARNING;
handler=handlers.errorhandler
Example 18.18: Example for a re-run definition
In the example above a failed node will be repeated up to three times until an attempt
succeeds. Failed attempts will be downgraded to warning in the run-log. In case all
attempts fail the last attempt will be reported as error or exception. After every failing
attempt QF-Test will execute the procedure handlers.errorhandler.
If you are interested in the number of the current re-run attempt, you can use the variable
(67)
reruncounter from the qftest variable group, see section 7.5 .
The @rerun doctag has parameters like attempts or errorlevel with possibilities
as follows:
attempts
The maximum number of attempts.
errorlevel (optional)
Defines the error states to be handled. Possible values are EXCEPTION, ERROR
or WARNING with = for an exact match or > or >= for a range. Specifying
errorlevel=ERROR means to re-run that node only in case of an error whereas
errorlevel>=ERROR triggers the re-run in case of errors or exceptions. If this
parameter isn’t specified the value errorlevel>=ERROR will be taken as
default.
18.3.
Re-execution of nodes (Re-run)
192
newerrorlevel (optional)
Specifies the error-level in the run-log for the initial run and possible additional
failed runs. You can again choose between EXCEPTION, ERROR or WARNING with
the additional options NOLOG and KEEP. The level NOLOG stands for removing the
failing results from the run-log entirely. NOLOG should be used with extreme care.
Using the level KEEP doesn’t override the original error level and reports it
unchanged. If this parameter isn’t specified the value WARNING will be taken as
default.
handler (optional)
The name of the procedure which should be called in case a caught error state
occurs. This procedure will be called after each failed attempt.
reusevariables (optional, default=true)
Specifies whether to reuse the variable values from the beginning of the first run.
When set to false the current variable values will be used.
logmessages (optional, default=true)
If that parameter is set to true a message will be written into the run-log, when
an attempt begins and when the execution of that sequence terminates. In
addition every node gets an annotation in the run-log with the current attempt.
logmessagesintoreport (optional, default=true)
If this parameter and the parameter logmessages are set to true, all messages
will be written to the report as well.
keepfirst (optional, default=false)
If this value is set to true the first failing attempt will be logged with its original
error level. In case of further failing attempts those errors will be logged with the
newerrorlevel level.
exceptiontype (optional)
In case you want to catch only one specific exception you can specify the
exception type here,
e.g.
CheckFailedException or just
ClientNotConnected for a ClientNotConnectedException.
This
parameter should only be used if you set Exception as value for the parameter
(493)
errorlevel. Please see the Catch node for details about exceptions.
18.3.
Re-execution of nodes (Re-run)
193
exceptionmessage (optional)
In case you want to catch only one specific exception with one text, you can
specify the text here. This parameter should only be used if you set Exception
(493)
as error level. Please see the Catch node for details about exceptions.
exceptionregex (optional)
If true, the value of exceptionmessage is a regular expression. This
parameter should only be used if you set Exception as error level and an
(493)
exception message. Please see the Catch node for details about exceptions.
exceptionlocalized (optional)
If true, the value of exceptionmessage should be the localized exception
message, e.g. mostly the full text. This parameter should only be used if you set
(493)
Exception as error level and an exception message. Please see the Catch
node for details about exceptions.
Chapter 19
Distributed test development
The previous chapters all focused on creating and organizing sets of reliable tests in a
single test-suite. However, when testing a large application, a single suite may not be
enough. There are at least two scenarios where splitting tests into multiple test-suites
becomes essential:
• Multiple developers are creating and editing tests. To avoid redundancy and duplication of code, separately developed tests should use common Procedures and
Components where possible, but only one person can edit a test-suite at a time.
• Tests are simply getting too large. Run-logs for extensive test-runs may cause the
system to run out of memory and organizing a large number of tests in a single
suite is difficult. Things may become unwieldy. It may also be desirable to be able
to run some of the tests as part of the whole test as well as standalone.
QF-Test provides a set of advanced features that make it possible to split and arrange
tests across a set of test-suites. Multiple developers can work on separate parts of
a test, then coordinate their efforts, merge the Components of their suites and create
libraries of shared Procedures.
This chapter first explains the various mechanisms for distributed test development and
how they interact. The final section then summarizes these in concise step-by-step
instructions on how to approach large testing efforts with QF-Test.
19.1
Referencing nodes in another test-suite
(463)
(644)
It is possible to reference Procedures and Components in a test-suite other than the
current one. These references can be explicit or implicit through included files:
19.1.
Referencing nodes in another test-suite
195
• Explicit references use a syntax similar to the one used in URLs to specify an
item inside a web page. The referenced suite must be prepended to the
(467)
(466)
Procedure name
attribute of a Procedure call
of the QF-Test component ID
attribute of a Component dependent node, separated by a ’#’ character. The usual
packagepath.procedure becomes suite#packagepath.procedure.
(396)
(395)
• Implicit references make use of the Include files attribute of the Test-suite node.
Whenever a node is not found in the current suite, QF-Test will search for a matching Procedure or Component within all the suite’s directly or indirectly included files
(a file is considered indirectly included by a suite if it is found as an included file
within one of the suite’s own included files; for example, if suite A includes B, and
suite B includes C, then C is indirectly included by A).
A test-suite that references a node in another test-suite becomes dependent on that
suite. If the Name of a Procedure or the QF-Test ID of a Component in the referenced suite
changes, the suite with the reference must get updated, otherwise the link is broken
and the suite will no longer work correctly. In such cases QF-Test will automatically
update references if it knows about them. The best way to ensure that is to have both
test-suites in a common project because QF-Test automatically tracks all includes and
all explicit references within a project. Alternatively you can list the calling suite in the
(396)
Dependencies (reverse includes) attribute of Test-suite root node of the referenced suite.
While implicit references are more convenient in most cases, they can make tests harder
to understand because it is not immediately obvious where the Procedure or Component
referenced by some node is actually located. One possibility
to find out is to select
”Locate procedure” (Ctrl-P ) or ”Locate component” (Ctrl-W ) from the context menu.
Additionally, QF-Test provides the menu items Operations→Make references explicit
and Operations→Make references implicit which let you toggle quickly between the
two modes without changing the actually referenced nodes.
In both cases, the referenced suite can either be given a relative or absolute filename.
Relative filenames will be resolved relatively to the directory of current suite, or - if that
fails - relatively to the directories on the library path (see option Directories holding
(331)
test-suite libraries ). Always use the forward ’/’ as the directory separator, even under
Windows. QF-Test will map it to the correct character for the system it runs on. This
keeps your test-suites independent from the operating system.
Note
Your Package and Procedure names as well as Component QF-Test IDs should not contain
any ’\’ or ’#’ characters. If they do, you need to include an escape character in the
(722)
Procedure call or the QF-Test component ID attribute. See section 40.6 for details about
escaping and quoting special characters.
When choosing the Procedure for a Procedure call or the Component for some event in
the dialog, QF-Test offers a selection of all currently opened test-suites. If a Procedure or
Component from another test-suite is selected, QF-Test automatically creates the correct
19.2.
Managing Components
196
reference, taking included suites into account. When the test is run at a later time, the
referenced test-suite is loaded automatically if necessary.
During execution QF-Test keeps a stack of currently executing suites. Whenever a Procedure is called in another suite, the called suite is pushed on to the top of this stack and
removed when execution returns to the calling suite. Whenever during the execution of
a Procedure a Window or Component is referenced by its QF-Test ID, QF-Test searches
through this stack of suites from the top to the bottom, i.e. first in the test-suite of the
called Procedure and then in the calling suite, always checking any included files along
the way. This process is quite complicated and you should take care to keep your include
hierarchies simple. In case you encounter problems anyway, a detailed explanation is
(723)
given in section 40.7 .
19.2
Managing Components
(40)
As we have emphasized in chapter 5 , the Components are the essential part of a
test-suite. If the SUT changes between releases, these will likely be affected most. If
changes are so massive that QF-Test cannot adapt automatically, the Components will
have to be updated manually. This is why you should try to avoid redundancy in the
Component hierarchy of your tests more than in any other part.
Therefore, when splitting your tests across multiple test-suites you should try to keep
the Components together in one central test-suite and include this suite from the other
suites. For very large applications you may want to split the Component hierarchy into
parts, each related to a separate part of the SUT’s GUI.
Maintaining this central Component library is not trivial. The problems that will arise can
be resolved with QF-Test as follows:
• When multiple test developers are recording new Components simultaneously, they
cannot be integrated immediately into the central suite, because only one user
can edit the central suite at a time. Instead, Components must be integrated later
by importing them into the central suite when the new tests have stabilized. This
is explained in the following section.
• When the SUT changes, Components in the central suite may need to be updated. If this involves changing any Component QF-Test IDs, this will break any
references to these Components from other suites. To avoid that, QF-Test must
update those references and it will do so, provided that the suites that depend on
the central suite are currently loaded, belong to the same project or are listed in
(396)
(395)
the Dependencies (reverse includes) attribute of the Test-suite node of the central
suite.
19.3.
Merging test-suites
19.3
197
Merging test-suites
Test-suites can be merged by importing one test-suite into another with the
File→Import... menu item.
You can select the areas of the test-suite, which should be imported.
You have to take care about a correct Include/Reverse-Include of your test-suites to
(288)
for
ensure, that all calls and component references are still valid. See chapter 28
details.
19.3.1
Importing Components
During import, all Windows and Components of the imported test-suite are integrated
into the component hierarchy of the importing suite. Components that already exist are
not copied. An QF-Test ID conflict (identical components with different QF-Test IDs or
differing components with identical QF-Test IDs) is resolved automatically by changing
the QF-Test ID of the imported component.
Afterwards, all Windows and Components are removed from the imported suite. All nodes
in the imported suite that referred to these Components are updated accordingly. Ideally,
the imported suite should include the importing suite so no explicit suite references will
have to be created.
19.3.2
Importing Procedures and Testcases
3.3+
As you can import Components QF-Test also allows to import Procedures, Packages, Dependencies and Test-cases as well as Test-sets by choosing ’Procedures’ or ’Tests’ in the
import dialog. You should take care about keeping all calls consistent, e.g. in most
cases it does not make sense to import Procedures without their required Components.
In case you only want to import one dedicated Procedure or Test-case you can use the
button ’Detailimport’ on the importdialog. Here you can choose any node you want to
import separately.
19.4
Strategies for distributed development
There is no single best way of test development or organization, but one approach that
works well is the following:
• Start with a central test-suite that has the functionality needed to start and stop the
19.4.
Strategies for distributed development
198
SUT and a basic set of Tests and Procedures. This will become your master suite
which will contain all Components.
• Make sure that your developers have understood the importance of assigning
names with setName() and that unique names are assigned consistently
where needed.
Where setName() is not an option, try to implement
(807)
ComponentNameResolvers to achieve this (see section 43.1.6 ). You should
be able to record and replay sequences without much ado and without ”polluting”
the Component hierarchy after trivial changes in the user interface.
• Move as much functionality as possible into Procedures, especially commonly-used
stuff and the setup and cleanup routines for the SUT.
• To create new tests, start with an empty test-suite. Include the master test-suite by
(396)
(395)
editing the Include files attribute of the Test-suite node of the new suite. Create
the Setup and Cleanup nodes to start and stop the SUT by calling the respective
Procedures in the master suite.
• Create your tests as required. When recording sequences, the Components of the
master-suite will be used if possible. New Components are added to the new suite,
so the master suite will not be modified at this stage.
• Where possible, call Procedures in the master suite for common operations.
• When your new set of tests is complete and you are satisfied that they work well,
import any required nodes of your new test-suite into the master suite. You have
to ensure that all new Component nodes that you recorded are imported into the
master suite’s Component hierarchy in any case. The master suite’s existing Components will not be affected by this, so other suites that depend on the master suite
will not need to be modified.
• After importing Components you can import all or only the required Procedures into
the master suite.
• You now have various options of how to arrange the actual sequences of events
and checks that form your tests. In any case it is a good idea to move everything to Procedures and Packages structured after your test-plan. Then the top-level
Test-set or Test-case nodes of the master suite and your new suite will only contain the required hierarchy of Test-set, Test-case, Test-step and Sequence nodes filled
with Procedure calls to the actual test-cases. Such an arrangement has several
advantages:
– All your tests are structured cleanly.
– You can easily create different sets of tests with varying complexity and runtime.
19.5.
Static validation of test-suites
199
– You have the option to keep the test-cases in separate test-suites and have
the master suite call them. These ”test-case-libraries” must include the
master-suite, so they need not contain any Components themselves. You can
organize your tests so that the master-suite will run the whole set of tests,
while each separate suite can also be run standalone.
– The tests can be maintained by several developers as long as modifications
to the master suite are coordinated.
• If you decide to keep your new tests in the newly created test-suite instead of
moving them to the master suite, modify the master suite to tell QF-Test that there
is a new test-suite that depends on it. To do so, either ensure that both test-suites
(396)
belong to the same project or add the new test-suite to the Dependencies attribute
of the master suite’s Test-suite node.
• If you need to modify or extend the new test-suite later, proceed as before. You
can record new sequences as needed. When you are done, merge any newly
created Components back into the master suite.
• If your SUT changes in a way that requires updates or adaptions to the mastersuite’s Component hierarchy, you must coordinate your test developers. Before you
start updating the Components, make sure that all suites that directly or indirectly
include the master suite belong to the same project as the master suite or are listed
in the Dependencies attribute of the master suite’s Test-suite node. If modifying the
Components of the master suite involves any QF-Test component ID changes, QFTest will update the depending test-suites accordingly, so they should not be edited
simultaneously by others.
• The file format for QF-Test test-suites is XML and thus plain text. As a result, testsuites can be managed very well by version control systems like CVS. Changes
to some QF-Test component ID attributes of the depending suites can typically be
merged with other changes without conflicts, alleviating the need for coordination.
Of course, the above scheme can be extended to have several master suites for testing
different parts or aspects of an application. It may be a good idea to ensure that the
component hierarchies in these suites don’t overlap too much though. This will save
you the effort of maintaining all these hierarchies in case the user interface of the SUT
changes significantly.
19.5
Static validation of test-suites
3.1+
Working in a project over time will cause modifications, refactoring or deletion of steps in
your test-suite structure, e.g. you may consider renaming Procedures or simply removing
them once they are not required anymore.
19.5.
Static validation of test-suites
19.5.1
200
Avoiding invalid references
In such cases it is quite important that you adapt all references of the according Procedure in order to guarantee that the remaining tests keep running. For this purpose
QF-Test automatically updates all references during the process of renaming or moving
elements on demand.
If you want to ensure that your created test structure doesn’t contain any call of nonexisting Procedures anymore, you can also use the ”Analyze references” command of
QF-Test in order to perform a static validation of your test-suite. This command will
open a dialog showing all references and whether they are still okay or something is
missing.
You can trigger the analysis via a right mouse-click and selecting
Additional node operations→Analyze references... or selecting the according entry
from the main menu under Operations . This method is also available in batch mode.
19.5.
Static validation of test-suites
201
Figure 19.1: Result of analyzing references
3.5+
QF-Test also provides features to search through your test-suites for duplicate nodes,
empty Packages or Procedures or to analyze for nodes having invalid characters in their
names.
This kind of static validation is available for Procedures, Dependencies, Test-cases, Test-sets
and Components and their references.
19.5.
Static validation of test-suites
19.5.2
202
Unused procedures
4.0.3+
During test development it could happen that procedures, which were used in the first
version of your tests will not be used in newer versions due to re-factoring of tests.
If those procedures won’t get deleted immediately they will stay in the test-suite and
the test-suite will grow and grow. Sometimes you could get the feeling that you have
too many procedures or that you have lost the overview of your procedures. In order to check for such unused procedures or dependencies in your test-suite you can
open the context menu via a right mouse click at Test-suite or Procedures and select
Additional node operations→Find unused callables... . This operation creates a report
showing any procedures or dependencies which had been created but haven’t been
used yet. Now you could decide what you want to do with those.
Sometimes you might simply remove all of those unused nodes immediately via
Additional node operations→Remove unused callables .
Chapter 20
Automated Creation of Basic
Procedures
3.0+
20.1
Introduction
At the beginning of a typical QF-Test project the tester records the first tests and starts
them. After a couple of such recordings and first success stories he notices that only
recording or performing copy/paste bares some hidden pitfalls in maintaining the tests.
Just think about a possible workflow change in a main panel, then the tester might have
to adapt all test-cases. That’s why we recommend to make use of the modularization
concept using procedures and variables as early as possible in a project. For more
(91)
information about the modularization concept, please see section 9.5 .
In projects containing a lot of dialogs and graphical components it might be sufficient to
split those procedures into component-specific ones, e.g ”press button ok” and separate
workflow procedures, e.g. ”create a vehicle” combining the component-specific steps
together. This approach enables the tester to create new test-cases very fast. However,
he has to put a lot of efforts into creating those basic procedures first.
QF-Test comes with a Procedure Builder, which will create those basic procedures for
you. Using the Procedure Builder will drastically decrease the efforts of recording and
creating procedures for graphical components. So the tester can solely concentrate on
his main focus, i.e. designing the workflow of the test itself and the according test-data.
20.2
How to use the Procedure Builder
For creating the basic procedure automatically, you have to perform following steps:
• Start the SUT from QF-Test.
20.2.
How to use the Procedure Builder
204
• Navigate to the window or frame you want to create procedures for in the SUT.
• Press the ’Record Procedures’
button, select the according menu-item in QF(348)
Test or use the Hotkey for procedure recording .
• Perform a right mouse-click on the respective component in the SUT.
• Select the according recording-mode.
• Stop procedure recording by releasing the ’Record Procedures’ button, by deselecting the according menu-item in QF-Test or using the Hotkey for procedure
(348)
recording .
(472)
Now you should be able to find a newly created package in the Procedures
node of
the test-suite where you stopped the recording, containing the created procedures for
the components. By default this is called procbuilder. If the package procbuilder
already exists, a package procbuilder1 will be created and so on. When you open
the package you will see a set of packages for functions like check, get, select, wait
and so on. These contain the procedures created for the components you selected when
running the procedure builder, apart from the check package, where another level with
packages for the various check modes is inserted. This is the default structure, which
you can adapt to your needs by modifying the definition file for the procedure builder as
described in the following section.
20.3.
Configuration of the Procedure Builder
205
Figure 20.1: Recorded procedures
20.3
Configuration of the Procedure Builder
The act of building procedures is controlled by a template suite, which is located at
qftest-4.2.2/include/procbuilderdef.qft. This file should be copied to any
project-specific location, if you want to adapt it to your project. You can define its location
(349)
in the options at Configuration file for recorded procedures .
The template suite contains procedures for the most common GUI elements and actions. If you require other test-steps, you can add the according procedure to this testsuite.
The file itself is a test-suite with a dedicated structure. You can find a detailed expla(205)
nation of this structure in the subsection section 20.3.1 . The definition file allows the
tester to define procedures for components of dedicated classes or to define procedures
for working with all components of one certain window.
You can find some demo configurations at qftest-4.2.2/demo/procbuilder.
20.3.
Configuration of the Procedure Builder
20.3.1
206
The Procedure Builder definition file
The automated creation of basic procedures delivers different procedures depending
on the components. A text-field requires a setter procedure for setting its text, a button
requires a press procedure for pressing it or a window could require a setter which calls
the setter procedures of all text-fields or combo-boxes on that window to call just one
procedure for using the window etc..
Figure 20.2: The Procedure Builder definition file
(472)
The topmost package in the Procedures node is the name of the target package for the
newly created packages. By default this is called procbuilder. This package will be
(472)
inserted below the Procedures , when you have finished recording procedures. If the
package procbuilder already exists, a package procbuilder1 will be created and
so on.
The next level is the class level. Here you can define a package per class. The package
name represents the full class name, but with ’_’ as separators instead of ’.’. That’s
because ’.’ is not allowed in package names. The Procedure Builder creates the procedures also for descendants of specified classes. In case the names of your classes
contain a ’_’, you have to mark this via ’\_’.
20.3.
Configuration of the Procedure Builder
207
The following levels can be chosen freely because those levels are intended to structure
the procedures.
At the last level you have to define the steps of the procedure itself.
Of course there are a lot of variable data in that definition, e.g. like <COMPID>.
Using those you can specify variables for the procedure names, like the QF-Test ID of
the current component or the component-name. You can also record the current value
of the text-field or the current selected status of a checkbox. It’s even possible to keep
the package structure variable. For an overview of all possible variables, please see
(899)
chapter 45 .
Chapter 21
Interaction with Test Management
Tools
3.0+
QF-Test contains some pragmatic test management approaches, like creating a testcase overview or documenting test-cases within QF-Test. In bigger projects it might
be necessary to make use of an own dedicated test management system to track the
development status of test-cases or to link test-cases and their results to defects, usecases or features. Besides support for planning of test-cases and tracking their results a
test management system could also contain a test execution engine, which supervises
the occupation of test-systems during different test-runs.
As QF-Test doesn’t come with all of those features though continuously improving in
that area, it is very easy to integrate QF-Test with such a test management or test
execution system using the QF-Test Batch mode or the QF-Test Daemon mode. For
(176)
more information about the Batch mode or the Daemon mode, please see chapter 18 .
The following chapters describe some exemplary solutions which we provide for established test management systems. If you cannot find your test management system in
that list, please contact us or our mailing-list to get hints about a possible integration
approach.
21.1
HP ALM - Quality Center
21.1.1
Introduction
The current integration of QF-Test and HP ALM - Quality Center utilizes the built-in
VAPI-XP-TEST type of Quality Center.
21.1.
HP ALM - Quality Center
209
Figure 21.1: Integration with ALM - QualityCenter
The VAPI-XP-TEST type is intended to be an automated test-case for any test-tool.
QF-Test comes with a template file for the VAPI-XP-TEST script, which is
qcVapiXPTemplate.txt, see qftest-4.2.2/ext/qualitycenter. This script
can be used as template for all QF-Test tests in Quality Center. Please see section
(209)
21.1.2 for a detailed step-by-step description.
The QF-Test VAPI-XP-TEST template script employs an external worker VBScript
script, called qcTemplate.vbs. This script is also part of the QF-Test distribution
(see qftest-4.2.2/ext/qualitycenter) and has to be adapted to your specific
needs. So we encourage you to copy that file to a project specific location and adapt it
according to your needs.
The worker script launches QF-Test in batch mode on each test system locally, i.e. it
has to be accessible for each test system. As the test suite files and the configuration
files have to be available on the test system too, we recommend to put all those files on
a shared network drive or into the version management system.
After the execution of the test the run-log of QF-Test will be appended to the test instance
as well as the status of the test will be set to the result.
You can also change the worker script to make use of a daemon call (for details about
(882)
the daemon mode, please see chapter 44 ). In this case QF-Test will establish the
network connection to the test system and launch the test by itself. In case of the normal
batch call Quality Center establishes the connection to the test-system and triggers the
local QF-Test installation to perform the test. If you make use of the daemon call, the
worker script has to be located on the Quality Center system, but the test suite still
needs to be accessible on each test system.
If you do not make use of VBScript in your project, feel free to port the QF-Test demo
21.1.
HP ALM - Quality Center
210
scripts to JScript or any other supported language.
The following figure shows the VAPI-XP-TEST test-case in Quality Center:
Figure 21.2: QF-Test VAPI-XP-TEST test-case in HP ALM - QualityCenter
21.1.2
Step-by-step integration guide
General steps to be performed on the the test system:
1. Copy
the
template
worker
script
from
qftest-4.2.2/ext/qualitycenter/qcTemplate.vbs to your project
location and rename it to a proper name. We recommend to use the same path
on all test-systems. Perhaps you should use a shared network drive.
2. Within the worker script you can define certain default option, e.g. whether the
batch or daemon mode should be used as default and what should be the name
21.1.
HP ALM - Quality Center
211
for the default run-log file. This also can be done at a later stage, which might be
recommendable when initially starting with the integration process to keep things
simple.
Steps in Quality Center to create a test-case:
1. Start Quality Center and log in to your project.
2. You might want to create a new test-set e.g. called ”DemoTestSet” in the ”Test
plan” area.
Figure 21.3: In Test plan create new Test-set
3. In this test-set create a new test with type VAPI-XP-TEST.
21.1.
HP ALM - Quality Center
212
Figure 21.4: Create new test of type VAPI-XP-TEST
Note
4. On the HP VAPI-XP Wizard window just press finish without any modifications.
(That means you have VBScript as script language and COM/DCOM Server Test
as test type).
21.1.
HP ALM - Quality Center
Figure 21.5: HP VAPI-XP Wizard
5. You will then get a new test as shown below.
213
21.1.
HP ALM - Quality Center
214
Figure 21.6: Test details
6. Change to the ’Test script’ tab of the test and copy the content of the template
file qftest-4.2.2/ext/qualitycenter/qcVapiXPTemplate.txt into the
Script Viewer’s text area.
21.1.
HP ALM - Quality Center
215
Figure 21.7: Copy template content to script text area
7. Within the script please do following adaptations:
• Change the pathToModule variable to the location you have copied the
worker script qcTemplate.vbs to.
• Change the testSuiteFile variable to your desired test-suite file.
• If you want to execute one specific test, you can also change the testCase
variable to the desired test-case name.
Please read the comments in the script carefully, because you can also use test-case
specific settings optionally.
Steps to be performed to run the sample test-case
1. Change to the ”Test lab” section in Quality Center.
21.1.
HP ALM - Quality Center
216
2. You might want to create a new sample test-set.
Figure 21.8: New test-set in Test lab section
3. Add the test-case to the new test-sets’ execution grid by choosing it from the test
plan structure.
21.1.
HP ALM - Quality Center
217
Figure 21.9: Add test to execution grid
4. Now you can launch the test-case. Ensure the ”Run all tests locally” checkbox
is activated unless you really have a remote system with a QF-Test environment
already set up.
21.1.
HP ALM - Quality Center
218
Figure 21.10: Run the test
5. Now Quality Center should start the test run - possibly on your machine, then
you should see the SUT coming up after some time, actions being performed and
closed at the end. When the run has finished, the result is noted down with the
test: Passed or Failed.
21.1.
HP ALM - Quality Center
219
Figure 21.11: Test result
6. After the test has terminated, in addition to the result the run-log of the test will be
uploaded as attachment to the test instance. To view the run-log, please doubleclick to the test in the execution grid, then change to ”Runs” and again double-click
at the paper-clip attachments symbol for the respective test-run.
21.1.
HP ALM - Quality Center
220
Figure 21.12: Uploaded run-log
21.1.3
Troubleshooting
First of all we need to state that we are not QualityCenter experts. Therefore there might
be better and advance options for troubleshooting. Hence we want to at least provide
some hints we used so far.
Unfortunately the process output during the test execution in QualityCenter is only visible
for a fraction of time, not allowing a direct analysis. Therefore we need to find a work
around.
The text editor of the VAPI-XP-TEST node in the ”Test plan” area allows to directly
execute the script. Then in the output area below the output gets visible permanently
showing possibly something helpful.
21.1.
HP ALM - Quality Center
221
Figure 21.13: Script debug run
But the direct execution from the script node needs to be handled with care. Of course
it is not considered as a real test run, so no run-log can be uploaded which results in a
respective Run-time error ”Object required” in the output. Don’t get confused by that!
For more debugging, additional statements like TDOutput.Print ”Some text” can
be added to both the test script and the worker script. By this you can see how far the
script runs until a possible error occurs.
The text script editor has a ”Syntax check” button which is helpful for validation after
every change.
21.2.
Imbus TestBench
21.2
Imbus TestBench
21.2.1
Introduction
222
The current integration of QF-Test and the TestBench consists of two parts:
• Creating a QF-Test template file using the TestBench interactions.
• Importing QF-Test results into TestBench.
You can find all required libraries and test-suites in the folder
qftest-4.2.2/ext/testbench/Version_1.1_TestBench_2.3. Please take
care to copy all test-suites to a project-related folder first and modify them there.
The following section provides a short overview about the integration concept.
21.2.2
Creating QF-Test template from interactions
After planning your tests and designing the interactions in the TestBench, you can create
a template QF-Test file using the QF-Test export plug-in for interactions. Imbus will
provide all required information, how to install this plugin.
After exporting the interactions you will find all interactions as procedures and their
structure as packages in the QF-Test file. Now you can start recording the respective
steps in QF-Test and fill the empty procedures.
The completed file has to be saved in your project-specific test-suite folder, because
this file should be used as input file for the test execution later. We recommend to use
a project-specific location, perhaps a shared network drive or the version management
system.
21.2.3
Importing test-execution results
For
running
tests
you
need
specific
test-cases
and
procedures,
which
can
be
found
in
the
provided
test-suites
in
the
folder
qftest-4.2.2/ext/testbench/Version_1.1_TestBench_2.3/suite.
Please take care to copy all test-suites to a project-related folder first and modify them
there.
You can also find a demo implementation for the JCarConfigurator in the folder
qftest-4.2.2/ext/testbench/Version_1.1_TestBench_2.3/suite/demo.
You have to take care that you need to include the test-suite
TestBench_Automation.qft to your test-suite. If you have created procedures via
21.3.
QMetry
223
(221)
the iTEP exporting as described in chapter section 21.2.2
that test-suite as well.
you will need to include
The next step is to adapt the configuration for the output files. This can be achieved by
modifying the files testaut.properties and user.properties.
Now you are ready to call the test-case Standalone test executor from
TestBench_Automation.qft.
When the test-run has been completed, you can import all those results using the iTEP
or iTORX import plug-in into the TestBench. The single QF-Test run-logs will then be
attached to the test-instances.
21.3
QMetry
21.3.1
Introduction
The current integration between QF-Test and QMetry relies on planning the tests and
its steps within QMetry and forwarding the actual test-execution to QF-Test. Once the
the test-run terminates the QF-Test run-log and its HTML report will be automatically
uploaded to QMetry to the respective result area as well as the state of the test-case in
QMetry will be set to the according result.
You need to prepare your test-system in order to run QF-Test tests. Please perform the
following steps:
• In the ’Admin’ area of the QMetry Testmanagement view install a test-execution
agent at the ’Agent’ view.
• Download the required agent and configuration files to install the QMetry execution
agent on your test-system.
• Install the respective QF-Test QMetry Launcher at your test-agent.
• Install and set-up a platform at the ’Platform’ view, which is also located in the
’Admin’ area of QMetry.
• Configure QMetryAgent.properties correctly to use the required environment
variables of QMetry’s QF-Test wrapper.
• Configure QMetryConfig.properties correctly to show to the right QF-Test
executable.
• Configure
additional
parameters
for
the
QF-Test
QMetryConfig.properties, see next section for details.
call
in
21.3.
QMetry
224
• Launch the QMetry agent. Please do not launch the agent as Windows-Service
to avoid running GUI-Tests within the service-session. If you launch the agent as
service you should run the QF-Test tests via the QF-Test daemon, which shouldn’t
run in a service session then.
After setting up the agent and launcher, you need to plan the test-execution. QMetry
supports several ways of integrating QF-Test test-cases. You can find all supported
integrations in QMetry’s integration guide document. Please perform following steps for
a simple integration:
• In the Testmanagement view change to ’Test Cases’ and plan the test-cases there.
• At the individual test-case you have to set the value ’Test Script Name’ to the
path of the required QF-Test test-suite holding the actual implementation of the
test-case.
• The name of the test-case must be exactly the same as the specified value for the
QF-Test ID attribute in QF-Test.
• Add the test-case to an executable test-suite in the ’Test Suites’ view.
Now you are ready to run the test-cases:
• Open the ’Test Suites’ view and select the required test-suite for execution.
• Select the ’Execute TestSuite’ tab.
• Run or schedule a test-run via assigning an agent to the ’Automation’ column.
• The next time when the local QMetry agent is polling the QMetry server it will get
the necessary information to run the test-case.
• Once the test-run terminates you will find the run-log of QF-Test and its HTML
report attached to the ’Execution History’ of the executed test-suite. The state of
the test-case will also be updated accordingly.
The following figure shows the ’Execution History’ tab in ’Test Suites’ holding the run-log:
21.3.
QMetry
225
Figure 21.14: QF-Test run-log in QMetry
You will find a more detailed description of how to setup QMetry in the manual of QMetry
and in QMetry’s integration guide document.
21.3.2
Sample Configuration
It’s recommended to set
QMetryConfig.properties:
following
values
in
the
configuration
file
• Set the value of generic.adapter.success.code to 0,1.
• Set qftest.additional.arguments to -test ${QMTestCaseName} in
case of local test execution.
• In case of using QF-Test’s daemon set qftest.additional.arguments
to
-test ${QMTestCaseName} -calldaemon -daemonhost
<testsystem> -daemonport <daemonport> .
As already mentioned in the previous section, you need to use the same name for the
test-case within QMetry and for the value of the QF-Test ID attribute within QF-Test.
Further ways for integrating QMetry and QF-Test can be found in the integration guide
document provided by QMetry.
21.4.
Klaros
226
21.4
Klaros
21.4.1
Introduction
Klaros is a test management tool developed and supported by verit Informationssysteme
GmbH, Kaiserslautern, Germany.
Klaros is available in two kinds of editions, a free community edition and an enterprise
edition with an extended set of functionality, individual configuration options and full
customer support.
The current integration of QF-Test with Klaros comprises:
• Import of QF-Test results into Klaros.
21.4.2
Importing QF-Test results into Klaros
(167)
After creating the XML report file as described in chapter 17 , you can upload the
results to Klaros. An example for a QF-Test import URL may look like this, where the
result file is contained in the HTTP request body.
http://localhost:18080/klaros-web/seam/resource/rest/importer?
config=P00001&env=ENV00001&sut=SUT00001&type=qftest&
time=05.02.2013_12:00&username=me&password=secret
Example 21.1: Importing test results into Klaros
The curl command line tool can be used on Linux or Windows/Cygwin to trigger an
import in a single command line.
curl -v -H "Content-Type: text/xml" -T "my_qftest_report.xml" \
"http://localhost:18080/klaros-web/seam/resource/rest/importer\
?config=P00001&env=ENV00001&sut=SUT00001&type=qftest\
&time=05.02.2013_12:00&user=me&password=secret"
Example 21.2: Using curl command to import test results into Klaros
Further information can be found within the Klaros online manual at https://www.klarostestmanagement.com/files/doc/html/User-Manual.Import-Export.html.
21.5.
SQS-TEST/Professional
21.5
SQS-TEST/Professional
21.5.1
Introduction
227
SQS is one of the leading international specialists for software quality with about 3000
employees world-wide.
With SQS-TEST/Professional Suite SQS offers an extensive test management framework, covering everything from requirement management through test-case development and execution to integration of test-results into a comprehensive report system.
QF-Test seamlessly interoperates with this system without special requirements for the
integration.
21.5.2
Integration
QF-Test execution is triggered by SQS-TEST/Professional and results are led back,
processed and administrated again in SQS-TEST/Professional.
The integration with QF-Test is performed via a public API interface of
SQS-TEST/Professional. Only the SQS-pilot has to be installed.
That given, scripts can be transferred to the target system, either manually or directly
addressed within the SQS tool.
Please ask SQS for more details on their SQS-TEST/Professional platform. Also for
technical questions on the integration with QF-Test contact the SQS support via support@sqs.com or by telephone +49 2203 9154 -11 (SQS test suite support).
21.6
TestLink
21.6.1
Introduction
The current integration of QF-Test with the open-source tool TestLink consists of two
parts:
• Generating template test-suites for QF-Test from the planned test-cases of
TestLink.
• Importing QF-Test results into TestLink.
3.5.1+
If you use TestLink 1.9.4 or newer you can use the TestLink API for interacting with
21.6.
TestLink
228
TestLink. The TestLink API requires a valid development key. Therefore open TestLink
and go to ’My Settings’. In the settings you can generate a development key by pressing
’Generate key’ under the ’API interface’ section.
For TestLink 1.9.3 or older versions the integration mechanism accesses the database
of TestLink directly. This approach requires a JDBC database driver to use the provided
scripts. You can download those drivers from the web page of their providers.
Exporting the planned test-cases including its test-steps from TestLink to QF-Test supports the test-creator to implement the test-cases exactly as planned.
Importing the test results into TestLink provides a better overview over all executed
manual and automated tests-cases in one tool.
Note
Test results can also be uploaded to TestLink without exporting them before. Therefore
you have to take care, that the ID of the test-case from TestLink is part of the test-case’s
name in QF-Test. The name has to be called like this: <TestLink-ID>: Name of
the test-case.
21.6.2
Generating template test-suites for QF-Test from test-cases
QF-Test offers the capability to generate template test-suites following the same structure as the planned tests in TestLink to guarantee a synchronized structure of automated
tests and test planning.
In the QF-Test file you can find one Test-case node per test-case and one Test-set node
per suite from TestLink. If you have specified the fields ”Steps” and ”Expected Results”
of a test-case, the generating-script will also create an empty Test-step for each test-step
in the according test-case. The expected result will be shown in the Comment attribute
of the Test-step node.
Now the template test-suite has to be filled by the test automation engineer with the
according steps by adding QF-Test steps to the generated Test-step nodes.
3.5.1+
In case you use TestLink 1.9.4 or newer you need to perform following steps:
1. Take care that test automation is enabled in TestLink. Therefore set the
respective enable_test_automation key to ENABLED in the configuration file
config.inc.php.
2. Copy the folder qftest-4.2.2/ext/testlink/api to a project-specific location.
3. Open the launcher script you want to use with a text editor. The launcher scripts
are exportTests.bat for Windows and exportTests.sh for Unix.
4. Adapt the paths of the variables JAVA, QFTDIR and TESTLINKINTEGRATOR.
21.6.
TestLink
229
5. Open the file TestLinkUserSpecifics.py with a text editor.
6. Adjust the variables serverurl and devkey.
7. If you want to export custom fields from TestLink, also adjust the variable
custom_fields.
8. Run the adapted export script, like shown below.
exportTests.bat --testproject projectname
--targetsuite /path/to/testsuite.qft
Example 21.3: Sample call of exporting test-cases from 1.9.4
If you use TestLink 1.9.3 or older, please perform those steps:
1. Copy the folder qftest-4.2.2/ext/testlink/export to a project-specific
location.
2. Open the launcher script you want to use with a text editor. The launcher
scripts
are
exportTestLinkToQFT.bat
for
Windows
and
exportTestLinkToQFT.sh for Unix.
3. Adapt the paths of the variables JAVA, QFTDIR and TESTLINKINTEGRATOR.
4. Open the file TestLinkDBIntegrator.py with a text editor.
5. Adjust the variables dbdriver, conncetionstr, dbuser and dbpass according to your database connection.
6. If you want to export custom fields from TestLink, also adjust the variable
custom_fields.
7. Run the adapted export script, like shown below.
exportTestLinkToQFT.bat --testproject projectname
--targetsuite /path/to/testsuite.qft
Example 21.4: Sample call of exporting test-cases till 1.9.3
21.6.
TestLink
21.6.3
230
Execution of test-cases
Executing the QF-Test tests can be performed as usual. But you should create a XMLreport at the end of the test-run, because the import mechanism is using this report.
Therefore you have to use the ’-report.xml’ parameter during test execution. If you
create the reports via the GUI, you have to check the checkbox ’Create XML report’.
Note
In case you did not export test-cases from TestLink the ID of the test-case from TestLink
has to be part of the test-case’s name in QF-Test. The name has to be called like this:
<TestLink-ID>: Name of the test-case.
qftest -batch -report.xml reportFolder testsuite.qft
Example 21.5: Sample execution to create a XML report
21.6.4
Importing QF-Test results into TestLink
After creating the XML report file, you can upload the results to TestLink.
Per default the import mechanism creates a new build for every test-run. The build
number of TestLink will be created by the run-ID of the QF-Test report. You can change
the run-ID, by setting the parameter ’-runid’ when launching the tests with QF-Test. But
you can also set the ’-build’ parameter during import to specify a custom build name.
3.5.1+
In case you use TestLink 1.9.4 or newer you need to perform following steps:
1. Take care that test automation is enabled in TestLink. Therefore set the
respective enable_test_automation key to ENABLED in the configuration file
config.inc.php.
2. Copy the folder qftest-4.2.2/ext/testlink/api to a project-specific location. (If you have copied them already for exporting you can use the same files.)
3. Open the launcher script you want to use with a text editor. The launcher scripts
are importResults.bat for Windows and importResults.sh for Unix.
4. Adapt the paths of the variables JAVA, QFTDIR and TESTLINKINTEGRATOR.
5. Open the file TestLinkUserSpecifics.py with a text editor.
6. Adjust the variables serverurl and devkey. (If you have adapted them already
for exporting you can use the same values.)
7. Run the adapted import script, like shown below.
21.6.
TestLink
231
importResults.bat --testproject projectname
--resultfile qftestReport.xml --testplan testplanname
--platform system1
Example 21.6: Importing test results into TestLink from 1.9.4
If you want to overwrite the build name you can use the ’-build’ parameter.>
importResults.bat --testproject projectname
--resultfile qftestReport.xml --testplan testplanname
--platform system1 --build myBuild
Example 21.7: Importing test results into TestLink from 1.9.4 with custom build
If you use TestLink 1.9.3 or an older version, please perform following steps:
1. Copy the folder qftest-4.2.2/ext/testlink/import to a project-specific
location.
2. Open the launcher script you want to use with a text editor. The launcher scripts
are importToTestLink.bat for Windows and importToTestLink.sh for
Unix.
3. Adapt the paths of the variables JAVA, QFTDIR and TESTLINKINTEGRATOR.
4. Open the file ReportParser.py with a text editor.
5. Adjust the variables dbdriver, conncetionstr, dbuser and dbpass according to your database connection.
6. If you want to export custom fields from TestLink, also adjust the variable
custom_fields.
7. Run the adapted import script, like shown below.
importToTestLink.bat --testproject projectname
--resultfile qftestReport.xml --testplan testplanname
--tester tester
Example 21.8: Importing test results into TestLink till 1.9.3
Chapter 22
Integration with Development Tools
Automating GUI tests is just one part of the development cycle. Requirements like
automating the compilation or build process, running tests, creating documentation or
providing a deliverable package led on to a variety of different development tools like
IDEs (e.g. Eclipse) or build tools (e.g. make, ant, maven) or so called continuous
integration systems (like Jenkins/Hudson, Cruise Control, Continuum).
(176)
In general, by use of QF-Test’s command line interface as documented in chapter 18
(671)
and chapter 36 a straight forward integration with those tools should be possible.
Note
(309)
GUI tests require an active user session. Chapter Hints on setting up test-systems
contains useful tips and tricks to set-up your test-systems. In FAQ 14 you can find
technical details.
The following sections contain examples for integrations with a some of the tools mentioned above.
22.1
Eclipse
Eclipse (http://eclipse.org) is an Open Source software developer tool for java applications.
QF-Test offers an Eclipse plugin enabling you to start an application directly from Eclipse
and run tests on it - anything from whole test sets, single test cases or even just a mouse
click.
22.1.1
Installation
For the installation please copy the Eclipse plugin file de.qfs.qftest_4.2.2.jar
from the subdirectory qftest-4.2.2/misc/ of the QF-Test installation directory to
22.1.
Eclipse
233
the subdirectory ’dropins’ of the Eclipse installation directory. Then (re-)start Eclipse
and the plugin will be available.
22.1.2
Configuration of the test nodes
Open the Eclipse menu Run→Run Configurations . Enter the QF-Test nodes to be
started in the tab ’Main’ and if necessary enter parameters in the tabs ’Settings’ and
’Initial Settings’. (The tabs ’Environment’ and ’Common’ are standard Eclipse tabs that
are not needed for the configuration of the QF-Test Plugin.)
Then save the configuration by pressing ’Apply’. To start a test run press ’Run’.
Tab ’Main’
Figure 22.1: Eclipse plugin configuration - tab ’Main’
Enter the fully qualified path to the QF-Test executable ’qftest.exe’ in
the
field
QF-Test
executable,
e.g.
C:\Program Files
(x86)\qfs\qftest\qftest-4.1.0\bin\qftest.exe.
’Run configuration to be used as SUT’ is an optional entry. You may enter an existing
Eclipse ’Run Configuration’ for starting the application to be tested. At the start of the
22.1.
Eclipse
234
application the QF-Test plugin sets up the connection to QF-Test so you can replay or
record tests on the application. Use this option when you specify QF-Test nodes in
the ’Startup nodes’ section which do not start the application themselves. Please be
aware that the run configuration to be used as SUT will be started and then right away
the listed ’startup nodes’ will be executed. So, to make sure the SUT is started when
executing the ’startup nodes’ the first action of the first ’startup nodes’ should be to wait
for the SUT. This can be done either by inserting a Wait for client to connect node at the
beginning of the first ’startup node’ or by adding a first ’startup node’ just calling a Wait
for client to connect node in QF-Test.
Enter all QF-Test nodes to be executed in the table ’Startup nodes’. You need to specify
the QF-Test ID of the node as well as its test-suite. Please be aware that the QF-Test ID
is a separate attribute of the node and not its name. The QF-Test ID attribute is empty
by default and has to be set before use.
Tab ’Settings’
Figure 22.2: Eclipse plugin configuration - Tab ’Settings’
Variables specified in this tab will be read each time before executing the run configuration.
22.1.
Eclipse
235
’Path to run-log folder’ specifies the directory where to save the run-logs of the test runs
of the run configuration. It is optional. When empty the run-logs are saved as configured
in QF-Test itself.
If required enter variables to be passed to QF-Test on command line level in the table
’Variables’. This will overwrite default values of the variable.
Tab ’Initial Settings’
Figure 22.3: Eclipse plugin configuration - Tab ’Initial Settings’
The values set in this tab are optional and only read in once before the start of QF-Test
When changing them you need to restart QF-Test before they take effect.
Path to license file: Path of the license file to be used.
Path to qftest system config file: Path of the qftest.cfg file to be used.
Path to qftest user config file: Path of the user configuration file to be used.
22.2.
22.2
Ant
236
Ant
People who are using Apache Ant (http://ant.apache.org) as build system may easily
integrate QF-Test in their build file:
<project name="QF-Test" default="runtest">
<property name="qftest"
location="c:\Program Files\qfs\qftest\qftest-4.2.2\bin\qftest.exe" />
<property name="logdir" value="c:\mylogs" />
<target name="runtest" description="Run a test in batchmode">
<echo message="Running ${suite} ..." />
<exec executable="${qftest}" failonerror="false"
resultproperty="returncode">
<arg value="-batch" />
<arg value="-compact" />
<arg value="-runlog" />
<arg value="${logdir}\+b" />
<arg value="${suite}" />
</exec>
<condition property="result"
value="Test terminated successfully.">
<equals arg1="${returncode}" arg2="0" />
</condition>
<condition property="result"
value="Test terminated with warnings.">
<equals arg1="${returncode}" arg2="1" />
</condition>
<condition property="result"
value="Test terminated with errors.">
<equals arg1="${returncode}" arg2="2" />
</condition>
<condition property="result"
value="Test terminated with exceptions.">
<equals arg1="${returncode}" arg2="3" />
</condition>
<echo message="${result}" />
</target>
</project>
Example 22.1: Ant build file build.xml to execute a test-suite
The
above
example
assumes
the
test-suite
to
be
defined
as
property
when
running
ant:
ant -Dsuite=”c:\Program
Files\qfs\qftest\qftest-4.2.2\demo\carconfig\carconfig_en.qft”.
22.3.
22.3
Maven
237
Maven
People who are using Apache Maven (http://maven.apache.org) as build system may
easily integrate QF-Test in their build. This can be achieved by using the antrun plugin
of Maven. A demo pom.xml file, where QF-Tests tests are executed in the test phase
could look like this:
22.3.
Maven
238
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<artifactId>testant</artifactId>
<packaging>jar</packaging>
<name>testant</name>
<groupId>de.qfs</groupId>
<version>1</version>
<properties>
<qf.exe>"C:\Program Files\qfs\qftest\qftest-4.2.2\bin\qftest.exe"</qf.exe>
<qf.reportfolder>qftest</qf.reportfolder>
<qf.log>logFile.qrz</qf.log>
<qf.suite>"c:\path\to\testsuite.qft"</qf.suite>
</properties>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-antrun-plugin</artifactId>
<executions>
<execution>
<phase>test</phase>
<configuration>
<tasks>
<exec executable="${qf.exe}">
<arg value="-batch"/>
<arg value="-report"/>
<arg value="${qf.reportfolder}"/>
<arg value="-runlog"/>
<arg value="${qf.log}"/>
<arg value="${qf.suite}"/>
</exec>
</tasks>
</configuration>
<goals>
<goal>run</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
Example 22.2: Maven build file pom.xml to execute a test-suite
In your project it might become required to run the tests during another build phase,
22.4.
Jenkins/Hudson
239
than the configured test phase in the example. In this case you have to configure the
plugin accordingly, like described in the Maven documentation.
22.4
Jenkins/Hudson
3.3+
Jenkins/Hudson (http://jenkins-ci.org or http://hudson-ci.org) are continuous integration
build tools. They are meant to control and monitor the build process within a software
project. One important step in this build process is automated testing.
There are number of benefits to be gained when integrating QF-Test with
Jenkins/Hudson:
• In case Jenkins/Hudson is already used for the continuous integration process,
integration of automated GUI tests can be easily achieved.
• Easy-to-use administration of scheduled test runs and notification of results via
email or RSS.
• Jenkins/Hudson’s web based GUI provides good overview and control of test results.
• By use of the HTML Publisher Plugin it is possible to embed QF-Test’s HTML
reports directly into the Jenkins/Hudson GUI.
• Results generated during the test run such as run-logs and reports can be archived
automatically. Therefore maintaining an own directory structure is not needed
anymore.
As Jenkins and Hudson share a common history, following sections will use Jenkins as
representative.
22.4.1
Note
Install and start Jenkins
For GUI tests, Jenkins must not be configured to run as a service but within a real
user session. On Windows the .msi installer unfortunately directly installs Jenkins as
service without any further inquiry. Please beware of it therefore and ensure Jenkins is
started as real user process as described below.
To install Jenkins download the war Archive (which can be found here) and start it via
java -jar jenkins.war.
As soon as Jenkins is started its web interface can be accessed via
http://localhost:8080. It should look like the following:
22.4.
Jenkins/Hudson
240
Figure 22.4: Jenkins after start-up.
22.4.2
Requirements for GUI tests
GUI testing requires an unlocked, active desktop. That is the only way to ensure that the
SUT behaves the same as if a normal user interacts with it. Chapter Hints on setting up
(309)
test-systems contains useful tips and tricks to set-up the Jenkins/Hudson process.
Jenkins allows execution of tasks on remote machines. This is of course also relevant for
GUI testing. Due to its nature GUI tests are typically not intended to run on the central
build server. In addition, tests might need to be executed for different environments,
operating systems and SUT versions.
On a remote machine, a Jenkins slave agent needs to be launched in order to connect
to the Jenkins server and wait for jobs to be processed. As described in the Jenkins
documentation, there are several options to launch this agent, but for the GUI tests to
properly work the only possible launch method is to use Java Web Start.
22.4.
Jenkins/Hudson
241
For GUI tests it is vital to have an active, unlocked user session. Therefore it is not
possible to start the agent via a windows service but a real (test) user must be logged in
(e.g. via auto login) using Windows Autostart to launch the Jenkins agent. Furthermore
screen locking needs to be disabled.
Note
Please see also FAQ 14 for more technical background details.
22.4.3
Install QF-Test Plugin
The QF-Test Plugin enables QF-Test to interact with Jenkins. To install the plugin open
the Jenkins dashboard and navigate via ”Manage Jenkins” to ”Manage Plugins”. Select
the QF-Test Plugin from the ”Available” tab. When installing the QF-Test Plugin the
JUNIT and HTML-Publisher Plugin will also be downloaded automatically, in case they
were not already installed. Finally restart Jenkins to complete the installation. Now the
QF-Test Plugin will show up under the Installed tab, as shown in Figure 20.2.
Note
For an in-depth look at the plugin and all of its available features, please have a look at
the Youtube Video about the installation and configuration of the plugin.
Note
Jenkins will automatically use the latest installed version of QF-Test. In case you want to
use a different version, you can provide its path under the QF-Test section in the Jenkins
configuration (Manage Jenkins -> Configure System).
Figure 22.5: Install QF-Test Plugin.
22.4.
Jenkins/Hudson
22.4.4
242
Configure a job for QF-Test
To create a new job open the Jenkins dashboard and click ”New Item”. The standard
project type is ”Freestyle project”. Once the job is created it has to be configured.
The first thing that needs to be done is to change the workspace of this job to the
directory that contains your test-suites. It can be changed in the ”Advanced Project
Options” under ”Advanced...” where you check ”Use custom workspace” and specify
the test-suites’ directory.
In the ”Display Name” field you have the option to provide a name to be shown instead
of your job name, which is typically left empty.
Figure 22.6: Set custom workspace.
Now proceed to the ”Build” section and add the ”Run QF-Test” build step by choosing it
from the ”Add build step” drop down. In the left field you can either enter the name of a
test-suite or a folder containing test-suites (e.g.
your workspace directory
being ”C:\MySuites\” and the path of the test-suite you want to run being
”C:\MySuites\qftest\mysuite.qft”, simply enter qftest/mysuite.qft in the first
field). In case you enter a directory, every test-suite in this directory will be run. In the
(676)
right field you can enter additional Command line arguments .
22.4.
Jenkins/Hudson
243
Figure 22.7: Add build step to run QF-Test
The advanced options of the ”Run QF-Test” build step are optional.
The first option allows you to provide a dedicated QF-Test version for this job.
The next option lets you change the directory for the temporary reports. The QF-Test
Plugin will create a folder called ”qftestJenkinsReports” in the selected workspace and
save the reports and logs for the post-build actions.
Finally, there is an option to run your test-suites on a daemon. You need to enter the
deamonhost and port to do so. To learn more about the daemon mode please see
(182)
section 18.2 of the manual.
Figure 22.8: Configure build step advanced options.
Next the post build actions need to be configured to process test results and archive
QF-Test run-logs and HTML reports.
22.4.
Jenkins/Hudson
244
To
archive
the
run-logs
add
the
post-build
action
”Archive
the
artifacts”
and
for
”Files
to
archive”
enter
this
path:
qftestJenkinsReports/$JOB_NAME/$BUILD_NUMBER/logs/*.q*
To publish the HTML Reports add the post-build action ”Publish
HTML
reports”.
In
the
HTML
directory
enter
this
path:
qftestJenkinsReports/$JOB_NAME/$BUILD_NUMBER/html/ . Also change the
index page from index.html to report.html. You can keep past HTML reports if
you check the respective box.
Finally
add
the
post-build
action
”Publish
JUnit
test
result
report”
and
for
”Test
report
XMLs”
enter
this
path:
qftestJenkinsReports/$JOB_NAME/$BUILD_NUMBER/junit/report_junit.xml
Note
There will be warnings that the path you entered doesn’t match anything, because of
the environment variables in the path. These warnings can be ignored.
Figure 22.9: Configure post build steps.
22.4.
Jenkins/Hudson
245
Note
This plugin supports concurrent builds, as long as your licence allows enough QF-Test
instances.
Note
You can use the additional command line argument ”-dbg” to get more debug information
in the console output of your build.
Chapter 23
Keyword-driven testing with QF-Test
23.1
Introduction
The concept of keyword-driven testing allows business analysts and testers to describe
test-cases without any deep QF-Test knowledge. Those test-cases can either be described in a meta-language or directly in a testmanagement system. QF-Test will then
read and execute those test-cases. The implementation of the underlying test-steps
inside QF-Test needs to be done by engineers having QF-Test knowledge.
Business testers and QF-Test engineers are free in defining the way how to describe
test-steps. They can define a wide range of keywords like simple action-related keywords (e.g. click a button) or more complex business-related keywords (e.g. create an
object inside the SUT). Combining test-steps into a test-case could be achieved by a
strict table-oriented approach like clickButton=OK or formulating the test-steps in continuous text like ”Close the dialog via click on ok”.
The following examples illustrate the most popular variants of defining test-cases in a
keyword-driven manner. The demo test-case will create a vehicle in the CarConfigurator
demo application of QF-Test.
(248)
Option 1: Business-related test-steps (see section 23.2.1
qftest-4.2.2/demo/keywords/simple_business.
). You find a sample at
23.1.
Introduction
247
Test-step
Launch SUT, if necessary
Open vehicles dialog via Options -> Vehicles
Enter data (name and price)
Press new in order to create the vehicle
Press OK in order to close the dialog
Check creation of vehicle in table
Table 23.1: Test-case using business-related keywords
(251)
Option 2: Atomic test-steps (see section 23.2.2 ).
qftest-4.2.2/demo/keywords/simple_atomic.
You find a sample at
Test-step
Launch SUT, if necessary
Select menu ’Options’
Select menu ’Vehicles...’
Fill text-field ’Name’
Fill text-field ’Price’
Press button ’New’ in order to create the vehicle
Press button ’OK’ in order to close the dialog
Check table, whether new created vehicle appears
Table 23.2: Test-case using atomic keywords
Option
3:
Behavior-Driven
Testing
(BDT)
(256)
section
23.4.1 ).
You
perspective
(see
qftest-4.2.2/demo/keywords/behaviordriven.
from
a
technical
find
a
sample
at
23.1.
Introduction
248
Test-step
Given SUT is running
Given vehicles dialog is opened
When vehicle name is set to <name>
And vehicle price is set to <price>
And button new clicked
And button ok clicked
Then row with <name> and <formatted-price> appears in table
And column model has value <name>
And column price has value <formatted-price>
Table 23.3: Test-case with Behavior-Driven Testing from a technical perspective
Option
4:
Behavior-Driven
Testing
(BDT)
with
business
(258)
section
23.4.2 ).
You
find
a
sample
at
keywords
(see
qftest-4.2.2/demo/keywords/behaviordriven_business.
Test-step
Given application is ready to enter vehicle data
When vehicle created with <model> and <price>
Then <model> with <formatted price> appears in table
Table 23.4: Test-case with Behavior-Driven Testing from a business perspective
In the subsequent sections you will find a more detailed description of those variants
of keyword-driven testing with QF-Test. The concept of Behavior-driven testing (BDT)
(255)
is described in section 23.4 . From a perspective of QF-Test BDT is just a special
variation of keyword-driven testing.
All samples use the CarConfigurator of QF-Test which is also provided by the installation. You can find all samples in the folder qftest-4.2.2/demo/keywords/. In
order to show the test planning aspects we provide Excel files containing the required
test-steps. Of course you can also go ahead and use your testmanagement tool to plan
test-steps. We are using Excel files as Excel is a very common piece of software and
reading Excel file is quite simple in QF-Test.
(252)
The section keywords with dynamic components (see section 23.3 ) describes how
to use QF-Test just as test-executor without really recording steps within QF-Test. You
need to interact with QF-Test only in rare cases when applying this scenario.
Please take care to copy all test-suites to a project-related folder first and modify them
there.
23.2.
Simple Keyword-driven testing with QF-Test
23.2
249
Simple Keyword-driven testing with QF-Test
The simplest way of using keywords is to use existing procedures. Procedures can be
designed as business-related procedures or as atomic component-oriented procedures.
Business-related procedures perform real workflow in the application, e.g. creating a
new vehicle. Atomic component-oriented procedures perform very basic steps like click
the ok-button.
23.2.1
Business-related Procedures
As stated in the previous section business-related procedures represent a
real business workflow in your application.
You can find a sample
at
qftest-4.2.2/demo/keywords/simple_business/
SimpleKeywords.qft.
The respective test-plan can be found at
qftest-4.2.2/demo/keywords/simple_business/ simple_keywords.xlsx.
Please take care to copy the demo folder to a project-related folder first and modify
them there.
The sample shows the ”Create vehicle” test-case of the QF-Test CarConfigurator. It
consists of following test-steps:
1. Launch SUT, if necessary
2. Open vehicles dialog via Options -> Vehicles
3. Enter data (name and price)
4. Press new in order to create the vehicle
5. Press OK in order to close the dialog
6. Check creation of vehicle in table
Let’s take a look at the Excel file now:
23.2.
Simple Keyword-driven testing with QF-Test
250
Figure 23.1: Excel file business-related keywords
As QF-Test can read excel files row by row, we have decided to go for that excel struc(440)
ture. Reading that file follows the data-driven concept (see section 34.4 ). It’s also
possible to use another structure of the excel file, but then we lose the advantage of
using the QF-Test functionality directly without any scripts or if-conditions.
In the first row we find the values teststep, value1 and value2. That row will
be interpreted as variable names by QF-Test. Every subsequent row will then contain
respective values for those variables. This mechanism allows QF-Test to walk through
that Excel file in order to execute the planned test-steps.
Now let’s take a look at the test-suite SimpleKeywords.qft. The test-suite looks like
this:
23.2.
Simple Keyword-driven testing with QF-Test
Figure 23.2: Test-suite business-related keywords
251
23.2.
Simple Keyword-driven testing with QF-Test
Node
Test-set ”Scenario”
Dependency reference dependencies.sutStarted
Test-case ”Create vehicle”
Teststep ”Walk through test-steps”
Data-driver ”steps from excel”
Excel file ”steps from excel”
Test-step ”Generic test-step”
Call procedure ”business.$(teststep)”
252
Purpose
Represents the container node for the testexecution. This node could be omitted theoretically.
The concept of dependencies allows QF-Test to
manage the starting and stopping process of your
application in an intelligent way. Dependencies
can contain a strategy in case of any unexpected
(426)
for
behavior during test-run. See section 34.3
details.
This node represents the implemented test-case.
This node is required to read the excel file with a
data-driver node.
A data-driver reads the content of the Excel file row
by row.
Points to the Excel file.
This node will be filled with the current test-step
name from Excel during execution. This approach
will create a readable report.
Here the respective procedure defined in Excel is
called. The variable teststep will be filled with
the planned data of Excel because of the data
driver mechanism.
Table 23.5: Structure of SimpleKeywords.qft
All required procedures are implemented in the package business. In order to allow
a simple variable definition any steps in Excel use the variables value1 and value2.
Every procedure maps those generic names to the specific parameters of the procedure
itself.
23.2.
Simple Keyword-driven testing with QF-Test
253
Figure 23.3: Procedure fillDialog
This concept requires that any used keyword has to be implemented in QF-Test already
before using it. If your steps require more than two parameters you need to extend the
excel file with more columns. In addition you need one test-case node in QF-Test per
(259)
test-case in Excel. You can make this more flexible, see section 23.5 .
23.2.2
Atomic component-oriented procedures
Besides
the
already
known
business-related
procedures
representing
workflows you can describe every action individually.
Applying this
concept results in a very detailed description.
You can find the sample
test-suite
at
qftest-4.2.2/demo/keywords/simple_atomic/
SimpleAtomicKeywords.qft.
The
respective
test-plan
can
be
found
at
qftest-4.2.2/demo/keywords/simple_atomic/
23.3.
Keyword-driven testing using dynamic or generic components
254
simple_atomic_keywords.xlsx. Please take care to copy the demo folder to a
project-related folder first and modify them there.
Let’s take a look at the ”Create vehicle” test-case of the CarConfigurator again. The
test-case now consists of the following steps:
1. Launch SUT, if necessary
2. Select menu ’Options’
3. Select menu ’Vehicles...’
4. Fill text-field ’Name’
5. Fill text-field ’Price’
6. Press button ’New’ in order to create the vehicle
7. Press button ’OK’ in order to close the dialog
8. Check table, whether new created vehicle appears
Similar to the business-related keywords you need to specify an Excel file containing
the planned test-steps and you need to create the procedures in QF-Test as well. You
can find the implemented procedures in the package atomic of the test-suite
qftest-4.2.2/demo/keywords/SimpleAtomicKeywords.qft.
In case you
choose this approach you can also think about using the automated procedure
(202)
generation (see chapter 20 ).
The next section describes how to use dynamic procedures. This means that we will
still write atomic component-oriented procedures in our test-plan, but there will be no
need to create a procedure for each and every step or component. Instead of individual
procedures we will create procedures like clickButton or setText. Those procedures will
then be re-used every time.
23.3
Keyword-driven testing using dynamic or generic
components
The previous section shows how we can apply keyword-driven testing to call various
procedures depending on the test-plan. But the graphical components and their recognition still stays in QF-Test and the respective procedures. This approach requires that
every procedure needs to be recorded or created before actually running the tests.
However,
information
it’s
also
directly in
possible
to
the test-plan.
specify
the
actual
component
This plan should then be
23.3.
Keyword-driven testing using dynamic or generic components
255
interpreted by QF-Test.
You can find a sample test-suite at
qftest-4.2.2/demo/keywords/generic/Keywords_With_Generics.qft.
The
respective
test-plan
can
be
found
at
qftest-4.2.2/demo/keywords/generic/keywords-generic.xlsx. Please
take care to copy the demo folder to a project-related folder first and modify them there.
This approach depends on the concept of generic component recognition in QF-Test.
Generic component recognition allows the user to apply variables to the recorded component information or to move components out of hierarchical component structure.
(307)
Please see section 30.6 for details.
Let’s go back to our sample test-case. The test-case ”Create vehicle” looks like this.
1. Launch SUT, if necessary
2. Select menu ’Options’
3. Select menu ’Vehicles...’
4. Fill text-field ’Name’
5. Fill text-field ’Price’
6. Press button ’New’ in order to create the vehicle
7. Press button ’OK’ in order to close the dialog
8. Check table, whether new created vehicle appears
As you can see the test-case follows the same description like in the previous section
about atomic keywords.
The Excel file looks like this:
Figure 23.4: Excel file of generic components
23.3.
Keyword-driven testing using dynamic or generic components
256
The
used
Excel
file
contains
values
like
selectMenu
or
dialog.clickButton for the teststep column. Additionally a new column
target was introduced.
That new variable will be explained later.
Like
in the previous samples you can find a demo implementation at
qftest-4.2.2/demo/keywords/generic/Keywords_With_Generics.qft.
You can find the respective procedures in the package generic. Please take care to
copy the demo folder to a project-related folder first and modify them there.
Figure 23.5: Test-suite for generic components
Let’s investigate the procedure selectMenu first. This procedure consists of a
mouse-click at the component GenericMenuItem. If we analyze that component in
the test-suite, we see that the attribute class is set to MenuItem. We also see that
the attributes Name and Feature are empty, but there is one entry for the Extra
features table. This entry has the name qfs:label with the state Must match
and the value $(target). The next attributes Structure is empty again and the
23.4.
Behavior-driven testing (BDT)
257
(307)
values for Geometry are set to ’-’. You can details about the ’-’ at section 30.6
.
This way of defining a component means that the recognition of that component relies
on the variable target. The variable itself is used in the extra feature qfs:label.
That extra feature qfs:label represents the best describing text of a component, e.g.
the text on a button or a label close to a text-field. The excel file got the column target
which contains the exact label of the respective target components. This method has
been applied to all other components as well.
Another noteworthy aspect is the package dialog under the package generic. This
has been introduced because QF-Test also takes the window or dialog objects into
account in order to recognize the graphical components correctly. QF-Test also distinguishes between windows and dialogs. Standard windows which allow the user to work
within a second window of the application as well and so-called modal windows preventing the user to work in a second window of your application. In most cases it’s simpler to
separate those two kinds of windows in several packages. If you want, it might be possible to unify them in one window, but that’s not shown in the current samples. By the way,
you don’t need to separate between those window types if you test web applications as
there every component is part of a web-page.
In this section we have seen how to make the component recognition more flexible
using variables. Additionally we have created one procedure per action and type of
target component. This concept allows us to define all test-cases within Excel. The
required procedures including the generic components have to be created at the beginning of the project. Of course you can also mix this approach with some recorded
procedures. Those recorded procedures can then be used like business-related proce(248)
dures described in section 23.2.1 )
23.4
Behavior-driven testing (BDT)
Besides the traditional concept of keyword-driven testing a second concept called
Behavior-driven testing (BDT) is widely being used. Using this approach allows testers
to describe test-cases more or less in continuous text and sentences. But the tester
needs to follow a predefined vocabulary at the beginning of the sentence. Test-cases
described like this can be more readable for persons without any knowledge of the
test-cases. Test-cases can be described from a technical perspective (see section
(256)
(258)
23.4.1 ) or from a business perspective (see section 23.4.2 ) like in keyword driven
testing. You find samples for both variants in the following sections.
23.4.
Behavior-driven testing (BDT)
23.4.1
258
Behavior-Driven Testing (BDT) from technical perspective
Describing a test-case from a technical perspective using Behavior-Driven Testing
(BDT) uses more or less elementary actions for designing a test-case. You can find a
sample
test-suite
at
qftest-4.2.2/demo/keywords/behaviordriven/
BehaviorDrivenTesting.qft.
The respective test-plan can be found at
qftest-4.2.2/demo/keywords/behaviordriven/
createvehicle.xlsx.
Please take care to copy the demo folder to a project-related folder first and modify
them there.
The ”Create vehicle” test-case looks like this if it’s described in the BDT manner from a
technical view:
1. Given SUT is running
2. Given vehicles dialog is opened
3. When vehicle name is set to <name>
4. And vehicle price is set to <price>
5. And button new clicked
6. And button ok clicked
7. Then row with <name> and <formatted-price> appears in table
8. And column model has value <name>
9. And column price has value <formatted-price>
BDT requires to use the terms Given, When, And and Then at the beginning of any
sentence. You will find more information about this approach in the testing literature.
QF-Test requires matching procedures for above test-steps, so we need to build respective procedures again. It’s an established method to divide the BDT-keywords in
separate packages. The provided test-suite therefore contains the packages Given,
When_And and Then.
23.4.
Behavior-driven testing (BDT)
259
Figure 23.6: Test-suite Behavior-driven testing technical
The provided sample test-suite contains all procedures in the respective packages, e.g.
a procedure vehicles dialog opened inside the package Given. In order to prevent annoying typos a Server-script read test-steps and fix typos formats any
steps to lower case and tries to replace multiple blanks by one. This script is called directly before the procedure call of $(teststep).
In order to run the test-case on multiple test-data the sample was extended.
Of course you can apply the concept of generic component recognition as described in
(252)
the previous section (see section 23.3 ). To that end you would need to specify a very
exact description or implement a script filtering the target components from the test-step
itself.
23.4.
Behavior-driven testing (BDT)
23.4.2
260
Behavior-Driven Testing (BDT) from business perspective
Describing test-cases from business perspective using Behavior-Driven Testing (BDT)
requires actions from a user’s point of view. So those actions contain several
interactions likes mouse-clicks or text-inputs. You can find a sample test-suite
at
qftest-4.2.2/demo/keywords/behaviordriven_business/
BehaviorDrivenTesting-Business.qft. The respective test-plan can be found
at
qftest-4.2.2/demo/keywords/behaviordriven_business/
createvehicle-business.xlsx. Please take care to copy the demo folder to a
project-related folder first and modify them there.
The ”Create vehicle” test-case looks like this if it’s described in the BDT manner from a
business perspective:
1. Given application is ready to enter vehicle data
2. When vehicle created with <model> and <price>
3. Then <model> with <formatted price> appears in table
This approach uses the keywords Given, When, And and Then at the beginning of any
sentence like the technical one. The provided test-suite therefore contains the packages
Given, When_And and Then.
23.5.
Scenario files
261
Figure 23.7: Test-suite Behavior-driven testing from business perspective
In order to prevent annoying typos a Server-script read test-steps and fix
typos formats any steps to lower case and tries to replace multiple blanks by one.
This script is called directly before the procedure call of $(teststep).
In order to run the test-case on multiple test-data the sample was extended.
23.5
Scenario files
Apart from defining single test-cases you can also specify the entire test scenario in
Excel files or even within your test management tool. In our sample we will go on using
Excel to keep things simple. Of course it’s also possible to use your testmanagement
tool there. For reasons of simplification we have used Excel files again. You can find a
sample test-suite at qftest-4.2.2/demo/keywords/generic_with_scenario/
23.5.
Scenario files
262
Keywords_With_Generics.qft.
The scenario itself can be found in
qftest-4.2.2/demo/keywords/generic_with_scenario/
scenario.xlsx.
All used test-cases are described in a separate excel file,
see
qftest-4.2.2/demo/keywords/generic_with_scenario/
keywords-generic-testcases.xlsx. Please take care to copy the demo folder to
a project-related folder first and modify them there.
The provided scenario consists of two test-cases using the concept of generic proce(252)
dures and components (see section 23.3 ). You can use any other approach, if you
want.
Let’s take a look at the scenario Excel file.
Figure 23.8: Excel file as scenario file
The worksheet ”Scenario” contains a column ”testcase”. This value will be used as variable later. Each subsequent row represents a test-case name. Those test-cases correspond with the worksheets in keywords-generic-testcases.xlsx. The worksheets ”tc_createvehicle” and ”tc_createvehicle_cheap” contain the respective test-case
description.
How does the test-suite look like?
23.5.
Scenario files
263
Figure 23.9: Test-suite scenario file
23.6.
Custom test-case description
Node
Test-set ”Generic samples with scenario”
Dependency reference dependencies.sutStarted
Data driver ”scenario”
Excel file ”test-cases”
Test-case ”Generic test-case”
Teststep ”Walk through test-steps”
Data driver ”steps from excel”
Excel file ”steps from excel”
Test-step ”Generic test-step”
Call procedure ”generic.$(teststep)”
264
Purpose
Represents the container node for the testexecution. This node could be omitted theoretically.
The concept of dependencies allows QF-Test to
manage the starting and stopping process of your
application in an intelligent way. Dependencies
can contain a strategy in case of any unexpected
(426)
for
behavior during test-run. See section 34.3
details.
Here we read the required test-cases to execute.
Points to the scenario excel file.
This node represents the implemented test-case.
This node is required to read the test-steps from
the excel file with a data-driver node.
A data-driver reads the content of the Excel file
describing the test-case row by row.
Points to the Excel file describing the test-case.
This node will be filled with the current test-step
name from Excel during execution. This approach
will create a readable report.
Here the respective procedure defined in Excel is
called. The variable teststep will be filled with
the test data of Excel because of the data driver
mechanism.
Table 23.6: Structure of Keywords_With_Generics.qft
23.6
Custom test-case description
The previous sections show all samples using Excel files. As already mentioned it’s also
possible to use different file-types, e.g. XML or CSV files. But you can also think about
evaluate the result of a web-service. Therefore, you will need to implement Serverscripts reading the required information like test-step name, component names or variables for QF-Test. Setting variables can be achieved by the methods rc.setLocal
bzw. rc.setGlobal.
Once those variables have been set it will become necessary to call test-case or procedures. Therefore, you can use the methods rc.callTest or rc.callProcedure.
(114)
You can find a full API description at chapter 12 .
You can also find a few samples in the provided test-suite of the manual tester
(qftest-4.2.2/demo/manualtester) or in the integration of the imbus TestBench
(qftest-4.2.2/ext/testbench).
23.7.
23.7
Adapting to your software
265
Adapting to your software
All examples make use of the CarConfigurator of QF-Test. You can use those samples
as templates to adapt the existing concept to your strategy of keyword-driven testing.
The provided samples can only act as templates because of the huge variety of ways of
creating applications and the many different testing strategies. They will never serve as
out-of-the-box solution without any need of adapting them. In order to find a matching
solution to your project you can also get in touch with our support team.
Nevertheless you can find a full sample for the CarConfigurator at
qftest-4.2.2/demo/keywords/ full_sample_for_carconfig. The sample
(259)
uses the concept of scenario files as described in section 23.5 .
23.7.
Adapting to your software
Technology
JavaFX
266
Necessary Adjustments
1. Replace the value awt with fx in the attribute GUI-Engine on all window components.
2. Perhaps you need to extend the recorded window components with additional
variables.
3. You need to adapt the procedure startStop.startSUT in order to start your
application. Simply copy the created steps from the quickstart wizard.
4. You might possibly have to create some resolver scripts to tune component
recognition.
Java/Swing
1. Perhaps you need to extend the recorded window components with additional
variables.
2. You need to adapt the procedure startStop.startSUT in order to start your
application. Simply copy the created steps from the quickstart wizard.
3. You might possibly have to create some resolver scripts to tune component
recognition.
Java/SWT
1. Replace the value awt with swt in the attribute GUI-Engine on all window components.
2. Perhaps you need to extend the recorded window components with additional
variables.
3. You need to adapt the procedure startStop.startSUT in order to start your
application. Simply copy the created steps from the quickstart wizard.
4. You might possibly have to create some resolver scripts to tune component
recognition.
Web
1. Replace the window component with a Webpage node.
2. As dialogs are already part of the webpage you don’t need to keep them separated and can create a component within the webpage..
3. You need to adapt the procedure startStop.startSUT in order to start your
application. Simply copy the created steps from the quickstart wizard.
4. You might possibly have to tune component recognition using the CustomWebResolver concept.
Table 23.7: Necessary adaptions to your SUT
Chapter 24
Performing GUI-based load tests
24.1
Background
techniques
and
comparison
with
other
In addition to functional and system tests, QF-Test can also be used to perform load
tests, stress tests or performance tests. The idea is to test the performance of some
server applications by running a number of GUI clients concurrently.
Performance is measured by running multiple GUI clients in parallel. QF-Test enables
you to measure the actual end-to-end response times (the time span from user action
until the result shows up). For the following paragraphs we will use the term load testing.
There are many different ways for setting up and performing load tests, most of which
are not using real GUI clients. Instead they directly make use of the protocol between
the client and server, e.g. by sending HTTP request or performing RMI or other kinds of
remote procedure calls.
There are a number of pros and cons for protocol-based or GUI-based load testing:
• Use of resources:
Protocol-based testing uses very little resources at the client side, so it can easily
scale up to the breaking point of the server without requiring too much hardware.
GUI-based tests incur the full memory and performance overhead for each client,
which can be quite significant, especially in case of Swing or JavaFX based rich
clients. In addition, every client creates a GUI and, therefore, a real active user
session is required.
• Efforts creating tests:
Rich clients typically represent a complex user interface, which correlates with a
certain complexity of the client/server API. Creating protocol-based tests that cover
most of that API can be quite an effort. On the other hand, GUI-based tests that
24.1.
Background and comparison with other techniques
268
have already been implemented for functional testing may be available for reuse.
If not, it is still much easier to automate complete use-cases with QF-Test than at
protocol level.
• Measuring response times:
With GUI-based testing, actual end-to-end response times (the time span from
user action until the result shows up) are measured, while protocol-based tests
measure only the times for the server call. Either can be useful, depending on the
situation.
In some cases it can be quite reasonable to combine both approaches. You can think
about running GUI tests on a few systems in order to measure those end-to-end times
and in parallel you can trigger protocol-based tests in order to create some load.
In summary, GUI-based load tests can be very useful and efficient - especially if functional tests can be reused - provided that either the number of clients that need to be
simulated is not too high, or that sufficient hardware is available for the client side.
At the end of this section here is a overview diagram showing all involved systems.
Figure 24.1: Load testing scenario
24.2.
24.2
Load testing with QF-Test
269
Load testing with QF-Test
As load testing is a sophisticated subject QF-Test provides a demo tests-suite which
can be used as initial point for your project. You can find that demo solution at
qftest-4.2.2/demo/loadtesting/. This folder contains the following files:
File
Systems.xlsx
carconfig_Loadtesting.qft
daemonController_twoPhases.qft
checkForRunningDaemons.qft
Purpose
You can configure which test-systems are involved in the test-run.
Furthermore you can configure global variables for the test-run
there.
This test-suite contains the GUI tests which will get executed on
the test-systems.
This test-suite represents the controlling test-suite for the entire
test-run. Using this test-suite allows you to launch and co-ordinate
the test-run on multiple systems.
This test-suite contains test-cases to check for running daemon
processes on individual test-systems.
Table 24.1: Content of load testing directory
The test-suites and files mentioned above can be applied to a load testing project which
makes use of multiple test-systems. Please take care to copy the demo folder to a
project-related folder first and modify them there. The subsequent figure shows an
illustration:
24.2.
Load testing with QF-Test
Figure 24.2: Overview load testing project
The provided sample test-suite for controlling the test-run looks like this:
270
24.2.
Load testing with QF-Test
271
Figure 24.3: Sample test-suite daemonController_twoPhases.qft
In order to execute load tests with QF-Test you should take care of the following:
1. Provision of test-systems
2. Conception of the test-run
3. Preparing test-systems prior to the test-run
4. Test execution
5. Evaluating results
You can find brief explanations as well as some hints for each item in the following
sections.
24.2.
Load testing with QF-Test
24.2.1
272
Provision of test-systems
You perform load tests with QF-Test via the GUI. GUI tests require an active user session
and shouldn’t get executed in parallel at the same desktop. That’s why we recommend
to set-up a virtual or physical system for every client involved. It’s indeed possible to run
multiple GUI tests in parallel on the same desktop, but this can end up in very subtle
problems, e.g. issues with the current focus. That’s why running multiple GUI tests on
the same desktop is not recommended and should only be taken into consideration in
exception.
QF-Test needs to be installed on every system. In addition, the required test-suites,
the configuration file of QF-Test and necessary test data files need to be deployed to
the test-systems as well. You can either copy those files to every system locally or you
establish a common network share. Furthermore, every test-system requires at least a
runtime license in order to run the tests. QFS offers to lease such runtime licenses even
for a certain period of time.
24.2.2
Conception of the test-run
The simplest case is to run the same test on all involved test-systems. However, many
load testing projects require different sets of GUI tests to be executed. You can think
about running tests for various roles of users or user groups. A possible group can
represent standard users another some kind of administrator users.
Besides designing the test-run for multiple roles load tests are often split into several
phases. A phase represents a certain thematic priority. As an example you can divide
your project into four phases. The first phase stands for the ”Launch” phase. There,
the SUT is getting launched on all involved test-systems and some initial actions as
the log-in can take place. During the second phase 50 clients perform their specific
test-scenario. The third phase is performed using 100 clients and the final fourth phase
downgrades to 50 clients again. This kind of scaling is also called ramp-up phase (incrementally increasing load) and ramp-down phase (incrementally decreasing load).
Such a conception using several phases increasing the load allows you to test the load
capacity of your application in several steps. Like this you will get the information that
your application was ok in phase one and problems occurred in the second phase, rather
than just a statement about all or nothing.
Using several phases makes sense if multiple roles are the actual focus of your tests.
In some cases launching the application on all involved test-system can break the environment. So you can think about splitting your project at least into a ”launch” phase and
”test” phase.
You should create one test-suite per role to keep track of your test-cases.
24.2.
Load testing with QF-Test
273
Implementation in the sample test-suite:
You can find a sample project with two phases in the provided controller test-suite
daemonController_twoPhases.qft The first phase (Launch Phase Phase)
launches the application. The second phase (Test Phase) represents the actual test
phase. You can configure the required test-suite in the corresponding Run...Phase
test nodes of each phase.
The provided sample focuses on several roles instead of phases. In case you would
like to create a third phase, simply copy the test node Test Phase and rename it
accordingly.
24.2.3
Preparing test-systems prior to the test-run
You need to launch the QF-Test daemon before you can start your test-run. This QFTest daemon requires a vacant network port. In order to work effectively we recommend
to use the same port on all systems, e.g. 5555.
You can launch the daemon like this:
qftest -batch -daemon -daemonport 5555
Example 24.1: Launching QF-Test daemon
Please note, that the daemon needs to be started in an active user session. You can
accomplish this using tools like the task planer. You can find further details about the
(182)
(309)
daemon at section 18.2 . Chapter Hints on setting up test-systems contains useful
tips and tricks to set-up the daemon process. In FAQ 14 you can find technical details.
If you want to check whether the daemons are up and running you can either run
individual ping commands of the daemon or you run the provided test-suite
checkForRunningDaemons.qft.
qftest -batch -calldaemon -ping -daemonhost localhost -daemonport 5555
Example 24.2: Ping of QF-Test daemon at localhost
Note
On Windows you should use the command qftestc.exe instead of qftest.exe for
every command.
24.2.4
Test execution
During test execution you will need some scripts that will contact the various QF-Test
daemons in order to co-ordinate the test run. Such scripts can use QF-Test’s daemon
24.3.
Hints on test execution
(883)
API (see section 44.2
274
(671)
) or its command line (see chapter 36
).
Implementation in the sample test-suite:
The provided test-suite daemonController_twoPhases.qft allows you to run such
a load testing scenario and collect the run-logs of the test-runs afterwards. In the provided Excel file Systems.xlsx you can configure which test-systems should be involved. That files also contains some variables to organize your tests in roles as de(270)
scribed in section 24.2.2 .
Once all test-systems have been correctly configured you can start the test-run via running the entire test-suite.
Besides the pure execution of such a load testing project you can also meet further
requirements. The provided test-suites shows samples for the following aspects:
(272)
1. Synchronizing the test-run on several test-systems, see section 24.3.1
(274)
2. Measure end-to-end times, see section 24.3.2
24.2.5
.
.
Evaluating results
Evaluating results can become quite challenging just because of that huge amount of
data. You can analyze the QF-Test run-logs as well as the QF-Test reports. Perhaps
you receive some measurements at server side or you find a couple of logs which you
can analyze by specific tools.
During test execution you can also create custom log-files with QF-Test as described in
(274)
section 24.3.2 for details.
24.3
Hints on test execution
24.3.1
Synchronization
To get consistent results, it may sometimes be necessary to coordinate the tests in the
parallel threads, either to make sure that all clients access the server simultaneously,
or to prevent just that. Furthermore a role (see previous section) might require all testsystems to be in a certain state before running a specific action.
Test-runs can be synchronized with the help of a Server script
contain the following:
(502)
node. That script should
rc.syncThreads("identifier", timeout, remote=3)
24.3.
Hints on test execution
275
identifier is a name for the synchronization point, timeout is the maximum time in
milliseconds to wait for all threads to reach the given synchronization point and remote
specifies how many systems should wait for that synchronization point.
If the timeout is exceeded without the expected number of threads reaching the syn(662)
chronization point, a TestException
is thrown. To log an error instead of raising
an exception, set the optional parameter throw to 0 (default value 1) or you pack that
(502)
Server script step into a Try step.
rc.syncThreads("case1", 120000, remote=3, throw=0)
You can find a sample implementation in carconfig_Loadtesting.qft.
Figure 24.4: Call of rc.syncThreads in demo test-suite
24.4.
Troubleshooting
24.3.2
276
Measuring end-to-end response times
It’s a very common requirement for GUI tests to measure end-to-end response times.
QF-Test logs those times into its run-log. Instead of having to parse that run-log in order
to retrieve those values you can implement a so-called TestRunListener to write a
dedicated log file, which just contains the required measurements.
In order to measure the interesting parts, you will need to mark your test-steps or sequence using a dedicated keyword. The provided sample implementation uses the keyword @transaction for that purpose. If you want to use another keyword, you have to
change the code of the provided TestRunListener.
In the provided sample test-suite all measurements will be logged into a simple CSV file.
That CSV file can be used later for the actual evaluation by another tool. Furthermore,
writing that CSV file doesn’t brake the test-run. If you want to create Excel files or fill
databases in order to evaluate the results you should do that after the test-run due to
performance reasons.
(869)
You can find details about the TestRunListener at section 43.7
mentation can be found in carconfig_Loadtesting.qft.
. The sample imple-
The created CSV file looks like this:
open vehicles;118;20150921145057;OK
close vehicles;84;20150921145057;OK
Example 24.3: CSV file for time measurements
In that CSV file the first value represents the name of the measurement, the second
value stands for the duration of the action in milliseconds, the third shows the time when
the step was performed, the fourth value shows whether the step was successful.
24.4
Troubleshooting
Due to the complexity of load testing projects you may face issues in several areas.
1. Why are wrong test-cases executed?
Adapt the variable testsuite in the respective test-case. You can also address
a test-case directly via testsuite#testset.testcase.
2. The QF-Test daemon cannot be started.
Is the network port vacant? You can check this using the netstat command.
Here is a sample for the port 5555.
24.4.
Troubleshooting
Windows
netstat -a -p tcp -n | findstr ”5555”
Linux
netstat -a --tcp --numeric-ports | grep 5555
277
3. The test-systems cannot be reached although the QF-Test daemon is running.
Check whether the QF-Test daemon can be reached on the test-sytem, see
(270)
section 24.2.1 . If the QF-Test daemon is running, please perform following
steps:
(a) Can you reach the QF-Test daemon on the local systems using the ping com(271)
mand, see section 24.2.3 ?
(b) Ensure that the daemon or its Java process is not blocked by your system
firewall.
(c) Perhaps there are issues resolving the host name of your test-system. Then
try to launch the daemon with the additional parameter -serverhost
localhost or -serverhost IP-Adresse or -serverhost <host
name>. In case you use the IP-address, please also access that system
using the IP-address, otherwise use the host name.
Chapter 25
Executing Manual Tests in QF-Test
3.0+
25.1
Introduction
QF-Test is primarily a tool for the creation and execution of automated tests. However, it
is rarely possible - or economical - to automate 100% of the required tests for a project.
In most projects some manual tests need to be performed as well. One of the biggest
challenges in testing a project is consolidating the different results and reports of automated and manual testing to get an overview about the execution status of all tests. To
facilitate reporting the results of manual test execution along with those of automated
testing, QF-Test now offers the capability of tracking manual tests from within itself.
The steps to be performed during a manual test have to be defined in an Excel file which
is read by a test-suite called ManualTestRunner.qft. This test-suite is provided
along with a sample specification file in the directory demo/manualtester below the
QF-Test installation directory. The test designer has to specify each step in that Excel file
including the expected result. After stepping through the manual tests QF-Test provides
the usual results - a run-log, HTML and XML reports. Additionally, a newly created
Excel file with the results of the respective test-run is created. For a detailed description
(277)
please see section 25.2 .
The dialog used for the test execution is called ManualStepDialog and looks like this:
25.2.
Step-by-step Guide
279
Figure 25.1: Example for a ManualStepDialog
The title of the dialog shows the name of the test-case. The detailed step description
and the expected result are shown in the first two text-boxes. After performing the test
the tester has to specify whether the test succeeded or not. In case the test failed the
tester also has to enter the received result which is intended to show the differences
between the actual and the expected result. This dialog can also be used for your own
(905)
purposes, see section 46.1 .
25.2
Step-by-step Guide
Please perform the following steps on your system to launch a manual test from QFTest.
• Copy
the
definition
Excel
file
from
qftest-4.2.2/demo/manualtester/SampleTestDescription.xlsx to
your project location and rename it to a suitable name. We recommend to use the
same path on all test-systems. Perhaps you can make use of a shared network
drive.
25.3.
Structure of the Excel file
• Also
copy
the
execution
test-suite
qftest-4.2.2/demo/manualtester/ManualTestRunner.qft
project location. You may want to rename it as well.
280
to
from
your
• Open the Excel file and define the test-steps.
• After saving the changes to the Excel file, open the execution test-suite and adapt
the global variable testFile variable to target your specific Excel file.
• Turn off the QF-Test debugger. It would only interfere with the steps of the manual
tester.
• Start the test-suite via selecting the test-suite node and pressing ”Start test-run”.
• QF-Test will now read the data from the Excel file and open a dialog containing the
first test-step.
• Enter the result of the test-step and proceed with executing each test-step.
• At the end of the test execution QF-Test will write a new Excel file containing the
test description and the according results. You can also store the run-log of that
execution or create an HTML report.
Please read the comments in the test-suite and Excel file carefully, because you can
adapt this concept according to your needs. It is even possible to start only specific
tests.
25.3
Structure of the Excel file
The Excel file has a specific structure which allows you to describe the manual teststeps quite flexibly. The meaning of the columns is explained in the following table:
Column
TestCase
Type of Test
Comment
Short Description
Step Description
Expected Result
Description
A unique identifier for each test-case. If the step belongs to the same test-case as
the previous step, just leave this column empty.
Optional definition of the kind or function of the test or step, e.g. a functional test
or a usability test, startup, etc.
An individual comment for the test-case. This comment will be shown in the runlog of QF-Test.
A short description about the content of the test.
The detailed description of the manual step.
The description of the expected result of that test-step.
Table 25.1: Description of the Excel file for the definition of manual tests
25.4.
The ManualTestRunner test-suite
281
The Excel file with the results of the manual test execution will contain two additional
columns as follows:
Column
Received Result
State
Description
The result the tester received during test-execution. If a test-step fails, the tester
must specify a received result.
The state of the test, i.e. PASSED, FAILED, CANCELED or SKIPPED.
Table 25.2: Description of the Excel file with the results of manual tests
25.4
The ManualTestRunner test-suite
The ManualTestRunner.qft test-suite contains some global variables at suite-level
which provide fine-grained control over test-run. These are explained in the following
table. All variables not listed here are used internally by the test-suite and should not be
changed.
Global Variable
testFile
testSheet
resultSheet
tests
defaultState
testCaseColumn
commentColumn
shortDescColumn
stepDescColumn
expResultColumn
recResultColumn
stateColumn
Description
The path to the test-step definition Excel file.
The worksheet of the Excel file containing the test-steps.
The name of the worksheet for the results.
A list of tests to be intended to execute. If this variable is empty, all tests will be
executed. If you want to execute only test 5 and 6, you can specify 5,6 or 5-6.
It is even possible to specify things like: 1,3-5,7 to execute the tests 1, 3, 4, 5
and 7.
The default selection of the state. You can set it either to PASSED or FAILED. All
other states will be converted to FAILED.
The heading of the column containing the test-case number.
The heading of the column containing the comment.
The heading of the column containing the short step description.
The heading of the column containing the full step description.
The heading of the column containing the expected result.
The heading of the column containing the received result.
The heading of the column containing the state of the test.
Table 25.3: Description of the global variables in the ManualTestRunner test-suite
25.5.
25.5
Results
282
Results
An executed test-step can be set to one of the following states:
Result
PASSED
FAILED
CANCELED
SKIPPED
Description
The test-step was successful.
The test-step failed.
The test-step was canceled.
The test-step was skipped.
Table 25.4: States of manual test-execution
Part II
Best Practices
Chapter 26
Introduction
This part of the manual describes best practices based on lessons learned from several
customer projects and user feedback. The concepts described should assist you in
finding the best strategy for using QF-Test in your projects.
Note
QF-Test is a very generic tool. The hints and experiences described here are just suggestions from our point of view, which we hope will support you in working efficiently
and successfully with QF-Test in your project. But they are just one way of doing things
and you will have to find your own solution that works best for your specific project.
In case you are interested in how other customers use QF-Test you can post a question
to our mailing list which you can reach via http://www.qfs.de/mailman/listinfo/qftest-list.
Chapter 27
How to start a testing project
This chapter talks about the most important aspects that should be considered before
you start to use QF-Test widely in your testing project. It mostly raises questions and
gives general answers with references to more detailed information.
The aim of this chapter is to provide hints about issues which you should take care of in
order to make your GUI tests reliable, stable, repeatable and especially maintainable.
27.1
Infrastructure and testing environment
Before you start creating and running automated tests you should think about some
general matters pertaining to the environment where the tests have to run. In order to
make tests reliable and repeatable you have to take into account that you must be able
to bring your SUT into a well-defined state, which includes the state of its backend, e.g.
a server and/or a database. If you do not think about such aspects it might become very
difficult and sometimes quite tricky to re-run a test or simply to analyze test-results and
maintenance of tests can become a nightmare.
Please consider the following topics:
1. What is the initial state of your SUT?
• Which user is the actual user running the tests in your SUT? Most projects
work with dedicated test-users for running tests. Another approach could be
to have one test-user per test engineer.
• Which language setting of your SUT is the primary one? Is it really required
to reach a full coverage of all supported languages or is it sufficient to run the
bulk of the tests in one primary language and create only a few tests to specifically test localization? In most cases repeating tests in several languages just
27.1.
Infrastructure and testing environment
286
covers the same functionality, so you gain no real new information after running them. However, unless you take precautions, the language setting will
(299)
influence the component recognition of QF-Test, please see chapter 30 for
details.
2. What is the initial state of your database?
• Can you work with an extra test database or do you have to use a production database? Test databases contain test-data with designed and planned
content whereas production databases contain real-life data. Is the latter predictable and reliable? What about the danger that tests can mess with and
possibly destroy production data? If at all possible you should avoid running
automated tests in a production environment.
• Can you clean up or reset the environment after one test-run for re-running
the test? Is it possible to undo changes in the database or is it required to
use new test-data for the next regression phase?
• How can you read or write test-data? Do you want to use standard SQL
scripts or can you reuse libraries from development? Some projects even reinstall the whole database before every test-run because they cannot reuse
any test-data or clean the database correctly.
3. Do you want to integrate QF-Test with other tools, e.g. build tools or test management tools?
• How to integrate QF-Test with a test management tool? If you can reuse
already planned test-steps you can avoid redundant work in planning tests
and creating them. For the standard integration for such tools, please see
(207)
chapter 21 .
• Should test’s be launched by a build-tool? If you have created tests you
can run them unattended and trigger the run by a build-system like Ant or
(176)
CruiseControl. Please see chapter 18 for details about test-execution.
• Should test-results be uploaded to a reporting system or into a
test-management system or is it more sufficient to put the HTML reports and
run-logs on a centralized HTTP-server?
4. Who will work with QF-Test?
• Do only one or two engineers work with QF-Test or do all developers and
business testers participate in test development? You can find some hints
(292)
about working in a team with different roles in section 28.5 .
• What are the skills of the engineers? It is recommended to have at least
one dedicated person with a good QF-Test knowledge in the team, who is
also capable of implementing scripts and understanding software development principles.
27.2.
Location of files
287
Of course there will be more issues to take care about which are specific for your project.
Try to figure them out.
27.2
Location of files
You should also think about following aspects of saving or installing files:
1. Where to install QF-Test to? QF-Test can be installed locally on every system but
this forces you to update every system manually whenever you need to upgrade
to a new version. You can also install QF-Test on a shared network drive, if your
(286)
network is reliable, see section 27.2.1 for details.
2. Where to store the configuration file qftest.cfg? Among other things that file
contains information about how QF-Test should recognize components or what
should go into the run-log. These options have to be the same for every QFTest user, otherwise you cannot share tests in your team. To ensure that you
can either use a shared network installation for QF-Test or specify the config file
via command-line parameters when launching QF-Test. Make sure the shared
config file is write-protected unless you explicitly want to change it. For details,
(9)
see section 1.6 .
3. Where to store the license file license? You should put the license file to a
central place in order to update the license only once when you receive an update
for it. Again, you can either have a shared network installation for QF-Test or can
use command-line parameters to specify the location of that file when launching
(8)
QF-Test. For details, see section 1.5 .
4. Where to store the test-suites? The best place to store test-suites is a version
management system where you can track the changes and access any version of
the files. If this is not possible you should store them on a shared network drive.
5. Where to store the test data files? Test data files are associated with test-suites
so you should store them closely to the suites, i.e. either in the same version
management system or on a shared network drive.
6. Where to store the HTML reports and run-logs? You should put those files in
a centralized place where any engineer can take a look at them to evaluate the
tests-results. Most people tend to use an HTTP server or a shared network drive
for that.
27.3.
Component Recognition
27.2.1
288
Network installation
If you plan to install QF-Test on a shared network drive you have to take care about
some specific things.
The main source of conflict is the system settings file qftest.cfg. It is actually a good
(and necessary) thing to have all users use the same system settings, especially for
the recognition options. Sharing the system settings file facilitates this. However, this
file should be made read-only so that one user will not inadvertently change the system
settings for everyone. If the file is read-only, QF-Test will not save the system settings
upon exit. Any change to these settings will have to be made by explicitly making that
file writable, then exiting QF-Test and then making it read-only again. Alternatively each
(691)
user could specify a different system settings file via -systemcfg <file> but that’s
not advisable.
The running QF-Test instances will also share the log directory (internal logging, not a
problem) and the Jython package cache which can occasionally cause problems so that
QF-Test cannot initialize its Jython interpreter. This doesn’t happen often and can be
fixed by clearing (not removing) the Jython cachedir.
For Windows, each user should also execute the setup.exe for the primary QF-Test
version, located in the installed qftest-x.y.z directory, to get proper registry settings and
documentation links on his machine.
In the rare case when a QF-Test patch overwrites existing jar files of QF-Test, running
instances based on those jars may crash on Windows.
27.3
Component Recognition
The most important aspect of a GUI testing tool is a stable and reliable recognition of
the graphical components. In that area QF-Test is very flexible and can be configured
in several ways. In most cases the default configuration for the component recognition
works well, but sometimes you may have to change it.
If you change the component recognition options after creating lots of test-cases, those
test-cases may break. Therefore you should try to find the most appropriate settings for
your project as early as possible. It is worth spending time in that area before starting
to implement a huge amount of tests because in the worst case you might have to rerecord or at least update all or most of the existing test-cases after a critical change of
the recognition options.
Best start by recording some demo test-cases and figure out how QF-Test recognizes
(40)
your SUT’s components. The recognition mechanism is described in chapter 5 and
(55)
chapter 6 . If you re-run those demo test-cases - ideally on different versions of your
SUT - and run into recognition problems, you have to ask yourself following questions
27.3.
Component Recognition
289
about those tests:
1. Are there enough synchronization points, like Wait for component to appear or Check
nodes with timeouts to execute test-steps only if the SUT is ready for them?
(a) Sample 1: After opening a window you can only work in that window, if it is
really there -> Use a Wait for component to appear node.
(b) Sample 2: After pressing on a search button, you can only continue with the
test when the search is really over -> Use a Check node with a timeout.
Another important aspect besides synchronization points is the correct approach of recognizing components. You have to ask yourself the following questions to determine,
which recognition approach might be the most appropriate one:
1. Do the developers use unique and stable names for their components? Please
(634)
take a closer look at section 34.13 .
2. Perhaps it is sufficient to use a regular-expression for the Feature attribute of the
component of the main window under the Windows and components node. Please
(303)
see section 30.3 for details.
3. If development did not set useful or even dynamic names it may be required to
(299)
implement a NameResolver. Please take a closer look at chapter 30 .
4. Do any of the QF-Test recognition options need to be changed? These are de(299)
scribed in chapter 30 .
(307)
5. Is it possible to use generic components? See section 30.6
for details.
In some cases it is sufficient to change the default configuration. Let us assume the
developers have set unique and stable names for the target components, i.e. buttons,
textfields, checkboxes etc. In such cases it may be sufficient to just change the ’Name
override mode’ setting of QF-Test to ’Name overrides everything’. This setting tells QFTest to ignore any changes in the component hierarchy and just work with the target
components and the window directly.
Note
You have to change this option in two places: Once at ’Record’ -> ’Components’ ->
’Name override mode’ and at ’Replay’ -> ’Recognition’ -> ’Name override mode’. See
(299)
chapter 30 for more details.
Chapter 28
Organizing test-suites
One of the most challenging tasks in a project is keeping the test-suites maintainable
over a long period of time. Especially if some window or workflow changes significantly
the maintenance effort should be kept to a minimum.
It is also worth thinking about how to minimize the creation efforts of tests that contain
a lot of similar or even the same steps. A typical use-case is launching the SUT or the
login process or a very important and basic workflow like navigating to a certain part of
the SUT.
Another aspect to think about is how to efficiently organize test-suites if different people
work in your testing project.
3.5+
In any case you should create your test-suites within a QF-Test project as described in
(110)
chapter 10 . This feature provides a better overview over your test-suites and directories.
The following sections show some best practices how to keep your tests maintainable,
extensible and well-organized.
28.1
Organizing tests
(87)
In chapter section 9.2 we describe the concepts of Test-set and Test-case nodes. A
Test-case node stands for one dedicated test-case and its test-data. A typical Test-case
could be derived from a use-case, a requirement or a defect description in your environment, e.g ’Calculate the price for vehicle xyz at 10% discount’ for the JCarConfigurator
application.
Test-set nodes are collections of Test-sets and Test-cases which can be used for organizing
test-cases, e.g. ’Tests for calculating prices’.
Test-step nodes represent the individual test-steps of a Test-case like ’Open window’ or
28.2.
Modularization
291
’Check calculation’.
If you have an external description of the Test-case or any other associated information which might be important for it, it is recommended to add an HTML link to it
in the Comment attribute of the Test-case. You will see that link in the report later.
It is even possible to create a separate test documentation using the menu action
File→Create testdoc documentation . More details about documentation can be found
(167)
in chapter 17 .
3.1+
The report and test documentation also contain the Test-step nodes which are used in a
Test-case.
If a Test-case consists of lots of Procedure calls or Sequences, you should organize the
single test-steps in Test-step nodes. Those Test-step nodes have the advantage that you
can really see every significant step in the QF-Test window and also in the report later.
If you want to put several nodes into a Test-step you can pack the respective nodes into
a Test-step via selecting them, perform a right-mouse click and selecting
Pack nodes→TestStep .
28.2
Modularization
One of the most important concepts for effective test automation is modularization. Modularization here means placing reusable sequences in a dedicated location and calling
these whenever possible. This concept enables you to create a sequence only once
and reuse it as often as you require it in your tests without re-recording the same steps
all the time. Changes in the SUT that require an update of the tests that rely on such a
sequence, e.g. a change to some basic workflow, can then be handled by updating just
the procedure in a single location instead of many identical sequences spread all over
the test-suites.
The modularization concept in QF-Test is implemented via Procedure nodes. Procedures
(91)
are well described in section 9.5 .
If you have lots of test cases, it is best to have almost every test-step as a Procedure and
create those procedures up front, if possible. With those procedures in place you can
fill your test-cases very fast by just adding the respective Procedure call nodes.
In larger projects it is useful to have Procedures at different level, e.g. component specific
procedures like ’Click OK’ and workflow oriented procedures like ’Create a vehicle’.
28.3.
28.3
Parameterization
292
Parameterization
The concept of modularization enables you to maintain test-steps at a single location in
your test-suites. But how to use different test data for different tests?
If you have a Procedure that can be called with different test data, e.g. a typical ’Login’
process with name and password or the ’Select accessory’ procedure of the JCarConfigurator, you can use variables within the QF-Test nodes. Those variables should be
used at places where test data is usually being accessed. In most cases variables will
be used for Text input nodes or for selections of items in a list or a table or tree nodes.
If a procedure requires variables you should define the required variables in its list of
(464)
Variable definitions . This is to ensure that you get a list of all required parameters
whenever you add a respective Procedure call node for that Procedure to your tests. Some
customers even set dummy default values for parameters so they can recognize immediately when a parameter has not been initialized by the calling test.
The next step is to move those variables from the Procedure call either into the
(403)
Variable definitions
section of the Test-case node or to put the test data into a Data
driver node with a Data table or using an external data source.
(91)
The usage of variables and parameters is well described in section 9.5 . Parameters
(93)
can also be created automatically, please see section 9.5.4 . You can find more details
(159)
about the data driver concept for loading test data from a data source in chapter 16 .
28.4
Working in multiple test-suites
Up to now you have read about using the modularization and parameterization concept
to avoid unnecessary and redundant work in the creation process of your tests. You
should have recognized that those concepts will reduce the maintenance efforts of your
test-suites by changing or updating only one single sequence instead of several ones.
But we still do not know how to organize our work for different test engineers or for a
very large project with a lot of GUI elements.
The answer for an effective working organization comes again from the software development area and it is to use several ’libraries’ for different areas and different responsibilities.
Importing other test-suites into a test-suite enables you to follow that encapsulation
approach. A typical organization of test-suites in your project could look like this:
28.4.
Working in multiple test-suites
293
Figure 28.1: Structure of multiple test-suites
Level 0 is the level that contains test-steps (i.e. Procedures) which are required for nearly
all test-cases in your projects. Such test-steps could be ’Launch SUT’ or ’Perform the
login’.
Level 1 contains test-steps for a specific part of the SUT. For the JCarConfigurator you
can think about a test-suite ’Vehicles’ containing Procedures like ’Create a vehicle’, ’Remove a vehicle’ and another test-suite ’Accessories’, which contains Procedures like ’Create an accessory’ or ’Remove an accessory’.
Level 2 is the test-case level. It contains Test-cases and Test-sets for the respective area
of your software, e.g. ’Tests for vehicle creation’ or ’Tests for accessory creation’. Of
course you could also have a test-suite like ’Integration tests’ which refers to test-steps
from different test-suites at level 1 and level 0.
Level 3 is the so called scenario level. Those test-suites usually just contain Test calls
to level 2 and stand for different scenarios within your test project, e.g. ’Nightly test
scenario’, ’Defect verification scenario’ or ’Quick-test build verification’.
Note
The structure described in this document is of course just one possible solution for
handling test-suites in a project and is not a strict rule you have to follow. You could also
think about splitting level 1 into a GUI-element level and a workflow level or merge level
28.5.
Roles and responsibilities
294
2 and level 3 to one level. Which structure you finally implement also depends on the
experience and knowledge of the test-engineers working in your project.
The including area of level 1 test-suites looks like this:
Figure 28.2: Including test-suites of level 1
You can find a detailed description of how to include test-suites in section 19.1
(723)
section 40.7 .
(193)
and
(297)
you can find a step-by-step description how to extend an already
In section 29.5
(296)
existing test-suite and in section 29.4 you can find strategies of handling components
in such a scenario.
28.5
Roles and responsibilities
(290)
If you take a closer look at the organization shown in the previous section section 28.4
you may recognize that it is also possible to organize your test-suites based on different
knowledge levels of test engineers.
28.5.
Roles and responsibilities
295
Figure 28.3: Structure of different test-suites with roles
Level 0 and level 1 require a good knowledge in working with QF-Test but not a deep
knowledge of the SUT. On the other hand level 2 and level 3 require a very good knowledge of the SUT and the planned test-cases but those engineers usually do not require
a very deep knowledge of QF-Test as long as they just use procedures from level 0 and
level 1.
Test engineers working in level 0 and level 1 should be capable of implementing scripts
or control structures like the Try/Catch concept which enables them to create strong
and powerful test libraries. At least the engineers working on level 0, but also recommended for engineers working on level 1, should have a good knowledge about compo(40)
(55)
(299)
nent recognition in QF-Test. Please see chapter 5 , chapter 6 and chapter 30 .
Note
Even if you are working alone on a project it is strongly recommended to split the tests
and procedures into different levels because maintenance will become easier than with
everything kept in one huge test-suite.
28.6.
28.6
Managing components at different levels
296
Managing components at different levels
If you follow the approach suggested in the previous section (section 28.4
to define where the components belong. There are two possibilities:
(290)
) you have
1. Put all components into level 0.
2. Split the components like the procedures into several levels.
Storing all components in level 0 is the most simple solution but this could cause you to
update level 0 very often, just because one single component in your project changes.
You have to assign responsible persons to keep that structure cleanly.
In big projects you may consider storing the common components like the login dialog or
the main frame menus, that are important for everyone, in level 0. Components specific
to a certain area, e.g. a dedicated vehicle dialog, appear only in the test-suite that holds
the procedures operating on those components.
(296)
The workflow for moving components between test-suites is described in section 29.4
(297)
and the workflow for extending existing test-suites is described in section 29.5 .
28.7
3.5+
Note
Reverse includes
(396)
of test-suites belonging to a QF-Test
You don’t need to care about Dependencies
(110)
project as described in chapter 10 , because QF-Test automatically resolves dependent suites. So, if you use the concept of projects, you can skip this chapter.
If you work in different test-suites in your project you might sometimes want to rename
a Procedure or a Test-case. If you do that you may encounter some troubles in updating
the references to that Procedure or Test-case in other test-suites. If you want to keep the
other files also being updated after renaming such an element you have to maintain the
(396)
Dependencies attribute of the root node of the library test-suite.
(290)
If you follow the approach described in section section 28.4 , you should ensure that
level 0 contains a reverse include to level 1, level 1 should contain one to level 2 and level
2 should contain another one to level 3. A sample from the provided demo test-suites is
(292)
shown in figure 28.2 .
Chapter 29
Efficient working techniques
This chapter will help you in optimizing your working techniques and avoid unnecessary
steps when working with QF-Test.
29.1
Using QF-Test projects
3.5+
The previous chapter describes the creation of several test-suites. As you can image
the amount of test-suites will increase over time. You can get a better overview over
them using a QF-Test project.
QF-Test projects show all involved test-suites in a very nice way. Furthermore projects
automatically take care of propagating modifications to referring test-suites. You can
(110)
find more information about projects at chapter 10 .
29.2
Creating test-suites from scratch
In the previous chapter we described the concept of creating maintainable test-suites
by utilizing procedures and variables within QF-Test. Normally people start recording
very long sequences and splitting those into smaller parts or even procedures later. It is
very hard to split up long sequences as you have really to walk through the whole one to
find proper boundaries. Another disadvantage is that you cannot see parts which have
already been implemented in existing test-cases or procedures.
Instead of that described workflow we recommend to plan the tests and their test-steps
including procedures first. Then you can start recording procedure by procedure. We
came to the conclusion that anticipatory recording and creation is very helpful especially
for working in bigger teams. A typical workflow for creating those procedures looks like
this:
29.3.
The standard library qfs.qft
298
1. Plan the required procedures first.
2. Also plan the required package structure first.
3. Record every procedure as a separate sequence.
4. Rename the recorded sequence like the procedure should be called.
5. Transform the recorded sequence into a Procedure,
Transform node into... action from the context-menu of QF-Test.
using
the
6. Move the Procedure to the correct location.
7. Replace test data with variables, either manually or by using the parameterizer
(93)
(see section 9.5.4 )
8. Add the variables to the Variable definitions of the Procedure node, possibly specifying default values.
(109)
9. Describe the procedure in its Comment attribute, see section 9.7
.
An alternative approach of creating procedures is the automated creation provided by
(202)
QF-Test. This concept is described in chapter 20 .
29.3
The standard library qfs.qft
QF-Test provides the standard library qfs.qft which is included per default in every
test-suite.
This suite contains many useful procedures for accessing components, the file system
or a database. Please take a look at that library before you begin implementing something that has already been solved by us.
29.4
Component storage
QF-Test records new components in the test-suite where you press the ’Stop recording’
button. Therefore it could happen that you record components in the wrong test-suite.
If you want to move those components into another test-suite you must always use
File→Import from the target suite. Take care to ensure that both suites belong to
the same project or to specify correct ’Include files’/’Dependencies relations in those test(195)
suites. This workflow is described in detail in section 19.2 .
3.1+
For cleaning your component structure in a test-suite you can first import the test-suite
29.5.
Extending test-suites
299
into itself. Then you can select the Windows and components step, open the context menu
via a right mouse-click and click at Mark unused components... . You will get a list of all
components which are not used in the project. If you are sure that those components
can be removed, then select the Remove unused components in the context menu of
the Windows and components step.
Note
As soon as a Component’s comment contains any text it is considered used, even if it
has no references.
29.5
Extending test-suites
Several workflows can be followed to extend existing test-suites:
1. Simply record and extend the target test-suite directly.
(193)
2. Work with a scratch suite as described in chapter 19
.
If you extend testsuites directly via clicking at ’Stop recording’ in that suite, then you
have to care about the recorded components. In case you have changed the recorded
component hierarchy under Windows and components the newly recorded components
will be recorded in the normal hierarchy again and you have to move the new components to the optimized hierarchy. Another aspect is that it could become very difficult for
moving single components into another test-suite under the right location there.
If you work in a scratch suite you can create the new test-steps temporarily in a completely new test-suite and import the recorded components and the created procedures
and test-cases into the target suites together.
(290)
A detailed workflow for extending a test-suite from level 1 (from section 28.4
look as follows:
) could
1. Create a new test-suite.
2. Add the test-suite to be extended to the Include files area of the new one.
3. Save the new test-suite.
4. Ensure that both test-suites belong to the same project or add the new test-suite
to the Dependencies area of the test-suite to extend.
5. Record the new test-steps in the scratch suite. Also create Procedures, if required.
6. Then import the components, procedures and tests into the target test-suite as
(296)
(196)
described in section 29.4 and section 19.3 .
A more detailed description of working in multiple test-suite can be found in chapter
(193)
19 .
29.6.
29.6
Working in the script editor
300
Working in the script editor
The script editor of QF-Test contains some fancy features to avoid too much typing
actions.
If
to call methods of the run-context rc, you can simply type rc. and press
you want Ctrl-Space
,
then you will get a list of all available methods.
The auto-completion is also working for several variable names, which are:
Variables
doc
frame
iw
node
Options
rc
resolvers
Just press Ctrl-Space without typing anything
Methods
Methods of a DocumentNode.
Methods of a FrameNode.
Methods of the ImageWrapper.
Methods of a DOMNode.
The keys and values of QF-Test options to set.
Methods of the Run-context.
Methods of the Resolvers module.
A list of all variables with auto-completion.
Table 29.1: List of variables with auto-completion.
Chapter 30
How to achieve robust component
recognition
The most important feature of a GUI test-tool is the recognition of the graphical components. QF-Test offers a lot of configuration capabilities in that area. This chapter gives
an overview over the most common strategies and settings to make the component
recognition as stable as possible in a project.
(634)
We also recommend to take a closer look at section 34.13
Note
and section 40.2
(718)
.
You have to figure out the component recognition strategy before you start utilizing QFTest for a large testing area in your project. Otherwise you can get into serious troubles
(283)
in maintaining your tests. Some hints for doing that are described in chapter 27 .
30.1
Using the default configuration
In most of the projects the default configuration works very well. The default configuration takes the name of a component, its class (type) and also its hierarchy (structure of
window) into account for recognizing a component.
You could get troubles using this configuration, if
• Developers use non-unique names.
• The component hierarchy changes significantly.
• Names change significantly (see section 30.4
(304)
).
If developers do not assign any names, QF-Test uses a combined probability calculation
of properties, like the label, the text of a button or the geometry for recognizing a component. That propability calculation is quite good if at least a few properties stay similar
30.1.
Using the default configuration
302
over time, e.g. the label text. However, you should ensure that the windows of the SUT
are always opened with or resized to the same size.
The default configuration for recording components looks like this:
Figure 30.1: Default configuration for component recording
The default configuration for recognizing a component during playback looks like this:
30.2.
Using the ’Name overrides everything’ mode
303
Figure 30.2: Default configuration for component recognition
30.2
Using the ’Name overrides everything’ mode
If your developers use stable and unique names for all important components which can
be accessed by a test, e.g. buttons, tables or text-fields, you can consider changing the
configuration to the ’Name overrides everything’ mode. This has the advantage that only
names of those components and the window they belong to will be used for recognizing
the component. Changes to the SUT’s component hierarchy will the have no effect on
the recognition of those components based on their unique name.
This setting will cause trouble if:
30.2.
Using the ’Name overrides everything’ mode
304
• Developers use non-unique names.
(304)
• Names are not stable (see section 30.4
).
The ’Name overrides everything’ configuration for component recording looks like this:
Figure 30.3: ’Name overrides everything’ configuration for component recording
The ’Name overrides everything’ configuration for recognizing a component during playback looks like this:
30.3.
Using regular expressions for working with dynamic window titles
305
Figure 30.4: ’Name overrides everything’ configuration for component recognition
30.3
Using regular expressions for working with
dynamic window titles
In a lot of applications you will face the situation that the developers do not use unique
names and QF-Test keeps recording the same components again and again in different places. Playback with previously recorded components may still works unless the
window geometry changes significantly.
In this case it is very likely that the title of the main window changes frequently, e.g. to
display a version string, a user name, a file name or some other variable information.
30.4.
Influencing the names by implementing a NameResolver
306
If you want to keep your tests working and prevent recording multiple variants of this
window and all its components, you have to select the respective Window node in the
and edit its Feature attribute to replace the dynamic parts of that title with a regular
expression. Be sure to check the ’Use regexp’ box. Now your tests should work again.
Here you see the use of a regular expression for a component of the JCarConfigurator.
Its Feature attribute has to start with ’Edit’ followed by an optional dynamic part:
Figure 30.5: Using a regular expression in the Feature attribute
QF-Test uses regular expressions at many places. You can find detailed information at
(720)
section 40.4 to learn more about how to use them.
Note
QF-Test assigns the QF-Test ID of a component automatically and may use the Feature
(43)
attribute to generate it like described in section 5.3 . The QF-Test ID attribute is an
artificial concept for QF-Test’s internal use to map actions onto recorded components.
Thus it can be pleasing to change it for better readability afterwards. When changing
the QF-Test ID QF-Test provides the opportunity to adapt all references automatically.
30.4
Influencing the
NameResolver
names
by
implementing
a
In GUI testing projects you can face a lot of interesting naming concepts. Sometimes
the components in an application have no names, but the testers know an algorithm
30.5.
Handling classes of components
307
how to name them reliably. Sometimes existing names change from time to time or
are completely dynamic, e.g. you can get a name ’button1’ after the first recording and
after the second recording you get ’button2’. Another situation could be that the current
version of the application is part of the name of a dialog window.
In such cases you should take a closer look at the NameResolver interface of QF-Test.
A NameResolver can be used to change or remove names set by developers for the
QF-Test perspective. They are only removed for QF-Test not from the real source code.
You can think about utilizing NameResolvers in following cases:
• The SUT has dynamically changing names.
• You know a method to set the names uniquely.
• You want to map names to other names (e.g. due to new versions or for testing
other languages.)
• You want to tune the names of components, e.g. to remove some parts and get
nicer QF-Test component IDs in QF-Test.
If you can achieve per-window uniqueness of names with the help of a NameResolver
you can also think about switching to the ’Name overrides everything’ setting described
(301)
in section section 30.2 .
Note
Whenever possible it is preferable that developers set the names directly in their source
code as they best know the context of that component. Implementing a NameResolver
can become an excruciating task if the developers change the content of the GUI a lot.
(807)
NameResolvers are well described in section 43.1.6
30.5
4.0+
.
Handling classes of components
QF-Test records so-called generic classes, which means common class names for objects independent of their technical class.
A sample recording for Swing components leads to the generic class Button for any
javax.swing.JButton even if you have extended that button class. For other engines you will get a similar recording with the respective classes of that toolkit. In addition this concept allows QF-Test to resolve obfuscated classes without any need to
change the default configuration.
During replay QF-Test compares the ’class’ attribute of the recorded component with
any class of the objects in the SUT. Therefore QF-Test can cope with class changes as
long as the standard type isn’t changed.
(45)
You can find a brief explanation of the generic class concept in section section 5.4.1 .
30.5.
Handling classes of components
30.5.1
4.0+
308
Handling class changes due to re-factoring by using
’Record system-classes only’
This chapter typically should be relevant just for QF-Test versions older than 4.0 where
no generic classes can be used.
By use of this option you can configure QF-Test to record the system base classes of
components instead of custom class names. System classes are javax.swing...
and java.awt... classes for Java/Swing and org.eclipse.swt.widgets...
classes for Java/SWT applications.
Figure 30.6: Option to configure recording of system classes only
Having just base classes, you will be safe against re-factoring steps of the developers
who may change the class structure or rename classes. This option will also protect you
from obfuscators which change classes names to protect them against re-engineering.
30.6.
Avoiding recording every component or using generic components
309
Having this option active should be helpful in most cases. Only if custom class names
are vital for a proper component recognition of components, i.e. if development did not
assign useful names but implemented different custom classes for specific containers
you might want to consider a deactivation.
3.1+
A mixed approach is also possible by implementing a ClassNameResolver. This can be
useful you only want to get system classes for some components, but not all, e.g. when
(809)
using a third-party library. Please see section 43.1.7 for more details.
30.6
Avoiding recording every component or using
generic components
By now you know how to optimize the component recognition for your needs, but you
still have to record every component before you can use it in your tests.
As you can imagine, it can be tiring to record each and every component required for
a test. Sometimes this may even become impossible, e.g. when you have changing
identifiers or changing labels. A typical situation could be localization testing of an
application.
Another situation could be the usage of a GUI framework of your developers. You could
have a lot of very similar dialogs, where only some components are different. But you
still have to record all parts, which are actually the same, like navigation buttons.
QF-Test offers a feature to reduce the recording of components to a minimum. This
concept is called ’Generalizing components’. It consists of making use of variables in
the component’s details or simply to remove dynamic parts of it.
The general approach for generalizing components is as follows:
1. Record some components that you want to generalize and compare them.
2. Create one generic component with an ID, which contains ’generic’ so that you
can recognize it later.
3. Remove all attributes that you do not want to use in the recognition in that generic
component.
4. Determine the main properties that you want to use for recognition, e.g. ’Name’,
’Feature’ or ’Class index’.
5. Set a variable for the respective attribute, e.g. $(name).
6. To prevent false positive matches, disable recognition based on geometry by specifying ’-’ for the ’X’ and ’Y’ attributes.
30.7.
Switching component recognition settings dynamically
310
7. Specify ’@generic’ in the Comment attribute for the generic component to prevent
it from inadvertently being removed by the ’Remove unused components’ action.
8. Create a procedure accessing the generic component and use the variable(s) from
the previous step as parameter(s).
Note
Generic components are very handy for replaying tests, but QF-Test will not use them
for recording. You will always get the real component recorded in the test-suite, but can
then change the created sequence to make use of the respective generic component.
30.7
Switching
dynamically
component
recognition
settings
3.1+
Since QF-Test version 3.1 it is even possible to change component recognition options
dynamically. Thus you are very flexible and you can combine different settings for your
SUT.
Setting options at runtime can be done via a Server script or SUT script - depending on the
option - and calling rc.setOption(OPTION_NAME, OPTION_VALUE). All possible
(333)
(349)
options can be found in section 33.2 for recording and in section 33.3 for replaying
tests.
Chapter 31
Hints on setting up test-systems
This chapter provides some hints how to set-up your test-systems and processes in
order to get a stable test execution.
31.1
Using the task scheduler
In order to execute tests or similar processes on a regular basis it is very common to setup a Windows service. Unfortunately those services have the disadvantage that they
don’t run in an active Windows user session. Because of this such processes should
never be started as services, because running GUI tests without an active user session
brings up very subtle problems during test-execution. You can find further technical
details at FAQ 14.
We recommend to define a scheduled task via the task scheduler instead of using services. This can be directly applied via the graphical UI of the task scheduler. The following instruction should work on Windows 7, Windows 8, Windows 8.1 and Windows
10. There might be a few differences depending on the exact operation system:
1. Start the Windows Task scheduler via ’System Control Panel’ -> ’Administrative
Tools’ -> ’Task Scheduler’.
2. Click ”Create Task” on the right.
3. At the ”General” tab define a ”Name”, e.g. QF-Test.
4. Now click at the ”Change User or Group” button and select the user for the session
which should be used to run QF-Test. Do not use the System- or Service session
but a real user session.
5. Press OK to close the dialog.
31.2.
Remote access to Windows systems
312
6. Now select ”Run only when user is logged in”.
7. Do not select ”Run with highest privileges”.
8. Choose the correct Windows version for your task.
9. Now select the ”Triggers” tab. Click ”New..” and define the desired time/trigger at
”Begin the task”.
10. Click ”OK” to close.
11. Now select the ”Actions” tab, click at ”New.” and specify ”Start a program” as
”Action” and ”Browse” to the .cmd or .bat file you’ve just created.
12. Press ”OK” to close this dialog.
13. You can now have a look at the ”Conditions” and ”Settings” tab if you need anything
else from your side.
14. If you have finished the setup press ”OK” and the task has been created.
It is very important that the user which is configured to run that process is logged in
(311)
correctly. This user can be logged in manually or automatically (see section 31.3 ). It
is recommended to use a virtual system for running GUI tests. On such virtual systems
the user needs to be logged in only on the guest system and the host can be locked.
31.2
Remote access to Windows systems
Accessing remote Windows systems via RDP is subject to some restrictions and requires a dedicated configuration of your systems. That’s because the implemented
RDP server of Windows desktop versions allows only one active user. Thus the (virtual)
monitor will be locked as soon as you access the system via RDP. After closing the
RDP session the monitor of the test-system remains locked. Usually you cannot use a
graphical user interface at a locked screen, thus you cannot test it, too.
Note
On Windows 10 or Windows Server 2016 systems you can make
use of RDP if you modify the Registry.
Therefore navigate to
HKEY_CURRENT_USER\Software\Microsoft\Terminal Server Client
or
HKEY_LOCAL_MACHINE\Software\Microsoft\Terminal Server Client and
add a new value RemoteDesktop_SuppressWhenMinimized as DWORD having the
value 2. Once that setting has been set you are allowed to minimize RDP connections,
but you have to keep the connection alive. The tests will still fail if you disconnect or
close the session.
31.3.
Automated logon on Windows systems
313
Instead of this approach you should use the capabilities your virtual server provides.
For VMware server the vSphere client would be the first choice. With VirtualBox you
can connect to VirtualBox with RDP (not with the Windows client). Of course this kind
of RDP connection has not the impacts on the test-system as explained above.
31.3
Automated logon on Windows systems
A simple possibility to get an active user session is to logon a test user automatically
after start-up of your system. This chapter describes how to configure your system for
that purpose.
Note
An automatic logon at Windows is always a security risk. Therefore you have to ensure
that the corresponding test-systems are not accessible from outside the test environment.
Although this guidance is generally valid it will be used commonly together with virtual
systems which will be accessed remotely. What to keep in mind for this remote access
(310)
will be explained at section 31.2 .
The following method is running with Windows 7, Windows 8, Windows 8.1 and Windows 10. There might be a few differences depending on the exact operation system:
1. Start the command line interface with administrator privileges.
2. Enter control userpasswords2.
3. Now the dialog ’User Accounts’ appears.
4. Remove the check at the checkbox ’Users must enter user name and password’.
5. In the next appearing dialog (’Automatic Logon’) enter the user password twice.
6. Click the ’Ok’ button to finish the configuration.
Of course also other methods exist to get the same result. Thus you could modify
the corresponding registry entry directly. Or you could download the ’Autologon’ tool
from Microsoft from https://technet.microsoft.com/en-us/sysinternals/bb963905. But all
those different methods lead to the same result, which is the modified registry entry. We
recommend to use the method explained above as in this way no download is needed
and a type mismatch in the registry entry is avoided. By the way, an automated logon
will never run for domain users. In fact this would be quite awkward in conjunction with
test-system. This may be an information which may calm your administrators down.
31.4.
31.4
Test execution on Unix
314
Test execution on Unix
On Unix systems you can set-up virtual displays using tools like VNC server. A very
useful window manager for those displays could be xfce.
Chapter 32
Test execution
This chapter gives some hints about how to implement your tests to get stable and
reliable test execution.
32.1
Dependencies
The ’Dependencies’ concept of QF-Test provides functionality to guarantee that all prerequisites for a test-case are fulfilled before running it. It is also capable of reacting to
unexpected behavior, e.g. closing an error dialog, which pops up and blocks your tests.
(426)
The concept is described in section 34.3
in the chapter ’Dependencies’.
and a use-case can be found in the tutorial
You should at least implement a Dependency which is responsible for launching the SUT,
containing a Setup for launching, a Cleanup for a normal exit and a Catch to react on any
unexpected behavior.
Note
If you implement a Cleanup, try to close the SUT normally first and only if the SUT does
not terminate correctly, kill it via Stop client.
For
SWING
and
SWT
applications
please
use
the
procedures
qfs.cleanup.swing.closeAllModalDialogs
and
qfs.cleanup.swt.closeAllModalDialogsAndShells from the standard library
qfs.qft for closing unexpected error dialogs.
32.2
Timeout vs. delay
Instead of using the ’Delay before’ and ’Delay after’ attributes you should try to use
QF-Test’s synchronization nodes to optimize test execution time.
32.3.
What to do if the run-log contains an error
316
The first kind of synchronization nodes are the ’waiter’ nodes like Wait for component to
appear, Wait for client to connect, Wait for document to load and Wait for process to terminate.
You can specify the Timeout attribute to wait for a component, process or document. The
Wait for component to appear node even provides the functionality to wait for the absence
of a component.
The second kind are the ’check’ nodes which allow you to specify the Timeout attribute
as well. Those nodes can be used to continue the test when a GUI element of your SUT
has reached a defined state.
32.3
What to do if the run-log contains an error
If the test report contains an error message or exceptions, the following steps should be
performed to find the source of that failure very fast:
1. Analyze the run-log, especially the screenshots and any other messages.
2. If you cannot find
the cause immediately, jump to the failing location in your testsuite by typing Ctrl-T in the run-log.
3. Set a breakpoint before or at the failing step.
4. Ensure that the debugger of QF-Test is enabled.
5. Run the failing test.
6. When QF-Test reaches the breakpoint and stops, open the debugger window and
check the active variable bindings to wee whether they contain any wrong values.
7. Perhaps at that time you can also see the error immediately in your SUT.
8. If you cannot see any source of that error, run the failing step.
9. If you still encounter errors you might have to re-debug some steps executed before the failing step. Use the ’Continue execution from here’ menu entry to jump to
previous steps instead of re-running the whole test again.
3.1+
Since QF-Test version 3.1 it is possible to mark nodes via the context menu-item
Set mark or setting bookmarks for specific nodes via the menu item Add bookmark .
These features enable you to find important nodes very fast again.
If you encounter problems with component recognition, please see section 5.9
(299)
chapter 30 .
(52)
and
Part III
Reference manual
Chapter 33
Options
Since QF-Test is a tool that is intended for a wide range of applications, the ”one size
fits all” approach doesn’t quite work. That’s why QF-Test has a great number of options
that control its functionality.
There are two kinds of options for QF-Test: user options and system options. User
options adjust the behavior of QF-Test’s own GUI while system options influence how
tests are recorded and replayed. Each user has its own set of user options whereas
(9)
system options are saved in a common system file. See section 1.6 for details about
configuration files.
3.1+
Many options can have their value changed at run time from a script via rc.setOption
(729)
as described in section 41.6 . Depending on whether the option takes effect in QFTest itself or in the SUT, the documentation for those options shows a ”Server script
name” or ”SUT script name” matching the constant from the Options class. Obviously
(502)
(504)
the option has to be set in a matching Server script or SUT script node. Where the
option’s value can be selected from a drop-down list, the documentation also lists the
constants that can be specified as the option’s value.
Though the number of options may look daunting, don’t let yourself be deterred by it. All
options have reasonable default values, so QF-Test works well out of the box for most
cases. However, if you find you need to change something or simply want to explore the
range of QF-Test’s abilities, this chapter is for you.
The options can be set in the dialog available through the menu item Edit→Options... .
The settings are saved in two configuration files, one for personal settings and one for
(9)
system-wide settings (see section 1.6 ).
Options
319
Figure 33.1: Options tree
33.1.
General options
320
To get at an option, first select the appropriate node of the tree. The options for that
topic are then displayed in the right part of the view. When switching from one group to
the other, the current values are verified but not adopted yet. This happens only after
confirmation with the OK button.
33.1
General options
This is the node for general QF-Test settings.
Figure 33.2: General options
Ask before closing (User)
When a test-suite or a test run-log has been modified, QF-Test asks whether it
should be saved before it closes its main window. That query can be suppressed
by turning off this option. Be warned that auto saving is not implemented yet, so
you may lose data if you forget to save before closing.
Ask before overwriting (User)
33.1.
General options
321
When you try to save a test-suite or a run-log or generate a report, pgkdoc or
(581)
testdoc or save the image of a Check image
over an existing file or directory,
QF-Test asks for confirmation unless you turn off the option for the respective
type of file.
Restore last session on startup (User)
If this option is set and QF-Test is opened in the workbench view, the previous
session is restored by loading previously opened test-suites and selecting the
previously selected node in each suite. If one or more test-suites are specified on
the command line, this are loaded in addition to the previous session and receive
the initial focus on startup.
Discard empty suite after loading (User)
A common situation in daily work is that a test-suite is loaded right after starting
QF-Test. In that case the initial empty test-suite is typically unwanted and even a
burden. Setting this option will cause the initial suite to be closed automatically.
Number of recent files in menu (User)
The File menu offers quick access to recently used test-suites or run-logs. This
option determines the maximum number of recent file entries in the menu.
Default script language for script nodes (User)
This option can be set to either ”Jython”, ”Groovy” or ”JavaScript” and determines
(503)
attribute of newly created
the default setting for the Script language
(502)
(504)
Server script or SUT script nodes.
Default character encoding for Jython (System)
(502)
This option sets the default encoding for Jython scripts in Server script
and
(504)
SUT script
nodes as well as the interactive Jython terminals for QF-Test and
the SUT. The default encoding for standard Jython is ’ascii’. To ease migration
and improve two-way-compatibility with older QF-Test versions, the default setting
for QF-Test is ’latin-1’. You can also experiment with other settings that better suit
your locale.
For background information please see http://docs.python.org/library/codecs.html.
Note
This setting does not affect the encoding for Python modules in separate files. If
you get an error that a module cannot be loaded because no encoding is
declared, you probably need to add a line of the form
33.1.
General options
# coding:
322
latin-1
close to the top of that file. Please see http://www.python.org/peps/pep-0263.html
for details.
Use native file chooser on Windows or macOS systems (User)
3.5+
Server script name: OPT_USE_NATIVE_FILECHOOSER
On Windows or macOS systems the native file chooser is more advanced and
more convenient to use than the Swing file chooser so QF-Test uses the native
one by default. In case you prefer the Swing file chooser you can get it back by
deactivating this option.
Show complete file path and QF-Test version in the title bar (User)
4.1.3+
If set, QF-Test shows the full path of the current test-suite and the QF-Test
version in the title bar of the main window.
33.1.1
Projects
3.5+
There are several options that influence the way QF-Test manages and displays
projects.
Figure 33.3: Projects
Project refresh interval (s) (User)
The interval at which a project automatically gets completely refreshed. You can
33.1.
General options
323
refresh a directory at any time by selecting it and pressing
F5 . To refresh the
complete hierarchy below the selected directory, press Shift-F5 instead.
Number of test-suite to open in one go without warning (User)
From the project tree you can open all test-suites contained in one directory
hierarchy in one go. If you accidentally select too many test-suites, QF-Test will
first issue a warning with the number of test-suites, allowing you to cancel that
action. This option determines the threshold for that warning.
Project files and directories to exclude (System)
In many cases a directory hierarchy holds files and directories that don’t really
belong to a project, most notably sub-directories created by version control
systems like subversion or cvs. In this option you can specify patterns for files
and directories to generally exclude from projects.
The patterns used here are not regular expressions but a simpler form often used
by development tools: An ’*’ stands for any number of characters up to the next
file separator - for compatibility reasons only forward ’/’ is used - while ’**’ means
0 or more characters of any kind, including ’/’. Every pattern is relative to the root
directory of the project. Some examples:
**/.svn
All directories named .svn at any depth.
**/.*
All directories starting with a ’.’ at any depth.
deprecated
A directory named deprecated directly below the project root.
33.1.2
Editing
These options are used to configure various settings regarding editing in the tree or
detail view.
33.1.
General options
324
Figure 33.4: Editing
Warn when modifying a test-suite that cannot be saved (User)
If saving test-suites is prohibited, for example when working without a license,
QF-Test will warn you that you will not be able to save your changes when a
test-suite is modified for the first time. Deactivating this option suppresses that
warning.
Ask before discarding detail modifications (User)
4.0+
When you have started making
changes
to an existing or newly inserted node
and then abort by pressing Escape or clicking the ”Cancel” button, QF-Test asks
for confirmation before discard your modifications. This dialog can be suppressed
by disabling this option. In this case, please be aware that - especially in case of
scripts - a lot of work may get lost in case of a mistake.
Ask before implicitly accepting detail modifications (User)
33.1.
General options
325
A very common mistake made while editing a test-suite is to forget pressing OK
after making changes in the detail view of a node before switching to some other
node, running a test, etc. If that happens QF-Test can either accept the modified
values automatically or ask for confirmation by popping up a dialog with the detail
view. The following options are available:
Always
Don’t accept values implicitly, always ask for confirmation.
Only if values are suspect or invalid
Try to accept values implicitly as long as they are valid and not suspect.
Currently ”being suspect” is defined as having leading or trailing whitespace
which can lead to subtle problems which are very hard to locate.
Never
Accept all valid values implicitly without asking for confirmation.
This option doesn’t change the effect of explicitly
discarding your modifications
.
with the Cancel button or by pressing Escape Number of undo levels per suite (User)
This option lets you set the number of edits that can be undone in a test-suite or
run-log.
Intelligent scrolling in trees (User)
The default methods for interacting with Swing trees are not ideal. Moving the
selection around causes a log of unnecessary horizontal scrolling and Swing has
the tendency to scroll trees to a position where little context is visible around the
selected node.
Because tree navigation is essential for QF-Test, some of these methods are implemented differently to provide a more natural interface and to make sure that
there is always enough context visible around the selected node. However, your
mileage may vary, so if you don’t like the alternative methods you can switch back
to the default Swing way of things by deactivating this option.
Syntax highlighting for tree nodes (User)
4.0+
This option controls activation of syntax highlighting for tree nodes within
test-suites and run-logs. If active, specific text parts of nodes (e.g. node name,
33.1.
General options
326
parameters, client) are outlined in different colors and styles. This significantly
improves readability.
Minimum font size (pt) (User)
This option lets you set the minimum font size (as point value) used within
QF-Test. A change in this value becomes operative after restarting QF-Test.
Show line numbers in script nodes (User)
(504)
If this option is set, line numbers are shown in SUT script
nodes.
(502)
and Server script
Show symbols for tabulator and linebreaks (User)
3.5+
If this option is set, QF-Test shows symbols for tabulator and linebreaks in tables
and relevant textareas.
Check references before deletion (User)
3.4+
If this option is set, QF-Test searches for references of nodes before nodes will be
deleted. If references can be found, they will be shown in a dialog.
Ask after changing QF-Test component IDs or use default (User)
3.5.3+
If this option is set, QF-Test asks whether the user wants to update the QF-Test
component IDs of any referring node after the QF-Test ID of a component has
been changed. If this option isn’t set QF-Test updates all references in case of
unique QF-Test component IDs.
Ask after changing callable names or use default (User)
3.5.3+
If this option is set, QF-Test asks whether the user wants to update the callable
names (i.e. procedures, packages, tests and dependencies) of any referring node
after the name of a callable node has been changed. If this option isn’t set
QF-Test updates all references in case of unique names.
Check if configuration files are writable (User)
4.1.2+
If this option is set, QF-Test checks whether the configuration files have writing
permissions once opening the ’Options’ dialog. If one configuration file has no
writing privileges, QF-Test will show a message.
33.1.
General options
33.1.3
327
Bookmarks
Here you can edit your bookmarks, a list of files and nodes that can be accessed quickly
via the menu File→Bookmarks .
4.0+
Instead of a file you can also specify a directory. When the respective bookmark is
selected, the file selection dialog is opened directly for this directory. The QF-Test ID for
the node is ignored in this case.
Figure 33.5: Bookmarks
Though you can also create new bookmarks manually, it is preferable to use the menu
item File→Add to bookmarks to add a bookmark for a whole test-suite or run-log or
to select Add to bookmarks in the context menu of a node in a test-suite to add a
bookmark for this specific node.
33.1.4
External tools
The following options determine which external programs are called by QF-Test.
33.1.
General options
328
Figure 33.6: External tools options
External editor command (User)
Scripts can be edited in an external editor by pressing Alt-Return or by clicking
the
button above the text area. The contents of the text area are then saved
to a temporary file and the external editor is run to edit that file. It is
recommended to define a name for the script before opening it in the external
(327)
editor (see also Warn when running external editor without file name ).
Otherwise a random number is chosen as file name, which makes it difficult to
distinguish several scripts opened in the external editor.
Changes made to an external file are picked up automatically by QF-Test. Depending on your settings, you may get a warning message when this happens (see
(327)
Warn when test-suite is changed by an external editor ). In case you are tempted
to edit your script code parallel in the internal QF-Test editor: These changes are
also saved in the temporary file. Editors like jEdit on their part are smart enough
to detect the change and reload the file automatically.
This option determines the external editor command to use. There are two variants, the plain name of an executable file or a complex command including options. The latter is distinguished by the string $(file) which is the placeholder for
the name of the temporary file. Additionally, $(line) may be used to pass the current line number to the editor as well.
Note
The $(file)/$(line) syntax is used simply to avoid yet another different convention
33.1.
General options
329
for variable attributes. No standard QF-Test $(...) variable expansion is taking
place.
Plain commands need never be quoted. Examples are:
• emacsclient
• notepad
• C:\Program Files\Crimson Editor\cedt.exe
Complex commands on the other hand may need to use quotes, especially on
windows. QF-Test takes care of quoting the $(file) argument itself:
• ”C:\Program Files\eclipse-3.6\eclipse.exe” –launcher.openFile $(file)
• javaw.exe -jar C:\Programs\jEdit4.2\jedit.jar -reuseview $(file)
• ”C:\Program Files\Crimson Editor\cedt.exe” $(file)
• xterm -e vi +$(line) $(file)
If this option is left empty, the value of the environment variable EDITOR is used,
if it is defined when QF-Test is started.
Directory passed temporary files to external editor (User)
4.1+
This option can be used to change the directory in which QF-Test saves
temporary files for opening in an external editor (see External editor
(326)
(9)
command ). If empty, the user configuration directory is used.
Warn when test-suite is changed by an external editor (User)
Display a warning message when changes to a script made by an external editor
(326)
are picked up by QF-Test (see also External editor command ).
Warn when running external editor without file name (User)
Display a warning message when a script without name is opened in an external
(326)
editor (see also External editor command ).
External imaging program (User)
(583)
(581)
The Image
of a Check image
node can be edited in an external imaging
program. The image is saved to a temporary PNG file and the external imaging
program is run to edit that file. When finished editing, the file must be saved and
the program exited. QF-Test will read the image back from the temporary file.
33.1.
General options
330
This option determines the program to use for the operation. There are two variants, the plain name of an executable file or a complex command including options.
The latter is distinguished by the string $(file) which is the placeholder for the name
of the temporary file.
Note
The $(file)/$(line) syntax is used simply to avoid yet another different convention
for variable attributes. No standard QF-Test $(...) variable expansion is taking
place.
Plain commands need never be quoted. Examples are:
• gimp
• mspaint
• C:\Windows\System32\mspaint.exe
Complex commands on the other hand may need to use quotes, especially on
windows. QF-Test takes care of quoting the $(file) argument itself:
• gimp –no-splash $(file)
• ”C:\Windows\System32\mspaint.exe” $(file)
HTML browser (User)
This option allows you to set the HTML browser used to open HTML pages (e.g.
report files or the context sensitive help). You can specify a complex command
using ’$url’ as placeholder for the URL to show, e.g.
netscape -remote openURL($url)
or just a simple command like
firefox
in which case the URL is passed as the last argument. If the option is empty the
system browser is used.
PDF reader (Unix only) (User)
This option allows you to set the program used to display PDF files. This is
necessary to open the PDF manual from the Help menu. If the option is empty,
the program associated with the .pdf extension is used.
33.1.5
Backup files
Unless told to do otherwise, QF-Test creates backups of existing files when saving testsuites or run-logs. These backup files are useful only in protecting against failures when
33.1.
General options
331
saving a file. They are by no means a replacement for proper system backups. The
following options determine if, when and how backup files are created.
Figure 33.7: Backup file options
Create backup files for test-suites (User)
Backup files for test-suites are created only if this option is activated. Please be
careful and don’t turn it off without a good reason, such as using a version control
system for test-suites which obviates the need to create backups. Just think
about the amount of work that goes into creating a useful test-suite and imagine
the frustration if it gets destroyed accidentally.
Create backup files for test run-logs (User)
Usually a run-log is far less valuable than a test-suite, so there is a separate
option that determines whether backups are created for run-logs.
Backup frequency (User)
There are two possibilities for the frequency with which backup files are created.
With the first option, ”One backup per session”, QF-Test creates a backup file only
the first time a file is saved. If you continue editing the suite or run-log and save
33.1.
General options
332
it again, the backup file is left unchanged. Only when you edit a different file or
restart QF-Test, a new backup is created. This setting is useful is you keep only
one backup per test-suite.
If, on the other hand, you keep multiple backups per suite, ”Backup on every save”
may be the preferred choice.
Name of the backup file (User)
Like many other things, the conventions for the names of backup files differ
between Unix and Windows. While the common extension for a backup file under
Windows is .bak, there are many variants under Unix. One of the most common
is appending a tilde character ’∼’.
Number of backup files to keep (User)
You can keep more than one backup file for each test-suite or run-log. If you do
so, backup files are named after the scheme .bak1, .bak2... for the .bak
naming style and ∼1∼, ∼2∼... for the other. The most recent backup is always
numbered 1. When a new backup is created, the old number 1 is renamed to 2, 2
renamed to 3 and so on. When the maximum is reached, the oldest files are
deleted.
Auto-save interval (s) (User)
Interval after which a modified test-suite is saved automatically. Setting this value
to 0 will disable auto-saving. Otherwise values less than about 20 seconds are
not useful. Run-logs are never saved automatically. Auto-save files are created in
the same directory as the test-suite or - in the case of new suites that have never
(9)
been saved - in the user configuration directory .
33.1.
General options
33.1.6
333
Library path
Figure 33.8: Library path option
Directories holding test-suite libraries (System)
This is a list of directories that are searched for test-suites whenever a suite
reference is given as a relative path that cannot be resolved relative to the current
(467)
attribute of a
suite. This includes direct suite references in the Procedure name
(466)
(644)
(645)
Procedure call
or a Component
QF-Test ID
reference as well as suites
(396)
(395)
included through the Include files attribute of the Test-suite node.
The include directory belonging to the current version of QF-Test is automatically
and invisibly placed at the end of the library path. This ensures that the common
library qfs.qft can always be included without knowing its actual location and
that its version is matching the version of QF-Test at all times.
Note
(683)
If the command line argument -libpath <path>
is given it will override the
settings of this option. In interactive mode, the value of the command line argument is displayed here, but it will not be saved with the system configuration unless
it is modified.
33.1.
General options
33.1.7
334
License
Figure 33.9: License options
Normally QF-Test license bundles contain a homogeneous mix of GUI engines. For
example, a bundle of QF-Test/swing licenses only supports the AWT/Swing engine, QFTest/suite licenses support both AWT/Swing and SWT for all included licenses. For
these kinds of simple licenses these license settings can be ignored.
A small problem arises in case of mixed engine licenses where some GUI engine is
included only for a part of the licenses. An example for this is a license bundle that was
formerly purchased for qftestJUI, upgraded to QF-Test/suite with QF-Test 2.0 and then
extended with further QF-Test/swing licenses, say two licenses for QF-Test/suite and
another two for QF-Test/swing. Such a license allows running four concurrent instances
of QF-Test, but only two of these can make use of the SWT engine. If more than two
instances are started with SWT support there will be a license conflict.
When QF-Test detects such a mixed license bundle for the first time it asks the user
which engine licenses to use. The choice made then can be changed here at any
33.2.
Recording options
335
time.
Besides, QF-Test can be started with the command line argument
(680)
-engine <engine> to override the supported GUI engines for this execution.
33.2
Recording options
The following options determine which kinds of events are recorded and which filters are
applied, how components are recorded and arranged, and how sub-items are handled.
Figure 33.10: Recording options
Show initial quickstart help on record button (User)
Controls the display of an initial question mark on the record button in order to
directly lead new users to the quickstart wizard.
Hotkey for recording (User)
SUT script name: OPT_RECORD_HOTKEY
Event recording can be directly started/stopped by pressing a key in the SUT.
The
key to be used for this function is set through this option. The default key is F11 .
Keep variables in recorded client names (System)
This option is very useful if the client name assigned to your SUT contains
variables (e.g. $(client)), which generally makes sense when creating
procedures. If this option is set, the client attribute of all recorded nodes is set to
(512)
(511)
the unexpanded Client
value of the Start SUT client
node through which the
SUT was started.
33.2.
Recording options
336
Insert recording at current selection (User)
Depending on what you are currently working on, it may or may not make sense
to add newly recorded sequences directly at the current insertion mark. If you
(415)
deactivate this option, new recordings are placed in a new Sequence
under
(425)
Extras .
Hotkey for checks (User)
SUT script name: OPT_RECORD_CHECK_HOTKEY
To simplify recording a sequence of events with interspersed checks, you can
switch between plain recording and recording checks by pressing a key in the
SUT.
The key that triggers this switch is set through this option. The default key is
F12 .
Highlight components when checking (User)
SUT script name: OPT_RECORD_CHECK_HIGHLIGHT
When QF-Test is recording checks, it can give visual feedback on the component
the mouse is currently over by inverting its foreground and background colors.
Rarely this may have unwanted visual side effects, so you can turn that feature off
with this option.
33.2.1
Events to record
These options specify which kinds of events are recorded and which aren’t. You should
not tinker with these unless you know what you are doing.
Figure 33.11: Options for events to record
33.2.
Recording options
337
Abstract ’Mouse click’ events (System)
Activating this option causes a sequence of MOUSE_MOVED, MOUSE_PRESSED,
MOUSE_RELEASED and MOUSE_CLICKED events to be recorded as a single
(538)
’Mouse click’ pseudo event (see section 34.8.1 ).
Simplified ’Mouse click’ recording (System)
4.1+
When recording ’Mouse click’ events, this option should also be activated. Except
for Drag&Drop and special case MOUSE_MOVED events, recording is then based
primarily on MOUSE_PRESSED events, turned into ’Mouse clicks’. This gives best
results in most cases, even when QF-Test receives too few or too many events
from the SUT. If this option is turned off, the algorithm from QF-Test 4.0 and older
is used. This is worth a try in case a recorded sequence cannot be replayed
directly.
Abstract ’Keystroke’ events (System)
This option lets you record a sequence of KEY_PRESSED, KEY_TYPED and
KEY_RELEASED events (or just KEY_PRESSED and KEY_RELEASED for function
(541)
keys) as a single ’Keystroke’ pseudo event (see section 34.8.2 ).
Record MouseEvents without coordinates where possible (System)
SUT script name: OPT_RECORD_REPOSITION_MOUSE_EVENTS
For many types of components and sub-items it doesn’t matter where exactly a
MouseEvent occurs. However, if large values are recorded for the X or Y
(538)
coordinate of a Mouse event , there’s the danger that the event might miss its
target upon replay if the component has shrunk a little due to font changes or
because the window has been resized. This is also a possible source for
(463)
with variable
problems when converting a recorded sequence to a Procedure
target components.
If this option is activated, QF-Test ignores the coordinates of recorded
MouseEvents if it thinks that the coordinates don’t matter for the target
component, e.g. for all kinds of buttons, for menu items, table cells, list items and
tree nodes. For the latter QF-Test distinguishes between clicks on the node itself
and on the expand/collapse toggle. When MouseEvents without coordinates are
played back, QF-Test targets the center of the respective component or item
except that the X coordinate for items is limited to 5 because item bounds cannot
always be calculated correctly.
(604)
Convert opening of a window into Wait for component to appear
(System)
33.2.
Recording options
338
When replaying a sequence during which a new window is opened, it may be
useful to allow for a longer than usual delay until the window is opened. By
activating this option, a recorded WINDOW_OPENED event will be turned into a
(604)
Wait for component to appear node automatically.
Web
(608)
For web clients this option causes a Wait for document to load node to be inserted
whenever loading of a document completes. This is important for proper synchronization when navigating to another page.
33.2.2
Events to pack
In order to keep the amount of raw event data generated during normal use of a Java
GUI manageable, QF-Test employs a set of recording filters and packers. These do
their best to keep everything needed for successful replay and throw away the rest.
To get an impression of the actual data behind a recorded sequence, try recording a
(334)
(335)
short sequence with all of these turned off, and with ’Mouse click’ and ’Keystroke’
pseudo events disabled.
Figure 33.12: Options for events to pack
MOUSE_MOVED events (System)
SUT script name: OPT_RECORD_PACK_MOUSE_MOVED
MOUSE_MOVED events are especially frequent. Every mouse cursor motion
generates a handful of these. Under most circumstances, only the last of a
33.2.
Recording options
339
consecutive series of MOUSE_MOVED events is actually useful, so all events
except the last one are dropped, if this option is activated. An example where this
is not advisable is recording some freehand drawing in a graphics application.
Note
One might think that MOUSE_MOVED events are completely useless, since
MOUSE_PRESSED or MOUSE_RELEASED events have their own set of coordinates,
but this is not the case. Some Java components require a MOUSE_MOVED event
before a MOUSE_PRESSED event is recognized.
MOUSE_DRAGGED events (System)
SUT script name: OPT_RECORD_PACK_MOUSE_DRAGGED
MOUSE_DRAGGED events are like MOUSE_MOVED events, but with one mouse
button held down. Similarly, only the last of a series of MOUSE_DRAGGED events is
recorded unless you turn off this option, except for special cases (see next
option).
Mouse drag hover delay (System)
SUT script name: OPT_RECORD_MOUSE_DRAGGED_HOVER
There are situations where not only the final target of a mouse drag is of interest,
but intermediate points as well. The most common is invoking a menu item in a
sub-menu.
Note
As of QF-Test 1.05.2, the following no longer applies because MOUSE_MOVED or
MOUSE_DRAGGED events that are required for opening sub-menus are not ”optimized away” anymore. However, there may be other situations where intermediate
stops are useful when recording drags.
Figure 33.13: Dragging to a sub-menu
33.2.
Recording options
340
As illustrated above, creating a new Test call node for a suite could be done
by clicking on the Insert menu button, dragging the mouse to the
Test and Sequence nodes item for the sub-menu, so the sub-menu pops up,
then dragging on to the Testcall menu item and releasing the mouse button.
Normally such a sequence would be reduced to a press, a drag to the final
Testcall item and a release. It would fail to replay, since the sub-menu
would never be popped up. To work correctly, an additional drag to the
Test and Sequence nodes item must be recorded.
For that reason QF-Test recognizes a MOUSE_DRAGGED event as important if you
hover over an intermediate component for a while during the drag. The delay (in
milliseconds) needed to recognize such an event is set through this option.
To record the above example correctly with this option set to 1000, you’d have to
click on the Insert menu button, drag to the Test and Sequence nodes item and
keep the mouse pointer stationary for one second, then move on to the TestCall
sub-menu item and release the mouse button.
Maximum drag distance for ’Mouse click’ event (System)
It sometimes happens unintentionally that the mouse cursor is moved between
pressing the mouse button and releasing it. This movement may be registered as
a MOUSE_DRAGGED event, depending on the JDK version and the distance of the
move. QF-Test is able to compensate for small movements and still convert the
click into an abstract ’Mouse click’ event. This option defines the maximum
distance between pressing and releasing the mouse button that QF-Test will
ignore. Every MOUSE_DRAGGED event above that distance will be left unchanged.
Collect key events into a Text input node (System)
Another example where a lot of events are generated is entering a short string of
text into a text field. Each character typed leads to at least one KEY_PRESSED,
one KEY_TYPED and one KEY_RELEASED event. For additional fun, the
KEY_RELEASED events may arrive out of order or not at all, depending on
operating system and JDK version.
If this option is activated, sequences of KeyEvents on a text component (to be
exact: a component whose class is derived from java.awt.TextField or
(545)
node.
javax.swing.text.JTextField) are converted into a Text input
Onlytrue character
input is packed, function or control keys or key combinations
with Control or Alt are left unchanged.
When the packed sequence is replayed, only KEY_TYPED events are generated.
KEY_PRESSED and KEY_RELEASED events cannot be generated, since the re-
33.2.
Recording options
341
quired key code is system dependent and cannot be determined from the character alone. This is not a problem however, since text components usually handle
only KEY_TYPED events and some special keys.
Automatically set ’Clear...’ attribute of recorded Text input nodes (System)
(547)
This option determines the value of the Clear target component first attribute of a
(545)
recorded Text input
node. If the option is not set, the attribute will not be set
either. Otherwise, the Clear target component first attribute is set if and only if the
text field or text area was empty before the input started.
Always set ’Replay single events’ attribute of recorded Text input nodes (System)
(547)
attribute of a
This option determines the value of the Replay single events
(545)
recorded Text input
node. If the option is set, the attribute will be set and vice
versa. The conservative way is to keep the option set, but for a typical application
that does not add its own KeyListeners to text fields it should be safe to turn it off
so as to speed up replay of Text input nodes.
33.2.
Recording options
33.2.3
342
Components
Figure 33.14: Options for recording components
Hotkey for components (User)
SUT script name: OPT_RECORD_COMPONENT_HOTKEY
This option defines the key for a very useful functionality: recording components
directly from the SUT. Pressing this key in the SUT will switch it to ”component
recording” mode, regardless
of whether event recording is currently activated or
not. The default key is Shift-F11 .
In this mode, clicking on a component with the mouse will cause the component
to be recorded and added to the test-suite if it was unknown before. If more than
one test-suite is currently opened, the menu item Record→Receive components
(645)
determines the suite that will receive the components. The QF-Test ID
of the
(644)
is put on the clipboard so it can be pasted into any text field with
Component
Control-V . This latter feature is very handy when creating an event or a check
from scratch.
33.2.
Recording options
343
Figure 33.15: Popup menu for recording components
You can also record a whole component hierarchy at once by clicking with the
right mouse button instead. This will bring up a popup menu with the following four
options:
Component only
This is similar to clicking with the left mouse button.
component is recorded.
Only the selected
Component and children
The selected component and all components contained therein are
recorded.
Whole window
Records every component in the whole window.
Suggest names
This is a special feature to improve the collaboration between testers and
developers in deciding which components should have names set with
setName(). All components in the whole window are recorded and put into
a test-suite of their own. Each unnamed component for which a name will
improve testability is marked with a name of the form ”SUGGESTED NAME
(n): suggestion”. The running count in braces is just used to avoid
duplicates. The suggested name is built from the component’s class and
other available information. It should be taken with a grain of salt.
Show methods
This is another special feature that brings up a component inspector window
showing the attributes and methods of the selected component’s class. See
(47)
section 5.5 for further information.
33.2.
Recording options
344
Normally ”component recording” mode is turned off by either pressing the hotkey
again or by selecting a component. If you want to record multiple single components, hold down an additional modifier key when switching to ”component recording” mode. That way selecting a component will not turn off the mode, only pressing the hotkey again will.
Record generic class names for components (System)
4.0+
SUT script name: OPT_RECORD_COMPONENT_GENERIC_CLASS
Where possible QF-Test assigns generic class names to components like
”Button”, ”Table” or ”Tree” in addition to the actual Java, DOM or AJAX specific
class names like ”javax.swing.JButton”, ”javafx.scene.control.Button”, ”INPUT” or
”X-BUTTON”. These generic class names are more descriptive and robust,
improve compatibility between different UIs and enable creation of generic utility
procedures. Generic class names can be used for component recognition or
registering resolvers. If this option is active, generic class names are recorded
where available.
Record system class only (System)
SUT script name: OPT_RECORD_COMPONENT_SYSTEM_CLASS_ONLY
If this option is set, QF-Test does not record any custom classes for
(644)
Components . Instead it moves up the class hierarchy until it encounters a
system class and records that. Set this option if the class names of your custom
GUI classes tend to differ between releases.
Note
You must activate this option if you intend to obfuscate the jar files of your application or if you are using a custom class loader to load your application’s GUI
classes.
Web
This option does not apply to web SUTs.
Match any class when recording components (System)
4.0+
SUT script name: OPT_RECORD_TOLERANT_CLASS_MATCH
For compatibility with older QF-Test versions that did not have generic classes,
QF-Test now matches against several classes of a component when recording,
the concrete class, the generic class and the system class. This is very useful if
you want to retain as many of your old components as possible. If you would
rather get new components based on generic classes in new recordings you
should deactivate this option. Components recorded for the first time will always
be recorded with the class determined by the preceding two options Record
(342)
(342)
generic class names for components and Record system class only .
33.2.
Recording options
345
Validate component recognition during recording (System)
3.5+
SUT script name: OPT_VALIDATE_RECORDED_COMPONENTS
In case non-unique names are assigned to components QF-Test can still
(646)
distinguish between these components with the help of the Extra feature
qfs:matchindex that specifies the index of the component with the given
name. If this option is set, QF-Test will check the name of the component during
recording and try to assign qfs:matchindex correctly.
Note
You should only deactivate this option if you are sure that component names
are ”reasonably unique” and the component validation significantly impacts performance during recording.
Convert HTML components to plain text (System)
Swing
SUT script name: OPT_RECORD_COMPONENT_CONVERT_HTML
Swing support HTML markup in various kinds of labels, buttons and
sub-elements of complex components.
For component identification and
validation, the HTML markup is often not useful and will clutter up things. If this
option is set, QF-Test converts HTML to normal text by removing all HTML
markup so only the actual text content is left.
Name override mode (record) (System)
Note
Server or SUT script name: OPT_RECORD_COMPONENT_NAME_OVERRIDE
Possible Values: VAL_NAME_OVERRIDE_EVERYTHING,
VAL_NAME_OVERRIDE_HIERARCHY,
VAL_NAME_OVERRIDE_PLAIN
There are two versions of this option which are closely related. This one is
(365)
effective during recording, the other one during replay. Obviously, both options
should always have the same value. There’s one exception though: When
migrating from one setting to another, QF-Test’s components have to be updated.
During that process, keep the replay option at the old setting and change this
option to the new one. Be sure to update the replay setting after updating the
components.
This option determines the weight given to the names of components for recording.
Possible choices are:
Web
Override everything
This is the most effective and adaptable way of searching components, but it requires that the names of the components are unique, at least within the same
window. If that uniqueness is given, use this choice.
Don’t use this value for a web page with frames. Use ”Hierarchical resolution”
33.2.
Recording options
346
instead.
Hierarchical resolution
This choice should be used if component names are not unique on a per-window
basis, but naming is still used consistently so that two components with identical
names have at least parent components or ancestors with distinct names. That
way, component recognition is still tolerant to a lot of change, but if a named component is moved to a different named parent in the SUT, the test-suite will have to
be updated to reflect the change.
Plain attribute
If there are components with identical names in the SUT within the same parent
component you must use this setting. The name will still play an important role in
(646)
component recognition, but not much more than the Feature attribute.
Automatic component names for Eclipse/RCP applications (System)
SWT
SUT script name: OPT_RECORD_COMPONENT_AUTOMATIC_RCP_NAMES
Eclipse and applications based on the Rich Client Platform (RCP) have a
complex GUI with support for changing perspectives. Such a change causes
components to be rearranged which can make it hard for QF-Test to recognize
them unless names are set at least on the major components. This is further
complicated by the fact that the structure of the components is not what it
appears to be - the main components are all arranged in a relatively flat hierarchy
within the workbench. On the upside, RCP based applications have a uniform
inner structure based on Views and Editors, many of which are named.
If this option is turned on, QF-Test will do its best to automatically associate GUI
elements with their RCP counterparts and assign names based on that association. This can drastically improve component recognition for such applications.
However, if some names thus assigned turn out not to be reliable over time, they
can also interfere. In such a case, names can be assigned to the affected com(40)
ponents either using setData as described in chapter 5 or with the help of a
(807)
NameResolver as described in section 43.1.6 . Both will override automatically
generated names.
Component hierarchy (System)
SUT script name: OPT_RECORD_COMPONENT_HIERARCHY
Possible Values: VAL_RECORD_HIERARCHY_INTELLIGENT,
VAL_RECORD_HIERARCHY_FULL,
VAL_RECORD_HIERARCHY_FLAT
QF-Test supports three different kinds of views for the components of the SUT.
For more information about their effect on component recognition, see section
(718)
40.2 .
33.2.
Recording options
347
The flat view collects all components of a window as direct child nodes of the
(634)
respective Window node. The advantage of this view is that structural changes
of the component hierarchy have little effect on component recognition. This is
also its greatest disadvantage: since structural information is not available, this
view gives reasonable recognition quality only if setName() is used ubiquitously.
Another drawback is the lack of clearness.
The complement to the flat view is the full hierarchy. It includes every single component of the SUT’s GUI, emulating all parent/child relationships. This view can
be a useful tool for developers or testers that want to gain insights into the SUT’s
structure, but is not very applicable for testing, since structural changes affect it
too much. As long as you don’t change the GUI however, it will give you excellent
recognition without the help of setName().
A compromise between flat and full hierarchy is available through the choice ”Intelligent”. For this view only the ”interesting” components of the SUT are recorded.
”Interesting” in this case means that either the user can interact with the component, or it is located at some important point in the hierarchy, like the children of a
split pane or a tabbed pane. In some later version of QF-Test this decision may be
made configurable as well.
Prepend QF-Test ID of window parent to component QF-Test ID (System)
Server script name: OPT_RECORD_COMPONENT_PREPEND_WINDOW_ID
(634)
parent of a
If selected, QF-Test prepends the QF-Test ID of the Window
(644)
(645)
Component to its QF-Test ID during recording. This is useful to disambiguate
QF-Test IDs of components with identical names in different windows.
Prepend parent QF-Test ID to component QF-Test ID (System)
Server script name: OPT_RECORD_COMPONENT_PREPEND_PARENT_ID
Possible Values: VAL_RECORD_COMPONENT_PREPEND_PARENT_ALWAYS,
VAL_RECORD_COMPONENT_PREPEND_PARENT_NAMED,
VAL_RECORD_COMPONENT_PREPEND_PARENT_FEATURE,
VAL_RECORD_COMPONENT_PREPEND_PARENT_NEVER
(644)
is recorded for the first time, QF-Test assigns an
When a Component
(645)
automatically generated QF-Test ID . The QF-Test ID of a direct or indirect
parent node may be prepended to this QF-Test ID. This is useful to distinguish
between similar components that don’t have a name of their own.
Example: Imagine two JScrollPanes, one named ”TreeScrollPane” and the
other named ”DetailScrollPane”. Without this functionality, their vertical scrollbars
would get the QF-Test IDs ”scrollbarVertical” and ”scrollbarVertical2”. With this
function turned on, the IDs would be ”TreeScrollPane.scrollbarVertical” and ”De-
33.2.
Recording options
348
tailScrollPane”.scrollbarVertical”. That way it is immediately obvious which component is the target of an event.
There are four possible settings:
• ”Never” turns this option off.
• ”Nearest named ancestor” is a useful setting, if your developers have assigned names to all major components with the Java method setName. A
component that doesn’t have a name of its own, gets the QF-Test ID of its
nearest named ancestor node prepended.
• If setName is used sparingly or not at all, it is better to set this option to
”Nearest ancestor with name or feature”. That way either the name or a
distinctive feature of an ancestor node will be applicable.
(344)
is set to ”Flat”.
• ”Always” is only useful if the option Component hierarchy
With this setting, every component gets the QF-Test ID of its parent node
prepended, which can lead to unusably long QF-Test IDs when components
are nested deeply.
Maximum QF-Test ID length (System)
If set, this option limits the length of automatically generated IDs. This can
improve clearness since QF-Test IDs can get very long, e.g. when generated
from a component’s feature.
33.2.4
Recording sub-items
Events on complex components like tables or trees can be recorded relative to a sub(650)
item of the component. For further information about sub-items see the Item node or
(57)
section 6.3 .
33.2.
Recording options
349
Figure 33.16: Options for recording sub-items
Record sub-items as event targets (System)
SUT script name: OPT_RECORD_SUBITEM
This option activates recording sub-items. When turned off, events on complex
components are no different from events on simple components.
Multi-level sub-items (System)
4.0+
Server or SUT script name: OPT_RECORD_SUBITEM_MULTILEVEL
Via this option you can completely disable multi-level sub-items (even for replay).
However, you should only turn this feature off in case you are running into
problems caused by test-suites that contain unquoted special characters like ’@’
or ’%’ in textual sub-item indexes. Even then it is preferable to update the
test-suites with properly quoted items, possibly using the special variable syntax
(67)
${quoteitem:...} (see section 7.5 ).
Sub-item format (System)
SUT script name: OPT_RECORD_SUBITEM_FORMAT
Possible Values: VAL_RECORD_SUBITEM_FORMAT_INTELLIGENT,
VAL_RECORD_SUBITEM_FORMAT_TEXT,
VAL_RECORD_SUBITEM_FORMAT_NUMBER
(650)
When recording an event for a sub-item, the Item’s
index can be defined
(652)
(652)
As string or As number .
The third choice, ”Intelligent”, causes QF-Test to record the index in the format
most appropriate for the item. If the name of the item is unique within the complex
component, a string index is recorded, a numeric index otherwise.
33.2.
Recording options
350
Sub-item type (System)
Server script name: OPT_RECORD_SUBITEM_TYPE
Possible Values: VAL_RECORD_SUBITEM_TYPE_INTELLIGENT,
VAL_RECORD_SUBITEM_TYPE_NODE,
VAL_RECORD_SUBITEM_TYPE_SYNTAX
(57)
As explained in section 6.3 , there are two ways to address a sub-item: either
(650)
node. This option determines
through a special syntax or by creating an Item
the method to use in recorded events.
(650)
Choosing ”Intelligent” will cause an Item node to be created if the index of the
sub-item is a string and if the sub-item itself is not editable. Otherwise the special
syntax is used.
Represent tree node as path (System)
SUT script name: OPT_RECORD_SUBITEM_TREE_PATH
It is not uncommon that trees have identically named nodes under different
parent nodes, e.g. a file system with the directories /tmp and /usr/tmp. By
(650)
using a path format in the Items for tree nodes QF-Test can make full use of the
hierarchical structure to distinguish between these nodes. The slash character ’/’
is used as separator.
If this option is deactivated, trees will be treated as flat lists.
33.2.5
Recording procedures
The following options determine the configuration of the Procedure Builder which is
(202)
described in detail in Automated Creation of Basic Procedures .
Figure 33.17: Procedure Builder options
Hotkey for procedure recording (User)
33.3.
Replay options
351
SUT script name: OPT_RECORD_PROCEDURE_HOTKEY
This option defines a key for turning on automatic procedure recording directly in
the SUT. The default key is Shift-F12 .
Configuration file for recorded procedures (System)
Here you can specify your own template file for the Procedure Builder. If a relative
path is given, QF-Test looks for the definition file in the directory that QF-Test was
started from and in the default include directory.
33.3
Replay options
The following settings change the way test-suites are executed.
Figure 33.18: Replay options
(Don’t) Panic key (User)
33.3.
Replay options
352
Server or SUT script name: OPT_PLAY_DONT_PANIC_HOTKEY
When running a test at full speed it can be rather difficult to get the focus to
QF-Test’s window and interrupt the test so you can do something different without
having all these windows flashing around the screen. This is all the more true
(361)
when the options Actually move mouse cursor
or Raise SUT windows
(360)
automatically are activated or when running in batch mode.
This option lets you define a key combination (the default being Alt-F12 ) that will
instantly pause all running tests if it is pressed in any SUT or QF-Test window
(unless multiple QF-Test instances are run simultaneously, of course). Pressing
the same key combination again will resume all tests, unless you manually resume
or stop any of them. In that case its effect is automatically reset to suspend tests.
Call stack size (System)
Server script name: OPT_PLAY_CALLSTACK_SIZE
(415)
or
The call stack size is a limit for the nesting depth of Sequences
(466)
Procedure calls during replay. This limit is needed to detect and handle endless
recursion.
When the nesting depth exceeds the call stack size, a
(668)
StackOverflowException
is thrown. The default value of 200 should be
sufficient but can be increased for very complex tests.
Log warning for nested test-cases (System)
Server script name: OPT_PLAY_WARN_NESTED_TEST_CASE
(398)
nodes should not be nested because such Test-cases
Execution of Test-case
cannot be listed properly in the report. If this option is active, a warning is logged
in case a Test-case is executed within another Test-case.
Mark nodes during replay (User)
If set, tree nodes that are currently executed are marked with an arrow.
Show replay messages in status line (User)
Determines whether the name of the currently executing node is shown in the
status line.
Raise test-suite window after replay (User)
This option is mainly used together with the option Raise SUT windows
(360)
automatically . It causes the window of a test-suite to be raised after a test run.
(360)
See also option Force window to the top when raising
.
33.3.
Replay options
353
Minimize test-suite window during replay (User)
If this option is set, QF-Test will minimize the window of a test-suite while its tests
are being executed. The window will pop back up automatically when the test is
stopped or suspended. This feature is especially useful on Windows 2000/XP
systems where programs are prohibited from bringing their windows to the top so
QF-Test cannot raise the windows of the SUT.
Show message dialog after (User)
After replay is finished, the status line shows the number of errors and warnings
that occurred. If an uncaught exception was thrown, an error dialog is displayed.
Additionally, a message dialog can be displayed in case of warnings or errors.
This options sets the minimum error level that triggers such a message dialog.
Always locate the source of an error (User)
When an exception is thrown during replay, the node that caused the exception
will be made visible and selected. If you don’t like this, you can turn this feature off
and locate the node via the Run→Find last error source... menu item instead.
Salt for crypting passwords (System)
3.0+
(546)
(545)
QF-Test can store encrypted passwords in the Text
attribute of a Text input
(556)
(553)
node for a password field or the Detail attribute of a Selection used for a login
dialog in a web SUT. When such passwords are en- or decrypted, QF-Test
combines the key with the salt specified in this option. Without this salt, anybody
with sufficient knowledge is able to decrypt your passwords to get the plain-text
version.
Note
Don’t let this option give you a false sense of security. Anybody that gains access
to this salt and anybody that can execute your tests can also gain access to the
plain-text version of the password. However, encrypting passwords is still useful
to prevent obvious plain-text passwords getting stored in test-suites and run-logs,
and encrypted passwords are reasonably safe from someone who only gets hold
of a test-suite or run-log without access to this salt.
How to handle disabled components (System)
4.0+
Server script name: OPT_PLAY_ERROR_STATE_DISABLED_COMPONENT
Possible Values: VAL_PLAY_DISABLED_COMPONENT_WARNING,
VAL_PLAY_DISABLED_COMPONENT_ERROR,
VAL_PLAY_DISABLED_COMPONENT_EXCEPTION
33.3.
Replay options
354
In case you replay an event on a component, which is disabled you can configure
QF-Test’s behavior for that case. You can
• Log a warning message
• Log an error message
(663)
• Throw a DisabledComponentStepException
How to handle exceeded execution timeout (System)
4.1+
Server script name: OPT_PLAY_ERROR_STATE_EXECUTION_TIMEOUT
Possible Values: VAL_PLAY_EXECUTION_TIMEOUT_WARNING,
VAL_PLAY_EXECUTION_TIMEOUT_ERROR,
VAL_PLAY_EXECUTION_TIMEOUT_EXCEPTION,
VAL_PLAY_EXECUTION_TIMEOUT_WARNING_IMMEDIATE,
VAL_PLAY_EXECUTION_TIMEOUT_ERROR_IMMEDIATE,
VAL_PLAY_EXECUTION_TIMEOUT_EXCEPTION_IMMEDIATE
In case of an exceeding execution timeout, you can configure QF-Test’s behavior
for that case. You can
• Log a warning message and run through possible cleanup nodes.
• Log an error message and run through possible cleanup nodes.
(663)
• Throw an ExecutionTimeoutExpiredException
possible cleanup nodes.
and run through
• Log a warning message and stop the node immediately.
• Log an error message and stop the node immediately.
(663)
• Throw an ExecutionTimeoutExpiredException
immediately.
and stop the node
The definition of running cleanup nodes includes that Cleanup and Catch nodes
get executed. Stopping the node immediately stands for not executing possible
Cleanup and Catch nodes at the end.
33.3.1
Client options
Various settings for process and SUT clients can be adjusted with the following options:
33.3.
Replay options
355
Figure 33.19: Client options
Ask whether to stop clients before exiting (User)
If there are still active clients upon exit of QF-Test, these are terminated after
asking for confirmation. If this option is turned off, the clients are terminated
unconditionally.
When terminating a process, kill its whole process tree (System)
Server script name: OPT_PLAY_KILL_PROCESS_TREE
The process of an SUT or a helper program started during a test can be
(532)
terminated via a Stop client node or manually via the Client menu. In case of
an SUT, QF-Test first tries to communicate with it and initiate a clean
System.exit call. Non-Java programs have to be killed. If the program has
started further child processes these may or may not get terminated by a normal
shutdown or kill, depending on circumstances.
It is normally undesirably to keep such processes around as they might interfere
with other tests or lock files that need to be removed or overwritten. Unless this
option is disabled, QF-Test will try to determine the whole process hierarchy for any
program it started and make sure that the main process and all child processes
get killed explicitly.
33.3.
Replay options
356
Number of terminated clients in menu (User)
Server script name: OPT_PLAY_MAX_CLIENTS
This option limits the number of menu items for terminated clients that are kept in
the Clients menu.
Maximum size of client terminal (kB) (User)
3.0+
Server or SUT script name: OPT_PLAY_TERMINAL_SIZE
The maximum amount of text (in kilobyte) that the individual client terminal will
hold. If the limit is exceeded, old text will be removed when new text arrives. A
value of 0 means no limit.
Note
This option also determines the amount of output available for the special variable
${qftest:client.output.<name>}.
Highlight selected component in the SUT (User)
Server script name: OPT_PLAY_HIGHLIGHT_COMPONENTS
If this option is set, QF-Test will highlight the associated component in the SUT
(644)
node or a node that references a Component is
whenever a Component
selected.
How to handle exceptions in the SUT (System)
SUT script name: OPT_PLAY_SUT_EXCEPTION_LEVEL
Possible Values: VAL_PLAY_EXCEPTION_LEVEL_WARNING,
VAL_PLAY_EXCEPTION_LEVEL_ERROR,
VAL_PLAY_EXCEPTION_LEVEL_EXCEPTION
Exceptions that are thrown during event handling in the SUT are typically a sure
sign for a bug in the SUT. This option determines what to do if such an exception
is caught. You can
• Log a warning message
• Log an error message
(667)
• Throw an UnexpectedClientException
Connect via QF-Test agent (System)
4+
Server script name: OPT_PLAY_CONNECT_VIA_AGENT
QF-Test version 4 introduced a new mechanism for connecting to an SUT, based
33.3.
Replay options
357
on Java agents. It is far more powerful and flexible than the older mechanisms
and should not be turned off without a very good reason.
Note
Without the agent an SUT based on Java 9 will not even start. Besides, the agent is
a prerequisite for access to JavaFX, embedded browsers and the full functionality
of live unit testing.
Instrument AWT EventQueue (System)
4.1+
Swing
Note
Server script name: OPT_PLAY_INSTRUMENT_EVENT_QUEUE
In earlier versions QF-Test replaced the AWT system EventQueue with its own,
taking pains to handle custom EventQueues and many other subtle details.
Starting with QF-Test 4.1 the default is to instrument the EventQueue class
instead which has less impact on the SUT, handles border cases well and is
preferable in almost all situations. In case of connection problems, the old
mechanism can be used by turning off this option.
Instrumenting the AWT EventQueue is only possible with the QF-Test agent.
Connect without JDK instrumentation (Swing) (System)
3.1.4+
Server script name: OPT_PLAY_IMPLICIT_INSTRUMENTATION
With this option activated there is no need to instrument standard JDKs/JREs in
order to connect to an AWT/Swing SUT. In cases where automatic connections
don’t work - or if you deactivate this option - you need to fall back to explicit JDK
(710)
(713)
instrumentation as described in section 39.1 and section 39.2 .
Note
If this option is activated, QF-Test sets the environment variable _JAVA_OPTIONS
(and also IBM_JAVA_OPTIONS for IBM JDKs) so that any JVM started directly
or indirectly by QF-Test will execute QF-Test’s accessibility hook when initializing
the AWT toolkit. In contrast to other tools QF-Test does not set this environment
variable globally and thus does not interfere with Java applications not started from
QF-Test. However, if the variable is already set globally and your Java application
relies on it, you must deactivate this option and use JDK instrumentation instead.
SWT
Pure Eclipse/SWT applications only require SWT instrumentation (see section
(714)
39.3 ). Some Eclipse/RCP applications show an AWT based splash screen on
startup and if this option is active that splash screen will trigger initialization of QFTest’s AWT/Swing engine, which will then become the primary engine for that SUT.
(504)
(529)
If you rely on the default engine for SUT script
and Wait for client to connect
nodes to be ’swt’, you need to deactivate this option.
Reuse IDs for SUT clients in nested sub-processes (System)
Server script name: OPT_PLAY_REUSE_SUT_IDS
33.3.
Replay options
358
This is a complex option which you should hopefully never care about. When an
SUT client launches another process that itself connects to QF-Test, the new
SUT client is identified by the name of the original SUT client with a ’:’ and a
numeric ID appended. The first ID will always be 2, with increasing numbers for
additional sub-processes.
When a sub-process terminates and another sub-process connects, QF-Test can
either reuse the ID of the terminated process or continue incrementing to create a
new ID.
In most cases it is preferable to reuse the sub-process ID. The most common case
is a single sub-process that is started, terminated, then started again. By activating
this option you can always address the single sub-process with the same client
name.
In a more complex situation, multiple sub-processes may be launched and terminated more or less at random, depending on progression of the test-run. In such
a case, always incrementing the ID for a new process is more deterministic.
In either case the ID counter will be reset when the original SUT client is started
anew.
Automatically perform garbage collection in the SUT (System)
SUT script name: OPT_PLAY_SUT_GARBAGE_COLLECTION
By default QF-Test performs a full garbage collection in the SUT once every few
(504)
executions. This is necessary due to a limitation in Java’s
hundred SUT script
default garbage collection mechanism that allows an OutOfMemoryError to
happen for the so called PermGen space, even though the required memory
could easily be reclaimed by a garbage collection.
When you are trying to analyze the memory performance of your application, these
explicit garbage collections might influence the results. For that case you can
disable QF-Test’s garbage collection via this option.
33.3.2
Terminal options
Various settings for the shared terminal can be adjusted with the following options:
33.3.
Replay options
359
Figure 33.20: Terminal options
Maximum size of shared terminal (kB) (User)
Server script name: OPT_PLAY_SHARED_TERMINAL_SIZE
The maximum amount of text (in kilobyte) that the shared terminal will hold. If the
limit is exceeded, old text will be removed when new text arrives. A value of 0
means no limit.
Regular expression to suppress display of certain text (User)
4.0+
By defining a regular expression for this option, certain text in the terminal output
can be suppressed.
Default value is empty.
(720)
See also Regular expressions .
Use rich text terminal (User)
4.0+
33.3.
Replay options
360
If activated the rich text terminal allowing monospaced font type and coloring of
given regular expressions. Deactivate this option if you want to switch back to the
simple terminal as it were before QF-Test version 4.
Note
QF-Test needs to be restarted in order to make a change in this option become
visible.
Terminal font size (pt) (User)
4.0+
This option lets you set the font size (as point value) used within the shared
terminal.
(357)
This option only has an effect if the Use rich text terminal
is active.
Use monospaced font (User)
4.0+
If activated the rich text terminal will use a monospaced font.
(357)
This option only has an effect if the Use rich text terminal
is active.
Regular expression coloring (User)
4.0+
If activated the shared terminal output is processed for given regular expressions
to be highlighted in different colors.
(357)
This option only has an effect if Use rich text terminal
(358)
coloring are active.
and Regular expression
Regular expression for red highlighting (User)
4.0+
This option allows to define a regular expression for output to be shown in red
color.
(357)
This option only has an effect if Use rich text terminal
(358)
coloring are active.
and Regular expression
Default value is:
(?md).+Exception\b.*\n(?>\n?ˆ(?>\sat|Caused
by:)\s.+\n)+|.*(?i)exception(?>s)?\b.*
With this also typical Java stack traces will be highlighted.
(720)
See also Regular expressions .
Regular expression for orange highlighting (User)
4.0+
This option allows to define a regular expression for output to be shown in orange
color. The default regular expression matches error log output and lines
containing respective text.
33.3.
Replay options
361
(357)
This option only has an effect if Use rich text terminal
(358)
coloring are active.
and Regular expression
Default value is:
(?md)ˆ[1-2] \(\d\d:\d\d:\d\d\.\d\d\d\) .*|.*(?i)(?>error(?>s)?|fehler)\b.*
With this also error log messages will be highlighted.
(720)
See also Regular expressions .
Regular expression for yellow highlighting (User)
4.0+
This option allows to define a regular expression for output to be shown in yellow
color. The default regular expression matches with typical stack traces and
exceptions.
(357)
This option only has an effect if Use rich text terminal
(358)
coloring are active.
and Regular expression
Default value is:
(?md)ˆ[3-4]
\(\d\d:\d\d:\d\d\.\d\d\d\)
.*|.*(?i)(?>warning(?>s)?|warnung(?>en)?)\b.*
With this also warning log messages will be highlighted.
(720)
See also Regular expressions .
Regular expression for blue highlighting (User)
4.0+
This option allows to define a regular expression for output to be shown in blue
color. The default regular expression matches with typical stack traces and
exceptions.
(357)
This option only has an effect if Use rich text terminal
(358)
coloring are active.
and Regular expression
Regular expression for green highlighting (User)
4.0+
This option allows to define a regular expression for output to be shown in green
color. The default regular expression matches with typical stack traces and
exceptions.
(357)
This option only has an effect if Use rich text terminal
(358)
coloring are active.
33.3.3
and Regular expression
Event handling
These options influence some details of how events are simulated in the SUT during
replay.
33.3.
Replay options
362
Figure 33.21: Event handling options
Raise SUT windows automatically (System)
SUT script name: OPT_PLAY_RAISE_SUT_WINDOWS
If this option is set, windows of the SUT for which a MouseEvent or KeyEvent is
replayed will be raised automatically when they get activated. This eases
switching between QF-Test and the SUT to visually verify that a sequence is
replaying correctly.
(350)
See also options Raise test-suite window after replay
(360)
top when raising .
and Force window to the
Force window to the top when raising (System)
3.4.1+
Note
SUT script name: OPT_PLAY_RAISE_SUT_WINDOWS_FORCED
This option only has an effect on Windows systems.
Windows only allows an application to bring one of its own windows to the front
if that application currently has the focus. This can make it difficult for QF-Test to
33.3.
Replay options
363
raise SUT windows and to automatically switch between the SUT and QF-Test. If
this option is activated, QF-Test temporarily sets the ”always on top” attribute to
force windows to the top.
See also options Raise test-suite window after replay
(360)
automatically .
(350)
and Raise SUT windows
Check for modal dialogs (System)
SUT script name: OPT_PLAY_CHECK_MODAL
A modal dialog is a window that blocks all input to other windows until it is closed.
It is often used to display an error message or request user input.
Because the events simulated by QF-Test are artificial, they are not blocked by a
modal dialog and can reach any window. This is normally not desirable, since the
existence of a modal dialog may signal an unexpected error. Activating this option
causes QF-Test to check for modal dialogs itself before replaying MouseEvents or
KeyEvents. If such an event is targeted to a window that is blocked by a modal
(663)
dialog, a ModalDialogException is thrown.
Actually move mouse cursor (System)
SUT script name: OPT_PLAY_MOVE_MOUSE_CURSOR
If this option is set, the mouse cursor is actually moved across the screen as
MouseEvents are simulated. This feature requires a working AWT robot.
While this option is mainly intended to give visual feedback, it can have a positive impact on test reliability because it reduces side-effects through events from
the underlying system that might interfere with the automated test. However, for
tests where precise mouse movement is essential, for example a drawing tool, this
option should be turned off.
Delay for hard events during replay (ms) (System)
SUT script name: OPT_PLAY_DELAY_HARD_EVENTS
During event replay QF-Test blocks or delays some ”hard” events, i.e. events that
come directly from the system, for example when the user is moving the mouse
cursor. This is done to prevent them from interfering with the SUT in an unlucky
moment. Popup windows, which are used for menus and combo boxes among
other things, are especially sensitive to such events which can cause them to pop
down accidentally. Therefore, these filters improve testing stability considerably.
This option sets the maximum time that such events may be delayed. In the unlikely case that the filters have unwanted side effects with your application, you
can turn them off by setting the value to 0.
33.3.
Replay options
364
Scroll automatically to display sub-items (System)
SUT script name: OPT_PLAY_SCROLL_ITEM
If this option is set, accessing a sub-item of a complex component inside a scroll
pane will automatically cause the sub-item to be scrolled into view. In that case
you can remove most recorded events on scroll bars or scroll buttons, which are
not required for correct replay.
Expand tree nodes as needed (System)
SUT script name: OPT_PLAY_EXPAND_TREE
When accessing nodes of a tree component as hierarchic sub-items
it is possible to select a node that is not visible because one of its
ancestral nodes is not expanded. If this option is set, all ancestors of the
node will be expanded as needed.
Otherwise this situation leads to a
(662)
ComponentNotFoundException .
Throw DisabledComponentException (System)
SUT script name: OPT_PLAY_THROW_DISABLED_EXCEPTION
If QF-Test replays events on a component that is not enabled, these events are
ignored silently. In most cases this indicates an error which is signaled by
(663)
throwing a DisabledComponentException .
Old test-suites may not be prepared to deal with this exception. These test-suites
should be fixed, but as a quick workaround DisabledComponentExceptions
can be suppressed by deactivating this option.
Tolerance for checking images (System)
SWT
Note
SUT script name: OPT_PLAY_IMAGE_TOLERANCE
Initially this option was intended for SWT/Gtk only but it turned out to be
universally applicable and useful.
Image rendering in Java applications and web browsers is not always fully deterministic. Even within the same session on a display with limited color depth the
RGB values of an icon image can vary slightly and it becomes worse when running
tests on different machines. Graphics driver, JDK version and operating system
settings also play a role. This makes strict image checks almost unusable in some
cases.
To work around the problem, this option defines a tolerance setting for default
image checks. For a pixel’s red, green and blue color values a deviation from the
expected value by the given amount is tolerated. Thus exact image checking for
33.3.
Replay options
365
the default algorithm can be enforced by setting this value to 0 but it is preferable
to use the ”identity” check algorithm instead (see Details about the algorithm for
(910)
image comparison ). The default setting of 5 is a good compromise, allowing
checks with differences that are normally not visually perceivable to succeed.
How to handle events coming from the wrong thread (System)
Swing
SUT script name: OPT_PLAY_WRONG_THREAD_ERROR_LEVEL
Possible Values: VAL_PLAY_THREAD_LEVEL_WARNING,
VAL_PLAY_THREAD_LEVEL_ERROR,
VAL_PLAY_THREAD_LEVEL_EXCEPTION
It is a rather common mistake in Swing based Java applications to access GUI
components from the wrong thread. Since Swing is not thread-safe, such calls
may only be made from the AWT event dispatch thread. Otherwise the potential
consequences are race conditions, leading to very subtle and hard-to-debug
errors, or deadlocks, freezing the application and making it unusable.
Background information about this topic is available from
http://download.oracle.com/javase/tutorial/uiswing/concurrency/index.html
,
specifically the sections on ”Initial Threads” and ”The Event Dispatch Thread”.
When QF-Test registers an event on a thread other than the AWT event dispatch
thread it issues an error message including a stack trace which can be useful in
fixing the problem. This set of options defines the severity of the message, whether
to perform strict checking and a maximum for the number of messages to log.
The possible choices for the option ”Error level” are ”Error” and ”Warning”. We
strongly suggest that you keep the default setting of ”Error” and make sure that
such problems are fixed sooner rather than later because they represent a serious
risk.
Strict checking (System)
Swing
SUT script name: OPT_PLAY_WRONG_THREAD_STRICT
If strict checking for wrong thread errors is activated, error or warning messages
will be issued for all kinds of events triggered from the wrong thread. Otherwise
the problem will be ignored for ”less relevant” events. Which events are
considered more or less relevant is arbitrary, based on the fact that there’s a lot of
Java literature (including early Java documentation from Sun) claiming that it is
safe to create components on any thread and that thread-safety only needs to be
enforced once a component is made visible. A lot of code follows this pattern and
the risk for causing problems in this case is indeed minimal. In the presence of
such code disabling ”Strict checking” will cause error messages to be logged only
for the more relevant problems. If you want to get rid of all thread violations - as
we recommend - you should turn strict checking on.
33.3.
Replay options
366
Maximum number of errors to log per SUT client (System)
Swing
SUT script name: OPT_PLAY_WRONG_THREAD_MAX_ERRORS
In case an SUT contains code that violates thread safety it is possible that a very
large number of events are triggered from the wrong thread. Logging all those
errors can significantly impact test performance, yet logging more than the first
few errors does not really contribute much. The option ”Maximum number of
errors to log per SUT client” limits the possible number of error messages for this
case.
33.3.4
Component recognition
How component recognition works - and the impact of these options on it - is explained
(718)
in section 40.2 . The pre-defined values should give good results. If you experience
problems with component recognition, you can try to improve it by adjusting the probabilities.
33.3.
Replay options
367
Figure 33.22: Component recognition options
The name of a component plays a special role. The following option affects the impact
of names:
Name override mode (replay) (System)
Note
SUT script name: OPT_PLAY_RECOGNITION_NAME_OVERRIDE
Possible Values: VAL_NAME_OVERRIDE_EVERYTHING,
VAL_NAME_OVERRIDE_HIERARCHY,
VAL_NAME_OVERRIDE_PLAIN
There are two versions of this option which are closely related. This one is
(343)
effective during replay, the other one during recording. Obviously, both options
should always have the same value. There’s one exception though: When
migrating from one setting to another, QF-Test’s components have to be updated.
During that process, keep this option at the old setting and change the record
option to the new one. Be sure to update the replay setting after updating the
components.
33.3.
Replay options
368
This option determines the weight given to the names of components for component recognition. Possible choices are:
Web
Override everything
This is the most effective and adaptable way of searching components, but it requires that the names of the components are unique, at least within the same
window. If that uniqueness is given, use this choice.
Don’t use this value for a web page with frames. Use ”Hierarchical resolution”
instead.
Hierarchical resolution
This choice should be used if component names are not unique on a per-window
basis, but naming is still used consistently so that two components with identical
names have at least parent components or ancestors with distinct names. That
way, component recognition is still tolerant to a lot of change, but if a named component is moved to a different named parent in the SUT, the test-suite will have to
be updated to reflect the change.
Plain attribute
If there are components with identical names in the SUT within the same parent
component you must use this setting. The name will still play an important role in
(646)
component recognition, but not much more than the Feature attribute.
The algorithm for component recognition is very tolerant and biased towards finding a
match. If the best match is not perfect QF-Test logs information about the encountered
differences, either as a warning or a plain message, depending on the following options:
Log missing name (System)
SUT script name: OPT_PLAY_WARN_MISSING_NAME
If this option is set, a message will be logged whenever a component is targeted
that does not have a name, but QF-Test ”thinks” it should have one. A plausible
name is suggested where possible.
Log ambiguous name (System)
SUT script name: OPT_PLAY_WARN_AMBIGUOUS_NAME
(365)
is set to ”Override everything” or
If the option Name override mode (replay)
”Hierarchical resolution”, a message is logged whenever QF-Test encounters
more than one potential target components with the same name. That message
can be suppressed with the help of this option.
Log feature mismatch (System)
SUT script name: OPT_PLAY_WARN_FEATURE_MISMATCH
33.3.
Replay options
369
A component is considered to have a ”feature mismatch” if it is determined by
QF-Test as the target best suited for an event or check even though at one or
(646)
attribute did not match the
more levels of the hierarchy the recorded Feature
current state of the component. If this option is activated, feature mismatches are
logged, notifying you that it may be a good idea to update the affected
components.
Log extra feature mismatch (System)
SUT script name: OPT_PLAY_WARN_EXTRA_FEATURE_MISMATCH
An ”extra feature mismatch” is similar to a feature mismatch as explained above,
except that it applies to extra features with status ”should match”. If this option is
activated, extra feature mismatches are logged, notifying you that it may be a
good idea to update the affected components.
Log structure mismatch (System)
SUT script name: OPT_PLAY_WARN_STRUCTURE_MISMATCH
A ”structure mismatch” is similar to a feature mismatch as explained above,
except that instead of the feature it is the structure information represented by the
(648)
(648)
and Class count
where the mismatch occurred. If this
attributes Class index
option is activated, structure mismatches are logged, notifying you that it may be
a good idea to update the affected components.
Log intermediate named ancestor (System)
SUT script name: OPT_PLAY_WARN_NAMED_ANCESTOR
An ”intermediate named ancestor” is a direct or indirect parent component of the
target component in the SUT which is not part of the component hierarchy in
(365)
QF-Test even though it has a name. If the option Name override mode (replay)
is set to ”Hierarchical resolution”, this is considered as a mismatch, comparable
to a feature or structure mismatch. If this option is activated, interferences
through intermediate named ancestors are logged, notifying you that it may be a
good idea to update the affected components.
Log warning instead of message (System)
4.2+
SUT script name: OPT_PLAY_COMPONENT_WARNINGS
If this option is activated, deviations from the expected values during component
recognition are logged as warnings instead of plain messages. It can be useful to
temporarily activate this option to increase the visibility of such deviations, either
to update the respective component information or to seek out false positive
33.3.
Replay options
370
matches. For normal testing this creates too much noise and tends to obscure
more interesting warnings.
For an explanation of the remaining options for component recognition please refer to
(718)
section 40.2 . The respective SUT script names for these options are:
OPT_PLAY_RECOGNITION_BONUS_NAME
OPT_PLAY_RECOGNITION_PENALTY_NAME
OPT_PLAY_RECOGNITION_BONUS_FEATURE
OPT_PLAY_RECOGNITION_PENALTY_FEATURE
OPT_PLAY_RECOGNITION_BONUS_EXTRAFEATURE
OPT_PLAY_RECOGNITION_PENALTY_EXTRAFEATURE
OPT_PLAY_RECOGNITION_BONUS_STRUCTURE
OPT_PLAY_RECOGNITION_PENALTY_STRUCTURE
OPT_PLAY_RECOGNITION_PENALTY_MODAL
OPT_PLAY_RECOGNITION_MINIMUM_PROBABILITY
33.3.5
Delays
Here values can be set for various delays.
Figure 33.23: Delay options
Default delays (ms) (System)
33.3.
Replay options
371
Server script name: OPT_PLAY_DELAY_BEFORE, OPT_PLAY_DELAY_AFTER
These two options set the delay in milliseconds before and after the execution of
(417)
a node. If a node defines its own Delay before/after , its value overrides this
default.
These options are useful to slow a test down so you’ll be able to follow it.
Drag&Drop and interpolation of mouse movement
Simulating Drag&Drop is non-trivial. It is made possible only by generating ”hard” mouse
events that actually move the mouse cursor. On Windows systems, some mouse drivers
(717)
can interfere with these ”hard” events. See section 40.1
for further details about
Drag&Drop.
To make Drag&Drop as reliable as possible, movement of the mouse cursor is highly
configurable. Since the requirements for Drag&Drop and hard mouse events differ from
those for general mouse moves which only provide visual feedback, two sets of options
are provided. The settings for demo mouse moves are ignored unless the respective
(361)
option Actually move mouse cursor is activated.
Typically movements for Drag&Drop and hard events should be slower and involve more
interpolation steps than those for demo moves, which could slow down a test considerably. All of the following values influence the overall speed of mouse moves. A little
experimentation may be required to get the desired effect.
Delay (ms) (System)
SUT script name: OPT_PLAY_DND_DELAY, OPT_PLAY_MOVEMOUSE_DELAY
After each single movement of the mouse cursor QF-Test will wait for the
specified number of milliseconds. This value should be between 2 and 20 if
interpolation is enabled and between 20 and 200 if interpolation is turned off. With
interpolation, 10 is a good value for Drag&Drop and 5 for demo mouse moves.
Interpolation step size (System)
SUT script name: OPT_PLAY_DND_STEP, OPT_PLAY_MOVEMOUSE_STEP
The size of the steps for interpolation of mouse movement. Set this to 0 to turn
interpolation off. Good values are between 1 and 3 for Drag&Drop and between 2
and 10 for demo mouse moves.
Acceleration (System)
SUT
script
name:
OPT_PLAY_DND_ACCELERATION,
OPT_PLAY_MOVEMOUSE_ACCELERATION
33.3.
Replay options
372
To avoid needless slowdown of tests, long distance mouse movement can be
accelerated. A value of 0 turns off acceleration. Useful values range from 1 for
very little acceleration to 10 or more for high acceleration. Good values are
between 3 and 5 for Drag&Drop and between 6 and 20 for demo mouse moves.
Acceleration threshold (System)
SUT
script
name:
OPT_PLAY_DND_THRESHOLD,
OPT_PLAY_MOVEMOUSE_THRESHOLD
To ensure that small movements as well as the start end end of each movement
remain precise, acceleration is turned off for movements that require less than
this threshold’s number of steps. Good values are between 4 and 8 for
Drag&Drop and between 0 and 6 for demo mouse moves.
33.3.6
Timeouts
These timeouts are essential for reliable replay of tests under varying conditions. They
define how long QF-Test waits for a component to be in the proper state for an event
before throwing an exception.
Don’t make these values too small, so a little hiccup due to high load won’t interrupt
a test needlessly. QF-Test continues as soon as the conditions for replaying an event
are met, so higher values for the timeouts won’t slow down execution (except for focus,
see below). On the other hand, don’t set any values higher than a few seconds or you’ll
have to wait too long until you finally get an error message when a component is truly
not found.
33.3.
Replay options
373
Figure 33.24: Timeout options
Deadlock detection (s) (System)
Server script name: OPT_PLAY_TIMEOUT_DEADLOCK
(664)
If the SUT does not react for the given time, a DeadlockTimeoutException
is thrown. Setting this value to 0 will suppress deadlock detection.
Wait for GUI engine (ms) (System)
SUT script name: OPT_PLAY_TIMEOUT_ENGINE
This option is useful only for multi-engine SUTs, like Eclipse with embedded
(529)
AWT/Swing components. A Wait for client to connect
node finishes as soon as
33.3.
Replay options
374
(530)
the first GUI engine connects to QF-Test, unless its GUI engine
attribute specifies to wait for a specific engine. To prevent a subsequent
(604)
Wait for component to appear
node for a component of the wrong engine from
failing immediately, QF-Test first waits for the time specified with this option to
give the second GUI engine a chance to connect also.
Wait for non-existent component (ms) (System)
SUT script name: OPT_PLAY_TIMEOUT_COMPONENT
The maximum time in milliseconds that QF-Test waits for the target component of
an event to become visible. When the connection between QF-Test and the SUT
is established, this option is temporarily set to at least 30000 ms so as to allow
the SUT time for its initialization.
Wait for non-existent item (ms) (System)
SUT script name: OPT_PLAY_TIMEOUT_ITEM
If an event is targeted on a sub-item of a complex component, QF-Test first waits
for the component to become visible. Then it gives the SUT the chance to make
the intended sub-item available before this timeout is exceeded.
Default timeout for checks (ms) (System)
Server script name: OPT_PLAY_CHECK_TIMEOUT
(564)
This option defines a default timeout for Check nodes that have no Timeout
attribute set and that represent an actual check in the report instead of being
used for test control flow, i.e. checks that don’t throw an exception and don’t set a
result variable or that have a @report doctag.
If your tests include a lot of Check nodes without explicitly defined Timeout that
are expected to fail - which is unlikely for the actual checks described above you may be able speed up test execution by setting this value to 0. However,
it would be preferable to set the Timeout attribute of the respective nodes to 0
instead and leave this option unchanged because it increases general stability of
check execution.
Wait for modal dialog (ms) (System)
SUT script name: OPT_PLAY_TIMEOUT_MODAL
If an event is targeted at a window that is blocked by a modal dialog, a
(663)
ModalDialogException
will be thrown. However, modal dialogs are often
temporary, informing the user about some ongoing processing. If this option is set
to a non-zero value, QF-Test will wait for the given time before it throws the
33.3.
Replay options
375
exception. If the modal dialog disappears before the time limit is exceeded, the
test will continue immediately. This greatly simplifies handling of temporary modal
dialogs.
Note
(604) (335)
If the option Convert opening of a window into Wait for component to appear
is activated, recording a sequence of events during which a temporary dialog is
(604)
displayed may result in a Wait for component to appear node for that dialog. If the
dialog is displayed for a short time only, it is best to remove such nodes to avoid
timing issues. If the SUT employs temporary modal dialogs often it may be best to
(604)
disable the option Convert opening of a window into Wait for component to appear
(335)
.
Wait for ’busy’ GlassPane (ms) (System)
Swing
SUT script name: OPT_PLAY_TIMEOUT_GLASSPANE
As an alternative to temporary modal dialogs some applications employ a so
called GlassPane together with a ’busy’ mouse cursor (typically in the form of an
hourglass) to inform the user that the application is busy. The GlassPane is an
invisible component that covers an entire window. If an event is delivered to such
a window, the GlassPane will typically intercept it, preventing normal event
processing, which can throw a test-run severely off course.
QF-Test handles this case automatically by waiting until the GlassPane disappears
before delivering an event, performing a check, etc. If the timeout given in this
(663)
option is exceeded and the GlassPane is still active, a BusyPaneException
is thrown.
If the value of this option is set to 0, GlassPane checking is disabled and the event
is delivered regardless. A BusyPaneException is never thrown in this case.
(604)
The Wait for component to appear node is a special case. When waiting for a com(606)
ponent (not its absence) covered by a busy GlassPane, the Timeout attribute of
the node overrides this option and is used to wait for both, the appearance of the
component and the removal of the busy GlassPane. Thus it is possible to handle
cases where the SUT is expected to be busy for an exceptionally long time on an
individual basis without changing the default timeout of this option.
Wait for button/menu enable (ms) (System)
SUT script name: OPT_PLAY_TIMEOUT_ENABLED
A MouseEvent on a button or menu item is simply ignored, if the component is
not enabled. This could lead to unwanted side effects during a test, so QF-Test
waits until the component is enabled or the specified timeout is exceeded. If
the component does not become activated within the given time, a
(663)
DisabledComponentException
is thrown unless the option Throw
(362)
DisabledComponentException is deactivated.
33.3.
Replay options
376
Wait for focus (ms) (System)
SUT script name: OPT_PLAY_TIMEOUT_FOCUS
If set, this timeout causes QF-Test to wait for a component to get the keyboard
focus before it simulates any KeyEvents on it. This option actually can slow down
a test noticeably if a component fails to get the focus, so don’t set it higher than
about 100. A good value is 20.
Poll interval for component wait (ms) (System)
SUT script name: OPT_PLAY_POLL_COMPONENT
When waiting for a component in the SUT to appear, QF-Test can’t always rely on
Java’s event mechanism. Instead it has to repeatedly scan the SUT for the
component. This option determines the interval between searches.
Sub-item poll interval (ms) (System)
SUT script name: OPT_PLAY_POLL_ITEM
When waiting for a non-existent sub-item in the SUT, QF-Test can’t rely on the
event mechanism. Instead it has to repeatedly search the complex component for
the sub-item. This option determines the interval between searches.
Retry check interval (ms) (System)
SUT script name: OPT_PLAY_POLL_CHECK
(560)
(564)
fails for which a Timeout
is defined, QF-Test repeatedly queries
If a Check
the component’s state until either it matches the given values or the time is up.
The interval to wait between queries is set with this option.
33.3.7
Backward compatibility
These options can re-set QF-Test to older behavior. Those settings have changed that
much that QF-Test cannot keep backward compatibility over all versions.
Figure 33.25: Options for replay backward compatibility
33.4.
Web options
377
Don’t evaluate variable boolean attributes (before 4.2) (System)
4.2+
Server script name: OPT_PLAY_DONT_EVALUATE_BOOLEAN_OPTIONS
Expressions in values of attributes with variable boolean values will be evaluated
by Jython since 4.2.0. Such attributes are attributes like the Replay as ”hard” event
attribute of Mouse event node or the Modal attribute of Window nodes.
Use old-style GNU regexps (before 3.1) (System)
Server or SUT script name: OPT_USE_GNU_REGEXP
Since version 3.1 QF-Test uses the standard Java regular expression syntax by
default. In case this causes problems with regular expressions in older tests, you
can switch back to using the GNU regexp package via this option. For further
(720)
information about regular expressions please see section 40.4 .
Old style image grab (before 2) (System)
Swing
SUT script name: OPT_RECORD_CHECK_IMAGE_OLD_STYLE
(581)
When recording images for Check image nodes, non-opaque components used
to be recorded with a black background. Thus, the image could deviate from what
the user was seeing. This error has been corrected and normally the background
is now recorded correctly. If this option is activated, the old, broken mechanism is
used instead which may be useful if many checks for non-opaque components
have already been recorded using that method.
33.4
Web options
Web
The following options are used specifically for web testing.
33.4.
Web options
378
Figure 33.26: Web options
Turn ’ID’ attribute into name where ”unique enough” (System)
SUT script name: OPT_WEB_ID_AS_NAME
Use the ID of a DOM node as the component name, provided the ID is sufficiently
unique. Uniqueness is determined per node type and depending on the setting of
(365)
(343)
the options Name override mode (replay) and Name override mode (record) .
Eliminate all numerals from ’ID’ attributes (System)
Note
SUT script name: OPT_WEB_SUPPRESS_NUMERALS
Of course this option only changes the way QF-Test treats ID attributes. The
attributes themselves are left unchanged or the application would most likely no
longer work.
With this option activated QF-Test removes all numerals from ’ID’ attributes to prevent problems caused by automatically generated IDs often found in Ajax frameworks like GWT. Such dynamic IDs can change after the slightest modification to
33.4.
Web options
379
a web application which causes tests to break, especially if names are based on
IDs. By removing the dynamic part from such IDs they become less useful, because they are no longer unique, but also less harmful. Uniqueness of names
(646)
is taken care of by QF-Test. Since IDs also serve as a basis for Feature
and
(646)
Extra features attributes, this option is helpful even if IDs are not used as names.
Limit URL feature of ’Web page’ node to host or file (System)
SUT script name: OPT_WEB_LIMIT_URL
If this option is active, all pages coming from the same host are recorded as the
same page by reducing the URL feature to the host part of the URL. This is often
useful when pages share a common look and navigation structure.
For file URLs, the URL is reduced to the filename, with intermediate directories
removed.
Retarget mouse event on trivial node to parent (System)
SUT script name: OPT_WEB_RETARGET_MOUSE_EVENT
When recording mouse events on DOM nodes in a web page it is often useful to
ignore ”trivial” nodes and concentrate on the important ones. For example, when
clicking on a text hyperlink it is typically not of interest whether part of the link is
formatted with a bold font. It is the link that is important.
If this option is active QF-Test does not simply record the event for the deepest
DOM node under the mouse cursor. Instead it moves up the hierarchy until it finds
an ”interesting” node. In the example above, QF-Test would record the event on
the A node with the option active and on the contained B node otherwise.
Tolerate intermediate parent components (System)
SUT script name: OPT_WEB_TOLERATE_INTERMEDIATE_PARENT
Normally QF-Test’s component recognition is tolerant to changes in the
component hierarchy. For web pages with deeply nested tables this can lead to
performance problems because the potential variants of determining the target
component grow exponentially with the nesting depth. If you experience such
problems, try to deactivate this option. It will reduce adaptability but should help
with performance.
Note
The by far preferable solution is to set unique ID attributes for the different tables
and other components so that QF-Test’s name override mechanism can apply.
This not only speeds up recognition drastically, it is also much more reliable and
tolerant to change.
33.4.
Web options
380
Take visibility of DOM nodes into account (System)
SUT script name: OPT_WEB_TEST_VISIBILITY
Similar to AWT/Swing or SWT, QF-Test normally only recognizes visible DOM
nodes as target components. However, visibility of DOM nodes is not as well
defined as that of components in a Java GUI. For example it is possible that an
invisible DOM node has visible child nodes. Also, if a web page contains illegal
HTML constructs it is possible that a DOM node is considered invisible, even
though it is displayed in the browser window. If you come across such a problem
you can turn off this option.
Let the browser determine the target element for check recording (System)
SUT script name: OPT_WEB_CHECK_VIA_BROWSER
When recording checks, components or procedures QF-Test needs to determine
the target element under the mouse cursor. In case of overlapping nodes there
are two different ways for calculating which one should be used. By default
QF-Test lets the browser decide, which is usually the best choice. Since the
different browsers don’t always behave in the same reliable way, this option can
be turned off in case of problems to use the older mechanism based on the
z-order of elements instead. This option has no effect on check replay.
How to handle JavaScript errors in a web application (System)
SUT script name: OPT_WEB_JAVASCRIPT_ERROR_LEVEL
Possible Values: VAL_WEB_JAVASCRIPT_LEVEL_WARNING,
VAL_WEB_JAVASCRIPT_LEVEL_ERROR
Dynamic HTML and AJAX are implemented via a lot of JavaScript code that is
executed in the Browser. If an error occurs in such a script, browsers either
ignore it or show an error dialog with some details about the error, depending on
user setting. Many of these errors are harmless, others can be severe. QF-Test
intercepts the error message and logs an error or warning in the run-log instead.
This set of options defines the severity of the message and a maximum for the
number of such kinds of messages to log.
The possible choices for the option ”Error level” are ”Error” and ”Warning”. We
advise that you set it to ”Error” and make sure that such problems are reported to
development and fixed sooner rather than later because they can represent a bug
in the application you are testing. Known messages that are not going to be fixed
by development can be excluded and ignored via the option Errors that should be
(379)
ignored .
Maximum number of errors to log per SUT client (System)
33.4.
Web options
381
SUT script name: OPT_WEB_JAVASCRIPT_MAX_ERRORS
In case a web page contains erroneous code it is possible that a lot of errors are
triggered. Logging all those errors can significantly impact test performance, yet
logging more than the first few errors does not really contribute much. The option
”Maximum number of errors to log per SUT client” limits the possible number of
error messages for this case.
Errors that should be ignored (System)
3.5+
SUT script name: OPT_WEB_JAVASCRIPT_ERROR_FILTERS
It is possible that some JavaScript errors cannot or will not be fixed, for example
when they are coming from third-party code. In such a case it is preferable to
ignore the known errors while still having QF-Test report unexpected ones. When
the browsers reports a JavaScript error, QF-Test searches its error message for
the occurrence of any of the regular expressions specified in this option. In case
of a match, the error is ignored. If no exceptions are defined or none match, the
error is reported in accordance with the previous options.
33.4.1
HTTP Requests
These options influence the behavior of HTTP requests.
Figure 33.27: Options for HTTP Requests
Synchronize with HTTP requests (System)
4.1+
Note
SUT script name: OPT_WEB_TRACK_HTTP_REQUESTS
HTTP request tracking is supported for browsers in QF-Driver mode only. It does
not apply to WebDriver based tests or to browsers embedded in Java like
WebView.
33.4.
Web options
382
Because most things in AJAX based web applications run asynchronously, one
of the main challenges in testing such applications is timing. QF-Test uses various means to synchronize with the SUT and this option controls one of them. If
turned on, QF-Test will keep track of all HTTP requests the browser sends to the
server. Before and after replaying an event, QF-Test will wait until no requests are
outstanding. The following two options Timeout for HTTP request synchronization
(380)
(380)
(ms) and Extended HTTP request timeout for new documents (ms) are used
for fine-tuning this feature.
Timeout for HTTP request synchronization (ms) (System)
4.1+
SUT script name: OPT_WEB_HTTP_REQUEST_TIMEOUT
When synchronizing with the SUT by tracking HTTP requests as explained for the
(379)
option Synchronize with HTTP requests , QF-Test cannot wait indefinitely for
outstanding requests as this would impact test performance too much. This
option defines the maximum time to wait for outstanding requests in normal
situations and the following option Extended HTTP request timeout for new
(380)
documents (ms) is used right after a page has finished loading.
Extended HTTP request timeout for new documents (ms) (System)
4.1+
SUT script name: OPT_WEB_HTTP_REQUEST_TIMEOUT_DC
When synchronizing with the SUT by tracking HTTP requests as explained for the
(379)
option Synchronize with HTTP requests , QF-Test cannot wait indefinitely for
outstanding requests as this would impact test performance too much. Right after
a page has finished loading, AJAX applications tend to send a lot of requests and
may take a while to build the final user interface via JavaScript. This options
defines the maximum time to wait for outstanding requests at that point. The
previous option ref=”opt_webrequesttimeout”/> is used to set the standard
timeout for other situations.
Record HTTP Request as browser request (System)
4.1+
SUT script name: OPT_WEB_RECORD_CLIENT_REQUEST_STEP
(631)
is created. This
When recording HTTP Requests a Browser HTTP request
request will be submitted directly via the browser so that the response is shown
afterwards and test execution can be continued directly in the browser. By
(626)
deactivating this option a Server HTTP request will be recorded. This request will
be submitted by QF-Test and doesn’t have any effect on the browser. The
response is only accessible in QF-Test.
33.5.
Debugger options
33.4.2
383
Backward compatibility
These options can re-set QF-Test to older behavior. Those settings have changed that
much that QF-Test cannot keep backward compatibility over all versions.
Figure 33.28: Options for web backward compatibility
4.2+
Don’t fetch text for generic classes with generic class approach (before 4.2) (System)
SUT
script
name:
OPT_WEB_TEXT_DONT_TRAVERSE_ALL_NODES_FOR_GENERICS
(561)
(591)
In versions older than 4.2.0 Check text
and Fetch text
nodes provided too
much or too less text for components of generic classes in some cases.
Especially SELECT or TableCell components containing text-fields returned
wrong text. Now the any child component with a generic class will be taken into
account.
Treat underscores like blanks in extra features (before 4.2) (System)
4.2+
SUT script name: OPT_WEB_TREAT_UNDERSCORES_AS_BLANKS_IN_EF
Older QF-Test versions than 4.2.0 turned all underscores into blanks once Extra
features were compared. This behavior could cause troubles in case you were
really searching for underscores.
33.5
Debugger options
These options modify the behavior of the debugger.
33.5.
Debugger options
384
Figure 33.29: Debugger options
Enable debugger (User)
By default the debugger is disabled unless this option, which can also be
modified through the menu item Debugger→Enable debugger , is activated.
If a test is interrupted by a breakpoint or be pressing the ”Pause” button, the debugger is activated automatically. Similarly, starting a test-run with ”Step in” or
”Step over” will activate the debugger for the duration of that test-run.
Always open debugger window (User)
When test execution is halted and the debugger entered QF-Test can optionally
open a separate window for the debugger. This option determines whether
debugging should happen in a separate window or the normal test-suite window.
Show variable bindings automatically (User)
When test execution is halted and the debugger entered QF-Test can display the
current variable bindings in the workbench window. If this option is active the
variables are shown automatically each time a test-run first enters the debugger.
Alternatively the variables can be viewed in the debugger window or shown in the
workbench window via the menu Debugger→Show variables .
Automatic breaks (User)
These options describe the situations in which execution of a test will be
suspended and the debugger entered:
33.6.
Run-log options
385
Break on uncaught exception
The debugger will break if an exception is thrown that will not be handled by
(493)
a Catch node.
Break on error
The debugger will break if an error occurs.
Break on warning
The debugger will break if a warning occurs.
33.6
Run-log options
These options let you control which information is collected in a run-log, if and when a
run-log is shown and how to locate errors.
33.6.
Run-log options
33.6.1
386
General run-log options
Figure 33.30: General run-log options
When to show run-logs (User)
A run-log is created for every execution of a test. A number of recent run-logs are
available from the Run menu, the most recent run-log can also be opened by
pressing Control-L . Additionally the run-log can be shown during execution or
after an error as follows:
At start
33.6.
Run-log options
387
This choice causes QF-Test to open the run-log when it begins executing a test
sequence. The nodes of the log will be added as execution proceeds.
After finish
With this choice the run-log is shown after replay is finished.
On exception
The run-log is shown only if an uncaught exception is thrown.
Don’t show
The run-log is not
displayed automatically. You have to open it via the Run menu
or by pressing Control-L .
Number of run-logs in menu (User)
A limit for the number of menu items for recent run-logs kept in the Run menu.
Automatically save run-logs (User)
3.0+
To prevent excessive memory use through run-logs and also to make the most
recent run-logs persistent between QF-Test sessions, the recent run-logs kept in
(9)
the Run menu are saved automatically to the user configuration directory or
(385)
the directory defined in the option Directory for run-logs . The filename for the
run-log is based on a timestamp. QF-Test uses file locks to prevent collisions and
accidental removal in case of parallel sessions and automatically keeps the user
(9)
configuration directory clean by removing unreferenced logs, so there should be
no reason to disable this feature. Still, you can do so by disabling this option.
Directory for run-logs (User)
4.0+
By default, run-logs created during interactive use of QF-Test are stored in the
(9)
user configuration directory . This option can be used to specify a different target
directory.
Note
This option is interpreted by QF-Test whenever a test is started. At that point,
test-suite and global variable bindings are already in place and in contrast to
other options it is possible to use QF-Test variable syntax here. This includes
special variables like ${env:HOME} to look up an environment variable or even
${qftest:suite.dir} to save the run-log next to the test-suite. If the directory is dynamic, like in the latter case, QF-Test may not be able to clean up old run-logs regularly. Errors in variable expansion are silently ignored and the user configuration
(9)
directory is used instead.
Show expanded variable values in tree nodes (User)
3.1+
33.6.
Run-log options
388
The nodes in the tree view of a run-log can either be displayed with variables
expanded to the value they had at run-time or with the original variables. Both
views have their use, so you can toggle between them via this option or more
quickly via the menu item View→Show nodes expanded .
Find next error function locates (User)
The search for an error in a run-log,
accessible through the Edit→Find next error
menu item or by pressing Control-N , is a very useful utility. This option
determines whether the search stops at exceptions only, errors and exceptions or
even warnings.
Skip suppressed errors (User)
Like the previous one, this option controls the search for errors in a run-log. If
activated, warnings, errors or exceptions that have not propagated to the top, are
(490)
(493)
clause or messages
not found. Thus exceptions caught by a Try /Catch
(416)
suppressed through the Maximum error level attribute are skipped.
This option is also accessible through the Edit→Skip suppressed errors menu
item.
Cleanup tree when jumping to next or previous error (User)
3.0+
When repeatedly jumping to errors in a run-log the tree can easily get cluttered
with many expanded nodes. If this option is activated, QF-Test will automatically
clean up the tree each time you navigate to an error so that only the parents of
the error node are expanded.
Note
When viewing split run-logs, partial run-logs containing an error will remain in
memory as long as their nodes are expanded. Keeping this option activated will
ensure that partial run-logs will be released as soon as possible, keeping memory
use manageable even viewing the errors of a very large run-logs.
Save compressed run-logs (*.qrz) (System)
Server script name: OPT_LOG_SAVE_COMPRESSED
Run-logs can either be saved as plain or as compressed XML files. For large
run-logs without screenshots the compression factor can be as high as 10, so it’s
advisable to use compressed logs where possible. The only reason not to use
compression is if you want to transform the XML run-log afterwards. But even
then compressed run-logs are an option because the compression method used
is standard gzip format, so converting to and from compressed run-logs can
easily be done using gzip.
33.6.
Run-log options
389
When saving a log-file interactively you can always switch between compressed
or non-compressed format by choosing the appropriate filter or by giving the file
the extension .qrz or .qrl.
In batch mode, the default run-log format is compressed. To create an
uncompressed run-log, simply specify the extension .qrl in the parameter for
(688)
the -runlog [<file>] command line argument.
Create split run-logs (User)
3.0+
Server script name: OPT_LOG_SAVE_SPLIT
(442)
A run-log can be split into several parts by setting the Name for separate run-log
(440)
attribute of a Data driver or any of the various test nodes. By turning this option
off you can temporarily disable support for split run-logs in order to get a normal,
single run-log without having to modify any Name for separate run-log attributes.
(78)
See section 8.1.4
for further information about split run-logs.
Save split run-logs as ZIP files (*.qzp) (User)
3.0+
Server script name: OPT_LOG_SAVE_SPLIT_ZIP
Split run-logs can either be saved as a single ZIP file with the extension .qzp,
containing the main run-log and all partial logs together, or as a normal .qrl or
.qrz run-log that is accompanied by a directory with the same base name and
the suffix _logs, e.g. the file runlog.qrz plus the directory runlog_logs.
This option determines the format in which split run-logs are created in interactive
(385)
mode. It has no effect if the option Automatically save run-logs is turned off.
(78)
See section 8.1.4
for further information about split run-logs.
Minimum size for automatic splitting (kB) (System)
3.4+
Note
Server script name: OPT_LOG_AUTO_SPLIT_SIZE
This option exclusively applies to Test-case and Test-set nodes. At other places
run-logs are split only when Name for separate run-log has explicitly been set.
Split run-logs are the only reliable way to prevent running out of memory during
very long running tests or when the run-log grows quickly due to screenshots or
output from the SUT. They are also more efficient when transforming run-logs into
reports. However, explicit setting of Name for separate run-log attributes requires
an understanding of the issues involved and either making decisions about where
best to split a run-log or tedious typing when trying to split into small pieces.
As a compromise, QF-Test makes a very rough calculation about the size of a runlog during executing, taking screenshots and program output into account. When
execution of a Test-case or Test-set has finished and the approximate size of the
33.6.
Run-log options
390
run-log pertaining to that node is larger than the threshold specified in this option,
the run-log is split off and saved automatically. A value of 0 prevents automatic
splitting.
(78)
See section 8.1.4
for further information about split run-logs.
Name for automatically split ’Test-case’ run-log (System)
3.4+
Server script name: OPT_LOG_AUTO_SPLIT_TESTCASE_NAME
This option specifies the name to use for an external log when it is split off
automatically after execution of a Test-case has finished as described in the
previous option. Variables can be used as well as the ’%...’ placeholders
(401)
documented for the attribute Name for separate run-log .
The special variable ${qftest:testcase.splitlogname} is a good base. It expands to
a path name created from the name of the Test-case with possible parent Test-set
nodes as directories.
(78)
See section 8.1.4
for further information about split run-logs.
Name for automatically split ’Test-set’ run-log (System)
3.4+
Server script name: OPT_LOG_AUTO_SPLIT_TESTSET_NAME
This option specifies the name to use for an external log when it is split off
automatically after execution of a Test-set has finished as described in the option
(387)
Minimum size for automatic splitting (kB) . Variables can be used as well as the
(408)
’%...’ placeholders documented for the attribute Name for separate run-log .
The special variable ${qftest:testset.splitlogname} is a good base. It expands to
a path name created from the name of the Test-set with possible parent Test-set
nodes as directories.
(78)
See section 8.1.4
for further information about split run-logs.
33.6.
Run-log options
33.6.2
391
Options determining run-log content
Figure 33.31: Options determining run-log content
Log variable expansion (System)
Server script name: OPT_LOG_CONTENT_VARIABLE_EXPANSION
(62)
If this option is activated, every variable expansion is logged.
Log parent nodes of components (System)
Server script name: OPT_LOG_CONTENT_PARENT_COMPONENTS
Setting this option will cause all direct and indirect parent nodes to be logged in
addition to the target component node for every event, check, etc.
Log level for the SUT (System)
Server script name: OPT_LOG_CONTENT_SUT_LEVEL
Possible Values: VAL_LOG_SUT_LEVEL_MESSAGE,
33.6.
Run-log options
392
VAL_LOG_SUT_LEVEL_WARNING,
VAL_LOG_SUT_LEVEL_ERROR
The level for automatically generated messages in the SUT during replay, e.g.
details for component recognition. Only messages with the respective level, i.e.
plain messages, warnings or errors will be logged. This option has no effect on
messages created explicitly via rc.logMessage or qf.logMessage.
Number of events to log for error diagnosis (System)
SUT script name: OPT_LOG_CONTENT_DIAGNOSIS
During replay of a test QF-Test logs various events and other things going on
behind the scenes. This information is quickly discarded except when an error
happens. In that case the most recent events are written to a special run-log
node. The information may also be useful to developers but is mostly required for
error diagnosis when requesting support from Quality First Software GmbH.
This option determines the number of recent internal events to keep. Setting it to
0 disables the feature altogether. You should not set this value to less than about
400 without a good reason. Because the information is logged only for errors, the
cost for gathering it is minimal.
Maximum number of errors with screenshots per run-log (System)
Server script name: OPT_LOG_CONTENT_SCREENSHOT_MAX
The maximum number of screenshots that QF-Test takes and stores in the
run-log during a test-run on situations of exception or errors. Setting this value to
0 disables taking screenshots entirely, a negative value means unlimited
screenshots.
Count screenshots individually for each split log (System)
Server script name: OPT_LOG_CONTENT_SCREENSHOT_PER_SPLIT_LOG
If this option is set, each partial log of a split run-log may contain the maximum
number of screenshots defined above without affecting the count for the main
run-log. Otherwise the limit applies for the sum of all parts belonging to the same
main run-log.
(78)
See section 8.1.4
for further information about split run-logs.
Create screenshots of the whole screen (System)
Server script name: OPT_LOG_CONTENT_SCREENSHOT_FULLSCREEN
Activating this option causes QF-Test to take an image of the whole screen and
save it in the run-log when a screenshot is triggered by an exception or error.
33.6.
Run-log options
393
Create screenshots of individual windows (System)
Server script name: OPT_LOG_CONTENT_SCREENSHOT_WINDOW
Activating this option causes QF-Test to record images of all windows and dialogs
of the SUT and store them in the run-log when screenshots are triggered due to
exceptions or errors. Most of the time this will work even for windows that are
covered by other windows or in cases where a full screenshot is not possible, for
example when a screen is locked.
Log successful advanced image checks (System)
3.4+
Server
script
name:
OPT_LOG_CONTENT_SCREENSHOT_ADVANCED_ALWAYS
If this option is activated, QF-Test will store the expected and actual images as
well as the transformed images for successful advanced image checks in the
run-log. Otherwise these details are kept for failed image checks only.
Activating this option can raise the size of the run-log drastically so be sure to use
it in combination with compact run-logs and split run-logs.
Create compact run-log (System)
Server script name: OPT_LOG_CONTENT_COMPACT
Activating this option causes QF-Test to discard every node from a run-log that is
neither relevant for error diagnosis, nor for the XML/HTML report. After an error
or exception, as well as at the end of a test-run, the 100 most recent nodes are
not discarded, so all the relevant information should remain available.
Large tests should almost always be run with this option enabled to reduce the risk
of running out of memory due to excessively large run-logs.
This option is used only when QF-Test is run in interactive mode. It is ignored in
(10)
batch mode (see section 1.7 ) to avoid accidental loss of information. To create
(679)
a compact run-log in batch mode, use the -compact command line argument.
Don’t create run-log (System)
Server script name: OPT_LOG_CONTENT_SUPPRESS
For very-long-running tests or demos that are run in an endless loop, memory
consumption of the run-log is an issue, but split run-logs are an ideal solution.
Before split run-logs were available, turning run-logs off completely via this option
was sometimes the only way to get long-running tests to work. Now this option is
only retained for backwards compatibility.
33.6.
Note
Run-log options
394
In batch mode this option is ignored. To suppress the run-log, use the argument
(684)
-nolog .
Log SUT output individually (System)
3.0+
SUT script name: OPT_LOG_CONTENT_SPLIT_IO
If set, any text that an SUT client prints to its stdout or stderr stream is also
logged in the run-log. For each interaction with the SUT QF-Test collects text
printed before the event and after the event during synchronization. This makes it
possible to associate output like an exception stacktrace that is triggered by an
event with the event itself, something that is impossible if all output is kept in a
single piece.
Compactify individually logged SUT output (System)
3.0+
Server script name: OPT_LOG_CONTENT_COMPACTIFY_SPLIT_IO
Output from an SUT client tends to accumulate and can consume a lot of
memory. If this option is activated, individually logged SUT output for events that
are no longer of interest can be removed along with the events in compact
(391)
run-logs. Please see the option Create compact run-log for further information
about compact run-logs.
33.6.3
Options for mapping between directories with test-suites
4.0+
Figure 33.32: Options for mapping between directories with test-suites
Directory map for test-suites (System)
When analyzing a run-log it is often necessary to quickly switch between the
run-log and the respective test-suite. However, when running automated tests on
different systems like Windows and Linux, the directories from which test-suites
are loaded during the test vary and there is no automatic way to map between
different directory layouts.
33.7.
Variables
395
With this option you can assist QF-Test in locating test-suites. The ’From’ column
is a glob pattern that must match from the beginning of the path of the test-suite
stored in the run-log to the end of some directory in that path. The ’To’ column is
the respective replacement for the matching part and can also contain a glob pattern. When searching for the test-suite, QF-Test processes this list top to bottom,
performing the replacement for every match found and the first match leading to
an actual test-suite is used.
Note
A glob patterns is a simpler form of a regular expression often used by development tools: An ’*’ stands for any number of characters up to the next file separator
while ’**’ means 0 or more characters of any kind, including ’/’. Some examples:
**/test-suites
All directories named test-suites at any depth.
T:/test/sut_*
All directories starting with sut_ in the T:/test directory.
33.7
Variables
The following options pertain to variable binding.
Figure 33.33: Variable options
When binding variables, expand values immediately (System)
3.0+
Server script name: OPT_VARIABLE_IMMEDIATE_BINDING
When a set of variable bindings is pushed onto a variable stack, any additional
variable references in the values of these variables can either be expanded
immediately, or the values can be left unchanged to be expanded lazily as
(72)
needed. This is explained in detail in section 7.7 .
For immediate expansion turn this option on, for lazy expansion turn it off.
33.7.
Variables
396
Fall back to lazy binding if immediate binding fails (System)
3.0+
Server script name: OPT_VARIABLE_LAZY_FALLBACK
Old tests that rely on lazy variable binding may fail with the new default of
immediate binding. If this option is activated, variables that cannot be bound
immediately due to references to not-yet-defined variables are bound lazily
instead and a warning is issued. To get rid of the warning simply change the
value of the respective variable to use explicit lazy binding with ’$_’. Please see
(72)
section 7.7 for further information.
Furthermore, various kinds of variables can be defined here. These are explained in
(62)
chapter 7 .
Chapter 34
Elements of a test-suite
34.1
The test-suite and its structure
There are more than 60 different kinds of nodes for a test-suite which are all listed in
this chapter. Each node type has a unique set of features. The attributes of a node are
displayed and edited in the detail view of the editor. The restrictions that apply to each
(62)
attribute are listed as well as whether it supports variable expansion (see chapter 7 ).
Additional features of a node include the behavior of the node during execution of a test
and the kinds of parent and child nodes allowed.
34.1.1 Test-suite
The root node of the tree represents the test-suite itself.
Its basic
structure is fixed.
The root node contains an arbitrary number of
(405)
(398)
(472)
(425)
Test-sets or Test-cases , followed by the Procedures the Extras and the
(655)
Windows and components . When executed, the top-level Test-sets are executed one by
one.
Contained in: None
Children: An arbitrary number of Test-set or Test-case nodes, followed by the Procedures,
Extras and Windows and components nodes.
Execution: The top-level Test-set nodes are executed one by one.
Attributes:
34.1.
The test-suite and its structure
398
Figure 34.1: Test-suite attributes
Include files
(644)
or
This is a list of test-suites that are included by the suite. If a Component
(463)
Procedure reference cannot be resolved in the current suite, the included suites
are searched for it. When recording new components, QF-Test will search the
included suites for a matching Component node before creating a new one.
Relative pathnames are treated relative to the directory of the suite, or to a direc(331)
tory on the library path (see option Directories holding test-suite libraries ).
When you change anything in this attribute QF-Test will offer to update all affected
nodes to compensate for the change. For example, if you add or remove a suite
from the includes, QF-Test will make all references to that suite’s Procedures or
Components implicit or absolute so that the actual referenced nodes remain unchanged. In such a case, choose ”Yes”. If, on the other hand, you renamed a suite
or moved it to some other directory and are simply updating the includes to reflect
that, chose ”No” so all former implicit references into the old suite will now point to
the new one.
Variable: No
Restrictions: None
34.1.
The test-suite and its structure
399
Dependencies (reverse includes)
This list of test-suites is the reverse of the Include files attribute. It has no impact
on test execution. Instead it serves a hint to tell QF-Test which test-suites depend
on Components in this suite, either because they (directly or indirectly) include this
suite or because they explicitly reference Components in it. This information is
used when QF-Test IDs of Components are manipulated (for example after
(50)
updating components, see (section 5.8 )) and the QF-Test component ID
attributes of nodes depending on these components have to be updated. QF-Test
always checks all currently loaded suites for dependencies. Additionally, it will
automatically load and check all suites listed here.
Relative pathnames are treated relative to the directory of the suite, or to a direc(331)
tory on the library path (see option Directories holding test-suite libraries ).
Note
Like the Include files, the Dependencies are also resolved indirectly. For example if
suite A has the dependency B which has the dependency C, both suites B and C
will be loaded and checked for references automatically when components in suite
A are manipulated.
Variable: No
Restrictions: None
Variable definitions
These variables definitions are identical to the suite variable bindings accessible
(393)
pane of the global options. A detailed explanation of variable
in the Variables
(62)
(16)
definition and lookup is given in chapter 7 . See section 2.2.5 about how to
work with the table.
Variable: Variable names no, values yes
Restrictions: None
Execution timeout
Time limit for the node’s execution in milliseconds.
execution of that node will get interrupted.
If that limit expires the
Variable: Yes
Restrictions: >= 0
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
are much better suited to this task. The option External editor command
lets
34.2.
Test and Sequence nodes
400
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.2
Test and Sequence nodes
(398)
Tests and sequences are the main structural elements of a test-suite. Test-case nodes
represent logical test-cases and are implemented as specialized sequences. A sequence is a container that executes its child nodes one by one. It can define variables
(62)
(see chapter 7 ) that remain bound during the execution of the children.
Other kinds of sequences differ either in the way their child nodes are executed, or in
the restrictions they impose on their child or parent nodes.
34.2.1 Test-case
A Test-case node represents one or more logical test cases. In a sense it is the
most important of all QF-Test nodes and everything else only serves to lend
structure to Test-cases or to implement their logic. Functionally it is a highly
(415)
specialized Sequence with a number of important extensions and special attributes.
A Test-case should focus on the actual test it supposed to perform. Setup and cleanup
tasks required to ensure that the Test-case executes in the required environment and
does not interfere with subsequent tests should be implemented in the form of
(427)
(94)
Dependencies
as described in section 9.6 . Alternatively - or in addition to
(433)
(435)
Dependencies - a Test-case can have Setup
and Cleanup
nodes to be executed
before and after the Test-case.
(411)
node it is also somewhat simiBecause a Test-case can be called from a Test call
(463)
(400)
lar to a Procedure
in that its Name
attribute is mandatory and that it has a list of
(403)
Parameter default values that can be overridden in the calling node.
Test-cases also play a central role in run-logs and test reports. During a test-run a Testcase node can be executed several times in different contexts and with different parameters. Logically these executions may represent the same or different test-cases.
(401)
By defining a set of Characteristic variables you can specify which variable values are
used to differentiate between executions, thus characterizing the run-time environment
of the test. The values of these variables at the time of entry to the Test-case are
34.2.
Test and Sequence nodes
401
stored in the run-log. To emphasize the fact that each execution of a Test-case node
may represent a separate logical test-case there is an alternative name attribute called
(401)
Name for run-log and report . Its value may contain references to the Characteristic variables of the test. In the run-log or report the test will then be listed with this name,
including the expanded run-time variable values.
Finally, there are situations in which a test cannot or should not be executed for specific
(402)
variable settings. If the Condition
attribute of the Test-case is defined, the Test-case
will only be executed if that expression expands to a true value. If the Test-case is not
executed due to the Condition it will be listed as skipped in the report.
(395)
Contained in: Test-suite
(405)
, Test-set
.
(427)
(430)
Children: Optional Dependency
or Dependency reference
as the first element. A
(433)
(435)
Setup
may be the next and a Cleanup
the last node with an arbitrary number of
normal child nodes in between. A Test-case that does not contain normal child nodes will
be listed as not implemented in the report.
(403)
of the Test-case are bound on the primary
Execution: First the Variable definitions
(403)
(62)
and its Parameter default values on the fallback variable stack (see chapter 7 ). With
(402)
these in place the Condition is evaluated and the node will be skipped if a non-empty
Condition evaluates to false. Next the dependency of the Test-case - possibly inherited
(94)
from its parent node - is determined and resolved as described in section 9.6 . Then
the optional Setup is executed once, followed by the normal child nodes and a single
execution of the optional Cleanup. An exception raised during the course of the Test-case
(493)
will be caught and passed to the Dependency for handling in a Catch node. Even if the
exception is not handled by a Dependency it is not propagated beyond the Test-case to
prevent aborting the whole test-run. The error state is duly noted in run-log and report
however.
Attributes:
34.2.
Test and Sequence nodes
Figure 34.2: Test-case attributes
Name
402
34.2.
Test and Sequence nodes
403
(405)
A Test-case is identified by its name and the names of its Test-set ancestors, so
you should assign a name with a meaning that is easy to recognize and
remember.
Variable: No
Restrictions: Must not be empty or contain the characters ’.’ or ’#’.
Name for run-log and report
A separate name to be used for run-log and report. This is useful to differentiate
between multiple executions with potentially different values for the Characteristic
variables.
Variable: Yes
Restrictions: None
Characteristic variables
These variables are part of the characteristics of a Test-set or Test-case. Two
executions of a Test-case are considered to represent the same logical test-case if
the run-time values of all Characteristic variables are identical. The run-time values
of the Characteristic variables are stored in the run-log.
Variable: No
Restrictions: None
Name for separate run-log
If this attribute is set it marks the node as a breaking point for split run-logs and
defines the filename for the partial log. When the node finishes, the respective
log entry is removed from the main run-log and saved as a separate, partial
run-log. This operation is completely transparent, the main run-log retains
(78)
references to the partial logs and is fully controllable. Please see section 8.1.4
for further information about split run-logs.
(387)
This attribute has no effect if the option Create split run-logs is disabled or split
(690)
run-logs are explicitly turned off for batch mode via the -splitlog command
line argument.
There is no need to keep the filename unique. If necessary, QF-Test appends a
number to the filename to avoid collisions. The filename may contain directories
and, similar to specifying the name of a run-log in batch mode on the command
line, the following placeholders can be used after a ’%’ or a ’+’ character:
34.2.
Test and Sequence nodes
Character
%
+
i
r
w
e
x
t
y
Y
M
d
h
m
s
404
Replacement
Literal ’%’ character.
Literal ’+’ character.
(688)
The current runid as specified with -runid <ID> .
The error level of the partial log.
The number of warnings in the partial log.
The number of errors in the partial log.
The number of exceptions in the partial log.
The thread index to which the partial log belongs (for tests run with parallel threads).
The current year (2 digits).
The current year (4 digits).
The current month (2 digits).
The current day (2 digits).
The current hour (2 digits).
The current minute (2 digits).
The current second (2 digits).
Table 34.1: Placeholders for the Name for separate run-log attribute
Variable: Yes
Restrictions: None, characters that are illegal for a filename will be replaced with
’_’.
Inherit dependency from parent node
(427)
This option allows inheriting the Dependency
from the parent node as a
replacement for or in addition to specifying a Dependency for the node itself.
Variable: No
Restrictions: None
Condition
If the condition is non-empty it will be evaluated and if the result is false the
execution of the current node is aborted. In this case the node will be reported as
skipped.
(482)
(481)
Like the Condition
of an If
node, the Condition is evaluated by the Jython
interpreter, so the same rules apply.
Variable: Yes
Restrictions: Valid syntax
Expected to fail if...
34.2.
Test and Sequence nodes
405
This attribute allows specifying a condition under which the Test-case is expected
to fail. This is helpful to distinguish new problems from already known ones. Of
course the latter should be fixed, but if and when that happens may be outside
the tester’s sphere of influence.
Most of the time it is sufficient to simply set this attribute to ’true’ to mark the Testcase as an expected failure. But if the Test-case is executed multiple times, e.g.
(440)
or on multiple systems and fails only in specific cases, the
within a Data driver
condition should be written so that it evaluates to true for exactly those cases, e.g.
${qftest:windows} for a test that fails on Windows but runs fine on other systems.
If this attribute evaluates to true and the Test-case fails with an error, it will be listed
separately in the run-log, report and on the status line. It still means that there
is an error in the application, so the overall percentage of successful tests is not
changed.
It is treated as an error if this attribute evaluates to true and the Test-case does not
fail because this means that either the test is not able to reproduce the problem
reliably or that the problem has been fixed and the Test-case must be updated
accordingly.
Variable: Yes
Restrictions: Valid syntax
Variable definitions
(62)
These variables are bound on the direct bindings stack (see chapter 7 ). They
remain valid during the execution of the Test-case’s child nodes and cannot be
(411)
(16)
overridden by a Test call node. See section 2.2.5 about how to work with the
table.
Variable: Variable names no, values yes
Restrictions: None
Parameter default values
Here you can define default or fallback values for the Test-case’s parameters (see
(62)
chapter 7 ). Defining these values also serves as documentation and is a
valuable time-saver when using the dialog to select the Test-case for the
(412)
(411)
(16)
Test name attribute of a Test call . See section 2.2.5 about how to work with
the table.
Variable: Variable names no, values yes
Restrictions: None
Maximum error level
When a warning, error or exception occurs during a test run, the state of the
corresponding node of the run-log is set accordingly. This state is normally
34.2.
Test and Sequence nodes
406
propagated to the parent node in a way that ensures that the error state of a
run-log node represents the worst of its child nodes’ states. Using this attribute,
the maximum error state that the run-log node for a sequence will propagate, can
be limited.
Note
This value has no effect on the way exceptions are handled. It only affects the
error states of the run-log nodes and by that the exit code of QF-Test when run in
(10)
batch mode (see. section 1.7 ). It also has no effect on the creation of compact
(679)
run-logs (see command line argument -compact ). The node for a sequence in
which a warning, error or exception occurs is never removed from a compact log,
even if the error is not propagated due to the setting of this attribute.
Variable: No
Restrictions: None
Execution timeout
Time limit for the node’s execution in milliseconds.
execution of that node will get interrupted.
If that limit expires the
Variable: Yes
Restrictions: >= 0
Border for relative calls
This flag determines whether relative procedure calls, test calls or dependency
references are allowed within that certain node. Relative calls passing that border
are not allowed. If that attribute is not specified in the hierarchy, no relative calls
are allowed.
Variable: No
Restrictions: None
QF-Test ID
(691)
for execution in
When using the command line argument -test <n>|<ID>
batch mode you can specify the QF-Test ID of the node as an alternative to its
qualified name.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
34.2.
Test and Sequence nodes
407
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
lets
are much better suited to this task. The option External editor command
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.2.2 Test-set
(398)
The main purpose of a Test-set is to give structure to the Test-cases
of a
(407)
of a Test-set is part of the
test-suite. Test-sets can be nested. The Name
(411)
Test-case’s fully qualified name to which a Test call node refers. Test-sets are
(409)
also callable themselves and thus have a set of Parameter default values
that can be
overridden in the Test call.
One way of structuring Test-cases is to collect those with similar setup and cleanup
(427)
or
requirements in the same Test-set which can provide a Dependency
(430)
Dependency reference
node to be inherited by the Test-case or nested Test-set child
(409)
nodes. If the Inherit dependency from parent node
attribute of the Test-set is set, the
(94)
Dependency nodes of the Test-set’s parent(s) are also inherited. See section 9.6 for
information about QF-Test’s dependency mechanism. Alternative or additional setup
(433)
(435)
and cleanup is available in the form of Setup
and Cleanup
nodes which will be
executed before and after each of the Test-set and Test-case child nodes.
Another important feature of a Test-set is data-driven testing. This can be achieved by
(440)
(159)
adding a Data driver node with one or more Data binders as described in chapter 16 .
Like a Test-case, a Test-set plays an important role for reports. As it can also be executed
(408)
several times with different parameter settings, it has a set of Characteristic variables
(407)
and an alternative Name for run-log and report
that work just like for a Test-case. The
(409)
same is true for the Condition which can be used to skip an entire Test-set depending
on current variable values.
(395)
Contained in: Test-suite
(405)
, Test-set
Children: Optional Dependency
(427)
.
(430)
or Dependency reference
as the first child, followed
34.2.
Test and Sequence nodes
(440)
408
(433)
(435)
by an optional Data driver . A Setup may come next and a Cleanup at the end with
(405)
(398)
(411)
an arbitrary number of Test-set , Test-case and Test call nodes in between.
(409)
Execution: First the Parameter default values of the Test-set are bound on the fallback
(62)
(409)
variable stack (see chapter 7 ). With these in place the Condition is evaluated and
(427)
the node will be skipped if a non-empty Condition evaluates to false. The Dependency
(430)
or Dependency reference
node will not be resolved at this time unless its
(429)
Always execute, even in test-suite and test-set nodes
attribute is set.
If there is a
(440)
Data driver
node, it is executed to create a data driving context, bind one or more
Data binders and then repeatedly execute the other child nodes as described in chapter
(159)
16 . For each loop iteration - or once in case no Data driver is present - the Test-set
(433)
or
executes each of its Test-set or Test-case child nodes. If an optional Setup
(435)
Cleanup
node are present, they are executed before and after each of these child
nodes respectively.
Attributes:
34.2.
Test and Sequence nodes
409
Figure 34.3: Test-set attributes
Name
(398)
The name of a Test-set is part of its own identification and of that of the Test-case
and Test-set nodes it contains, so you should assign a name with a meaning that
is easy to recognize and remember.
Variable: No
Restrictions: Must not be empty or contain the characters ’.’ or ’#’.
34.2.
Test and Sequence nodes
410
Name for run-log and report
A separate name to be used for run-log and report. This is useful to differentiate
between multiple executions with potentially different values for the Characteristic
variables.
Variable: Yes
Restrictions: None
Characteristic variables
These variables are part of the characteristics of a Test-set or Test-case. Two
executions of a Test-case are considered to represent the same logical test-case if
the run-time values of all Characteristic variables are identical. The run-time values
of the Characteristic variables are stored in the run-log.
Variable: No
Restrictions: None
Name for separate run-log
If this attribute is set it marks the node as a breaking point for split run-logs and
defines the filename for the partial log. When the node finishes, the respective
log entry is removed from the main run-log and saved as a separate, partial
run-log. This operation is completely transparent, the main run-log retains
(78)
references to the partial logs and is fully controllable. Please see section 8.1.4
for further information about split run-logs.
(387)
This attribute has no effect if the option Create split run-logs is disabled or split
(690)
run-logs are explicitly turned off for batch mode via the -splitlog command
line argument.
There is no need to keep the filename unique. If necessary, QF-Test appends a
number to the filename to avoid collisions. The filename may contain directories
and, similar to specifying the name of a run-log in batch mode on the command
line, the following placeholders can be used after a ’%’ or a ’+’ character:
34.2.
Test and Sequence nodes
Character
%
+
i
r
w
e
x
t
y
Y
M
d
h
m
s
411
Replacement
Literal ’%’ character.
Literal ’+’ character.
(688)
The current runid as specified with -runid <ID> .
The error level of the partial log.
The number of warnings in the partial log.
The number of errors in the partial log.
The number of exceptions in the partial log.
The thread index to which the partial log belongs (for tests run with parallel threads).
The current year (2 digits).
The current year (4 digits).
The current month (2 digits).
The current day (2 digits).
The current hour (2 digits).
The current minute (2 digits).
The current second (2 digits).
Table 34.2: Placeholders for the Name for separate run-log attribute
Variable: Yes
Restrictions: None, characters that are illegal for a filename will be replaced with
’_’.
Inherit dependency from parent node
(427)
This option allows inheriting the Dependency
from the parent node as a
replacement for or in addition to specifying a Dependency for the node itself.
Variable: No
Restrictions: None
Condition
If the condition is non-empty it will be evaluated and if the result is false the
execution of the current node is aborted. In this case the node will be reported as
skipped.
(482)
(481)
Like the Condition
of an If
node, the Condition is evaluated by the Jython
interpreter, so the same rules apply.
Variable: Yes
Restrictions: Valid syntax
Parameter default values
34.2.
Test and Sequence nodes
412
Here you can define default or fallback values for the Test-set’s parameters (see
(62)
chapter 7 ). Defining these values also serves as documentation and is a
(412)
valuable time-saver when using the dialog to select the Test-set for the Test name
(411)
(16)
attribute of a Test call . See section 2.2.5 about how to work with the table.
Variable: Variable names no, values yes
Restrictions: None
Maximum error level
When a warning, error or exception occurs during a test run, the state of the
corresponding node of the run-log is set accordingly. This state is normally
propagated to the parent node in a way that ensures that the error state of a
run-log node represents the worst of its child nodes’ states. Using this attribute,
the maximum error state that the run-log node for a sequence will propagate, can
be limited.
Note
This value has no effect on the way exceptions are handled. It only affects the
error states of the run-log nodes and by that the exit code of QF-Test when run in
(10)
batch mode (see. section 1.7 ). It also has no effect on the creation of compact
(679)
run-logs (see command line argument -compact ). The node for a sequence in
which a warning, error or exception occurs is never removed from a compact log,
even if the error is not propagated due to the setting of this attribute.
Variable: No
Restrictions: None
Execution timeout
Time limit for the node’s execution in milliseconds.
execution of that node will get interrupted.
If that limit expires the
Variable: Yes
Restrictions: >= 0
Border for relative calls
This flag determines whether relative procedure calls, test calls or dependency
references are allowed within that certain node. Relative calls passing that border
are not allowed. If that attribute is not specified in the hierarchy, no relative calls
are allowed.
Variable: No
Restrictions: None
QF-Test ID
(691)
When using the command line argument -test <n>|<ID>
for execution in
34.2.
Test and Sequence nodes
413
batch mode you can specify the QF-Test ID of the node as an alternative to its
qualified name.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
are much better suited to this task. The option External editor command
lets
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.2.3 Test call
With this node a call to some other executable test node can be made. Pos(395)
(405)
(398)
sible targets are Test-suite , Test-set
and Test-case
nodes in the same
or a different test-suite. Execution will continue in the called node and when
finished return to the Test call and thus to its parent node.
(400)
(407)
The name of a Test-case or Test-set to call is determined by its Name and the Names
of its Test-set ancestors. These are concatenated with a dot (’.’) as separator, starting
with the outermost Test-set and ending in the Test-case’s name. Thus to call a Test-case
named nodeTest in a Test-set named Tree that is itself a child of a Test-set named
(412)
Main, the Test name
attribute would be set to ’Main.Tree.nodeTest’. A node in
a different test-suite is addressed by prepending the filename of the test-suite followed
by a ’#’ to the Test name. A Test-suite node is addressed by a single ’.’, so calling a
34.2.
Test and Sequence nodes
414
whole test-suite is done with a Test name attribute of the form ’suitename.qft#.’. It
is generally easiest to pick the target node interactively from a dialog by clicking on the
button above the Test name attribute.
See also section 19.1
(193)
for further information about cross-suite calls.
(398)
Contained in: All kinds of sequences
.
Children: None
(414)
Execution: The Variable definitions
of the Test call are bound, the target node is determined and execution is passed to it. After the Test call returns, the Test call’s variables
are unbound again.
Attributes:
Figure 34.4: Test call Attributes
Name
(407)
In case of a Test-case or Test-set it is the full name, created from the Name of its
(400)
Test-set parents and its own Name , joined by a dot. A Test-suite node is
addressed by a single ’.’ character. The button above the attribute brings up a
dialog in which you can select the target node interactively. You can also get to
34.2.
Test and Sequence nodes
415
this dialog by pressing Shift-Return or Alt-Return when the focus is in the text
field. By selecting the ”Copy parameters” checkbox you can adopt the Test-case’s
or Test-set’s Parameter default values as parameters for the Test call node to save
typing.
Variable: Yes
Restrictions: Must not be empty.
Name for separate run-log
If this attribute is set it marks the node as a breaking point for split run-logs and
defines the filename for the partial log. When the node finishes, the respective
log entry is removed from the main run-log and saved as a separate, partial
run-log. This operation is completely transparent, the main run-log retains
(78)
references to the partial logs and is fully controllable. Please see section 8.1.4
for further information about split run-logs.
(387)
This attribute has no effect if the option Create split run-logs is disabled or split
(690)
run-logs are explicitly turned off for batch mode via the -splitlog command
line argument.
There is no need to keep the filename unique. If necessary, QF-Test appends a
number to the filename to avoid collisions. The filename may contain directories
and, similar to specifying the name of a run-log in batch mode on the command
line, the following placeholders can be used after a ’%’ or a ’+’ character:
Character
%
+
i
r
w
e
x
t
y
Y
M
d
h
m
s
Replacement
Literal ’%’ character.
Literal ’+’ character.
(688)
The current runid as specified with -runid <ID> .
The error level of the partial log.
The number of warnings in the partial log.
The number of errors in the partial log.
The number of exceptions in the partial log.
The thread index to which the partial log belongs (for tests run with parallel threads).
The current year (2 digits).
The current year (4 digits).
The current month (2 digits).
The current day (2 digits).
The current hour (2 digits).
The current minute (2 digits).
The current second (2 digits).
Table 34.3: Placeholders for the Name for separate run-log attribute
34.2.
Test and Sequence nodes
416
Variable: Yes
Restrictions: None, characters that are illegal for a filename will be replaced with
’_’.
Variable definitions
This is where you define the parameter values for the target node. These
(62)
variables are bound on the primary variable stack (see chapter 7 ) so they
(16)
override any Parameter default values. See section 2.2.5 about how to work with
the table.
4.2+
In case you want to re-set the order of the parameters like they are sorted in the
called test-case or test-set, you can select Re-set parameter order .
Variable: Variable names no, values yes
Restrictions: None
Act like a procedure call
If the Test call node is executed inside a Test-case, this attribute determines how
exceptions are handled in the called node(s). If it is activated a single exception
terminates the whole call irrespective of Test-set and Test-case nesting, just like it
would for a regular Procedure call node. If this attribute is unset the special roles of
Test-set and Test-case nodes with their local exception handling is maintained.
Variable: Yes
Restrictions: None
Execution timeout
Time limit for the node’s execution in milliseconds.
execution of that node will get interrupted.
If that limit expires the
Variable: Yes
Restrictions: >= 0
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
34.2.
Test and Sequence nodes
417
Restrictions: Valid number >= 0
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
are much better suited to this task. The option External editor command
lets
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.2.4 Sequence
This is the most general kind of sequence. Its children are executed one by
one and their number or type is not limited in any way.
(398)
Contained in: All kinds of sequences
.
Children: Any
(416)
Execution: The Variable definitions
of the Sequence are bound and its child nodes
executed one by one. After the execution of the last child is complete, the variables are
unbound again.
Attributes:
34.2.
Test and Sequence nodes
418
Figure 34.5: Sequence attributes
Name
The name of a sequence is a kind of short description. It is displayed in the tree
view, so it should be concise and tell something about the function of the
sequence.
Variable: No
Restrictions: None
Variable definitions
This is where you define the values of the variables that remain bound during the
(62)
(16)
execution of the sequence’s child nodes (see chapter 7 ). See section 2.2.5
about how to work with the table.
Variable: Variable names no, values yes
Restrictions: None
Maximum error level
When a warning, error or exception occurs during a test run, the state of the
corresponding node of the run-log is set accordingly. This state is normally
propagated to the parent node in a way that ensures that the error state of a
run-log node represents the worst of its child nodes’ states. Using this attribute,
34.2.
Test and Sequence nodes
419
the maximum error state that the run-log node for a sequence will propagate, can
be limited.
Note
This value has no effect on the way exceptions are handled. It only affects the
error states of the run-log nodes and by that the exit code of QF-Test when run in
(10)
batch mode (see. section 1.7 ). It also has no effect on the creation of compact
(679)
run-logs (see command line argument -compact ). The node for a sequence in
which a warning, error or exception occurs is never removed from a compact log,
even if the error is not propagated due to the setting of this attribute.
Variable: No
Restrictions: None
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
are much better suited to this task. The option External editor command
lets
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.2.5 Test-step
34.2.
Test and Sequence nodes
420
A Test-step is a special Sequence that serves to divide a Test-case into steps
that can be documented individually and will show up in the report or testdoc
documentation. In contrast to Test-cases, which should not be nested, Teststeps can be nested to any depth.
(398)
Contained in: All kinds of sequences
.
Children: Any
(420)
Execution: The Variable definitions
of the Test-step are bound and its child nodes
executed one by one. After the execution of the last child is complete, the variables are
unbound again.
Attributes:
Figure 34.6: Test-step attributes
Name
The name of a sequence is a kind of short description. It is displayed in the tree
34.2.
Test and Sequence nodes
421
view, so it should be concise and tell something about the function of the
sequence.
Variable: No
Restrictions: None
Name for run-log and report
A separate name to be used for run-log and report. This is useful to differentiate
between multiple executions of the surrounding Test-case with potentially different
values for its Characteristic variables.
Variable: Yes
Restrictions: None
Name for separate run-log
If this attribute is set it marks the node as a breaking point for split run-logs and
defines the filename for the partial log. When the node finishes, the respective
log entry is removed from the main run-log and saved as a separate, partial
run-log. This operation is completely transparent, the main run-log retains
(78)
references to the partial logs and is fully controllable. Please see section 8.1.4
for further information about split run-logs.
(387)
This attribute has no effect if the option Create split run-logs is disabled or split
(690)
run-logs are explicitly turned off for batch mode via the -splitlog command
line argument.
There is no need to keep the filename unique. If necessary, QF-Test appends a
number to the filename to avoid collisions. The filename may contain directories
and, similar to specifying the name of a run-log in batch mode on the command
line, the following placeholders can be used after a ’%’ or a ’+’ character:
34.2.
Test and Sequence nodes
Character
%
+
i
r
w
e
x
t
y
Y
M
d
h
m
s
422
Replacement
Literal ’%’ character.
Literal ’+’ character.
(688)
The current runid as specified with -runid <ID> .
The error level of the partial log.
The number of warnings in the partial log.
The number of errors in the partial log.
The number of exceptions in the partial log.
The thread index to which the partial log belongs (for tests run with parallel threads).
The current year (2 digits).
The current year (4 digits).
The current month (2 digits).
The current day (2 digits).
The current hour (2 digits).
The current minute (2 digits).
The current second (2 digits).
Table 34.4: Placeholders for the Name for separate run-log attribute
Variable: Yes
Restrictions: None, characters that are illegal for a filename will be replaced with
’_’.
Variable definitions
This is where you define the values of the variables that remain bound during the
(62)
(16)
execution of the sequence’s child nodes (see chapter 7 ). See section 2.2.5
about how to work with the table.
Variable: Variable names no, values yes
Restrictions: None
Maximum error level
When a warning, error or exception occurs during a test run, the state of the
corresponding node of the run-log is set accordingly. This state is normally
propagated to the parent node in a way that ensures that the error state of a
run-log node represents the worst of its child nodes’ states. Using this attribute,
the maximum error state that the run-log node for a sequence will propagate, can
be limited.
Note
This value has no effect on the way exceptions are handled. It only affects the
error states of the run-log nodes and by that the exit code of QF-Test when run in
(10)
batch mode (see. section 1.7 ). It also has no effect on the creation of compact
34.2.
Test and Sequence nodes
423
(679)
run-logs (see command line argument -compact ). The node for a sequence in
which a warning, error or exception occurs is never removed from a compact log,
even if the error is not propagated due to the setting of this attribute.
Variable: No
Restrictions: None
Execution timeout
Time limit for the node’s execution in milliseconds.
execution of that node will get interrupted.
If that limit expires the
Variable: Yes
Restrictions: >= 0
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
are much better suited to this task. The option External editor command
lets
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.2.
Test and Sequence nodes
424
34.2.6 Sequence with time limit
(415)
This node extends the basic Sequence node with time-constraint checking.
Child nodes are executed as usual, but upon completion of the sequence
the elapsed time is compared to the time-limit. Exceeding the time limit
will result in a warning, error or exception, depending on the value of the attribute
(424)
(425)
Error level if time limit exceeded . Explicit delays like Delay before/after
or user interrupts are deducted from the duration before the constraints are checked, unless
(423)
Check realtime is activated.
(425)
For report generation, time-constraints are treated like checks. If the Comment
tribute starts with an ’!’ character, the result will be logged in the report.
Note
at-
The function of this node is to check time constraints in the user-acceptance range, i.e.
between a few hundred milliseconds and a few seconds. Real-time constraints of a few
milliseconds or less are beyond its limits.
(398)
Contained in: All kinds of sequences
.
Children: Any
(424)
Execution: The Variable definitions
of the Sequence with time limit are bound and its
child nodes executed one by one. After the execution of the last child is complete, the
variables are unbound again. The elapsed time is compared to the given time limit.
Attributes:
34.2.
Test and Sequence nodes
425
Figure 34.7: Sequence with time limit attributes
Name
The name of a sequence is a kind of short description. It is displayed in the tree
view, so it should be concise and tell something about the function of the
sequence.
Variable: No
Restrictions: None
Time limit for execution
The time (in milliseconds) allowed for the execution of the sequence.
Variable: Yes
Restrictions: Must not be negative.
34.2.
Test and Sequence nodes
426
Check realtime
(425)
Normally explicit delays like Delay before/after
or user interrupts are deducted
from the duration, before the time constraints are checked. To prevent this
deduction and therefore check the real-time, this attribute can be activated.
Variable: No
Restrictions: None
Error level if time limit exceeded
This attribute determines what happens in case the time limit is exceeded. If set
(665)
will be thrown. Otherwise a
to ”exception”, a CheckFailedException
message with the respective error-level will be logged in the run-log.
Variable: No
Restrictions: None
Variable definitions
This is where you define the values of the variables that remain bound during the
(62)
(16)
execution of the sequence’s child nodes (see chapter 7 ). See section 2.2.5
about how to work with the table.
Variable: Variable names no, values yes
Restrictions: None
Maximum error level
When a warning, error or exception occurs during a test run, the state of the
corresponding node of the run-log is set accordingly. This state is normally
propagated to the parent node in a way that ensures that the error state of a
run-log node represents the worst of its child nodes’ states. Using this attribute,
the maximum error state that the run-log node for a sequence will propagate, can
be limited.
Note
This value has no effect on the way exceptions are handled. It only affects the
error states of the run-log nodes and by that the exit code of QF-Test when run in
(10)
batch mode (see. section 1.7 ). It also has no effect on the creation of compact
(679)
run-logs (see command line argument -compact ). The node for a sequence in
which a warning, error or exception occurs is never removed from a compact log,
even if the error is not propagated due to the setting of this attribute.
Variable: No
Restrictions: None
Execution timeout
34.2.
Test and Sequence nodes
427
Time limit for the node’s execution in milliseconds.
execution of that node will get interrupted.
If that limit expires the
Variable: Yes
Restrictions: >= 0
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
lets
are much better suited to this task. The option External editor command
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.2.7 Extras
This node is a kind of clipboard or playground where the usual restrictions
on the parent of a node don’t apply. You can add any kind of node here to
assemble and try out some test sequences.
Contained in: Root node
Children: Any
34.3.
Dependencies
428
Execution: Cannot be executed
Attributes:
Figure 34.8: Extras attributes
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
lets
are much better suited to this task. The option External editor command
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.3
Dependencies
Dependencies are a very powerful feature for handling setup and cleanup requirements
for test-cases automatically. The goal is to isolate test-cases so that each one can be
34.3.
Dependencies
429
run independently without interfering with others. This is a very important requirement
for things like testing arbitrary sub-sets of test-cases, for example to re-test only failed
tests, or during test-development where it must be possible to quickly run and debug
any given test-case.
34.3.1 Dependency
Dependency nodes are used to implement advanced, automatic handling of
(405)
(398)
and Test-cases . A detailed
setup and cleanup requirements for Test-sets
(94)
description of the Dependency mechanism is given in section 9.6 . This section focuses on formal requirements for the Dependency node and its attributes.
As Dependencies are complex, they should be reused as much as possible. This can
be done by grouping Test-cases with identical dependencies in a Test-set and have them
inherit the Dependency of the Test-set. However, this mechanism alone is not flexible
(463)
and placed
enough, so a Dependency can also be implemented just like a Procedure
(472)
among the Procedures
of a test-suite to be referenced from a Dependency reference
(400)
attribute is mandatory and it also has a list of
node. For this to work, the Name
(429)
Parameter default values that can be overridden in the referencing node.
(428)
The Characteristic variables of a Dependency are part of its identity and play an important role in the dependency resolution mechanism.
(395)
Contained in: Test-suite
(405)
, Test-set
(398)
, Test-case
, Procedures
(472)
(470)
, Package
.
(430)
Children: Zero or more Dependency references
on which the Dependency is based,
(433)
(435)
(438)
optional Setup and Cleanup nodes and an optional Error handler followed by zero
(493)
or more Catch nodes.
Execution: Normally Dependencies are executed only indirectly in the setup phase of
a Test-set or Test-case. If a Dependency node is executed interactively, the dependency
(94)
stack is resolved as described in section 9.6 .
Attributes:
34.3.
Dependencies
430
Figure 34.9: Dependency attributes
Name
A Dependency is identified by its name, so you should assign a name with a
meaning that is easy to recognize and remember.
Variable: No
Restrictions: Must not be empty or contain the characters ’.’ or ’#’.
Name for run-log and report
A separate name to be used for run-log and report. This is useful to differentiate
between multiple executions with potentially different values for the Characteristic
variables.
Variable: Yes
Restrictions: None
34.3.
Dependencies
431
Characteristic variables
These variables are part of a Dependency’s identity. During Dependency resolution
(94)
as described in section 9.6 two Dependencies are considered equal only if they
are one and the same node and the run-time values of all their Characteristic
variables are identical. Additionally, the values of the Characteristic variables are
stored during the setup phase of the Dependency. Later, when the Dependency is
rolled back, these settings will be temporarily restored for the cleanup phase.
Variable: No
Restrictions: None
Always execute, even in test-suite and test-set nodes
Normally a Dependency is only executed if it belongs to a Test-case node.
Dependencies in Test-suite or Test-set nodes are simply inherited by the Test-case
descendants of these nodes. However, in some cases it is useful to resolve a
Dependency early, for example when the Dependency provides parameters for a
(402)
of a subsequent Test-case.
test-run that are required to evaluate a Condition
This can be achieved by activating this option.
Variable: No
Restrictions: None
Forced cleanup
Normally Dependencies are only rolled back and their cleanup code executed as
(94)
required by the dependency resolution mechanism described in section 9.6 . In
some cases it makes sense to force partial cleanup of the dependency stack
immediately after a Test-case finishes. This is what the Forced cleanup attribute is
for. If this option is activated, the dependency stack will be rolled back at least up
to and including this Dependency .
Variable: No
Restrictions: None
Parameter default values
Here you can define default or fallback values for the Dependency’s parameters
(62)
(see chapter 7 ). Defining these values also serves as documentation and is a
valuable time-saver when using the dialog to select the Dependency for the
(431)
(430)
Referenced dependency
attribute of a Dependency reference .
See section
(16)
2.2.5 about how to work with the table.
Variable: Variable names no, values yes
Restrictions: None
34.3.
Dependencies
432
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
lets
are much better suited to this task. The option External editor command
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.3.2 Dependency reference
(427)
A Dependency reference is simply a stand-in for a Dependency
defined in
(470)
some other place, typically a Package . The name of the referenced De(428)
(471)
pendency is determined by its Name
and the Names
of its Package parents. These are concatenated with a dot (’.’) as separator, starting with the outermost
Package and ending in the Dependency’s name. Thus to reference a Dependency named
demoStarted in a Package named Demo that is itself a child of a Package named Main,
(431)
set the Referenced dependency attribute to ’Main.Demo.demoStarted’.
See section 19.1
(193)
about how to reference a Dependency in a different test-suite.
(395)
Contained in: Test-suite
(405)
, Test-set
(398)
, Test-case
and Dependency
(427)
34.3.
Dependencies
433
Children: None
Execution: Normally Dependency references are executed only indirectly in the setup
phase of a Test-set or Test-case. If a Dependency reference node is executed interactively,
the referenced Dependency is determined and the dependency stack resolved accord(94)
ingly as described in section 9.6 .
Attributes:
Figure 34.10: Dependency reference attributes
Referenced dependency
(427)
(471)
(470)
The full name of the Dependency , created from the Names
of its Package
(428)
parents and its own Name , joined by a dot. The button above the attribute
brings up a dialog in which you can select the Dependency interactively. By
selecting the ”Copy parameters” checkbox you can adopt the Dependency’s
(429)
Parameter default values as parameters for the Dependency reference node to save
typing.
Variable: Yes
Restrictions: Must not be empty.
Dependency namespace
34.3.
Dependencies
434
Normally there is only a single stack of dependencies, but in some cases, e.g.
when mixing tests for independent SUT clients, it can be useful to have
independent sets of dependencies for different parts of the test, so that resolving
the dependencies for one part doesn’t necessarily tear down everything required
for a different part.
By setting the Dependency namespace attribute of a Dependency reference node you
can tell QF-Test to resolve the target dependency in that namespace. The default
dependency stack will be completely ignored in that case. If there is a dependency
stack remaining from a previous dependency resolution in the same namespace it
will be used for comparison instead, otherwise a new stack will be created.
Variable: Yes
Restrictions: None
Always execute, even in test-suite and test-set nodes
Normally a Dependency is only executed if it belongs to a Test-case node.
Dependencies in Test-suite or Test-set nodes are simply inherited by the Test-case
descendants of these nodes. However, in some cases it is useful to resolve a
Dependency early, for example when the Dependency provides parameters for a
(402)
of a subsequent Test-case.
test-run that are required to evaluate a Condition
This can be achieved by activating this option.
Variable: No
Restrictions: None
Variable definitions
(429)
These variables override the Parameter default values
of the Dependency
(16)
referenced by this Dependency reference. See section 2.2.5 about how to work
with the table.
4.2+
In case you want to re-set the order of the parameters like they are sorted in the
called dependency, you can select Re-set parameter order .
Variable: Variable names no, values yes
Restrictions: None
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
34.3.
Dependencies
435
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
lets
are much better suited to this task. The option External editor command
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.3.3 Setup
(415)
This node is just like a Sequence
except for its special place
(427)
(405)
(398)
in a Dependency , Test-set or Test-case node.
Contained in: Dependency
(427)
(405)
, Test-set
(398)
or Test-case
Children: Any
(434)
Execution: The Variable definitions
of the Setup are bound and its child nodes executed one by one. After the execution of the last child is complete, the variables are
unbound again.
Attributes:
34.3.
Dependencies
436
Figure 34.11: Setup attributes
Name
The name of a sequence is a kind of short description. It is displayed in the tree
view, so it should be concise and tell something about the function of the
sequence.
Variable: No
Restrictions: None
Variable definitions
This is where you define the values of the variables that remain bound during the
(62)
(16)
execution of the sequence’s child nodes (see chapter 7 ). See section 2.2.5
about how to work with the table.
Variable: Variable names no, values yes
Restrictions: None
Maximum error level
When a warning, error or exception occurs during a test run, the state of the
corresponding node of the run-log is set accordingly. This state is normally
propagated to the parent node in a way that ensures that the error state of a
run-log node represents the worst of its child nodes’ states. Using this attribute,
34.3.
Dependencies
437
the maximum error state that the run-log node for a sequence will propagate, can
be limited.
Note
This value has no effect on the way exceptions are handled. It only affects the
error states of the run-log nodes and by that the exit code of QF-Test when run in
(10)
batch mode (see. section 1.7 ). It also has no effect on the creation of compact
(679)
run-logs (see command line argument -compact ). The node for a sequence in
which a warning, error or exception occurs is never removed from a compact log,
even if the error is not propagated due to the setting of this attribute.
Variable: No
Restrictions: None
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
are much better suited to this task. The option External editor command
lets
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.3.4 Cleanup
34.3.
Dependencies
438
(415)
This node is just like a Sequence
except for its special place
(427)
(405)
(398)
in a Dependency , Test-set or Test-case node.
Contained in: Dependency
(427)
(405)
, Test-set
(398)
or Test-case
Children: Any
(436)
Execution: The Variable definitions of the Cleanup are bound and its child nodes executed one by one. After the execution of the last child is complete, the variables are
unbound again.
Attributes:
Figure 34.12: Cleanup attributes
Name
The name of a sequence is a kind of short description. It is displayed in the tree
view, so it should be concise and tell something about the function of the
sequence.
Variable: No
Restrictions: None
Variable definitions
This is where you define the values of the variables that remain bound during the
(62)
(16)
execution of the sequence’s child nodes (see chapter 7 ). See section 2.2.5
about how to work with the table.
34.3.
Dependencies
439
Variable: Variable names no, values yes
Restrictions: None
Maximum error level
When a warning, error or exception occurs during a test run, the state of the
corresponding node of the run-log is set accordingly. This state is normally
propagated to the parent node in a way that ensures that the error state of a
run-log node represents the worst of its child nodes’ states. Using this attribute,
the maximum error state that the run-log node for a sequence will propagate, can
be limited.
Note
This value has no effect on the way exceptions are handled. It only affects the
error states of the run-log nodes and by that the exit code of QF-Test when run in
(10)
batch mode (see. section 1.7 ). It also has no effect on the creation of compact
(679)
run-logs (see command line argument -compact ). The node for a sequence in
which a warning, error or exception occurs is never removed from a compact log,
even if the error is not propagated due to the setting of this attribute.
Variable: No
Restrictions: None
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
are much better suited to this task. The option External editor command
lets
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
34.3.
Dependencies
440
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.3.5 Error handler
(415)
This node is just like a Sequence
(427)
(94)
in a Dependency (see section 9.6 ).
Contained in: Dependency
except for its special place
(427)
Children: Any
(439)
Execution: The Variable definitions of the Error handler are bound and its child nodes
executed one by one. After the execution of the last child is complete, the variables are
unbound again.
Attributes:
Figure 34.13: Error handler attributes
Name
The name of a sequence is a kind of short description. It is displayed in the tree
34.3.
Dependencies
441
view, so it should be concise and tell something about the function of the
sequence.
Variable: No
Restrictions: None
Variable definitions
This is where you define the values of the variables that remain bound during the
(62)
(16)
execution of the sequence’s child nodes (see chapter 7 ). See section 2.2.5
about how to work with the table.
Variable: Variable names no, values yes
Restrictions: None
Maximum error level
When a warning, error or exception occurs during a test run, the state of the
corresponding node of the run-log is set accordingly. This state is normally
propagated to the parent node in a way that ensures that the error state of a
run-log node represents the worst of its child nodes’ states. Using this attribute,
the maximum error state that the run-log node for a sequence will propagate, can
be limited.
Note
This value has no effect on the way exceptions are handled. It only affects the
error states of the run-log nodes and by that the exit code of QF-Test when run in
(10)
batch mode (see. section 1.7 ). It also has no effect on the creation of compact
(679)
run-logs (see command line argument -compact ). The node for a sequence in
which a warning, error or exception occurs is never removed from a compact log,
even if the error is not propagated due to the setting of this attribute.
Variable: No
Restrictions: None
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
34.4.
Data driver
442
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
lets
are much better suited to this task. The option External editor command
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.4
Data driver
The term Data-driven Testing refers to a common method in automated testing where
test-cases are executed several times with different sets of data defined. With QF-Test’s
highly flexible variables there is no limit on how this data can be used but the most
common case is for event input values and expected check values.
(440)
QF-Test’s data-driving mechanism consists of the Data driver
node used to provide
a data-driving context and several kinds of Data binders. Currently available are the
(443)
(456)
Data table node that stores data internally within QF-Test the CSV data file node that
(446)
reads data from a CSV file, the Database node that reads data from a database and
(452)
the Excel data file that reads data from an Excel file. An extension API for plugging in
arbitrary external data is also available.
Further information about how the various parts of the data driver mechanism are work(159)
ing together is provided in chapter 16 .
34.4.1 Data driver
(405)
(417)
Except for its special place in a Test-set or Test-step , a Data driver is just
(415)
like a normal Sequence . It provides a context for one or more Data binders to
register themselves during the execution of the Data driver. The Test-set then
iterates over the sets of data provided by the registered Data binders and executes its
(159)
child nodes as described in chapter 16 . For this purpose a Data driver node needs to
(405)
(427)
(433)
be placed in a Test-set node, between the optional Dependency and Setup nodes.
(417)
Data driver nodes can also be placed in a Test-step as first steps.
34.4.
Data driver
443
(405)
Contained in: Test-set
(417)
, Test-step
.
Children: Any
Execution: When a Test-set or Test-step is executed it checks for a Data driver and runs
it. The contents of the Data driver node are not limited to Data binders, but can hold any
executable node so that they can perform any setup that may be required to retrieve the
(463)
data. Thus it is also possible to share Data binders by placing them inside a Procedure
and calling the Procedure from inside the Data driver. Any Data binders registered within
this Data driver’s context will then be queried for data by the Test-set or Test-step.
Attributes:
Figure 34.14: Data driver attributes
Name
The name of a Data driver is a kind of short description. It is displayed in the tree
view, so it should be concise and tell something about the kind of data provided.
Variable: No
Restrictions: None
Name for loop pass in the run-log
34.4.
Data driver
444
A separate name for each iteration to be used in the run-log. It can make use of
the variables bound as a result of the data-driving which makes it easier to locate
a specific step of the iteration.
Variable: Yes
Restrictions: None
Name for separate run-log
If this attribute is set it marks the node as a breaking point for split run-logs and
defines the filename for the partial log. Every time an iteration of the Data driver
finishes, the respective log entry is removed from the main run-log and saved as
a separate, partial run-log. This operation is completely transparent, the main
run-log retains references to the partial logs and is fully controllable. Please see
(78)
section 8.1.4 for further information about split run-logs.
(387)
This attribute has no effect if the option Create split run-logs is disabled or split
(690)
run-logs are explicitly turned off for batch mode via the -splitlog command
line argument.
There is no need to keep the filename unique. If necessary, QF-Test appends a
number to the filename to avoid collisions. The filename may contain directories
and, similar to specifying the name of a run-log in batch mode on the command
line, the following placeholders can be used after a ’%’ or a ’+’ character:
Character
%
+
i
r
w
e
x
t
y
Y
M
d
h
m
s
Replacement
Literal ’%’ character.
Literal ’+’ character.
(688)
The current runid as specified with -runid <ID> .
The error level of the partial log.
The number of warnings in the partial log.
The number of errors in the partial log.
The number of exceptions in the partial log.
The thread index to which the partial log belongs (for tests run with parallel threads).
The current year (2 digits).
The current year (4 digits).
The current month (2 digits).
The current day (2 digits).
The current hour (2 digits).
The current minute (2 digits).
The current second (2 digits).
Table 34.5: Placeholders for the Name for separate run-log attribute
Variable: Yes
34.4.
Data driver
445
Restrictions: None, characters that are illegal for a filename will be replaced with
’_’.
Variable definitions
This is where you define the values of the variables that remain bound during the
(62)
(16)
execution of the sequence’s child nodes (see chapter 7 ). See section 2.2.5
about how to work with the table.
Variable: Variable names no, values yes
Restrictions: None
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
lets
are much better suited to this task. The option External editor command
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.4.2 Data table
34.4.
Data driver
446
A Data table provides a convenient interface for storing test-data in tabular
form directly inside QF-Test. For details about the data-driving mechanism
(159)
please see chapter 16 .
Contained in: Any
Children: None
Execution: The Data table expands variable values in the table according to the option
(393)
When binding variables, expand values immediately . Each row is expanded individually left to right, meaning that - within the same row - a cell may refer to a variable
bound in a column further to the left. Then the Data table registers itself with the Data
driver context. A property group called like the node will be created additionally. That
group contains the variables size and totalsize. size shows the number of data
rows with taking care about iteration intervals. The variable totalsize shows the total
number of data rows without taking care about iteration intervals. When all Data binders
(405)
have been registered the Test-set will query the Data table in order to iterate over the
available sets of data. If no such context is available the respective property group will
be extended with all variables.
Attributes:
Figure 34.15: Data table attributes
Name
34.4.
Data driver
447
The Name of a Data binder is mandatory. It is used to distinguish Data binders in the
(479)
same Data driver context. A Break node executed during data-driven testing can
be used to break out of a specific loop by referring to the Data binder’s Name.
Variable: Yes
Restrictions: None
Iteration counter
The name of the variable that the iteration counter will be bound to.
Variable: Yes
Restrictions: None
Iteration ranges
An optional set of indexes or ranges to use from the bound data. This is
especially useful during test development in order to run sample tests with just a
single index or a subset of the given data.
Ranges are separated by ’,’. Each range is either a single index or an inclusive
range of the form ’from-to’ or ’from:to’ where ’to’ is optional. Indexes or ranges
may be specified multiple times, overlap or be given in descending order. Indexes are 0-based, negative indexes are counted from the end, -1 being the
last item. An invalid syntax or an index outside the valid data range will cause
(665)
a BadRangeException .
The following table shows some example range specifications and the resulting
indexes, based on a set of 20 entries.
Iteration ranges
0
-2, -1
1-2,4:5
18:,-33-2,16:15
Resulting indexes
[0]
[18, 19]
[1, 2, 4, 5]
[18, 19, 17, 18, 19]
[3, 2, 16, 15]
Table 34.6: Iteration range examples
Note
The value bound for the Iteration counter reflects the index in the current interval,
not the counter of actual iterations, e.g. if you specify ’2’ there will be a single
iteration with the Iteration counter bound to ’2’, not ’0’.
Variable: Yes
Restrictions: Valid syntax and index values
34.4.
Data driver
448
Data bindings
This is where the actual test data is defined. Each column of the table represents
one variable with its name specified in the column header. Each row is a set of
data, one value per variable. Thus the number of rows determines the number of
iterations of the data-driven loop. To start entering the data you first need to add
columns to the table to define the variables, then add rows to fill in the values.
(16)
See also section 2.2.5 about how to work with the table.
Variable: Yes, even the column headers
Restrictions: None
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
lets
are much better suited to this task. The option External editor command
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.4.3 Database
34.4.
Data driver
449
A Database node is used to load external data from a database. To access the
database it is required to have the jar file with the database driver in QF-Test’s
(727)
plugin-directory (section 41.2 ) or in the CLASSPATH environment variable
at QF-Test’s start-up time. For more information about the connection mechanism to
your database ask the developers or see www.connectionstrings.com.
(159)
For further details about the data-driving mechanism please see chapter 16
.
Contained in: Any
Children: None
Execution: The Database loads the data from the database and expands variable values
(393)
according to the option When binding variables, expand values immediately . Each
row is expanded individually left to right, meaning that - within the same row - a cell may
refer to a variable bound in a column further to the left. Then the Database node registers
itself with the Data driver context. The result columns of the used SQL statement will
be used for the variable names. The capitalization depends on the database driver,
e.g. completely capitalized letters for Oracle. A property group called like the node
will be created additionally. That group contains the variables size and totalsize.
size shows the number of data rows with taking care about iteration intervals. The
variable totalsize shows the total number of data rows without taking care about
(405)
iteration intervals. When all Data binders have been registered the Test-set will query
the Database in order to iterate over the available sets of data. If no such context is
available the respective property group will be extended with all variables.
Attributes:
34.4.
Data driver
450
Figure 34.16: Database attributes
Name
The Name of a Data binder is mandatory. It is used to distinguish Data binders in the
(479)
same Data driver context. A Break node executed during data-driven testing can
be used to break out of a specific loop by referring to the Data binder’s Name.
Variable: Yes
Restrictions: None
Iteration counter
The name of the variable that the iteration counter will be bound to.
Variable: Yes
Restrictions: None
Iteration ranges
An optional set of indexes or ranges to use from the bound data.
This is
34.4.
Data driver
451
especially useful during test development in order to run sample tests with just a
single index or a subset of the given data.
Ranges are separated by ’,’. Each range is either a single index or an inclusive
range of the form ’from-to’ or ’from:to’ where ’to’ is optional. Indexes or ranges
may be specified multiple times, overlap or be given in descending order. Indexes are 0-based, negative indexes are counted from the end, -1 being the
last item. An invalid syntax or an index outside the valid data range will cause
(665)
a BadRangeException .
The following table shows some example range specifications and the resulting
indexes, based on a set of 20 entries.
Iteration ranges
0
-2, -1
1-2,4:5
18:,-33-2,16:15
Resulting indexes
[0]
[18, 19]
[1, 2, 4, 5]
[18, 19, 17, 18, 19]
[3, 2, 16, 15]
Table 34.7: Iteration range examples
Note
The value bound for the Iteration counter reflects the index in the current interval,
not the counter of actual iterations, e.g. if you specify ’2’ there will be a single
iteration with the Iteration counter bound to ’2’, not ’0’.
Variable: Yes
Restrictions: Valid syntax and index values
SQL statement
The SQL query that should be executed to get the desired test-data. This
statement is supposed to be a select statement. Each column will stand for a
variable. The capitalization of the columns depends on the kind of the used
database driver, e.g. most of the Oracle drivers return completely capitalized
variables.
Variable: Yes
Restrictions: Must not be empty
Driver class
The class name of the database driver.
Note
The jar file with the database driver has to be placed in the CLASSPATH or in
QF-Test’s plugin directory before launching QF-Test.
34.4.
Data driver
452
Here is a list of the most common database drivers:
Database
Borland Interbase
DB2
Informix
IDS Server
MS SQL Server 2000
MS SQL Server 2005
mSQL
MySQL
Oracle
Pointbase
PostgreSQL
Standard Driver
Sybase
Classname of JDBC-driver
interbase.interclient.Driver
com.ibm.db2.jcc.DB2Driver
com.informix.jdbc.IfxDriver
ids.sql.IDSDriver
com.microsoft.jdbc.sqlserver.SQLServerDriver
com.microsoft.sqlserver.jdbc.SQLServerDriver
COM.imaginary.sql.msql.MsqlDriver
com.mysql.jdbc.Driver
oracle.jdbc.driver.OracleDriver
com.pointbase.jdbc.jdbcUniversalDriver
org.postgresql.Driver
sun.jdbc.odbc.JdbcOdbcDriver
com.sybase.jdbc2.jdbc.SybDriver
Table 34.8: Database drivers
Variable: Yes
Restrictions: Must not be empty
Connection string
The connection string for database connection, typically something like:
jdbc:databasetype://databasehost/databasename.
Here is a list of the most common database connection strings:
34.4.
Data driver
Database
Derby
IBM DB2
HSQLB
Interbase
MS SQL Server 2000
MS SQL Server 2005
MySQL
PostgreSQL
ODBC Data Sources
Oracle Thin
Sybase
453
Example
jdbc:derby:net://databaseserver:port/
jdbc:db2://database
jdbc:hsqldb:file:database
jdbc:interbase://databaseserver/database.gdb
jdbc:microsoft:sqlserver://databaseserver:
port;DatabaseName=database;
jdbc:sqlserver://databaseserver:
port;DatabaseName=database;
jdbc:mysql://databaseserver/database
jdbc:postgresql://databaseserver/database
jdbc:odbc:database
jdbc:oracle:thin:@databaseserver:port:
database
jdbc:sybase:tds:databaseserver:port/database
Table 34.9: Database connection strings
Variable: Yes
Restrictions: Must not be empty
Database user
The name of the user to use when connecting to the database. If your database
connection doesn’t require a user you can leave this field empty.
Variable: Yes
Restrictions: None
Database password
The password to use when connecting to the database. If your database
connection doesn’t require a password you can leave this field empty. To that end
the password can be encrypted by inserting the plain-text password, right-clicking
and selecting Crypt password from the popup menu. Please be sure to specify
(351)
a password salt before encrypting via the option Salt for crypting passwords .
Variable: Yes
Restrictions: None
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
34.4.
Data driver
454
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
lets
are much better suited to this task. The option External editor command
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.4.4 Excel data file
An Excel data file is used to load external data from an Excel-file to provide
test-data for data-driven testing. The first row must contain the names of the
variables to bind. The rest of the rows should contain the values to be used
for the iteration steps.
Note
It is possible that the contents of some cells are not read correctly. This can happen
especially for cells of the type ”date”, ”time” or ”currency”. The reason is that Excel
stores the value and the format of a cell separately and the Java package used to parse
Excel files doesn’t support all possibilities. Still, the Excel data file should read most cells
correctly, but in case of problems please change the type of the problematic cells in the
Excel file to ”text”. Once read by QF-Test, all values are treated as strings anyway.
(159)
For further details about the data-driving mechanism please see chapter 16
Contained in: Any
Children: None
.
34.4.
Data driver
455
Execution: The Excel data file node loads the data from the Excel file and expands
variable values in the table according to the option When binding variables, expand
(393)
values immediately . Each row is expanded individually left to right, meaning that within the same row - a cell may refer to a variable bound in a column further to the
left. Then the Excel data file registers itself with the Data driver context. A property group
called like the node will be created additionally. That group contains the variables size
and totalsize. size shows the number of data rows with taking care about iteration
intervals. The variable totalsize shows the total number of data rows without taking
(405)
care about iteration intervals. When all Data binders have been registered the Test-set
will query the Excel data file in order to iterate over the available sets of data. If no such
context is available the respective property group will be extended with all variables.
Attributes:
Figure 34.17: Excel data file attributes
Name
The Name of a Data binder is mandatory. It is used to distinguish Data binders in the
(479)
same Data driver context. A Break node executed during data-driven testing can
be used to break out of a specific loop by referring to the Data binder’s Name.
34.4.
Data driver
456
Variable: Yes
Restrictions: None
Iteration counter
The name of the variable that the iteration counter will be bound to.
Variable: Yes
Restrictions: None
Iteration ranges
An optional set of indexes or ranges to use from the bound data. This is
especially useful during test development in order to run sample tests with just a
single index or a subset of the given data.
Ranges are separated by ’,’. Each range is either a single index or an inclusive
range of the form ’from-to’ or ’from:to’ where ’to’ is optional. Indexes or ranges
may be specified multiple times, overlap or be given in descending order. Indexes are 0-based, negative indexes are counted from the end, -1 being the
last item. An invalid syntax or an index outside the valid data range will cause
(665)
a BadRangeException .
The following table shows some example range specifications and the resulting
indexes, based on a set of 20 entries.
Iteration ranges
0
-2, -1
1-2,4:5
18:,-33-2,16:15
Resulting indexes
[0]
[18, 19]
[1, 2, 4, 5]
[18, 19, 17, 18, 19]
[3, 2, 16, 15]
Table 34.10: Iteration range examples
Note
The value bound for the Iteration counter reflects the index in the current interval,
not the counter of actual iterations, e.g. if you specify ’2’ there will be a single
iteration with the Iteration counter bound to ’2’, not ’0’.
Variable: Yes
Restrictions: Valid syntax and index values
Excel file name
The name of the Excel file to read the test data.
resolved relative to the directory of the test-suite.
Relative path names are
34.4.
Data driver
457
The button above the attribute brings up a dialog in which you can select the
Excel file interactively. You can also get to this dialog by pressing Shift-Return or
Alt-Return when the focus is in the text field.
Variable: Yes
Restrictions: Must not be empty
Worksheet name
The name of the worksheet that contains the test-data. If empty, the first sheet
will be used.
Variable: Yes
Restrictions: None
Override date format
This value specifies a date format which will be used for all date values specified
in the Excel file.
The format must be specified as for the Java class
SimpleDateFormat, i.e. ’d’ for day, ’M’ for month, ’y’ for year. Thus ’dd’ stands
for two digits of a day and similar for ’MM’, ”yy’ or ’yyyy’.
Variable: Yes
Restrictions: None
Variables in rows
If this attribute is checked, the names of the variables will be taken from the first
row. If it’s not checked the names will come from the first column.
Variable: Yes
Restrictions: None
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
34.4.
Data driver
458
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
lets
are much better suited to this task. The option External editor command
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.4.5 CSV data file
A CSV data file node is used to load external data from a file and make it
available for data-driven testing. CSV stands for Comma-separated Values,
a more or less standard plain text file format. Each line in the file contains
one set of data with the values separated by a separator character, often but not always
a comma (’,’). For use with a CSV data file node the first line of the CSV file must contain
the names of the variables to bind. The rest of the lines should contain the values to be
used for the iteration steps.
Unfortunately, with CSV files there are non-uniform definitions for things like quoting,
white-space, multi-line values or embedded separator characters. Two de-facto standards exist, one used by Microsoft Excel and one by the rest of the world. QF-Test
supports both of these. For further details about the data-driving mechanism please
(159)
see chapter 16 .
Contained in: Any
Children: None
Execution: The CSV data file node loads the data from the CSV file and expands variable values in the table according to the option When binding variables, expand values
(393)
immediately . Each row is expanded individually left to right, meaning that - within the
same row - a cell may refer to a variable bound in a column further to the left. Then
the CSV data file registers itself with the Data driver context. A property group called
like the node will be created additionally. That group contains the variables size and
totalsize. size shows the number of data rows with taking care about iteration intervals. The variable totalsize shows the total number of data rows without taking
(405)
care about iteration intervals. When all Data binders have been registered the Test-set
34.4.
Data driver
459
will query the CSV data file in order to iterate over the available sets of data. If no such
context is available the respective property group will be extended with all variables.
Attributes:
Figure 34.18: CSV data file attributes
Name
The Name of a Data binder is mandatory. It is used to distinguish Data binders in the
(479)
same Data driver context. A Break node executed during data-driven testing can
be used to break out of a specific loop by referring to the Data binder’s Name.
Variable: Yes
Restrictions: None
Iteration counter
The name of the variable that the iteration counter will be bound to.
Variable: Yes
Restrictions: None
34.4.
Data driver
460
Iteration ranges
An optional set of indexes or ranges to use from the bound data. This is
especially useful during test development in order to run sample tests with just a
single index or a subset of the given data.
Ranges are separated by ’,’. Each range is either a single index or an inclusive
range of the form ’from-to’ or ’from:to’ where ’to’ is optional. Indexes or ranges
may be specified multiple times, overlap or be given in descending order. Indexes are 0-based, negative indexes are counted from the end, -1 being the
last item. An invalid syntax or an index outside the valid data range will cause
(665)
a BadRangeException .
The following table shows some example range specifications and the resulting
indexes, based on a set of 20 entries.
Iteration ranges
0
-2, -1
1-2,4:5
18:,-33-2,16:15
Resulting indexes
[0]
[18, 19]
[1, 2, 4, 5]
[18, 19, 17, 18, 19]
[3, 2, 16, 15]
Table 34.11: Iteration range examples
Note
The value bound for the Iteration counter reflects the index in the current interval,
not the counter of actual iterations, e.g. if you specify ’2’ there will be a single
iteration with the Iteration counter bound to ’2’, not ’0’.
Variable: Yes
Restrictions: Valid syntax and index values
CSV file name
The name of the CSV file to get the test data from. Relative path names are
resolved relative to the directory of the test-suite.
The button above the attribute brings up a dialog in which you can select the
CSV
file interactively. You can also get to this dialog by pressing Shift-Return or
Alt-Return
when the focus is in the text field.
Variable: Yes
Restrictions: Must not be empty
File encoding
34.4.
Data driver
461
An optional encoding for the CSV file, ”UTF-8” for example. If no encoding is
specified, the file will be read with the default encoding of the Java VM.
Variable: Yes
Restrictions: The encoding must be supported by the Java VM.
Read Microsoft Excel CSV format
If this option is active QF-Test will try to parse the CSV file using the format used
by Microsoft Excel.
Variable: Yes
Restrictions: None
Separator character
In this attribute you can specify the character to be used as separator for data
values within the CSV file. If no separator is defined a comma (’,’) is used as the
(459)
default value. If Read Microsoft Excel CSV format
is activated this attribute is
ignored.
Variable: Yes
Restrictions: None
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
are much better suited to this task. The option External editor command
lets
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
34.4.
Data driver
462
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.4.6 Data loop
A Data loop node is a simple loop with a single variable bound as the iteration
(398)
counter. It is useful for executing Test-cases multiple times by placing them
(405)
(440)
inside a Test-set with a Data driver holding a Data loop.
Contained in: Any
Children: None
Execution: During execution all the Data loop node does is register itself with the Data
driver context. The iteration counter will be used as variable. A property group called
like the node will be created additionally. That group contains the variables size and
totalsize. size shows the number of data rows with taking care about iteration
intervals. The variable totalsize shows the total number of data rows without taking
(405)
care about iteration intervals. When all Data binders have been registered the Test-set
will query the Data loop in order to iterate over the available sets of data. If no such
context is available the respective property group will be extended with all variables.
Attributes:
Figure 34.19: Data loop attributes
34.4.
Data driver
463
Name
The Name of a Data binder is mandatory. It is used to distinguish Data binders in the
(479)
same Data driver context. A Break node executed during data-driven testing can
be used to break out of a specific loop by referring to the Data binder’s Name.
Variable: Yes
Restrictions: None
Iteration counter
The name of the variable that the iteration counter will be bound to.
Variable: Yes
Restrictions: None
Iteration ranges
An optional set of indexes or ranges to use from the bound data. This is
especially useful during test development in order to run sample tests with just a
single index or a subset of the given data.
Ranges are separated by ’,’. Each range is either a single index or an inclusive
range of the form ’from-to’ or ’from:to’ where ’to’ is optional. Indexes or ranges
may be specified multiple times, overlap or be given in descending order. Indexes are 0-based, negative indexes are counted from the end, -1 being the
last item. An invalid syntax or an index outside the valid data range will cause
(665)
a BadRangeException .
The following table shows some example range specifications and the resulting
indexes, based on a set of 20 entries.
Iteration ranges
0
-2, -1
1-2,4:5
18:,-33-2,16:15
Resulting indexes
[0]
[18, 19]
[1, 2, 4, 5]
[18, 19, 17, 18, 19]
[3, 2, 16, 15]
Table 34.12: Iteration range examples
Note
The value bound for the Iteration counter reflects the index in the current interval,
not the counter of actual iterations, e.g. if you specify ’2’ there will be a single
iteration with the Iteration counter bound to ’2’, not ’0’.
Variable: Yes
Restrictions: Valid syntax and index values
34.5.
Procedures
464
Number of iterations
The number of iterations of the loop.
Variable: Yes
Restrictions: > 0
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
lets
are much better suited to this task. The option External editor command
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.5
Procedures
Procedures are a means to collect some of the basic building blocks of a test-suite like
(537)
(560)
events and checks into a larger, reusable structure.
Procedures can be called from any other part of the test-suite and even from different
(193)
suites (see section 19.1 ). You can pass parameters to a procedure in the form of
(62)
variable definitions (see chapter 7 ).
34.5.
Procedures
465
A typical example would be a procedure that selects a menu item in a menu. Its parameters could be the client name of the SUT, the name of the menu and the name of the
menu item.
34.5.1 Procedure
(415)
A Procedure is a Sequence
(466)
Procedure call .
that is executed from some other place by a
The parameters of the procedure are not defined explicitly. Instead they are
a consequence of the variable references in the children of the Procedure. You may want
(464)
to define fallback values for some or all of the parameters in the Variable definitions .
(465)
In any case it is a good idea to document the required parameters in the Comment
attribute.
(469)
A Procedure can return a value to the calling node with the help of a Return
Without such a node a Procedure implicitly returns the empty string.
(470)
Contained in: Package
node.
(472)
, Procedures
Children: Any
Execution: The Procedure’s variables are bound as fallback values. The child nodes are
executed one by one, then the fallback values are unbound again.
Attributes:
34.5.
Procedures
466
Figure 34.20: Procedure Attributes
Name
(470)
A Procedure is identified by its name and the names of its Package ancestors, so
you should assign a name with a meaning that is easy to recognize and
remember.
Variable: No
Restrictions: Must not be empty or contain the characters ’.’ or ’#’.
Variable definitions
Here you can define default or fallback values for the Procedure’s parameters (see
(62)
chapter 7 ). Defining these values also serves as documentation and is a
valuable time-saver when using the dialog to select the Procedure for the
(467)
(466)
(16)
Procedure name attribute of a Procedure call . See section 2.2.5 about how to
work with the table.
4.2+
In case you want to re-set the order of the parameters like they are sorted in the
called procedure, you can select Re-set parameter order .
Variable: Variable names no, values yes
Restrictions: None
34.5.
Procedures
467
Maximum error level
When a warning, error or exception occurs during a test run, the state of the
corresponding node of the run-log is set accordingly. This state is normally
propagated to the parent node in a way that ensures that the error state of a
run-log node represents the worst of its child nodes’ states. Using this attribute,
the maximum error state that the run-log node for a sequence will propagate, can
be limited.
Note
This value has no effect on the way exceptions are handled. It only affects the
error states of the run-log nodes and by that the exit code of QF-Test when run in
(10)
batch mode (see. section 1.7 ). It also has no effect on the creation of compact
(679)
run-logs (see command line argument -compact ). The node for a sequence in
which a warning, error or exception occurs is never removed from a compact log,
even if the error is not propagated due to the setting of this attribute.
Variable: No
Restrictions: None
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
are much better suited to this task. The option External editor command
lets
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
34.5.
Procedures
468
Variable: Yes
Restrictions: None
34.5.2 Procedure call
(463)
This node calls a Procedure
in the same or a different test-suite, meaning
that execution continues in the Procedure. When the Procedure is finished,
the value returned by the Procedure is bound to the variable defined in the
(467)
Variable for return value attribute and execution returns to the Procedure call and thus to
its parent node.
(463)
(464)
(471)
The name of the Procedure
to call is determined by its Name
and the Names
of
(470)
its Package
parents. These are concatenated with a dot (’.’) as separator, starting
with the outermost Package and ending in the Procedure’s name. Thus to call a Procedure
named expandNode in a Package named tree that is itself a child of a Package named
(467)
main, set the Procedure name attribute to main.tree.expandNode.
See section 19.1
(193)
about how to call a Procedure in a different test-suite.
(398)
Contained in: All kinds of sequences
.
Children: None
(468)
Execution: The Variable definitions
of the Procedure call are bound, the target
(463)
Procedure
is determined and execution passed to it. After the Procedure returns, the
Procedure call’s variables are unbound again.
Attributes:
34.5.
Procedures
469
Figure 34.21: Procedure call Attributes
Name
(463)
The full name of the Procedure
(471)
, created from the Names
(470)
of its Package
(464)
parents and its own Name , joined by a dot. The
button brings up a dialog
in which you can select the Procedure interactively. By selecting the ”Copy
parameters” checkbox you can adopt the Procedure’s default values as
parameters for the Procedure call node to save typing.
Variable: Yes
Restrictions: Must not be empty.
Variable for return value
(469)
The value returned by the Procedure, either through a Return node or the empty
string, is bound to the variable defined in this optional attribute. Additionally, the
most recent return value is always available as the special variable
${qftest:return}.
Variable: Yes
Restrictions: None
34.5.
Procedures
470
Local variable
This flag determines whether to create a local or global variable binding. If unset,
the variable is bound in the global variables. If set, the topmost current binding for
the variable is replaced with the new value, provided this binding is within the
(463)
(427)
(398)
context of the currently executing Procedure , Dependency or Test-case node.
If no such binding exists, a new binding is created in the currently executing
Procedure, Dependency or Test-case node or, if there is no such node, in the global
(62)
bindings. See chapter 7 for a detailed explanation of variable binding and
lookup.
Variable: No
Restrictions: None
Variable definitions
(463)
This is where you define the parameter values for the Procedure
(62)
(16)
7 ). See section 2.2.5 about how to work with the table.
(see chapter
Variable: Variable names no, values yes
Restrictions: None
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
are much better suited to this task. The option External editor command
lets
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
34.5.
Procedures
471
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.5.3 Return
(463)
This node can be used to return from a Procedure
pass a return value to the calling node.
From a script,
ReturnException.
prematurely and also to
the same effect can be achieved by raising a
(398)
Contained in: All kinds of sequences
.
Children: None
Execution: A ReturnException is thrown. If it is caught by a matching Procedure, the
Procedure is terminated and the return value passed to the caller. If the node is executed
outside a Procedure it will lead to an error.
Attributes:
Figure 34.22: Return Attributes
Return value
The value to return from the Procedure. May be empty in which case the empty
string is returned.
Variable: Yes
Restrictions: None
34.5.
Procedures
472
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
lets
are much better suited to this task. The option External editor command
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.5.4 Package
(463)
The only use of Packages is to structure the Procedures of a test-suite. The
(471)
Name of a Package is part of the Procedure’s fully qualified name, as required
(466)
by a Procedure call .
(470)
Contained in: Package
(470)
Children: Package
(472)
, Procedures
(463)
, Procedure
Execution: Cannot be executed.
Attributes:
34.5.
Procedures
473
Figure 34.23: Package Attributes
Name
(463)
The name of a Package is part of the identification of the Procedures it contains,
so you should assign a name with a meaning that is easy to recognize and
remember.
Variable: No
Restrictions: Must not be empty or contain the characters ’.’ or ’#’.
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Border for relative calls
This flag determines whether relative procedure calls, test calls or dependency
references are allowed within that certain node. Relative calls passing that border
are not allowed. If that attribute is not specified in the hierarchy, no relative calls
are allowed.
Variable: No
Restrictions: None
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
34.5.
Procedures
474
this text area might not be the right place. There are many excellent editors that
(326)
are much better suited to this task. The option External editor command
lets
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.5.5 Procedures
This is the root of all Packages
(470)
and Procedures
(463)
.
Contained in: Root node
(470)
Children: Package
(463)
, Procedure
Execution: Cannot be executed.
Attributes:
Figure 34.24: Procedures Attributes
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Border for relative calls
This flag determines whether relative procedure calls, test calls or dependency
34.6.
Control structures
475
references are allowed within that certain node. Relative calls passing that border
are not allowed. If that attribute is not specified in the hierarchy, no relative calls
are allowed.
Variable: No
Restrictions: None
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
lets
are much better suited to this task. The option External editor command
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.6
Control structures
(398)
Besides the standard sequence types QF-Test has a set of special control structures.
(481)
(484)
(488)
(473)
and Else
nodes. Loops
and
Conditional processing is handled via If , Elseif
(476)
(479)
(490)
While nodes can be aborted with a Break node. Exceptions are handled by Try ,
(493)
(496)
Catch and Finally nodes.
Beyond that, full scripting is available for the Jython language (formerly called JPython),
(114)
Groovy and JavaScript as documented in chapter 12 .
34.6.1 Loop
(415)
This node is basically the same as a Sequence except that its children can
be executed more than once. This is useful in two ways. For one thing, a
test sequence that executes OK a hundred times is more trustworthy than a
sequence that only runs once. The other use is to run a number of similar jobs with
slight variations. To that end, the count of the current iteration is bound as a variable
during execution.
34.6.
Control structures
476
Special Loops with varying increments can be achieved by changing the value of the
(475)
Iteration counter during execution.
(479)
Execution of a Loop can be terminated prematurely with the help of a Break node. An
(488)
optional Else node may be placed at the end of the Loop. It is executed if all iterations
of the Loop are run through completely without hitting a Break.
(398)
Contained in: All kinds of sequences
.
Children: Any
(475)
(475)
Execution: The Variable definitions of the Loop are bound. The Iteration counter is
initialized to 0 and the child nodes are executed one by one. For each iteration the
Iteration counter is increased by one and the children are executed again. After the final
execution of the last child is complete, the Iteration counter and the Variable definitions are
unbound again.
Attributes:
Figure 34.25: Loop attributes
Name
The name of a sequence is a kind of short description. It is displayed in the tree
view, so it should be concise and tell something about the function of the
sequence.
34.6.
Control structures
477
Variable: No
Restrictions: None
Number of iterations
The number of iterations of the loop.
Variable: Yes
Restrictions: > 0
Iteration counter
The name of the variable that will hold the iteration count during the execution.
Make sure to use different Iteration counter names for nested loops.
Variable: Yes
Restrictions: None
Variable definitions
This is where you define the values of the variables that remain bound during the
(62)
(16)
execution of the sequence’s child nodes (see chapter 7 ). See section 2.2.5
about how to work with the table.
Variable: Variable names no, values yes
Restrictions: None
Maximum error level
When a warning, error or exception occurs during a test run, the state of the
corresponding node of the run-log is set accordingly. This state is normally
propagated to the parent node in a way that ensures that the error state of a
run-log node represents the worst of its child nodes’ states. Using this attribute,
the maximum error state that the run-log node for a sequence will propagate, can
be limited.
Note
This value has no effect on the way exceptions are handled. It only affects the
error states of the run-log nodes and by that the exit code of QF-Test when run in
(10)
batch mode (see. section 1.7 ). It also has no effect on the creation of compact
(679)
run-logs (see command line argument -compact ). The node for a sequence in
which a warning, error or exception occurs is never removed from a compact log,
even if the error is not propagated due to the setting of this attribute.
Variable: No
Restrictions: None
QF-Test ID
34.6.
Control structures
478
(479)
The QF-Test ID of the Loop node can be used in a Break
outer loop explicitly when loops are nested.
node to terminate an
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
are much better suited to this task. The option External editor command
lets
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.6.2 While
This is a sequence that is executed repeatedly as long as a condition is fulfilled.
(479)
The loop can be terminated prematurely with the help of a Break
node.
(398)
Contained in: All kinds of sequences
.
Children: Any
(478)
Execution: The Variable definitions
of the While node are bound. If the condition
evaluates to true, the child nodes are executed one by one. This is repeated until the
(479)
condition evaluates to false or the loop is terminated through a Break
node or an
Exception. Finally the Variable definitions are unbound again.
34.6.
Control structures
479
Attributes:
Figure 34.26: While attributes
Condition
A condition is an expression that evaluates to either true or false. QF-Test
discriminates between simple expression that it evaluates itself and complex
expressions that are passed to the Jython script language to evaluate.
An empty string or the string false (regardless of case) is interpreted as false,
the string true as true. Whole numbers are true if and only if they are non-zero.
Evaluating expressions in Jython opens the way for powerful expression handling.
Jython supports the standard operators ==, !=, >, >=, < and <=. You can combine
expressions with and and or and define their priority with braces.
Note
Accessing QF-Test variables in a condition follows the same rules as in Jython
(118)
scripts (see section 12.2.3 ). You can use the standard QF-Test syntax $(...)
and ${...:...} for numeric or boolean values. String values should be accessed with rc.lookup(...).
Some examples:
34.6.
Control structures
480
Expression
Empty String
0
21
False
True
abc abc
25 > 0
${qftest:batch}
not ${qftest:batch}
rc.lookup(”system”, ”java.version”) == ”1.3.1”
rc.lookup(”system”, ”java.version”)[0] == ”1”
(1 > 0 and 0 == 0) or 2 < 1
Value
False
False
True
False
True
Syntax error
True
True if QF-Test is run in batch mode
True if QF-Test is run in interactive mode
True if JDK Version is 1.3.1
True is JDK Version starts with 1
True
Table 34.13: Condition examples
Variable: Yes
Restrictions: Valid syntax
Name
The name of a sequence is a kind of short description. It is displayed in the tree
view, so it should be concise and tell something about the function of the
sequence.
Variable: No
Restrictions: None
Variable definitions
This is where you define the values of the variables that remain bound during the
(62)
(16)
execution of the sequence’s child nodes (see chapter 7 ). See section 2.2.5
about how to work with the table.
Variable: Variable names no, values yes
Restrictions: None
Maximum error level
When a warning, error or exception occurs during a test run, the state of the
corresponding node of the run-log is set accordingly. This state is normally
propagated to the parent node in a way that ensures that the error state of a
run-log node represents the worst of its child nodes’ states. Using this attribute,
the maximum error state that the run-log node for a sequence will propagate, can
be limited.
34.6.
Note
Control structures
481
This value has no effect on the way exceptions are handled. It only affects the
error states of the run-log nodes and by that the exit code of QF-Test when run in
(10)
batch mode (see. section 1.7 ). It also has no effect on the creation of compact
(679)
run-logs (see command line argument -compact ). The node for a sequence in
which a warning, error or exception occurs is never removed from a compact log,
even if the error is not propagated due to the setting of this attribute.
Variable: No
Restrictions: None
QF-Test ID
(479)
The QF-Test ID of the While node can be used in a Break
outer loop explicitly when loops are nested.
node to terminate an
Variable: No
Restrictions: Must not be empty, contain any of the characters ’#’, ’$’, ’@’, ’&’, or
’%’ or start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
are much better suited to this task. The option External editor command
lets
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.6.3 Break
34.6.
Control structures
482
(473)
This node is used to terminate a Loops
(476)
or a While
node prematurely.
(398)
Contained in: All kinds of sequences
.
Children: None
Execution: A BreakException is thrown. If it is caught by a matching loop, the loop
is terminated, otherwise it will lead to an error.
Attributes:
Figure 34.27: Break attributes
QF-Test loop ID
For nested loops you can specify the loop to terminate by specifying the
(475)
(473)
(479)
QF-Test ID of a Loop node and refer to it here. It works with QF-Test ID for
(476)
a While
node respectively. In case this field is empty the innermost loop is
terminated.
Variable: Yes
Restrictions: None
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
34.6.
Control structures
483
Variable: Yes
Restrictions: Valid number >= 0
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
are much better suited to this task. The option External editor command
lets
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.6.4 If
Like in Java the child nodes of this node are executed only if a condition evaluates to true. However QF-Test differs from common programming languages
in the way alternative branches are arranged.
(398)
Contained in: All kinds of sequences
.
(484)
Children: Any executable node, followed by an arbitrary number of Elseif
(488)
an optional Else at the end.
(483)
nodes with
Execution: The Variable definitions of the If node are bound. If the condition evaluates
to true, the normal child nodes are executed one by one. Otherwise the conditions of
(484)
the Elseif nodes are evaluated and the first Elseif node whose condition evaluates to
(488)
true is executed. If none of the conditions are true of no Elseif nodes exist, the Else
node is executed, if one exists. Finally the Variable definitions are unbound again.
Attributes:
34.6.
Control structures
484
Figure 34.28: If attributes
Condition
A condition is an expression that evaluates to either true or false. QF-Test
discriminates between simple expression that it evaluates itself and complex
expressions that are passed to the Jython script language to evaluate.
An empty string or the string false (regardless of case) is interpreted as false,
the string true as true. Whole numbers are true if and only if they are non-zero.
Evaluating expressions in Jython opens the way for powerful expression handling.
Jython supports the standard operators ==, !=, >, >=, < and <=. You can combine
expressions with and and or and define their priority with braces.
Note
Accessing QF-Test variables in a condition follows the same rules as in Jython
(118)
scripts (see section 12.2.3 ). You can use the standard QF-Test syntax $(...)
and ${...:...} for numeric or boolean values. String values should be accessed with rc.lookup(...).
Some examples:
34.6.
Control structures
485
Expression
Empty String
0
21
False
True
abc abc
25 > 0
${qftest:batch}
not ${qftest:batch}
rc.lookup(”system”, ”java.version”) == ”1.3.1”
rc.lookup(”system”, ”java.version”)[0] == ”1”
(1 > 0 and 0 == 0) or 2 < 1
Value
False
False
True
False
True
Syntax error
True
True if QF-Test is run in batch mode
True if QF-Test is run in interactive mode
True if JDK Version is 1.3.1
True is JDK Version starts with 1
True
Table 34.14: Condition examples
Variable: Yes
Restrictions: Valid syntax
Name
The name of a sequence is a kind of short description. It is displayed in the tree
view, so it should be concise and tell something about the function of the
sequence.
Variable: No
Restrictions: None
Variable definitions
This is where you define the values of the variables that remain bound during the
(62)
(16)
execution of the sequence’s child nodes (see chapter 7 ). See section 2.2.5
about how to work with the table.
Variable: Variable names no, values yes
Restrictions: None
Maximum error level
When a warning, error or exception occurs during a test run, the state of the
corresponding node of the run-log is set accordingly. This state is normally
propagated to the parent node in a way that ensures that the error state of a
run-log node represents the worst of its child nodes’ states. Using this attribute,
the maximum error state that the run-log node for a sequence will propagate, can
be limited.
34.6.
Note
Control structures
486
This value has no effect on the way exceptions are handled. It only affects the
error states of the run-log nodes and by that the exit code of QF-Test when run in
(10)
batch mode (see. section 1.7 ). It also has no effect on the creation of compact
(679)
run-logs (see command line argument -compact ). The node for a sequence in
which a warning, error or exception occurs is never removed from a compact log,
even if the error is not propagated due to the setting of this attribute.
Variable: No
Restrictions: None
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
are much better suited to this task. The option External editor command
lets
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.6.5 Elseif
34.6.
Control structures
487
(481)
This node is an alternative branch in an If node. If the condition of the If
node evaluates to false, the first Elseif node whose condition is true is executed.
(481)
Contained in: If
Children: Any
(486)
of the Elseif are bound and its child nodes exeExecution: The Variable definitions
cuted one by one. After the execution of the last child is complete, the variables are
unbound again.
Attributes:
Figure 34.29: Elseif attributes
Condition
A condition is an expression that evaluates to either true or false. QF-Test
discriminates between simple expression that it evaluates itself and complex
expressions that are passed to the Jython script language to evaluate.
An empty string or the string false (regardless of case) is interpreted as false,
the string true as true. Whole numbers are true if and only if they are non-zero.
Evaluating expressions in Jython opens the way for powerful expression handling.
34.6.
Control structures
488
Jython supports the standard operators ==, !=, >, >=, < and <=. You can combine
expressions with and and or and define their priority with braces.
Note
Accessing QF-Test variables in a condition follows the same rules as in Jython
(118)
scripts (see section 12.2.3 ). You can use the standard QF-Test syntax $(...)
and ${...:...} for numeric or boolean values. String values should be accessed with rc.lookup(...).
Some examples:
Expression
Empty String
0
21
False
True
abc abc
25 > 0
${qftest:batch}
not ${qftest:batch}
rc.lookup(”system”, ”java.version”) == ”1.3.1”
rc.lookup(”system”, ”java.version”)[0] == ”1”
(1 > 0 and 0 == 0) or 2 < 1
Value
False
False
True
False
True
Syntax error
True
True if QF-Test is run in batch mode
True if QF-Test is run in interactive mode
True if JDK Version is 1.3.1
True is JDK Version starts with 1
True
Table 34.15: Condition examples
Variable: Yes
Restrictions: Valid syntax
Name
The name of a sequence is a kind of short description. It is displayed in the tree
view, so it should be concise and tell something about the function of the
sequence.
Variable: No
Restrictions: None
Variable definitions
This is where you define the values of the variables that remain bound during the
(62)
(16)
execution of the sequence’s child nodes (see chapter 7 ). See section 2.2.5
about how to work with the table.
Variable: Variable names no, values yes
Restrictions: None
34.6.
Control structures
489
Maximum error level
When a warning, error or exception occurs during a test run, the state of the
corresponding node of the run-log is set accordingly. This state is normally
propagated to the parent node in a way that ensures that the error state of a
run-log node represents the worst of its child nodes’ states. Using this attribute,
the maximum error state that the run-log node for a sequence will propagate, can
be limited.
Note
This value has no effect on the way exceptions are handled. It only affects the
error states of the run-log nodes and by that the exit code of QF-Test when run in
(10)
batch mode (see. section 1.7 ). It also has no effect on the creation of compact
(679)
run-logs (see command line argument -compact ). The node for a sequence in
which a warning, error or exception occurs is never removed from a compact log,
even if the error is not propagated due to the setting of this attribute.
Variable: No
Restrictions: None
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
are much better suited to this task. The option External editor command
lets
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
34.6.
Control structures
490
Variable: Yes
Restrictions: None
34.6.6 Else
(481)
An Else node is executed if neither the condition of its If
(484)
condition of its Elseif siblings evaluate to true.
(481)
Contained in: If
(473)
, Loop
, Try
parent, nor the
(490)
Children: Any
(489)
Execution: The Variable definitions of the Else are bound and its child nodes executed
one by one. After the execution of the last child is complete, the variables are unbound
again.
Attributes:
Figure 34.30: Else attributes
Name
The name of a sequence is a kind of short description. It is displayed in the tree
view, so it should be concise and tell something about the function of the
sequence.
34.6.
Control structures
491
Variable: No
Restrictions: None
Variable definitions
This is where you define the values of the variables that remain bound during the
(62)
(16)
execution of the sequence’s child nodes (see chapter 7 ). See section 2.2.5
about how to work with the table.
Variable: Variable names no, values yes
Restrictions: None
Maximum error level
When a warning, error or exception occurs during a test run, the state of the
corresponding node of the run-log is set accordingly. This state is normally
propagated to the parent node in a way that ensures that the error state of a
run-log node represents the worst of its child nodes’ states. Using this attribute,
the maximum error state that the run-log node for a sequence will propagate, can
be limited.
Note
This value has no effect on the way exceptions are handled. It only affects the
error states of the run-log nodes and by that the exit code of QF-Test when run in
(10)
batch mode (see. section 1.7 ). It also has no effect on the creation of compact
(679)
run-logs (see command line argument -compact ). The node for a sequence in
which a warning, error or exception occurs is never removed from a compact log,
even if the error is not propagated due to the setting of this attribute.
Variable: No
Restrictions: None
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
34.6.
Control structures
492
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
lets
are much better suited to this task. The option External editor command
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.6.7 Try
A Try is a special sequence whose semantic equates the Java
try-catch-finally composition. As in Python, this composition was
extended to support optional else blocks. A Try behaves the same way as a
(415)
with the extension, that exception handling is possible. Like a
special Sequence
Sequence it has a set of normal child nodes that it executes one by one. After these may
(493)
(488)
nodes with an optional Else
node followed by
come an arbitrary number of Catch
(496)
an optional Finally node at the end.
If an exception is thrown during the execution of one of the normal child nodes, the
Catch nodes are tested for whether they are able to catch that exception. The first one
found is executed and the Try will be exited normally afterwards without continuing with
the execution of the normal child nodes and without passing the exception on. If no
matching Catch is found, the exception will terminate the Try immediately (almost, see
below) and be passed onto the Try’s parent.
(488)
A possible Else child node at the end of a try node will be executed, if and only if no
(493)
Catch
nodes had been executed. This means, it is executed, when no exception in
the try block was thrown.
(496)
If the Try has a Finally
child node, this node will be executed just before the Try
finishes, no matter whether an exception is thrown and whether it is handled or not.
Note
Exceptions can also be caught implicitly by a Test
convenient.
(398)
Contained in: All kinds of sequences
.
(656)
, which is less flexible but often more
34.6.
Control structures
493
(493)
Children: Any executable node, followed by an arbitrary number of Catch
(488)
(496)
an optional Else node and/or an optional Finally node at the end.
nodes with
(491)
Execution: The Variable definitions
of the Try are bound and its normal child nodes
executed one by one. If an exception is thrown, execution of the normal children is
(494)
terminated. If a Catch node with a matching Exception class
is found it is executed.
Before exiting the Try its Finally node is executed unconditionally. After unbinding the
Variable definitions, the Try is either exited cleanly if no exception was thrown or the
exception was caught, or it passes on the uncaught exception.
Attributes:
Figure 34.31: Try attributes
Name
The name of a sequence is a kind of short description. It is displayed in the tree
view, so it should be concise and tell something about the function of the
sequence.
Variable: No
Restrictions: None
Variable definitions
This is where you define the values of the variables that remain bound during the
34.6.
Control structures
494
(62)
(16)
execution of the sequence’s child nodes (see chapter 7 ). See section 2.2.5
about how to work with the table.
Variable: Variable names no, values yes
Restrictions: None
Maximum error level
When a warning, error or exception occurs during a test run, the state of the
corresponding node of the run-log is set accordingly. This state is normally
propagated to the parent node in a way that ensures that the error state of a
run-log node represents the worst of its child nodes’ states. Using this attribute,
the maximum error state that the run-log node for a sequence will propagate, can
be limited.
Note
This value has no effect on the way exceptions are handled. It only affects the
error states of the run-log nodes and by that the exit code of QF-Test when run in
(10)
batch mode (see. section 1.7 ). It also has no effect on the creation of compact
(679)
run-logs (see command line argument -compact ). The node for a sequence in
which a warning, error or exception occurs is never removed from a compact log,
even if the error is not propagated due to the setting of this attribute.
For a Try node, the error state of the run-log is additionally affected by the
(495)
(493)
Maximum error level of the Catch node that handles an exception.
Variable: No
Restrictions: None
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
34.6.
Control structures
495
this text area might not be the right place. There are many excellent editors that
(326)
are much better suited to this task. The option External editor command
lets
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.6.8 Catch
(415)
(490)
A Catch is a special Sequence that can only be placed inside a Try node
(427)
or a Dependency . Its job is to catch exceptions that may arise during the
(398)
execution of a Try’s or a Test-case with a Dependency.
A Catch can handle an exception if the class of the exception is the same as the Catch
(494)
node’s Exception class attribute or a derived class thereof, just as in Java.
Contained in: Try
(490)
, Dependency
(427)
Children: Any
(495)
Execution: The Variable definitions
of the Catch are bound and its child nodes executed one by one. After the execution of the last child is complete, the variables are
unbound again.
Attributes:
34.6.
Control structures
496
Figure 34.32: Catch attributes
Exception class
This ComboBox lets you select the class of the exception that is to be caught. All
(662)
QF-Test exceptions are derived from the class TestException . For details
(662)
about the possible exceptions see chapter 35 .
Variable: No
Restrictions: None
Expected message
You can further qualify the exception to catch by specifying a message to look for.
If this attribute is empty, all exceptions of the specified class are caught.
Otherwise it is compared to exception.getMessage() and the exception is
caught only in case of a match.
Variable: Yes
34.6.
Control structures
497
Restrictions: Valid regexp if required.
As regexp
If this attribute is set, the exception message is matched against a regexp (see
(720)
section 40.4 ) instead of comparing plain strings.
Variable: Yes
Restrictions: None
Match against localized message
Most exceptions have two kinds of error message: The raw message is typically
some short English text whereas the localized message contains more details
and is either English or German, depending on the current language settings of
QF-Test. Both are shown in the run-log. If this attribute is set, the localized
exception message is used for comparison, otherwise the raw message. The
latter is usually preferable as it doesn’t depend on language settings so no regexp
is needed in order to ensure the correct handling of the different languages.
Variable: Yes
Restrictions: None
Name
The name of a sequence is a kind of short description. It is displayed in the tree
view, so it should be concise and tell something about the function of the
sequence.
Variable: No
Restrictions: None
Variable definitions
This is where you define the values of the variables that remain bound during the
(62)
(16)
execution of the sequence’s child nodes (see chapter 7 ). See section 2.2.5
about how to work with the table.
Variable: Variable names no, values yes
Restrictions: None
Maximum error level
(416)
(398)
In contrast to the Maximum error level of other sequences , this attribute does
not determine the error state propagated by the run-log node for the Catch node
(490)
itself, but for the log of its parent Try node, provided that the Catch is executed
in order to handle an exception.
34.6.
Control structures
498
The error state for any warnings, errors or exceptions that happen during the execution of the Catch node are not limited by the setting of this attribute. Otherwise
problems occurring during exception handling might accidentally go unnoticed.
Variable: No
Restrictions: None
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
lets
are much better suited to this task. The option External editor command
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.6.9 Finally
A Finally node, which can only be placed at the end of a Try node, will always
be executed as the last thing just before exiting the Try, no matter what happened there. This is used primarily to ensure that cleanup code like removing
a temporary file or terminating a process is executed under any conditions.
34.6.
Control structures
Contained in: Try
499
(490)
Children: Any
(497)
Execution: The Variable definitions
of the Finally are bound and its child nodes executed one by one. After the execution of the last child is complete, the variables are
unbound again.
Attributes:
Figure 34.33: Finally attributes
Name
The name of a sequence is a kind of short description. It is displayed in the tree
view, so it should be concise and tell something about the function of the
sequence.
Variable: No
Restrictions: None
Variable definitions
This is where you define the values of the variables that remain bound during the
(62)
(16)
execution of the sequence’s child nodes (see chapter 7 ). See section 2.2.5
about how to work with the table.
Variable: Variable names no, values yes
34.6.
Control structures
500
Restrictions: None
Maximum error level
When a warning, error or exception occurs during a test run, the state of the
corresponding node of the run-log is set accordingly. This state is normally
propagated to the parent node in a way that ensures that the error state of a
run-log node represents the worst of its child nodes’ states. Using this attribute,
the maximum error state that the run-log node for a sequence will propagate, can
be limited.
Note
This value has no effect on the way exceptions are handled. It only affects the
error states of the run-log nodes and by that the exit code of QF-Test when run in
(10)
batch mode (see. section 1.7 ). It also has no effect on the creation of compact
(679)
run-logs (see command line argument -compact ). The node for a sequence in
which a warning, error or exception occurs is never removed from a compact log,
even if the error is not propagated due to the setting of this attribute.
Variable: No
Restrictions: None
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
are much better suited to this task. The option External editor command
lets
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
34.6.
Control structures
501
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.6.10 Throw
If you need to handle an exceptional situation, you can use this node to throw
an explicit Exception.
(398)
Contained in: All kinds of sequences
.
Children: None
(668)
Execution:
A UserException
(499)
Exception message attribute.
is thrown,
its message taken from the
Attributes:
Figure 34.34: Throw attributes
Exception message
(668)
An arbitrary message for the UserException
to throw.
Variable: Yes
Restrictions: Must not be empty.
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
34.6.
Control structures
502
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
lets
are much better suited to this task. The option External editor command
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.6.11 Rethrow
(493)
An exception that was caught by a Catch
node can be thrown again with
the help of a Rethrow node. This is especially useful if you need to catch all
(490)
kinds of exceptions except one. To handle that case, create a Try node with
(662)
a Catch for the special exception followed by a Catch for a TestException . Then
place a Rethrow in the first Catch node.
(398)
Contained in: All kinds of sequences
directly below a Catch node.
. The Rethrow node doesn’t have to be placed
Children: None
(493)
Execution: The last exception caught by a Catch
node is thrown again. If no such
(668)
exception exists, a CannotRethrowException is thrown.
Attributes:
34.6.
Control structures
503
Figure 34.35: Rethrow attributes
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
lets
are much better suited to this task. The option External editor command
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.6.
Control structures
504
34.6.12 Server script
Server side scripts are executed by an interpreter (Jython, Groovy or
(114)
JavaScript) embedded into QF-Test. Scripting is explained in chapter 12
(726)
and chapter 41 . As server side scripts run embedded to QF-Test those
scripts cannot interact with the SUT. It is recommended that Server side scripts be
used for cases that work without the SUT or time consuming operations like accessing
databases or files.
(398)
Contained in: All kinds of sequences
.
Children: None
Execution: The script is executed by an embedded interpreter.
Attributes:
Figure 34.36: Server script attributes
Script
The script to execute.
Note
You may use QF-Test variables of the syntax $(var) or ${group:name} in Jython
34.6.
Control structures
505
scripts. They will be expanded before the script is passed to the Jython interpreter.
This can lead to unexpected behavior. rc.lookup(...) is the preferred method
(118)
in this case (see section 12.2.3 for details).
Note
In spite of syntax highlighting and automatical indentation this attribute might not
be the right place to write complex scripts. There are many excellent editors that
(326)
are much better suited to this task. The option External editor command lets you
define an external editor in which scripts can be edited conveniently by pressing
Alt-Return or
by clicking the
button. Complex scripts can also be written as
separate modules which can then be imported for use in this attribute. See chapter
(726)
41 for details.
Variable: Yes
Restrictions: Valid syntax
Script language
This attribute determines the interpreter in which to run the script, or in other
words, the scripting language to use. Possible values are ”Jython”, ”Groovy” and
”JavaScript”.
Variable: No
Restrictions: None
Name
The name of a script is a kind of short description. It is displayed in the tree view,
so it should be concise and tell something about the function of the script.
Variable: No
Restrictions: None
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
34.6.
Control structures
506
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
lets
are much better suited to this task. The option External editor command
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.6.13 SUT script
Client side scripts are executed by an interpreter (Jython, Groovy
or JavaScript) that QF-Test embeds into the SUT. Scripting is explained
(114)
(726)
in chapter 12 and chapter 41 . As client side scripts run in the SUT you
should use them to access the components and properties of the SUT.
(398)
Contained in: All kinds of sequences
.
Children: None
Execution: The script is executed inside the SUT by an embedded interpreter.
Attributes:
34.6.
Control structures
507
Figure 34.37: SUT script attributes
Client
The name of the SUT client process in which to execute the script.
Variable: Yes
Restrictions: Must not be empty.
Script
The script to execute.
Note
You may use QF-Test variables of the syntax $(var) or ${group:name} in Jython
scripts. They will be expanded before the script is passed to the Jython interpreter.
This can lead to unexpected behavior. rc.lookup(...) is the preferred method
(118)
in this case (see section 12.2.3 for details).
Note
In spite of syntax highlighting and automatical indentation this attribute might not
be the right place to write complex scripts. There are many excellent editors that
34.6.
Control structures
508
(326)
are much better suited to this task. The option External editor command lets you
define an external editor in which scripts can be edited conveniently by pressing
Alt-Return or
by clicking the
button. Complex scripts can also be written as
separate modules which can then be imported for use in this attribute. See chapter
(726)
41 for details.
Variable: Yes
Restrictions: Valid syntax
Script language
This attribute determines the interpreter in which to run the script, or in other
words, the scripting language to use. Possible values are ”Jython”, ”Groovy” and
”JavaScript”.
Variable: No
Restrictions: None
GUI engine
The GUI engine in which to execute the script. Only relevant for SUTs with more
(697)
than one GUI engine as described in chapter 37 .
Variable: Yes
(697)
Restrictions: See chapter 37
Name
The name of a script is a kind of short description. It is displayed in the tree view,
so it should be concise and tell something about the function of the script.
Variable: No
Restrictions: None
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
34.7.
Processes
509
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
lets
are much better suited to this task. The option External editor command
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.7
Processes
In order to enable QF-Test to communicate with the SUT, the SUT’s process must be
started by QF-Test. For details about how the application is started by QF-Test and
(699)
which kind of node is most applicable in your case, see chapter 38 .
The following nodes are used to start or stop processes, to wait for a connection with
the SUT or to wait for the termination of a process and check its exit value. The number
of concurrent processes managed by QF-Test is limited only by the underlying system,
but you have to assign a unique name to each process by which other nodes will identify
it.
We will refer to processes run by QF-Test as clients. QF-Test distinguishes between
two types of clients: arbitrary processes that are simply started and stopped and SUT
clients, the actual Java applications that QF-Test interacts with.
Standard input and output of a client is redirected to a terminal that you can open from
the Clients menu. The client’s output is also stored in the log of a test run.
34.7.1 Start Java SUT client
This node provides the standard and most flexible way of starting an SUT
client. To use it, you must know the Java command that starts your applica(511)
tion. If the SUT is run through a script, use a Start SUT client node instead
(699)
(see chapter 38 ).
(398)
Contained in: All kinds of sequences
.
34.7.
Processes
510
Children: None
Execution: The command line for the program is built from the attributes and the process is started. Its input and output are redirected to QF-Test.
Attributes:
Figure 34.38: Start Java SUT client attributes
Client
This is the identifier for the SUT client. It must remain unique as long as the
process is alive. Other nodes will refer to the client by this name.
34.7.
Processes
511
Variable: Yes
Restrictions: Must not be empty
Executable
The Java program to run, typically java or javaw on Windows. If you want to run
a specific Java virtual machine you should give the full path name of the
executable.
brings up a dialog in which you can
file inThe file button
select the program
teractively. You can also get to this dialog by pressing Shift-Return or Alt-Return ,
when the focus is in the text field.
Variable: Yes
Restrictions: Must not be empty
Directory
Here you can set the working directory for the program. If you leave this value
empty, the program will inherit QF-Test’s working directory.
brings up a dialog in which you can select the directory
inThe directory button
teractively. You can also get to this dialog by pressing Shift-Return or Alt-Return ,
when the focus is in the text field.
Variable: Yes
Restrictions: Must be empty or an existing directory
Class name
The fully qualified name of the Java class of the SUT whose main(String[])
method will be called. If your application is run from an executable jar archive with
the -jar parameter, you must leave this attribute empty.
Note
QF-Test can only run the main method of the class if both the class itself and the
main method are declared public.
Please verify this if you get an
IllegalAccessException when you try to start your application.
Variable: Yes
Restrictions: Valid class name
Executable parameters
There is one tab for each of the following parameter lists:
Executable
The command line arguments for the Java executable. Put each parameter
34.7.
Processes
512
on a line of its own. Special quoting of whitespace or symbols is not
required.
For example to set the classpath, set one line to -classpath and the following line to the desired value of the classpath. You don’t need to concern
yourself with QF-Test’s jar files.
Class
These are the arguments that are passed to the main(String[]) method
of the class to be started. Again, each argument requires a row of its own
and special quoting is not required.
Extra
Extra
parameters
for
the
class
de.qfs.apps.qftest.start.StartClient which is the wrapper class
started by QF-Test in order to gain access to the EventQueue, establish an
RMI connection between the SUT and QF-Test and then start the main class
of the SUT. These parameters are intended for future extensions, scripting
and internal logging.
(16)
See section 2.2.5
about how to work with the tables.
Variable: Yes
Restrictions: None
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
34.7.
Processes
513
(326)
are much better suited to this task. The option External editor command
lets
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.7.2 Start SUT client
If your application is normally started through a shell script or a special executable starter, the most convenient way to run it from QF-Test is through
this kind of node. Depending on how the application is actually started by this
script, some modifications may be required. A typical symptom is that the SUT starts up
(699)
fine, but the connection to QF-Test is not established. In that case, read chapter 38 .
(398)
Contained in: All kinds of sequences
.
Children: None
Execution: The command line for the program is built from the attributes and the process is started. Its input and output are redirected to QF-Test.
Attributes:
34.7.
Processes
514
Figure 34.39: Start SUT client attributes
Client
This is the identifier for the SUT client. It must remain unique as long as the
process is alive. Other nodes will refer to the client by this name.
Variable: Yes
Restrictions: Must not be empty
Executable
The program to run. If the executable file is not located in a directory on the PATH
you must give the full path name.
brings up a dialog in which you can
file inThe file button
select the program
teractively. You can also get to this dialog by pressing Shift-Return or Alt-Return ,
when the focus is in the text field.
Variable: Yes
34.7.
Processes
515
Restrictions: Must not be empty
Directory
Here you can set the working directory for the program. If you leave this value
empty, the program will inherit QF-Test’s working directory.
The directory button
brings up a dialog in which you can select the directory
in
teractively. You can also get to this dialog by pressing Shift-Return or Alt-Return ,
when the focus is in the text field.
Note
This directory will become the current working directory after executing the program. As a result, a script named, say, ./copy_data will be looked up relative
to QF-Test’s working directory and not the directory given. Only the path names
referred to in the script itself will be resolved relative to the new directory.
Variable: Yes
Restrictions: Must be empty or an existing directory
Executable parameters
There is one tab for each of the following parameter lists:
Executable
The command line arguments for the executable. Put each parameter on a
line of its own. Special quoting of whitespace or symbols is not required.
Extra
Extra
parameters
for
the
class
de.qfs.apps.qftest.start.StartClient which is used to gain
access to the EventQueue, establish an RMI connection between the SUT
and QF-Test and then start the main class of the SUT. These parameters are
intended for future extensions, scripting and internal logging.
(16)
See section 2.2.5
about how to work with the tables.
Variable: Yes
Restrictions: None
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
34.7.
Processes
516
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
lets
are much better suited to this task. The option External editor command
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.7.3 Start process
To run an arbitrary program during a test, you can either use this node or an
(524)
Execute shell command
node. This node is preferable if you need to pass
possibly complex arguments to the executable.
(398)
Contained in: All kinds of sequences
.
Children: None
Execution: The command line for the program is built from the attributes and the process is started. Its input and output are redirected to QF-Test.
Attributes:
34.7.
Processes
517
Figure 34.40: Start process attributes
Client
This is the identifier for the SUT client. It must remain unique as long as the
process is alive. Other nodes will refer to the client by this name.
Variable: Yes
Restrictions: Must not be empty
Executable
The program to run. If the executable file is not located in a directory on the PATH
you must give the full path name.
The file button
brings up a dialog in which you can
file in select the program
teractively. You can also get to this dialog by pressing Shift-Return or Alt-Return ,
when the focus is in the text field.
Variable: Yes
Restrictions: Must not be empty
Directory
34.7.
Processes
518
Here you can set the working directory for the program here. If you leave this
value empty, the program will inherit QF-Test’s working directory.
The directory button
brings up a dialog in which you can select the directory
in
teractively. You can also get to this dialog by pressing Shift-Return or Alt-Return ,
when the focus is in the text field.
Note
This directory will become the working directory of the newly started process. It
does not affect the working directory of QF-Test. As a result, a program named,
say, ./startserver will be looked up relative to QF-Test’s working directory and
not the directory given. Only the path names referred to in the program itself will
be resolved relative to the given directory.
Variable: Yes
Restrictions: Must be empty or an existing directory
Executable parameters
The command line arguments for the executable. Put each parameter on a line of
its own. Special quoting of whitespace or symbols is not required.
(16)
See section 2.2.5
about how to work with the tables.
Variable: Yes
Restrictions: None
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
are much better suited to this task. The option External editor command
lets
34.7.
Processes
519
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.7.4 Start web engine
This node is used to start a web browser specifically for web testing. To
(511)
test a Java applet in a browser, use a Start SUT client node instead. If the
process for the SUT is already running, this node can also be used to open
an additional browser window in the same process.
(398)
Contained in: All kinds of sequences
.
Children: None
Execution: The command line for the browser program is built from the attributes and
the process is started. Its input and output are redirected to QF-Test.
Attributes:
34.7.
Processes
520
Figure 34.41: Start web engine attributes
Client
This is the identifier for the SUT client. It must remain unique as long as the
process is alive. Other nodes will refer to the client by this name.
Variable: Yes
Restrictions: Must not be empty
Browser type
The kind of browser to start. Officially supported values are currently ”ie” for
Internet Explorer, ”firefox” (or ”mozilla”) for Mozilla Firefox, ”chrome” for Google
Chrome, ”edge” for Microsoft Edge, ”safari” for Apple Safari, ”headless-firefox” for
34.7.
Processes
521
Mozilla Firefox without visible window, and ”headless-chrome” (or ”headless”) for
Google Chrome without visible window.
Variable: Yes
Restrictions: Legal values are ”ie”, ”firefox” (or ”mozilla”), ”chrome”, ”edge”, ”safari”, ”headless-firefox”, and ”headless-chrome” (or ”headless”)”
Directory of browser installation
For most browsers this field can be left empty. For Firefox with QF-Driver
connection mode the installation directory of the browser must be specified here.
When connecting via WebDriver, the default installation of the given browser is
used. If several versions of that browser can be installed in parallel, a specific
version may be chosen be specifying its installation directory here.
Variable: Yes
Restrictions: Must be empty or an existing directory.
Browser connection mode
Since version 4.1 QF-Test can connect to a browser in two different ways: By
embedding the browser in a Java VM (the traditional QF-Driver mode) or via
Selenium WebDriver. Further information about these can be found in chapter
(148)
14 . For most browsers only one variant is supported, but for Chrome on
Windows or Firefox up to version 43 on Windows and Linux both modes are
possible. This attribute determines what to do in such cases. Possible values are:
Prefer QF-Driver
Prefer QF-Driver mode if possible, otherwise use WebDriver. This is the
standard, used also if this attribute is left empty.
QF-Driver only
Exclusively use QF-Driver mode.
Prefer WebDriver
Prefer WebDriver mode if possible, otherwise use QF-Driver.
WebDriver only
Exclusively use WebDriver mode.
Variable: Yes
Restrictions: Must be empty or one of the values ’Prefer QF-Driver’, ’QF-Driver
only’, ’Prefer WebDriver’ or ’WebDriver only’
34.7.
Processes
522
Executable
The Java program to start the browser with, typically java or javaw on
Windows. If you want to run a specific Java virtual machine you should give the
full path name of the executable.
Note
If the web application does include a Java Applet, make sure that you specify the
System JRE here that is used for the Applet plugin, otherwise the browser will
crash.
The default value is ${qftest:java} which is the Java program that QF-Test itself
runs on. When using QF-Driver (only Windows/Linux) please ensure that the Java
program used here is of the same 32bit or 64bit variant as the Firefox browser
specified in the Directory of browser installation attribute.
brings up a dialog in which you can
file inThe file button
select the program
teractively. You can also get to this dialog by pressing Shift-Return or Alt-Return ,
when the focus is in the text field.
Variable: Yes
Restrictions: Must not be empty
Executable parameters
There is one tab for each of the following parameter lists:
Java VM
Special command line arguments for the java executable in which the
browser is embedded. Can be used to specify the available memory (default
is 200MB), define system properties, enable debugging, etc. Put each
parameter on a line of its own. Special quoting of whitespace or symbols is
not required.
Extra
Extra parameters for the QF-Test wrapper that embeds the browser and
establishes an RMI connection between the SUT and QF-Test. These
parameters are intended for future extensions, scripting and internal logging.
(16)
See section 2.2.5
about how to work with the tables.
Variable: Yes
Restrictions: None
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
34.7.
Processes
523
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
lets
are much better suited to this task. The option External editor command
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.7.5 Open browser window
This node opens a particular web page in a new browser window in a running
web engine process. You should start that process via a Start web engine node.
(398)
Contained in: All kinds of sequences
.
Children: None
Execution: Open a given web-page in the already running web engine process.
Attributes:
34.7.
Processes
524
Figure 34.42: Open browser window attributes
Client
This is the identifier for the SUT client. It must remain unique as long as the
process is alive. Other nodes will refer to the client by this name.
Variable: Yes
Restrictions: Must not be empty
URL
The URL of the web page to be shown in the browser. As a special case the URL
”about:nowindow” can be used to start the SUT process without opening an initial
browser window. In that state, cache or preference settings for the browser can
be modified which may be a prerequisite for opening a given URL. The browser
window with the actual target URL can then be opened via another Open browser
window node.
Variable: Yes
Restrictions: Must not be empty
34.7.
Processes
525
Name of the browser window
This attribute can be ignored unless you need to test a web application with
multiple open browser windows holding similar documents. In that case the Name
of the browser window attribute can be used to identify the browser window. Here
you can specify the name for the browser window to be opened. You find a brief
description how to handle multiple browser windows in FAQ 24.
Variable: Yes
Restrictions: None
Geometry of browser window
These optional attributes for the X/Y coordinate, width and height can be used to
specify the geometry of the browser window to open.
Variable: Yes
Restrictions: Width and height must not be negative.
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
lets
are much better suited to this task. The option External editor command
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
34.7.
Processes
526
Restrictions: None
34.7.6 Execute shell command
This node is a convenient way to execute shell commands during a test. The
shell that will execute the command can be specified with the command line
(689)
(689)
and -shellarg <argument>
arguments -shell <executable>
for QF-Test. The default for Unix is /bin/sh, on Windows either command.com or
cmd.exe is used, the value of the COMSPEC environment variable.
When the shell is started it is treated like every other process started by QF-Test, so
you can kill it or wait for it to terminate and check its exit code.
(398)
Contained in: All kinds of sequences
.
Children: None
Execution: A shell is started to execute the command. Its input and output are redirected to QF-Test.
Attributes:
Figure 34.43: Execute shell command attributes
Client
This is the identifier for the SUT client. It must remain unique as long as the
process is alive. Other nodes will refer to the client by this name.
34.7.
Processes
527
Variable: Yes
Restrictions: Must not be empty
Shell command
The command to execute in the shell.
command prompt.
Windows
Enter this just as you would at the
On Windows systems, quoting of arguments with blanks can be a little tricky. If
you’re using the standard Windows shell, simply use double quotes as always, for
example dir ”C:\Program Files”. If you’re using a Unix shell on Windows
(689)
by specifying the -shell <executable> command line argument, use single
quotes instead, i.e. ls ’C:/Program Files’.
Variable: Yes
Restrictions: Must not be empty
Directory
Here you can set the working directory for the shell. If you leave this value empty,
the shell will inherit QF-Test’s working directory.
brings up a dialog in which you can select the directory
inThe directory button
teractively. You can also get to this dialog by pressing Shift-Return or Alt-Return ,
when the focus is in the text field.
Variable: Yes
Restrictions: Must be empty or an existing directory
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
34.7.
Note
Processes
528
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
lets
are much better suited to this task. The option External editor command
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.7.7 Start PDF client
Use this node to load the PDF document to be tested into a viewer, which QF-Test will
(152)
start as a client process. Please refer to chapter 15 for information about the testing
of PDF documents.
(398)
Contained in: All kinds of sequences
.
Children: None
Execution: The viewer is started and the PDF document loaded.
Attributes:
34.7.
Processes
529
Figure 34.44: Start PDF client attributes
Client
This is the identifier for the SUT client. It must remain unique as long as the
process is alive. Other nodes will refer to the client by this name.
Variable: Yes
Restrictions: Must not be empty
PDF document
The PDF document to be opened. If a relative path is specified it is resolved
relative to the directory containing the current test-suite.
The file button
brings up a dialog in which you can
file in select the program
teractively. You can also get to this dialog by pressing Shift-Return or Alt-Return ,
when the focus is in the text field.
Variable: Yes
Restrictions: Must not be empty
Page of PDF document
The page to show.
34.7.
Processes
530
An integer number is interpreted as page number. A string in quotation marks is
interpreted as page name.
Example:
5 opens page number 5
”Index IV” opens the page with the name Index IV
Variable: Yes
Restrictions: Must be a integer number or valid page name in quotation marks.
Password
If the document requires a password, it can be specified here.
Note
When using a password it may be desirable to avoid having the password show
up as plain text in the test-suite or a run-log. To that end the password can be
encrypted by inserting the plain-text password, right-clicking and selecting
Crypt password from the popup menu. Please be sure to specify a password
(351)
salt before encrypting via the option Salt for crypting passwords .
Variable: Yes
Restrictions: Empty or a valid password, if required only.
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
are much better suited to this task. The option External editor command
lets
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
34.7.
Processes
531
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.7.8 Wait for client to connect
This node is used to make sure that the connection to a client is established. If after a configurable timeout the client is still not connected, a
(666)
ClientNotConnectedException is thrown. You can also use the Variable for result attribute to store the result into a variable and the Throw exception on failure
attribute to suppress the exception.
(398)
Contained in: All kinds of sequences
.
Children: None
Execution: QF-Test waits until either the connection to the Java client is established or
the timeout is up.
Attributes:
34.7.
Processes
532
Figure 34.45: Wait for client to connect attributes
Client
The name of the Java client to wait for.
Variable: Yes
Restrictions: Must not be empty
Timeout
Time limit in milliseconds.
Variable: Yes
Restrictions: >= 0
GUI engine
The GUI engine to wait for. Only relevant for SUTs with more than one GUI
(697)
engine as described in chapter 37 .
34.7.
Processes
533
Variable: Yes
(697)
Restrictions: See chapter 37
Variable for result
This optional attribute determines the name for the result variable of the action. If
set, the respective variable will be set to ’true’ for a successful check or wait and
to ’false’ in case of failure.
Note
If this attribute is set, the attribute Error level of message is ignored and no error is
reported. The attribute Throw exception on failure always remains effective, so it is
possible to set a result variable and still throw an exception.
Variable: Yes
Restrictions: None
Local variable
This flag determines whether to create a local or global variable binding. If unset,
the variable is bound in the global variables. If set, the topmost current binding for
the variable is replaced with the new value, provided this binding is within the
(463)
(427)
(398)
context of the currently executing Procedure , Dependency or Test-case node.
If no such binding exists, a new binding is created in the currently executing
Procedure, Dependency or Test-case node or, if there is no such node, in the global
(62)
bindings. See chapter 7 for a detailed explanation of variable binding and
lookup.
Variable: No
Restrictions: None
Error level of message
This attribute determines the error level of the message that is logged in case of
failure. Possible choices are message, warning and error.
Note
If the attribute Throw exception on failure is set, this attribute is irrelevant and if
Variable for result is set this attribute is ignored.
Variable: No
Restrictions: None
Throw exception on failure
Throw an exception in case of failure.
For ’Check...’
nodes a
(665)
CheckFailedException
is thrown, for ’Wait for...’ nodes the respective
specific exception.
Variable: No
Restrictions: None
34.7.
Processes
534
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
lets
are much better suited to this task. The option External editor command
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.7.9 Stop client
This node forcibly terminates a client process started by QF-Test. If the process has already terminated, nothing is changed. Otherwise, if the client is
an SUT client, QF-Test first tries to call the Java method System.exit(-1)
in the SUT to achieve a clean shutdown. If the client is not an SUT client or the process
is still alive after the exit call, the process is killed and its exit code is set to -1.
Windows
Note: On Windows child processes started by a process are not terminated when the
(699)
parent process is killed. As explained in chapter 38 , the java program for the SUT
is not started directly by QF-Test, but through an intermediate program. This means
that when the System.exit(-1) call fails to terminate the SUT for some reason, the
process for the SUT’s Java VM will be left hanging around.
34.7.
Processes
535
(398)
Contained in: All kinds of sequences
.
Children: None
Execution: Terminates the last process that was started under the given name.
Attributes:
Figure 34.46: Stop client attributes
Client
The client name of the process that is to be killed.
Variable: Yes
Restrictions: Must not be empty
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
Comment
34.7.
Processes
536
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
are much better suited to this task. The option External editor command
lets
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.7.10 Wait for process to terminate
This node waits until a process that was started by QF-Test terminates. If
a given time limit is exceeded and the process is still alive. Per default a
(667)
ClientNotTerminatedException is thrown. Otherwise the client’s exit
code is read and tested against a given value. You can also use the Variable for result
attribute to store the result into a variable and the Throw exception on failure attribute to
suppress the exception.
(398)
Contained in: All kinds of sequences
.
Children: None
Execution: Waits for the end of a process and checks its exit value unless the timeout
is exceeded.
Attributes:
34.7.
Processes
537
Figure 34.47: Wait for process to terminate attributes
Client
The client name of the process to wait for.
Variable: Yes
Restrictions: Must not be empty
Timeout
Time limit in milliseconds.
Variable: Yes
Restrictions: >= 0
Expected exit code
If this attribute is set it is used to validate the exit code of the process. The type of
the comparison is defined by prepending one of the four operators ==, !=, < and
34.7.
Processes
538
>. Just a number without preceding operation test for equality. If the test fails, an
(667)
UnexpectedExitCodeException
is thrown.
The exception won’t be
thrown, if the attribute Throw exception on failure is not set.
Examples: If this attribute is set to 0, every exit code that is not 0 causes an
exception. This is the same as for ==0. A value of >0 causes an exception for
every exit code equal to or less than 0. This exception can be suppressed by
un-checking the attribute Throw exception on failure.
Variable: Yes
Restrictions: See above
Variable for result
This optional attribute determines the name for the result variable of the action. If
set, the respective variable will be set to ’true’ for a successful check or wait and
to ’false’ in case of failure.
Note
If this attribute is set, the attribute Error level of message is ignored and no error is
reported. The attribute Throw exception on failure always remains effective, so it is
possible to set a result variable and still throw an exception.
Variable: Yes
Restrictions: None
Local variable
This flag determines whether to create a local or global variable binding. If unset,
the variable is bound in the global variables. If set, the topmost current binding for
the variable is replaced with the new value, provided this binding is within the
(463)
(427)
(398)
context of the currently executing Procedure , Dependency or Test-case node.
If no such binding exists, a new binding is created in the currently executing
Procedure, Dependency or Test-case node or, if there is no such node, in the global
(62)
bindings. See chapter 7 for a detailed explanation of variable binding and
lookup.
Variable: No
Restrictions: None
Error level of message
This attribute determines the error level of the message that is logged in case of
failure. Possible choices are message, warning and error.
Note
If the attribute Throw exception on failure is set, this attribute is irrelevant and if
Variable for result is set this attribute is ignored.
Variable: No
Restrictions: None
34.8.
Events
539
Throw exception on failure
Throw an exception in case of failure.
For ’Check...’
nodes a
(665)
CheckFailedException
is thrown, for ’Wait for...’ nodes the respective
specific exception.
Variable: No
Restrictions: None
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
lets
are much better suited to this task. The option External editor command
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.8
Events
This section lists all kinds of nodes that trigger actions in the SUT. Besides true Java
events these include pseudo events with special behavior.
34.8.
Events
540
34.8.1 Mouse event
Mouse events simulate mouse movement and clicks as well as drag and drop
operations.
(398)
Contained in: All kinds of sequences
.
Children: None
Execution: The coordinates and other attributes of the event are sent to the SUT together with the data about the target component. The TestEventQueue determines
the corresponding component in the SUT, adjusts the coordinates and triggers the resulting event.
Attributes:
Figure 34.48: Mouse event attributes
Client
The name of the SUT client process to which the event is sent.
34.8.
Events
541
Variable: Yes
Restrictions: Must not be empty.
QF-Test component ID
(635)
(634)
The QF-Test ID
of the Window
target of the event.
, Component
(644)
(650)
or Item
node that is the
The
button brings up a dialog in which you can select the component
inter
actively. You can also get to this dialog by pressing Shift-Return or Alt-Return ,
when the focus is in the text field. As an alternative you can copy the target node
with Ctrl-C or Edit→Copy and insert its QF-Test component ID into the text field
by pressing Ctrl-V .
(650)
By using a special syntax you can target Items
directly or access components
(57)
(193)
from another suite (see section 6.3 and section 19.1 for details).
Variable: Yes
Restrictions: Must not be empty.
Event
This ComboBox lets you choose the type of the event.
MOUSE_MOVED,
MOUSE_PRESSED, MOUSE_RELEASED, MOUSE_CLICKED and MOUSE_DRAGGED
are the standard event IDs of the Java class MouseEvent.
The abstract ’Mouse click’ event is a compound of the events MOUSE_MOVED,
MOUSE_PRESSED, MOUSE_RELEASED and MOUSE_CLICKED. During replay the
pseudo event is simulated as four separate events. This adds to the clarity of
a test-suite and simplifies editing.
The special ’Double click’ event comprises all the individual events required to
simulate a complete double click.
4.1.3+
The events MOUSE_DRAG_FROM, MOUSE_DRAG_OVER and MOUSE_DROP_TO are
(717)
used to simulate Drag&Drop in the SUT. See section 40.1 for details.
Variable: No
Restrictions: None
X/Y
These are the coordinates of the MouseEvent. They are relative to the upper left
(634)
(644)
(650)
or Item
that is the target of the event.
corner of the Window , Component
They can be negative, e.g. to simulate a click on the expansion toggle of a node
in a JTree.
34.8.
Events
542
Most of the time the exact coordinates for a mouse don’t really matter, any place
within the target will do. In this case you should leave the X and Y values empty to
tell QF-Test to aim at the center of the target. Where possible QF-Test will leave the
values empty when recording, provided the option Record MouseEvents without
(335)
is active.
coordinates where possible
Variable: Yes
Restrictions: Valid number or empty
Modifiers
This value reflects the state of the mouse buttons and the modifier keys Shift ,
Control , Alt and Meta during a mouse or key event. States are combined by
adding up their values.
Value
1
2
4
8
16
Key/Button
Shift
Control
Meta or right mouse button
Alt or middle mouse button
Left mouse button
Table 34.16: Modifier values
Variable: Yes
Restrictions: Valid number
Click count
This value lets a Java program distinguish between a single and a double (or
even multiple) click.
Variable: Yes
Restrictions: Valid number
Popup trigger
If this attribute is set, the event can trigger a PopupMenu. This is Java’s
somewhat peculiar way of supporting different conventions for triggering
PopupMenus on different platforms.
Variable: Yes
Restrictions: None
34.8.
Events
543
Replay as ”hard” event
If this attribute is set the event is replayed as a hard event, meaning it is triggered
as a real system event that moves the mouse around and not just inserted as soft
event into the event queue. Soft events are are typically better because they
avoid impact of concurrent user mouse actions and are less likely to break due to
interference from an overlapping window. Nevertheless there are certain special
situations where hard events are helpful.
Variable: Yes
Restrictions: None
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
lets
are much better suited to this task. The option External editor command
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.8.2 Key event
34.8.
Events
544
Key events simulate keyboard input in the SUT. These are used mainly for
(545)
control and function keys. Input of text is better represented as a Text input .
The special keyboard event InputMethodEvent supports international
character codes. QF-Test doesn’t support InputMethodEvents directly. Instead, it
converts events of type INPUT_METHOD_TEXT_CHANGED into Key events of type
KEY_TYPED.
(398)
Contained in: All kinds of sequences
.
Children: None
Execution: The key codes of the event are sent to the SUT together with the data
about the target component. The TestEventQueue determines the corresponding
component in the SUT and triggers the resulting event.
Attributes:
Figure 34.49: Key event attributes
Client
The name of the SUT client process to which the event is sent.
Variable: Yes
Restrictions: Must not be empty.
34.8.
Events
545
QF-Test component ID
(635)
(634)
The QF-Test ID
of the Window
target of the event.
, Component
(644)
(650)
or Item
node that is the
button brings up a dialog in which you can select the component
interThe
actively. You can also get to this dialog by pressing Shift-Return or Alt-Return ,
when the focus is in the text field. As an alternative you can copy the target node
with Ctrl-C or Edit→Copy and insert its QF-Test component ID into the text field
by pressing Ctrl-V .
(650)
directly or access components
By using a special syntax you can target Items
(57)
(193)
from another suite (see section 6.3 and section 19.1 for details).
Variable: Yes
Restrictions: Must not be empty.
Event
This ComboBox lets you choose the type of the event.
KEY_PRESSED,
KEY_TYPED and KEY_RELEASED are the standard event IDs of the Java class
KeyEvent.
The ’Keystroke’ pseudo event is a compound of the events KEY_PRESSED,
KEY_TYPED and KEY_RELEASED. During replay the pseudo event is simulated as
two or three separate events, depending on whether it is a printable character key,
or a control or function key. In the latter case, no KEY_TYPED event is generated.
Variable: No
Restrictions: None
Key
(543)
(543)
(544)
This is a convenience method to set Key code , Key char
and Modifiers
directly by pressing the corresponding key while this component has the
(543)
keyboard focus. For KEY_TYPED events Key char is set to 0.
Unfortunately you can’t select the Tab key this
way since it is used for keyboard
(543)
(543)
traversal. Key code and Key char for the Tab key are both 9.
Variable: No
Restrictions: None
Key code
This is a Java specific code for the key, the keyCode member of the Java class
KeyEvent.
Variable: Yes
34.8.
Events
546
Restrictions: Valid number
Key char
This is the keyChar member of the Java class KeyEvent. Its value is
the
character generated created by the last key press, taking the state of the Shift key into account. Control and function keys always have a Key char value of
65535.
Variable: Yes
Restrictions: Valid number
Modifiers
This
value reflects the state of the mouse buttons and the modifier keys Shift ,
Control , Alt and Meta during a mouse or key event. States are combined by
adding up their values.
Value
1
2
4
8
16
Key/Button
Shift
Control
Meta or right mouse button
Alt or middle mouse button
Left mouse button
Table 34.17: Modifier values
Variable: Yes
Restrictions: Valid number
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
34.8.
Events
547
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
lets
are much better suited to this task. The option External editor command
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.8.3 Text input
This is a pseudo event for simulating text input from the keyboard. A single
(541)
Text input node replaces a whole sequence of Key event nodes. To achieve
this, QF-Test takes advantage of the fact that AWT and Swing text fields listen only to KEY_TYPED events, so input of text can be simulated by triggering one
KEY_TYPED event per text character. If your SUT uses custom text components that actually require the KEY_PRESSED and KEY_RELEASED events, you cannot use this node
(541)
type but have to resort to plain Key events .
(398)
Contained in: All kinds of sequences
.
Children: None
Execution: The text is sent to the SUT together with the data about the target component. The TestEventQueue determines the corresponding component in the SUT and
triggers the resulting events.
Attributes:
34.8.
Events
548
Figure 34.50: Text input attributes
Client
The name of the SUT client process to which the event is sent.
Variable: Yes
Restrictions: Must not be empty.
QF-Test component ID
(635)
(634)
The QF-Test ID
of the Window
target of the event.
, Component
(644)
(650)
or Item
node that is the
button brings up a dialog in which you can select the component
interThe
actively. You can also get to this dialog by pressing Shift-Return or Alt-Return ,
when the focus is in the text field. As an alternative you can copy the target node
with Ctrl-C or Edit→Copy and insert its QF-Test component ID into the text field
by pressing Ctrl-V .
(650)
By using a special syntax you can target Items
directly or access components
(57)
(193)
from another suite (see section 6.3 and section 19.1 for details).
Variable: Yes
Restrictions: Must not be empty.
34.8.
Events
549
Text
The text that is to be sent to the SUT.
Note
When inserting text into a password field it may be desirable to avoid having the
password show up as plain text in the test-suite or a run-log. To that end the
password can be encrypted by inserting the plain-text password, right-clicking and
selecting Crypt password from the popup menu. Please be sure to specify a
(351)
password salt before encrypting via the option Salt for crypting passwords .
Variable: Yes
Restrictions: No line breaks are possible.
Clear target component first
If this attribute is set, and the target component is a text field or text area, the
contents of the target component are removed before the new text is inserted.
Variable: Yes
Restrictions: None
Replay single events
If the target component is a text field or a text area, the text can optionally be
inserted by manipulating the component directly through its API. This is much
faster, but if KeyListeners are registered on the component they will not be
notified of the change. If this attribute is set, key events are simulated for every
single character.
Variable: Yes
Restrictions: None
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
34.8.
Events
550
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
lets
are much better suited to this task. The option External editor command
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.8.4 Window event
WindowEvents are of limited use for a test-suite since most of them are
not generated as a direct consequence of some user interaction. The only
exception is the WINDOW_CLOSING event that is triggered when the user
closes a window.
It is also possible to simulate WINDOW_ICONIFIED and
WINDOW_DEICONIFIED events.
(398)
Contained in: All kinds of sequences
.
Children: None
Execution: The event is sent to the SUT together with the data about the target window.
The TestEventQueue determines the corresponding window in the SUT and triggers
the resulting event.
Attributes:
34.8.
Events
551
Figure 34.51: Window event attributes
Client
The name of the SUT client process to which the event is sent.
Variable: Yes
Restrictions: Must not be empty.
QF-Test component ID
(635)
(634)
The QF-Test ID
of the Window
target of the event.
, Component
(644)
(650)
or Item
node that is the
button brings up a dialog in which you can select the component
interThe
actively. You can also get to this dialog by pressing Shift-Return or Alt-Return ,
when the focus is in the text field. As an alternative you can copy the target node
with Ctrl-C or Edit→Copy and insert its QF-Test component ID into the text field
by pressing Ctrl-V .
(650)
By using a special syntax you can target Items
directly or access components
(57)
(193)
from another suite (see section 6.3 and section 19.1 for details).
Variable: Yes
Restrictions: Must not be empty.
34.8.
Events
552
Event
This ComboBox lets you choose the type of the event. Possible values are
WINDOW_OPENED, WINDOW_CLOSING, WINDOW_CLOSED, WINDOW_ACTIVATED,
WINDOW_DEACTIVATED, WINDOW_ICONIFIED and WINDOW_DEICONIFIED, the
standard event IDs of the Java class WindowEvent.
Variable: No
Restrictions: None
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
lets
are much better suited to this task. The option External editor command
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.8.5 Component event
34.8.
Events
553
The name Component event for this node may be misleading, since QF-Test
filters all of these events except for windows, but since they represent the
Java class ComponentEvent, it is better to stick to that name. Except for
COMPONENT_MOVED and COMPONENT_SIZED events on a window, which are the result of the user moving or resizing the window interactively, all ComponentEvents are
artificial and thus ignored.
(398)
Contained in: All kinds of sequences
.
Children: None
Execution: The event is sent to the SUT together with the data about the target window.
The TestEventQueue determines the corresponding window in the SUT and triggers
the resulting event.
Attributes:
Figure 34.52: Component event attributes
Client
The name of the SUT client process to which the event is sent.
Variable: Yes
Restrictions: Must not be empty.
QF-Test component ID
34.8.
Events
554
(635)
(634)
The QF-Test ID
of the Window
target of the event.
, Component
(644)
(650)
or Item
node that is the
The
button brings up a dialog in which you can select the component
inter
actively. You can also get to this dialog by pressing Shift-Return or Alt-Return ,
when the focus is in the text field. As an alternative you can copy the target node
with Ctrl-C or Edit→Copy and insert its QF-Test component ID into the text field
by pressing Ctrl-V .
(650)
directly or access components
By using a special syntax you can target Items
(57)
(193)
from another suite (see section 6.3 and section 19.1 for details).
Variable: Yes
Restrictions: Must not be empty.
Event
This ComboBox lets you choose the type of the event. Possible values are
COMPONENT_SIZED and COMPONENT_MOVED, the standard event IDs of the Java
class ComponentEvent.
Variable: No
Restrictions: None
Y/Height
For a COMPONENT_MOVED event set this to the new Y-coordinate of the window,
for a COMPONENT_SIZED event to its new height.
Variable: Yes
Restrictions: Valid number, height > 0
X/Width
For a COMPONENT_MOVED event set this to the new X-coordinate of the window,
for a COMPONENT_SIZED event to its new width.
Variable: Yes
Restrictions: Valid number, width > 0
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
34.8.
Events
555
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
lets
are much better suited to this task. The option External editor command
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.8.6 Selection
A Selection node represents an abstract event like selecting a menu item,
choosing an entry in a combo box or selecting something from or closing a
system dialog. Currently this event node is used only for SWT and web SUT
clients, where some events cannot be triggered by ”soft” mouse events, . The
alternative of using ”hard” events has some disadvantages as described for the
(540)
(538)
Replay as ”hard” event attribute of a Mouse event .
(556)
The Detail attribute determines the kind of operation to perform, or the value to select,
depending on the target component.
SWT
The following combinations of widgets and Detail values are currently supported for
SWT:
34.8.
Events
556
Class
CCombo sub-item
ColorDialog
ColorDialog
Combo sub-item
CTabFolder sub-item
CTabFolder sub-item
DirectoryDialog
DirectoryDialog
FileDialog
FileDialog
FontDialog
FontDialog
Menu
MenuItem
MessageBox
ProgressBar
Scale
Slider
Spinner
TabFolder sub-item
Detail attribute
Empty
Color value in hexadecimal
#rrggbb format
CANCEL
Empty
Empty
close
Directory name
CANCEL
File name, including directory
CANCEL
Font description, system specific
CANCEL
close
Empty
Any of OK YES NO CANCEL
ABORT RETRY IGNORE OPEN
SAVE
ProgressBar value
Scale value
Slider value
Spinner value
Empty
Action
Choose item as current value
Select given color
Abort color selection
Choose item as current value
Select tab
Close tab
Select given directory
Abort directory selection
Select given file
Abort file selection
Select given font
Abort file selection
Cancel the menu (close without
selection)
Select item
Close with the given value as
the user’s choice
Set the given value
Set the given value
Set the given value
Set the given value
Select tab
Table 34.18: Supported SWT widgets for a Selection event
Note
Eclipse/RCP makes heavy use of dynamic CTabFolders where the items
can be moved between folders.
The items represent the actual business
objects whereas the folders are just scaffolding to hold them.
Besides,
the layout of the folders and items can change drastically when switching
perspectives. Thus it is often desirable to be able to select or close an item
independent of the CTabFolder it currently resides in. This can be done by
(463)
using the Procedures
qfs.qft#qfs.swt.ctabfolder.selectTab and
qfs.qft#qfs.swt.ctabfolder.closeTab, provided in the qfs.qft standard
library. Besides the ubiquitous client parameter, the only other parameter tabname
must be set to the name of the item to select or close.
Web
The following combinations of DOM nodes and Detail values are currently supported for
web SUT clients:
34.8.
Events
557
Node type
Confirmation dialog
File dialog for download
Detail attribute
Any of OK YES NO CANCEL
RETRY
The file to save to or CANCEL
Login dialog
Username|Password
CANCEL
Prompt dialog
The text to enter or CANCEL
Top-level DOCUMENT
back
Top-level DOCUMENT
forward
Top-level DOCUMENT
Top-level DOCUMENT
Top-level DOCUMENT
OPTION or SELECT sub-item
goto:URL
refresh
stop
0
OPTION or SELECT sub-item
1
OPTION or SELECT sub-item
-1
or
Action
Close the dialog with the given
value as the user’s choice
Close the dialog and save to the
specified file or abort the download
Close the dialog and login with
the specified data or abort.
The password can by encrypted
by right-clicking the Detail field
and selecting Crypt password .
Please be sure to specify a
password salt before encrypting
via the option Salt for crypting
(351)
passwords .
Close the dialog and return the
specified text or abort
Navigate back to the previous
page
Navigate forward to the next
page
Navigate to the specified URL
Reload the current page
Stop loading the current page
Choose the OPTION as current
value
Add the OPTION to the selection
Remove the OPTION from the
selection
Table 34.19: Supported DOM nodes for a Selection event
(398)
Contained in: All kinds of sequences
.
Children: None
Execution: The event is sent to the SUT together with the data about the target component. The component is resolved and an action performed which depends on the type
of component as listed in the table above.
Attributes:
34.8.
Events
558
Figure 34.53: Selection attributes
Client
The name of the SUT client process to which the event is sent.
Variable: Yes
Restrictions: Must not be empty.
QF-Test component ID
(635)
(634)
of the Window
The QF-Test ID
target of the event.
, Component
(644)
(650)
or Item
node that is the
The
button brings up a dialog in which you can select the component
inter
actively. You can also get to this dialog by pressing Shift-Return or Alt-Return ,
when the focus is in the text field. As an alternative you can copy the target node
with Ctrl-C or Edit→Copy and insert its QF-Test component ID into the text field
by pressing Ctrl-V .
(650)
By using a special syntax you can target Items
directly or access components
(57)
(193)
from another suite (see section 6.3 and section 19.1 for details).
Variable: Yes
Restrictions: Must not be empty.
Detail
34.8.
Events
559
The Detail attribute specifies the kind of operation to perform or the value to
select, depending on the target component. The possible combinations are listed
in detail above.
Variable: Yes
Restrictions: None
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
lets
are much better suited to this task. The option External editor command
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.8.7 File selection
A File selection is a pseudo event that you only need in a special case.
If your SUT uses the standard AWT file selection dialog, implemented by the
class java.awt.FileDialog, QF-Test has no chance to record the events
the user generates while selecting a file, since they are all handled by the underlying
34.8.
Events
560
system and never passed on to Java. For the same reason, the selection cannot be
simulated as a sequence of mouse and key events. This is not the case with Swing’s
javax.swing.JFileChooser which is implemented as a normal dialog.
Therefore QF-Test just records the result of the file selection in the form of this node and
stores that data in the FileDialog upon replay before closing it. For the SUT there is no
difference to an actual selection by the user.
(398)
Contained in: All kinds of sequences
.
Children: None
Execution: File and directory are stored in an open java.awt.FileDialog and
(662)
the dialog is closed.If no FileDialog is open, a ComponentNotFoundException
is thrown.
Attributes:
Figure 34.54: File selection attributes
Client
The name of the SUT client process to which the event is sent.
Variable: Yes
Restrictions: Must not be empty.
34.8.
Events
561
File
The name of the file (without the directory part) that is to be selected.
Variable: Yes
Restrictions: Must not be empty
Directory
The directory of the file that is to be selected.
Variable: Yes
Restrictions: Must not be empty
GUI engine
The GUI engine in which to look for a file selection dialog. Only relevant for SUTs
(697)
with more than one GUI engine as described in chapter 37 .
Variable: Yes
(697)
Restrictions: See chapter 37
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
are much better suited to this task. The option External editor command
lets
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
34.9.
Checks
562
Variable: Yes
Restrictions: None
34.9
Checks
Checks are the means by which QF-Test validates correctness of the SUT. If the current
state of a component doesn’t match the expected value of the check node a message
with a configurable error level is written to the run-log. Additionally or alternatively a
(665)
CheckFailedException
can be thrown and the result of the check can be assigned to a variable.
(564)
checks can also be used to wait for a component to attain a
By setting a Timeout
certain state, e.g. to wait for a MenuItem to become enable or for a CheckBox to
become selected.
(537)
Like events each check must have a target window, component or sub-item. Depending on that target, different kinds of checks are supported which can be recorded in
(36)
check-mode as described in section 4.3 via right-clicking the target component and
selecting the check from the resulting menu. In case you need a check for a special component that is not offered by default you can implement custom checks via the Checker
(856)
extension API described in section 43.5 .
There are six different data types available for checks and each of these corresponds to
a specific check node. Because it is possible to have different checks of the same data
type for the same target component, e.g. for the enabled state and the editable state of a
text field, both boolean, checks are further qualified by their Check type identifier attribute.
In most cases, when data type and target component are sufficient to uniquely identify
the kind of check, the value of this attribute is default. In case the specified check is
(666)
not supported for the given target component a CheckNotSupportedException
is thrown.
The following check nodes are available (if the respective component supports it):
(561)
• Check text
(566)
• Boolean check
(572)
• Check items
(576)
• Check selectable items
(581)
• Check image
(586)
• Check geometry
34.9.
Checks
563
34.9.1 Check text
Validates the text displayed in a component or sub-item.
(398)
Contained in: All kinds of sequences
.
Children: None
Execution: The text is sent to the SUT together with the data about the target component. The TestEventQueue determines the corresponding component in the SUT,
reads its displayed text and compares it to the required value.
Attributes:
34.9.
Checks
564
Figure 34.55: Check text attributes
Client
The name of the SUT client process to which the check is sent.
Variable: Yes
Restrictions: Must not be empty.
34.9.
Checks
565
QF-Test component ID
(635)
(634)
The QF-Test ID
of the Window
target of the check.
, Component
(644)
(650)
or Item
node that is the
button brings up a dialog in which you can select the component
interThe
actively. You can also get to this dialog by pressing Shift-Return or Alt-Return ,
when the focus is in the text field. As an alternative you can copy the target node
with Ctrl-C or Edit→Copy and insert its QF-Test component ID into the text field
by pressing Ctrl-V .
(650)
By using a special syntax you can target Items
directly or access components
(57)
(193)
from another suite (see section 6.3 and section 19.1 for details).
Variable: Yes
Restrictions: Must not be empty.
Text
The value to which the text displayed by the component or sub-item is compared.
4.0+
You can select Escape text for regular expressions from the context menu for escaping special characters of regular expressions of that text.
Variable: Yes
Restrictions: Valid regexp if required.
As regexp
If this attribute is set, the component’s text is matched against a regexp (see
(720)
section 40.4 ) instead of comparing plain strings.
Variable: Yes
Restrictions: None
Check type identifier
This attribute specifies the kind of check to perform. This makes it possible to
support different kinds of checks of the same data type for a given target
component without any ambiguity. With the help of a Checker additional check
(856)
types can be implemented as shown in section 43.5 . By default a Check text
node provides the following Check types (if the respective component supports it):
34.9.
Checks
Check
default
tooltip
text_positioned
text_font
text_fontsize
class
id
name
value
href
attribute:NAME
566
Description
Text of component
Tooltip of component. In some cases
you need to run a Mouse event step before that check to initialize the tooltip.
(155)
For details see PDF Check text .
The text font of the component. For de(157)
tails see PDF ’Check Font’ .
The text font size of the component. For
(157)
details see PDF ’Check Font size’ .
CSS class(es) of component
’id’ attribute of component
’name’ attribute of component
’value’ attribute of component
’href’ attribute of component
attribute named NAME of component
Engines
All
All
PDF only
PDF only
PDF only
Web only
Web only
Web only
Web only
Web only
Web only
Table 34.20: Provided Check types of Check text
Variable: Yes
Restrictions: Must not be empty
Negate
This flag determines whether to execute a positive or a negative check. If it is set,
a negative check will be performed, i.e. the checked property must not match the
expected value.
Variable: Yes
Restrictions: None
Timeout
Time limit in milliseconds until the check must succeed. To disable waiting, leave
this value empty or set it to 0.
Variable: Yes
Restrictions: Must not be negative.
Variable for result
This optional attribute determines the name for the result variable of the action. If
set, the respective variable will be set to ’true’ for a successful check or wait and
to ’false’ in case of failure.
Note
If this attribute is set, the attribute Error level of message is ignored and no error is
34.9.
Checks
567
reported. The attribute Throw exception on failure always remains effective, so it is
possible to set a result variable and still throw an exception.
Variable: Yes
Restrictions: None
Local variable
This flag determines whether to create a local or global variable binding. If unset,
the variable is bound in the global variables. If set, the topmost current binding for
the variable is replaced with the new value, provided this binding is within the
(463)
(427)
(398)
context of the currently executing Procedure , Dependency or Test-case node.
If no such binding exists, a new binding is created in the currently executing
Procedure, Dependency or Test-case node or, if there is no such node, in the global
(62)
bindings. See chapter 7 for a detailed explanation of variable binding and
lookup.
Variable: No
Restrictions: None
Error level of message
This attribute determines the error level of the message that is logged in case of
failure. Possible choices are message, warning and error.
Note
If the attribute Throw exception on failure is set, this attribute is irrelevant and if
Variable for result is set this attribute is ignored.
Variable: No
Restrictions: None
Throw exception on failure
Throw an exception in case of failure.
For ’Check...’
nodes a
(665)
CheckFailedException
is thrown, for ’Wait for...’ nodes the respective
specific exception.
Variable: No
Restrictions: None
Name
An optional name for the Check, mostly useful for the report.
Variable: Yes
Restrictions: None
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
34.9.
Checks
568
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
lets
are much better suited to this task. The option External editor command
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.9.2 Boolean check
Compares the current state of a component or sub-item with an expected
value.
(398)
Contained in: All kinds of sequences
.
Children: None
Execution: The expected state is sent to the SUT together with the data about the
target component. The TestEventQueue determines the corresponding component in
the SUT, determines its state and compares it to the required value.
Attributes:
34.9.
Checks
569
Figure 34.56: Boolean check attributes
Client
The name of the SUT client process to which the check is sent.
Variable: Yes
Restrictions: Must not be empty.
QF-Test component ID
(635)
(634)
The QF-Test ID
of the Window
target of the check.
, Component
(644)
(650)
or Item
node that is the
34.9.
Checks
570
The
button brings up a dialog in which you can select the component
inter
actively. You can also get to this dialog by pressing Shift-Return or Alt-Return ,
when the focus is in the text field. As an alternative you can copy the target node
with Ctrl-C or Edit→Copy and insert its QF-Test component ID into the text field
by pressing Ctrl-V .
(650)
directly or access components
By using a special syntax you can target Items
(57)
(193)
from another suite (see section 6.3 and section 19.1 for details).
Variable: Yes
Restrictions: Must not be empty.
Expected state
The value to which the current state of the component or sub-item is compared.
Variable: Yes
Restrictions: None
Check type identifier
This attribute specifies the kind of check to perform. This makes it possible to
support different kinds of checks of the same data type for a given target
component without any ambiguity. Boolean check nodes in particular are often
used to check different kinds of states like ’enabled’, ’editable’ or ’selected’ (like
(560)
described in section 34.9 ). The following table explains some of the check
types. It depends on the component class which check types are actually
available.
With the help of a Checker additional check types can be
(856)
implemented as shown in section 43.5 .
34.9.
Checks
571
Check
Example 1
Example 2
Details
visible
Example 1 shows a invisible Textfield and CheckBox..
Both are visible at Example
2.
editable
Example 1 shows a
Textfield which is not editable.
The Textfield at Example 2
is editable.
The CheckBox does not
support this kind of check.
enabled
Neither the Textfield nor the
CheckBox is enabled at Example 1, which means you
cannot interact with them.
Example 2 shows enabled
components.
checked (former selected)
Example 1 shows a CheckBox which is not selected.
It is selected at Example 2.
The Textfield does not support this kind of check.
selected (Table)
Example 1 shows a Table
where no cell is selected.
The lower left cell is selected at Example 2, indicated by a frame.
The check concerns the selection, not the CheckBox.
focused
Example 1 shows a focused Textfield (indicated
by the cursor).
The CheckBox is focused at
Example 2 (indicated by a
frame).
Table 34.21: Provided Check types of Boolean check
34.9.
Checks
572
Variable: Yes
Restrictions: Must not be empty
Timeout
Time limit in milliseconds until the check must succeed. To disable waiting, leave
this value empty or set it to 0.
Variable: Yes
Restrictions: Must not be negative.
Variable for result
This optional attribute determines the name for the result variable of the action. If
set, the respective variable will be set to ’true’ for a successful check or wait and
to ’false’ in case of failure.
Note
If this attribute is set, the attribute Error level of message is ignored and no error is
reported. The attribute Throw exception on failure always remains effective, so it is
possible to set a result variable and still throw an exception.
Variable: Yes
Restrictions: None
Local variable
This flag determines whether to create a local or global variable binding. If unset,
the variable is bound in the global variables. If set, the topmost current binding for
the variable is replaced with the new value, provided this binding is within the
(463)
(427)
(398)
context of the currently executing Procedure , Dependency or Test-case node.
If no such binding exists, a new binding is created in the currently executing
Procedure, Dependency or Test-case node or, if there is no such node, in the global
(62)
bindings. See chapter 7 for a detailed explanation of variable binding and
lookup.
Variable: No
Restrictions: None
Error level of message
This attribute determines the error level of the message that is logged in case of
failure. Possible choices are message, warning and error.
Note
If the attribute Throw exception on failure is set, this attribute is irrelevant and if
Variable for result is set this attribute is ignored.
Variable: No
Restrictions: None
34.9.
Checks
573
Throw exception on failure
Throw an exception in case of failure.
For ’Check...’
nodes a
(665)
CheckFailedException
is thrown, for ’Wait for...’ nodes the respective
specific exception.
Variable: No
Restrictions: None
Name
An optional name for the Check, mostly useful for the report.
Variable: Yes
Restrictions: None
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
are much better suited to this task. The option External editor command
lets
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.9.
Checks
574
34.9.3 Check items
Validates multiple displayed text strings in a component or sub-item.
(398)
Contained in: All kinds of sequences
.
Children: None
Execution: The text strings are sent to the SUT together with the data about the target
component. The TestEventQueue determines the corresponding component in the
SUT, reads its displayed values and compares them to the required values.
Attributes:
34.9.
Checks
575
Figure 34.57: Check items attributes
Client
The name of the SUT client process to which the check is sent.
Variable: Yes
Restrictions: Must not be empty.
34.9.
Checks
576
QF-Test component ID
(635)
(634)
The QF-Test ID
of the Window
target of the check.
, Component
(644)
(650)
or Item
node that is the
button brings up a dialog in which you can select the component
interThe
actively. You can also get to this dialog by pressing Shift-Return or Alt-Return ,
when the focus is in the text field. As an alternative you can copy the target node
with Ctrl-C or Edit→Copy and insert its QF-Test component ID into the text field
by pressing Ctrl-V .
(650)
By using a special syntax you can target Items
directly or access components
(57)
(193)
from another suite (see section 6.3 and section 19.1 for details).
Variable: Yes
Restrictions: Must not be empty.
Items
This table holds the values to which the text strings displayed by the component
or sub-item are compared. Each row represents one sub-item of the target
component. The ”Text” column holds the actual value while the ”Regexp” column
determines if this value is treated as a plain string or as a regexp (see section
(720)
40.4 ). The ”Regexp” column allows using a variable via performing a double
click on the respective cell.
(16)
See section 2.2.5
4.0+
about how to work with tables in QF-Test.
You can select Escape text for regular expressions from the context menu for escaping special characters of regular expressions of that cell text.
Variable: Yes for the ”Text” column, otherwise no.
Restrictions: Valid regexp if required.
Check type identifier
This attribute specifies the kind of check to perform. This makes it possible to
support different kinds of checks of the same data type for a given target
component without any ambiguity. With the help of a Checker additional check
(856)
types can be implemented as shown in section 43.5 .
Variable: Yes
Restrictions: Must not be empty
Timeout
Time limit in milliseconds until the check must succeed. To disable waiting, leave
this value empty or set it to 0.
34.9.
Checks
577
Variable: Yes
Restrictions: Must not be negative.
Variable for result
This optional attribute determines the name for the result variable of the action. If
set, the respective variable will be set to ’true’ for a successful check or wait and
to ’false’ in case of failure.
Note
If this attribute is set, the attribute Error level of message is ignored and no error is
reported. The attribute Throw exception on failure always remains effective, so it is
possible to set a result variable and still throw an exception.
Variable: Yes
Restrictions: None
Local variable
This flag determines whether to create a local or global variable binding. If unset,
the variable is bound in the global variables. If set, the topmost current binding for
the variable is replaced with the new value, provided this binding is within the
(463)
(427)
(398)
context of the currently executing Procedure , Dependency or Test-case node.
If no such binding exists, a new binding is created in the currently executing
Procedure, Dependency or Test-case node or, if there is no such node, in the global
(62)
bindings. See chapter 7 for a detailed explanation of variable binding and
lookup.
Variable: No
Restrictions: None
Error level of message
This attribute determines the error level of the message that is logged in case of
failure. Possible choices are message, warning and error.
Note
If the attribute Throw exception on failure is set, this attribute is irrelevant and if
Variable for result is set this attribute is ignored.
Variable: No
Restrictions: None
Throw exception on failure
Throw an exception in case of failure.
For ’Check...’
nodes a
(665)
CheckFailedException
is thrown, for ’Wait for...’ nodes the respective
specific exception.
Variable: No
Restrictions: None
34.9.
Checks
578
Name
An optional name for the Check, mostly useful for the report.
Variable: Yes
Restrictions: None
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
are much better suited to this task. The option External editor command
lets
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.9.4 Check selectable items
Validates multiple displayed text strings in a component or sub-item and additionally checks their selection state.
(398)
Contained in: All kinds of sequences
Children: None
.
34.9.
Checks
579
Execution: The text strings and selection state data are sent to the SUT together with
the data about the target component. The TestEventQueue determines the corresponding component in the SUT, reads its displayed values, compares them to the required values and checks the required selection state.
Attributes:
Figure 34.58: Check selectable items attributes
Client
34.9.
Checks
580
The name of the SUT client process to which the check is sent.
Variable: Yes
Restrictions: Must not be empty.
QF-Test component ID
(635)
(634)
The QF-Test ID
of the Window
target of the check.
, Component
(644)
(650)
or Item
node that is the
button brings up a dialog in which you can select the component
interThe
actively. You can also get to this dialog by pressing Shift-Return or Alt-Return ,
when the focus is in the text field. As an alternative you can copy the target node
with Ctrl-C or Edit→Copy and insert its QF-Test component ID into the text field
by pressing Ctrl-V .
(650)
By using a special syntax you can target Items
directly or access components
(57)
(193)
from another suite (see section 6.3 and section 19.1 for details).
Variable: Yes
Restrictions: Must not be empty.
Items
This table holds the values to which the text strings displayed by the component
or sub-item are compared. Each row represents one sub-item of the target
component. The ”Text” column holds the actual value while the ”Regexp” column
determines if this value is treated as a plain string or as a regexp (see section
(720)
40.4 ). The ”Selected” column holds the required selection state. The ”Regexp”
and the ”Selected” column allow using a variable via performing a double click on
the respective cell.
(16)
See section 2.2.5
4.0+
about how to work with tables in QF-Test.
You can select Escape text for regular expressions from the context menu for escaping special characters of regular expressions of that cell text.
Variable: Yes for the ”Text” column, otherwise no.
Restrictions: Valid regexp if required.
Check type identifier
This attribute specifies the kind of check to perform. This makes it possible to
support different kinds of checks of the same data type for a given target
component without any ambiguity. With the help of a Checker additional check
(856)
types can be implemented as shown in section 43.5 .
Variable: Yes
34.9.
Checks
581
Restrictions: Must not be empty
Timeout
Time limit in milliseconds until the check must succeed. To disable waiting, leave
this value empty or set it to 0.
Variable: Yes
Restrictions: Must not be negative.
Variable for result
This optional attribute determines the name for the result variable of the action. If
set, the respective variable will be set to ’true’ for a successful check or wait and
to ’false’ in case of failure.
Note
If this attribute is set, the attribute Error level of message is ignored and no error is
reported. The attribute Throw exception on failure always remains effective, so it is
possible to set a result variable and still throw an exception.
Variable: Yes
Restrictions: None
Local variable
This flag determines whether to create a local or global variable binding. If unset,
the variable is bound in the global variables. If set, the topmost current binding for
the variable is replaced with the new value, provided this binding is within the
(463)
(427)
(398)
context of the currently executing Procedure , Dependency or Test-case node.
If no such binding exists, a new binding is created in the currently executing
Procedure, Dependency or Test-case node or, if there is no such node, in the global
(62)
bindings. See chapter 7 for a detailed explanation of variable binding and
lookup.
Variable: No
Restrictions: None
Error level of message
This attribute determines the error level of the message that is logged in case of
failure. Possible choices are message, warning and error.
Note
If the attribute Throw exception on failure is set, this attribute is irrelevant and if
Variable for result is set this attribute is ignored.
Variable: No
Restrictions: None
34.9.
Checks
582
Throw exception on failure
Throw an exception in case of failure.
For ’Check...’
nodes a
(665)
CheckFailedException
is thrown, for ’Wait for...’ nodes the respective
specific exception.
Variable: No
Restrictions: None
Name
An optional name for the Check, mostly useful for the report.
Variable: Yes
Restrictions: None
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
are much better suited to this task. The option External editor command
lets
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.9.
Checks
583
34.9.5 Check image
Checks the displayed image of a component. This check is supported for all
kinds of components and for sub-items.
It is possible to check only parts of an image. To that end, a region within the
image can be defined, either by dragging a rectangle or by editing the region attributes.
Also, if the check image is smaller than the component, an offset into the component
can be given. When recording only the visible part of a component or when cropping
the check image to size after defining a region, the offset is determined automatically.
(398)
Contained in: All kinds of sequences
.
Children: None
Execution: The image data is sent to the SUT together with the data about the target
component. The TestEventQueue determines the corresponding component in the
SUT and compares its current image to the required value.
Attributes:
34.9.
Checks
584
Figure 34.59: Check image attributes
34.9.
Checks
585
Client
The name of the SUT client process to which the check is sent.
Variable: Yes
Restrictions: Must not be empty.
QF-Test component ID
(635)
(634)
of the Window
The QF-Test ID
target of the check.
, Component
(644)
(650)
or Item
node that is the
The
button brings up a dialog in which you can select the component
inter
actively. You can also get to this dialog by pressing Shift-Return or Alt-Return ,
when the focus is in the text field. As an alternative you can copy the target node
with Ctrl-C or Edit→Copy and insert its QF-Test component ID into the text field
by pressing Ctrl-V .
(650)
directly or access components
By using a special syntax you can target Items
(57)
(193)
from another suite (see section 6.3 and section 19.1 for details).
Variable: Yes
Restrictions: Must not be empty.
’Position of image relative to component’
If the image is smaller than the component, these offset coordinates together with
the size of the image determine the sub-region of the component that is checked.
Variable: Yes
Restrictions: Must not be negative.
Image
The recorded image of the component. It can be displayed at different zoom
levels, saved to or loaded from disk in the PNG format, or edited in an external
imaging tool. The tool to use must be defined with the option External imaging
(327)
program .
Variable: Yes
Restrictions: Must not be negative.
’Actual check region inside image’
If only some part of the displayed image of a component needs to be checked, a
region within the image can be defined that limits the comparison to that area.
Variable: Yes
Restrictions: Must not be negative.
34.9.
Checks
586
’Algorithm for image comparison’
This attribute defines a special algorithm for image comparison. See Details
(910)
about the algorithm for image comparison for further information.
Variable: Yes
Restrictions: Must match a special syntax.
Check type identifier
This attribute specifies the kind of check to perform. This makes it possible to
support different kinds of checks of the same data type for a given target
component without any ambiguity. The standard check type is ’default’. There are
two special check types for PDF, ’scaled’ and ’unscaled’, which are described in
(155)
the PDF section 15.3.2 . With the help of a Checker additional check types
(856)
can be implemented as shown in section 43.5 .
Variable: Yes
Restrictions: Must not be empty
Negate
This flag determines whether to execute a positive or a negative check. If it is set,
a negative check will be performed, i.e. the checked property must not match the
expected value.
Variable: Yes
Restrictions: None
Timeout
Time limit in milliseconds until the check must succeed. To disable waiting, leave
this value empty or set it to 0.
Variable: Yes
Restrictions: Must not be negative.
Variable for result
This optional attribute determines the name for the result variable of the action. If
set, the respective variable will be set to ’true’ for a successful check or wait and
to ’false’ in case of failure.
Note
If this attribute is set, the attribute Error level of message is ignored and no error is
reported. The attribute Throw exception on failure always remains effective, so it is
possible to set a result variable and still throw an exception.
Variable: Yes
Restrictions: None
34.9.
Checks
587
Local variable
This flag determines whether to create a local or global variable binding. If unset,
the variable is bound in the global variables. If set, the topmost current binding for
the variable is replaced with the new value, provided this binding is within the
(463)
(427)
(398)
context of the currently executing Procedure , Dependency or Test-case node.
If no such binding exists, a new binding is created in the currently executing
Procedure, Dependency or Test-case node or, if there is no such node, in the global
(62)
bindings. See chapter 7 for a detailed explanation of variable binding and
lookup.
Variable: No
Restrictions: None
Error level of message
This attribute determines the error level of the message that is logged in case of
failure. Possible choices are message, warning and error.
Note
If the attribute Throw exception on failure is set, this attribute is irrelevant and if
Variable for result is set this attribute is ignored.
Variable: No
Restrictions: None
Throw exception on failure
Throw an exception in case of failure.
For ’Check...’
nodes a
(665)
CheckFailedException
is thrown, for ’Wait for...’ nodes the respective
specific exception.
Variable: No
Restrictions: None
Name
An optional name for the Check, mostly useful for the report.
Variable: Yes
Restrictions: None
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
34.9.
Checks
588
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
lets
are much better suited to this task. The option External editor command
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.9.6 Check geometry
Checks the location and size of a component. This check is supported for all
kinds of components but not for sub-items.
(398)
Contained in: All kinds of sequences
.
Children: None
Execution: The geometry data is sent to the SUT together with the data about the
target component. The TestEventQueue determines the corresponding component in
the SUT and compares location and size to the required values.
Attributes:
34.9.
Checks
589
Figure 34.60: Check geometry attributes
Client
The name of the SUT client process to which the check is sent.
Variable: Yes
Restrictions: Must not be empty.
34.9.
Checks
590
QF-Test component ID
(635)
(634)
The QF-Test ID
of the Window
target of the check.
, Component
(644)
(650)
or Item
node that is the
button brings up a dialog in which you can select the component
interThe
actively. You can also get to this dialog by pressing Shift-Return or Alt-Return ,
when the focus is in the text field. As an alternative you can copy the target node
with Ctrl-C or Edit→Copy and insert its QF-Test component ID into the text field
by pressing Ctrl-V .
(650)
By using a special syntax you can target Items
directly or access components
(57)
(193)
from another suite (see section 6.3 and section 19.1 for details).
Variable: Yes
Restrictions: Must not be empty.
Geometry
The X/Y coordinates, width and height to which the location and size of the
component are compared. To check only some of these, e.g. just the location or
just the size, leave the others empty.
Variable: Yes
Restrictions: Valid number, width and height > 0
Check type identifier
This attribute specifies the kind of check to perform. This makes it possible to
support different kinds of checks of the same data type for a given target
component without any ambiguity. With the help of a Checker additional check
(856)
types can be implemented as shown in section 43.5 .
Variable: Yes
Restrictions: Must not be empty
Negate
This flag determines whether to execute a positive or a negative check. If it is set,
a negative check will be performed, i.e. the checked property must not match the
expected value.
Variable: Yes
Restrictions: None
Timeout
34.9.
Checks
591
Time limit in milliseconds until the check must succeed. To disable waiting, leave
this value empty or set it to 0.
Variable: Yes
Restrictions: Must not be negative.
Variable for result
This optional attribute determines the name for the result variable of the action. If
set, the respective variable will be set to ’true’ for a successful check or wait and
to ’false’ in case of failure.
Note
If this attribute is set, the attribute Error level of message is ignored and no error is
reported. The attribute Throw exception on failure always remains effective, so it is
possible to set a result variable and still throw an exception.
Variable: Yes
Restrictions: None
Local variable
This flag determines whether to create a local or global variable binding. If unset,
the variable is bound in the global variables. If set, the topmost current binding for
the variable is replaced with the new value, provided this binding is within the
(463)
(427)
(398)
context of the currently executing Procedure , Dependency or Test-case node.
If no such binding exists, a new binding is created in the currently executing
Procedure, Dependency or Test-case node or, if there is no such node, in the global
(62)
bindings. See chapter 7 for a detailed explanation of variable binding and
lookup.
Variable: No
Restrictions: None
Error level of message
This attribute determines the error level of the message that is logged in case of
failure. Possible choices are message, warning and error.
Note
If the attribute Throw exception on failure is set, this attribute is irrelevant and if
Variable for result is set this attribute is ignored.
Variable: No
Restrictions: None
Throw exception on failure
Throw an exception in case of failure.
For ’Check...’
nodes a
(665)
CheckFailedException
is thrown, for ’Wait for...’ nodes the respective
specific exception.
34.10.
Queries
592
Variable: No
Restrictions: None
Name
An optional name for the Check, mostly useful for the report.
Variable: Yes
Restrictions: None
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
are much better suited to this task. The option External editor command
lets
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.10
Queries
Automating a GUI test gets difficult whenever the SUT shows dynamic or unpredictable
behavior, i.e. when it displays values that change with every run of the program. This is
34.10.
Queries
593
typically the case for fields that are filled automatically with things like the current time,
an automatic ID from a database, etc.
QF-Test addresses this issue with means to read values from the SUT’s components or
to determine the numerical index of a sub-item when given its name. These values are
stored in variables to be used again later as the test proceeds.
34.10.1 Fetch text
This node lets you read a value from the SUT during the execution of a test
(62)
run. The text is assigned to the local or global variable (see chapter 7 )
(593)
named by the Variable name attribute.
Not all components display text and some complex components contain multiple
textual items, so this operation is only supported for certain components
or sub-items.
If you try to fetch the text from the wrong component, an
(666)
OperationNotSupportedException
is thrown, while an unsupported sub-item
(666)
leads to an UnexpectedIndexException . The following table lists the supported
component and sub-item targets for this operation. (P/S) means primary/secondary
index.
Web
In web applications every node could contain some text, so QF-Test returns either the
text or an empty value, but never throws an OperationNotSupportedException.
Class
AbstractButton
Dialog
Frame
ComboBox
ComboBox
JEditorPane
Label
List
TabbedPane
Table
JTableHeader
TextArea
JTextComponent
Tree
Label
TextField
Index (P/S)
-/-/-/-/List item/Character index/-/Item/Tab/Column/Row
Column/Line number/-/Node/-/-/-
Result
getText()
getTitle()
getTitle()
Current value (use renderer)
List item (use renderer)
Structural element at index (experimental)
getText()
Item (use renderer)
Title of tab
Cell contents (use renderer)
Column title (use renderer)
Line of text
getText()
Node (use renderer)
getText()
getText()
Table 34.22: Components supported by Fetch text
34.10.
Queries
594
(398)
Contained in: All kinds of sequences
.
Children: None
Execution:
The data of the target component is sent to the SUT. The
TestEventQueue determines the corresponding component, retrieves the requested
value and sends it back to QF-Test, where it is stored in a global variable.
Attributes:
Figure 34.61: Fetch text attributes
Client
The name of the SUT client process from which to query the data.
Variable: Yes
Restrictions: Must not be empty.
QF-Test component ID
(635)
The QF-Test ID
queried.
(634)
of the Window
(644)
, Component
or Item
(650)
node that is to be
The
button brings up a dialog in which you can select the component
inter
actively. You can also get to this dialog by pressing Shift-Return or Alt-Return ,
when the focus is in the text field. As an alternative you can copy the target node
34.10.
Queries
595
with Ctrl-C or Edit→Copy and insert its QF-Test component ID into the text field
by pressing Ctrl-V .
(650)
By using a special syntax you can target Items
directly or access components
(57)
(193)
from another suite (see section 6.3 and section 19.1 for details).
Variable: Yes
Restrictions: Must not be empty.
Variable name
The name of the global variable to which the result of the query is assigned (see
(62)
chapter 7 ).
Variable: Yes
Restrictions: Must not be empty.
Local variable
This flag determines whether to create a local or global variable binding. If unset,
the variable is bound in the global variables. If set, the topmost current binding for
the variable is replaced with the new value, provided this binding is within the
(463)
(427)
(398)
context of the currently executing Procedure , Dependency or Test-case node.
If no such binding exists, a new binding is created in the currently executing
Procedure, Dependency or Test-case node or, if there is no such node, in the global
(62)
bindings. See chapter 7 for a detailed explanation of variable binding and
lookup.
Variable: No
Restrictions: None
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
Comment
34.10.
Queries
596
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
are much better suited to this task. The option External editor command
lets
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.10.2 Fetch index
With the help of this node you can determine the index of a sub-item during
the execution of a test, provided its displayed text is known. Obviously only
(650)
is supported as the target component. The result is assigned to
an Item
(62)
(595)
the local or global variable (see chapter 7 ) named by the Variable name attribute.
(398)
Contained in: All kinds of sequences
.
Children: None
Execution:
The data of the target component is sent to the SUT. The
TestEventQueue determines the corresponding component, searches for the
requested sub-item and sends its index back to QF-Test, where it is stored in a global
variable.
Attributes:
34.10.
Queries
597
Figure 34.62: Fetch index attributes
Client
The name of the SUT client process from which to query the data.
Variable: Yes
Restrictions: Must not be empty.
QF-Test component ID
(635)
The QF-Test ID
queried.
(634)
of the Window
(644)
, Component
or Item
(650)
node that is to be
button brings up a dialog in which you can select the component
interThe
actively. You can also get to this dialog by pressing Shift-Return or Alt-Return ,
when the focus is in the text field. As an alternative you can copy the target node
with Ctrl-C or Edit→Copy and insert its QF-Test component ID into the text field
by pressing Ctrl-V .
(650)
By using a special syntax you can target Items
directly or access components
(57)
(193)
from another suite (see section 6.3 and section 19.1 for details).
Variable: Yes
Restrictions: Must not be empty.
34.10.
Queries
598
Variable name
The name of the global variable to which the result of the query is assigned (see
(62)
chapter 7 ).
Variable: Yes
Restrictions: Must not be empty.
Local variable
This flag determines whether to create a local or global variable binding. If unset,
the variable is bound in the global variables. If set, the topmost current binding for
the variable is replaced with the new value, provided this binding is within the
(463)
(427)
(398)
context of the currently executing Procedure , Dependency or Test-case node.
If no such binding exists, a new binding is created in the currently executing
Procedure, Dependency or Test-case node or, if there is no such node, in the global
(62)
bindings. See chapter 7 for a detailed explanation of variable binding and
lookup.
Variable: No
Restrictions: None
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
are much better suited to this task. The option External editor command
lets
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
34.10.
Queries
599
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.10.3 Fetch geometry
Use this node to find out the geometry of a window, component or sub-item in
the SUT. The result is stored in up to four local or global variables, one each
for the X and Y coordinates, width and height.
(539)
(539)
(538)
This node is useful if you want to set the X and Y coordinates of a Mouse event
relative to the right or bottom border of the target component. Simply fetch the component’s width and height and define the coordinates using the extended variable syntax
(71)
for expressions (see section 7.6 ).
The following table lists the supported sub-item targets for this operation. (P/S) means
primary/secondary index.
Class
JList
JTabbedPane
JTable
JTable
JTableHeader
JTree
Index (P/S)
Item/Tab/Column/Column/Row
Column/Node/-
Result
Item
Tab
Column
Cell
Column title
Node
Table 34.23: Components supported by Fetch geometry
(398)
Contained in: All kinds of sequences
.
Children: None
Execution:
The data of the target component is sent to the SUT. The
TestEventQueue determines the corresponding component, retrieves its geometry
and sends it back to QF-Test, where the values are stored in global variables.
Attributes:
34.10.
Queries
600
Figure 34.63: Fetch geometry attributes
Client
The name of the SUT client process from which to query the data.
Variable: Yes
Restrictions: Must not be empty.
QF-Test component ID
(635)
The QF-Test ID
queried.
(634)
of the Window
(644)
, Component
or Item
(650)
node that is to be
The
button brings up a dialog in which you can select the component
inter
actively. You can also get to this dialog by pressing Shift-Return or Alt-Return ,
when the focus is in the text field. As an alternative you can copy the target node
with Ctrl-C or Edit→Copy and insert its QF-Test component ID into the text field
by pressing Ctrl-V .
(650)
By using a special syntax you can target Items
directly or access components
(57)
(193)
from another suite (see section 6.3 and section 19.1 for details).
34.10.
Queries
601
Variable: Yes
Restrictions: Must not be empty.
Location relative to window
This attribute determines whether the X and Y coordinate of a component or
sub-item is calculated relative to its parent component or relative to its parent
window. For windows there is no difference.
Variable: No
Restrictions: None
Variable for x
The name of the global variable to which the X coordinate of the window,
(62)
component or sub-item is assigned (see chapter 7 ). If you are not interested in
the X coordinate, leave this value empty.
Variable: Yes
Restrictions: None
Variable for y
The name of the global variable to which the Y coordinate of the window,
(62)
component or sub-item is assigned (see chapter 7 ). If you are not interested in
the Y coordinate, leave this value empty.
Variable: Yes
Restrictions: None
Variable for width
The name of the global variable to which the width of the window, component or
(62)
sub-item is assigned (see chapter 7 ). If you are not interested in the width,
leave this value empty.
Variable: Yes
Restrictions: None
Variable for height
The name of the global variable to which the height of the window, component or
(62)
sub-item is assigned (see chapter 7 ). If you are not interested in the height,
leave this value empty.
Variable: Yes
Restrictions: None
34.10.
Queries
602
Local variable
This flag determines whether to create local or global variable bindings. If unset,
the variables are bound in the global variables. If set, the topmost current binding
for a variable is replaced with the new value, provided this binding is within the
(463)
(427)
(398)
context of the currently executing Procedure , Dependency or Test-case node.
If no such binding exists, a new binding is created in the currently executing
Procedure, Dependency or Test-case node or, if there is no such node, in the global
(62)
bindings. See chapter 7 for a detailed explanation of variable binding and
lookup.
Variable: No
Restrictions: None
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
lets
are much better suited to this task. The option External editor command
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.11.
Miscellaneous
34.11
603
Miscellaneous
This section lists the remaining nodes that don’t fit in well with any of the other sections.
34.11.1 Set variable
This node lets you set the value of a global variable. If the test is run in(10)
teractively from QF-Test and not in batch mode (see section 1.7 ) you can
optionally set the value interactively.
(398)
Contained in: All kinds of sequences
.
Children: None
(603)
Execution: If the test is run interactively and the Interactive attribute is set, a dialog is
(603)
shown in which the value for the variable can be entered. If the Timeout is exceeded
or the value is confirmed with the OK button, the variable is bound accordingly in the
global variables. If the dialog is canceled, the test run is stopped. In the non-interactive
(602)
case the variable is bound directly to the Default value .
Attributes:
34.11.
Miscellaneous
604
Figure 34.64: Set variable attributes
Variable name
(62)
The name of the global variable to which the value is assigned (see chapter 7 ).
Variable: Yes
Restrictions: Must not be empty.
Local variable
This flag determines whether to create a local or global variable binding. If unset,
the variable is bound in the global variables. If set, the topmost current binding for
the variable is replaced with the new value, provided this binding is within the
(463)
(427)
(398)
context of the currently executing Procedure , Dependency or Test-case node.
If no such binding exists, a new binding is created in the currently executing
Procedure, Dependency or Test-case node or, if there is no such node, in the global
(62)
bindings. See chapter 7 for a detailed explanation of variable binding and
lookup.
Variable: No
Restrictions: None
34.11.
Miscellaneous
605
Default value
The default value for the variable if the test is run non-interactively, the
(603)
(603)
Interactive attribute is not set or the Timeout is exceeded.
Variable: Yes
Restrictions: None
Interactive
Whether a dialog should be shown in which the value for the global variable can
be entered. Ignored if the test is run non-interactively.
Variable: Yes
Restrictions: None
Description
A short description to display in the dialog. If you leave this value empty, the
description Value for <Variable name> will be used.
Variable: Yes
Restrictions: None
Timeout
An optional timeout value for the dialog. If the dialog is shown and the value is left
unchanged for the specified period of time, the dialog is closed automatically and
the default value is used. This avoids blocking a test that is started interactively
from QF-Test and then left to run unattended.
Variable: Yes
Restrictions: Empty or > 0.
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
34.11.
Miscellaneous
606
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
lets
are much better suited to this task. The option External editor command
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.11.2 Wait for component to appear
This node is very important for the timing of a test run. The reaction time
of the SUT varies depending on system and memory load, so it may take
a while until, say, a complex dialog is opened. This node will delay further
execution of the test until the desired component or sub-item is available. If the time
(662)
is thrown.
limit is exceeded without success, a ComponentNotFoundException
You can also use the Variable for result attribute to store the result into a variable and the
Throw exception on failure attribute to suppress the exception.
This node is intended only for relatively long delays. Short delays are handled automat(370)
ically by the Timeout options .
(606)
By setting the Wait for absence
absence of a component.
attribute this node can also be used to ensure the
(398)
Contained in: All kinds of sequences
.
Children: None
Execution: The data of the target component are sent to the SUT. The
TestEventQueue waits until either the corresponding component becomes available
or the time limit is exceeded.
Attributes:
34.11.
Miscellaneous
607
Figure 34.65: Wait for component to appear attributes
Client
The name of the Java process in which to wait.
Variable: Yes
Restrictions: Must not be empty.
QF-Test component ID
(635)
The QF-Test ID
of the Window
(634)
(644)
, Component
(650)
or Item
to wait for.
The
button brings up a dialog in which you can select the component
inter
actively. You can also get to this dialog by pressing Shift-Return or Alt-Return ,
when the focus is in the text field. As an alternative you can copy the target node
34.11.
Miscellaneous
608
with Ctrl-C or Edit→Copy and insert its QF-Test component ID into the text field
by pressing Ctrl-V .
(650)
By using a special syntax you can target Items
directly or access components
(57)
(193)
from another suite (see section 6.3 and section 19.1 for details).
Variable: Yes
Restrictions: Must not be empty.
Timeout
Time limit in milliseconds.
Variable: Yes
Restrictions: >= 0
QF-Test ID
At the moment the QF-Test ID attribute has no meaning for this type of node.
Variable: No
Restrictions: Must not contain any of the characters ’\’, ’#’, ’$’, ’@’, ’&’, or ’%’ or
start with an underscore (’_’).
Wait for absence
If this attribute is set, QF-Test waits for the absence of a component. This is
useful e.g. to ensure that a dialog has closed or was never opened in the first
(663)
place. If the component does exist, a ComponentFoundException is thrown.
It is possible to check for the absence of a sub-item. In this case, the
complex component that holds the item must be present, otherwise a
(662)
ComponentNotFoundException is thrown.
Variable: Yes
Restrictions: None
Variable for result
This optional attribute determines the name for the result variable of the action. If
set, the respective variable will be set to ’true’ for a successful check or wait and
to ’false’ in case of failure.
Note
If this attribute is set, the attribute Error level of message is ignored and no error is
reported. The attribute Throw exception on failure always remains effective, so it is
possible to set a result variable and still throw an exception.
Variable: Yes
Restrictions: None
34.11.
Miscellaneous
609
Local variable
This flag determines whether to create a local or global variable binding. If unset,
the variable is bound in the global variables. If set, the topmost current binding for
the variable is replaced with the new value, provided this binding is within the
(463)
(427)
(398)
context of the currently executing Procedure , Dependency or Test-case node.
If no such binding exists, a new binding is created in the currently executing
Procedure, Dependency or Test-case node or, if there is no such node, in the global
(62)
bindings. See chapter 7 for a detailed explanation of variable binding and
lookup.
Variable: No
Restrictions: None
Error level of message
This attribute determines the error level of the message that is logged in case of
failure. Possible choices are message, warning and error.
Note
If the attribute Throw exception on failure is set, this attribute is irrelevant and if
Variable for result is set this attribute is ignored.
Variable: No
Restrictions: None
Throw exception on failure
Throw an exception in case of failure.
For ’Check...’
nodes a
(665)
CheckFailedException
is thrown, for ’Wait for...’ nodes the respective
specific exception.
Variable: No
Restrictions: None
Delay before/after
These attributes cause a delay before or after the execution of the node. If a
(368)
value is empty, the Default delay from the global options is used.
Variable: Yes
Restrictions: Valid number >= 0
Comment
Here you can enter a comment that explains the purpose of this node. This is the
preferred way of documenting the test-suite.
Note
For detailed documentation, especially for Test-set, Test-case or Procedure nodes,
this text area might not be the right place. There are many excellent editors that
(326)
are much better suited to this task. The option External editor command
lets
34.11.
Miscellaneous
610
you define an external editor in which comments can be edited conveniently by
pressing Alt-Return or by clicking the
button.
You can trigger a special behavior of some nodes using doctags, please see
(965)
Doctags .
Variable: Yes
Restrictions: None
34.11.3 Wait for document to load
Web
(604)
This node is a variant of the Wait for component to appear
node specifically
for web pages. It not only checks the existence of a given document. If
the target document was already known to exist the last time an event was
replayed, this node waits for the document to get reloaded. When working with web
pages it is often the case that the same or very similar documents are loaded many
times. Without this node’s functionality QF-Test could not discern the case where the
old document is still around from the one where the document is already reloaded. In
the former case, replaying an event could cause it to have no effect at all because at the
same time reloading of the document begins.
The Name of the browser window attribute can be used to limit the search to a given
browser window or to define a name for a new window. If Stop loading if timeout exceeded
is set, QF-Test will abort loading th