TTCN-3@Ericsson
Ludmila Ohlsson, Discipline Manager
Zsolt Szendrei, Unit manager
Make sure you have the right tools!
Make sure you have the right tools!
TTCN-3@ericsson | Ericsson Internal | EAB/OEP-10:0185 Uen, Rev PA3 | 2010-05-29 | Page 2
Ericsson
Technology Leadership
› Ericsson is a world-leading provider
of telecommunications equipment to
both fixed and mobile networks.
› Headquarters: Stockholm, Sweden
› Employees: 86,500 (May, 2010)
TTCN-3@ericsson | Ericsson Internal | EAB/OEP-10:0185 Uen, Rev PA3 | 2010-05-29 | Page 4
SUT: Mobile Networks
Radio
Networks
Core & Packet
Networks
Internet
Mobile
Terminals
This is an example of GSM and
WCDMA networks.
Fixed Telephony
Networks
 Test Automation is a key factor when testing in a complex
environment.
 Shorter lead time, higher quality and more efficient testing
are always goals for test organizations today.
TTCN-3@ericsson | Ericsson Internal | EAB/OEP-10:0185 Uen, Rev PA3 | 2010-05-29 | Page 5
Ericsson history
133 years of continuous INNOVATION
›
›
›
›
›
1878
1923
1968
1981
1991
› 1998
›
›
›
›
›
2000
2001
2004
2005
2006
› 2008
› 2009
From telegraph to telephone
From manual to automatic switching
From electro mechanics to computer control
Mobile communication begins
From 1G analog to 2G digital mobile
technology
Integration of voice and data in mobile
networks
Moving toward 3G and mobile Internet
Sony Ericsson joint venture launched
Breakthrough of WCDMA
Expanded in IP and fiber transmission
Launches of HSPA mobile broadband
globally
Break-out of mobile broadband and LTE
technology
ST-Ericsson joint venture launched
Early automatic switch
Multi-standard radio base station RBS 6000
TTCN-3@ericsson | Ericsson Internal | EAB/OEP-10:0185 Uen, Rev PA3 | 2010-05-29 | Page 6
Ericsson drives telecommunication
evolution
2010
2005
2004
2003
2002
2001
1986-96
1878
1880
1910
1947
TTCN-3@ericsson | Ericsson Internal | EAB/OEP-10:0185 Uen, Rev PA3 | 2010-05-29 | Page 7
Test Automation
“Truly Smart Test Automation”
TTCN-3@ericsson | Ericsson Internal | EAB/OEP-10:0185 Uen, Rev PA3 | 2010-05-29 | Page 9
Vision
› By providing state-of-the-art Test
Automation frameworks/tools,
methods and processes, we
support organizations to exceed
their productivity and quality goals.
TA will also enable a much higher
Test Equipment Utilization by using
unattended tests, for example
during nights and weekends.
TTCN-3@ericsson | Ericsson Internal | EAB/OEP-10:0185 Uen, Rev PA3 | 2010-05-29 | Page 10
Objective
› Our main objective is to support testers in Ericsson’s I&V activities by
providing access to the necessary integrated tools, methods and
processes.
› Test Automation is a key factor when testing in a complex
environment.
› Shorter lead time, higher quality and more efficient testing are always
goals for test organizations today.
› Test Automation supports more frequent test runs, mainly regression
tests, that are crucial in any incremental development, like
Streamline/Agile.
TTCN-3@ericsson | Ericsson Internal | EAB/OEP-10:0185 Uen, Rev PA3 | 2010-05-29 | Page 11
Statement of direction
› To reach the vision there is a number of things we believe are
useful in most cases. This means that
– Use a dedicated Test Automation Team with the right competence:
› To get attention, focus and long term responsibility
› Try to avoid a ”one-man-show”
– The Test Automation Framework must be solid, user friendly and reliable
to enable successful deployment
– ”Keep it simple” - Start in a small scale to prove the concept (A few test
cases…)
– Use an existing framework for automation if possible and Don’t invent the
wheel...
– Only do manual tests where there is a business case (some tests might be
too expensive or complicated to automate, pulling boards, cables etc.)
TTCN-3@ericsson | Ericsson Internal | EAB/OEP-10:0185 Uen, Rev PA3 | 2010-05-29 | Page 12
Think more than test case execution...
Test Automation Frameworks can also be used to:
› Run Quality checks
› Control and program test equipment (for example: Link break generators).
› Create Background Activity Traffic (BAT) and monitor behaviour and
performance.
› Measure, calculate and present Test Equipment Utilization.
› TCM related activities, for example loading SW on nodes after builds.
› Automation of test management tasks, like reporting of statistics.
TTCN-3@ericsson | Ericsson Internal | EAB/OEP-10:0185 Uen, Rev PA3 | 2010-05-29 | Page 13
Scope
Test Design
Test Implementation
and preparation
•Generate test cases
•Test case development
automatically from the
•Check equipment
requirement documentation, • Launch of tools
model or database.
•Test automation
•Architecture
Test Execution
• Run test cases
automatically
• Robustness
and recovery
•Logging of test results
•Test Plan
•Management
TTCN-3@ericsson | Ericsson Internal | EAB/OEP-10:0185 Uen, Rev PA3 | 2010-05-29 | Page 14
Test Evaluation
• analyze test results,
• analyze log files,
• make test case statistics
(passed, not passed),
• create reports out of log files
• FST
The testing challenge
A lot more to test…
› Telecom represents an ever
increasing network complexity
› Test are expected under
customer like (load test)
behaviour
› Remote testing
› Agile development methods
drives more testing (daily test)
› New platforms and processors
(multi-core) are non-deterministic
TTCN-3@ericsson | Ericsson Internal | EAB/OEP-10:0185 Uen, Rev PA3 | 2010-05-29 | Page 15
The test tool challenge
Once upon a time…
› Lack of good tools for test
automation (true automation)
› Several scripting languages with
different logics, capabilities and
syntax
› Each tool had its own user
interface, formats of logging and
configuration
› Tools were stand alone
monoliths incapable of
communicating
…but now with
›
›
›
›
›
!
One scripting language
One user interface
One logging format
One configuration format
One tool integration technology
TTCN-3@ericsson | Ericsson Internal | EAB/OEP-10:0185 Uen, Rev PA3 | 2010-05-29 | Page 16
From Titan to TitanSim &
Model Based Testing
What is Titan?
› TTCN-3 Test Design and Execution Environment
– Complete tool chain to develop TTCN-3 test suites
– Test Environment to configure and to execute TTCN-3 test cases
– Log analyzing tools
– Provides graphical interfaces for all the tasks above
– Command line interface for unattended automatic test execution (nightly build
and test)
› A software test tool capable of load
–Supports multiple platforms: Solaris, Linux, Windows (over cygwin) and
proprietary platforms
–Load capabilities thanks to an efficient (C/C++)
run-time-environment
› An Ericsson proprietary tool
–Not available outside Ericsson
TTCN-3@ericsson | Ericsson Internal | EAB/OEP-10:0185 Uen, Rev PA3 | 2010-05-29 | Page 18
Traffic Simulator
UE
MSC
RBS
BSC
Titan
TTCN-3@ericsson | Ericsson Internal | EAB/OEP-10:0185 Uen, Rev PA3 | 2010-05-29 | Page 19
MGW
What is TitanSim?
› TitanSim is…
–SW library written in TTCN-3
–Like the set of packages in the java domain
› Goal: Provide centrally the most commonly used features to
support the development of Performance Test applications
› Provides commonly used features, such as
–Handling protocol specific data and logic
› Data definitions, basic protocol functions, etc.
–Generic data structures with algorithms
› Hashmaps, Free-Busy Queue, etc
–Atomic behaviors of protocols that can be used to compose traffic scenarios
› Currently supports 24 protocols
TTCN-3@ericsson | Ericsson Internal | EAB/OEP-10:0185 Uen, Rev PA3 | 2010-05-29 | Page 20
Traffic Load Simulator
HLR
Uu
UE
Uu
CCN
RBS
BSC
MSC
MGW
Uu
SCP
RNC
TitanSim
TTCN-3@ericsson | Ericsson Internal | EAB/OEP-10:0185 Uen, Rev PA3 | 2010-05-29 | Page 21
SM-SC
More than just a test tool
Applications
Test Suites
(ready-to-use
load tools)
(ETSI SIP CTS, 3GPP benchmark,
3GPP performance)
Libraries
(useful functions, 27 libraries for load)
Servers
Protocol emulations
11
4
Test Ports (adapters)
Protocol support
~70
~200
TTCN-3 Executor (compilers, RTE,
Developer IDEs, Execution GUIs, utilities)
TTCN-3@ericsson | Ericsson Internal | EAB/OEP-10:0185 Uen, Rev PA3 | 2010-05-29 | Page 22
Deployment in test phases
Used in end-2-end
testing scenarios
Network
level
Key usage in load and
performance testing
System
level
Function
level
Unit
level
The #1 most
used function
testing method
in Ericsson
Model based
testing
Used as one out of
several unit (basic)
testing techniques
TTCN-3@ericsson | Ericsson Internal | EAB/OEP-10:0185 Uen, Rev PA3 | 2010-05-29 | Page 23
Titan, TitanSim & MBT
Services
MBT
TitanSim
Titan
TTCN-3@ericsson | Ericsson Internal | EAB/OEP-10:0185 Uen, Rev PA3 | 2010-05-29 | Page 24
Success story
TTCN-3@ericsson | Ericsson Internal | EAB/OEP-10:0185 Uen, Rev PA3 | 2010-05-29 | Page 26
host
2009-12-22
3,000
2009-08-29
2009-05-06
2009-01-11
2008-09-18
2008-05-26
2008-02-01
2007-10-09
2007-06-16
2007-02-21
2006-10-29
2006-07-06
2006-03-13
500
2005-11-18
2005-07-26
2005-04-02
2004-12-08
2004-08-15
2004-04-22
2003-12-29
2003-09-05
2003-05-13
2003-01-18
2002-09-25
2002-06-02
no.
2002-02-07
2001-10-15
No. of active Titan licenses
host
user
host+user
2,500
2,000
1,500
1,000
host+user
user
host
0
date
How to get 3000 TTCN users?
› Long TTCN history generating high maturity
–Research even before commercial vendors
› Pushing for the use of a standard test notation
–Easy reuse, competence build-up, a single tool
› Active participation in ETSI
–To grant the Ericsson needs
› In-house tool development
–For fast provisioning, test port development, training, etc
› Dedicated support group
–The Test Competence Centre and local groups
› Reference network
–Truly committed TTCN champions, yearly internal TTCN Conference
› An R&D all management commitment
–Long term granted funding
› Cooperation with external organizations: Eclipse, NTA Forum ….
TTCN-3@ericsson | Ericsson Internal | EAB/OEP-10:0185 Uen, Rev PA3 | 2010-05-29 | Page 27
Pushing for the use…
Is TTCN-3 an obvious choice for a test organizations?
In most of the cases NO, probably not for the first sight
› Usually no competence about TTCN-3, but JAVA, C…
› Test tool needs to be purchased, not always an easy
choice, open source tools for other languages may exist
› Available in house built “good enough” solutions
› Available test series?
TTCN-3@ericsson | Ericsson Internal | EAB/OEP-10:0185 Uen, Rev PA3 | 2010-05-29 | Page 28
Education is an ISSUE
› SW testing is not considered as a career option for a SW engineer “I want
something that is creative”
› Usually no separate subject/course about SW testing is available at most of the
universities
› TTCN-3 is not a common language for most of the SW engineers
› It is not evident that you can hire test engineers from the street with TTCN3
knowledge, though more and more consultant companies provide such services
The need should be raised from the industry towards the universities
› Within Ericsson there is a dedicated organization
called Test Competence Center that is
responsible for TTCN-3 education, support and
tool development
› Test Competence Center has a cooperation with
the Technical University at Budapest, our experts
are teaching at the university and providing the
possibility of on-the-job trainings for the students.
TTCN-3@ericsson | Ericsson Internal | EAB/OEP-10:0185 Uen, Rev PA3 | 2010-05-29 | Page 29
TTCN-3 Tools & test suites
› Availability of tools is not an issue anymore, you can select tools from
different vendors
› More and more standard test suites in TTCN-3
Ericsson’s solution:
› Ericsson’s Test Competence Center provides the toolset for “free”
internally
› The reuse of existing solutions is secured (e.g. more than 180 protocol
implementations stored centrally and accessible)
› Function and load test frameworks help reuse
› Requirements are handled quickly by using agile
methodology in toolset development
TTCN-3@ericsson | Ericsson Internal | EAB/OEP-10:0185 Uen, Rev PA3 | 2010-05-29 | Page 30
Key factors to success story
› Close cooperation with customers
› Architecture & framework supporting
reuse
› Financial Model
› Deployment, maintenance & support
› Dedicated organization
TTCN-3@ericsson | Ericsson Internal | EAB/OEP-10:0185 Uen, Rev PA3 | 2010-05-29 | Page 31