OPNFV Documentation

OPNFV Documentation
OPNFV Documentation
Release Danube
OPNFV Project
August 15, 2017
Contents
1
Release
1.1 Platform overview . . . . . . . . .
1.2 Installation . . . . . . . . . . . . .
1.3 User Guide & Configuration Guide
1.4 Release Notes . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
3
3
8
10
11
2
Test Frameworks
13
2.1 Test Framework Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.2 Testing User Guides . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.3 Testing Developer Guides . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219
3
Developer
313
3.1 Documentation Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313
3.2 OPNFV Projects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 320
Bibliography
707
i
ii
OPNFV Documentation, Release Danube
Open Platform for NFV (OPNFV) facilitates the development and evolution of NFV components across various open
source ecosystems. Through system level integration, deployment and testing, OPNFV creates a reference NFV
platform to accelerate the transformation of enterprise and service provider networks. Participation is open to anyone,
whether you are an employee of a member company or just passionate about network transformation.
Contents
1
OPNFV Documentation, Release Danube
2
Contents
CHAPTER 1
Release
Platform overview
Introduction
Network Functions Virtualization (NFV) is transforming the networking industry via software-defined infrastructures
and open source is the proven method for quickly developing software for commercial products and services that
can move markets. Open Platform for NFV (OPNFV) facilitates the development and evolution of NFV components
across various open source ecosystems. Through system level integration, deployment and testing, OPNFV constructs
a reference NFV platform to accelerate the transformation of enterprise and service provider networks. As an open
source project, OPNFV is uniquely positioned to bring together the work of standards bodies, open source communities, service providers and commercial suppliers to deliver a de facto NFV platform for the industry.
By integrating components from upstream projects, the community is able to conduct performance and use case-based
testing on a variety of solutions to ensure the platform’s suitability for NFV use cases. OPNFV also works upstream
with other open source communities to bring contributions and learnings from its work directly to those communities
in the form of blueprints, patches, bugs, and new code.
OPNFV initially focused on building NFV Infrastructure (NFVI) and Virtualised Infrastructure Management (VIM)
by integrating components from upstream projects such as OpenDaylight, OpenStack, Ceph Storage, KVM, Open
vSwitch, and Linux. More recently, OPNFV has extended its portfolio of forwarding solutions to include fd.io
and ODP, is able to run on both Intel and ARM commercial and white-box hardware, support VM, Container and
BareMetal workloads, and includes Management and Network Orchestration MANO components primarily for application composition and management in the Danube release.
These capabilities, along with application programmable interfaces (APIs) to other NFV elements, form the basic
infrastructure required for Virtualized Network Functions (VNF) and MANO components.
Concentrating on these components while also considering proposed projects on additional topics (such as the MANO
components and applications themselves), OPNFV aims to enhance NFV services by increasing performance and
power efficiency improving reliability, availability and serviceability, and delivering comprehensive platform instrumentation.
OPNFV Platform Architecture
The OPNFV project addresses a number of aspects in the development of a consistent virtualisation platform including
common hardware requirements, software architecture, MANO and applications.
OPNFV Platform Overview Diagram
3
OPNFV Documentation, Release Danube
To address these areas effectively, the OPNFV platform architecture can be decomposed into the following basic
building blocks:
• Hardware: with the Infra working group, Pharos project and associated activities
• Software Platform: through the platform integration and deployment projects
• MANO: through the MANO working group and associated projects
• Applications: which affect all other areas and drive requirements for OPNFV
OPNFV Lab Infrastructure
The infrastructure working group oversees such topics as lab management, workflow, definitions, metrics and tools for
OPNFV infrastructure.
Fundamental to the WG is the Pharos Specification which provides a set of defined lab infrastructures over a geographically and technically diverse federated global OPNFV lab.
Labs may instantiate bare-metal and virtual environments that are accessed remotely by the community and used for
OPNFV platform and feature development, build, deploy and testing. No two labs are the same and the heterogeneity
of the Pharos environment provides the ideal platform for establishing hardware and software abstractions providing
well understood performance characteristics.
Community labs are hosted by OPNFV member companies on a voluntary basis. The Linux Foundation also hosts an
OPNFV lab that provides centralized CI and other production resources which are linked to community labs. Future
lab capabilities will include the ability easily automate deploy and test of any OPNFV install scenario in any lab
environment as well as on a nested “lab as a service” virtual infrastructure.
4
Chapter 1. Release
OPNFV Documentation, Release Danube
OPNFV Software Platform Architecture
The OPNFV software platform is comprised exclusively of open source implementations of platform component
pieces. OPNFV is able to draw from the rich ecosystem of NFV related technologies available in open-source then
integrate, test, measure and improve these components in conjunction with our source communities.
While the composition of the OPNFV software platform is highly complex and constituted of many projects and
components, a subset of these projects gain the most attention from the OPNFV community to drive the development
of new technologies and capabilities.
Virtual Infrastructure Management
OPNFV derives it’s virtual infrastructure management from one of our largest upstream ecosystems OpenStack. OpenStack provides a complete reference cloud management system and associated technologies. While the OpenStack
community sustains a broad set of projects, not all technologies are relevant in an NFV domain, the OPNFV community consumes a sub-set of OpenStack projects where the usage and composition may vary depending on the installer
and scenario.
For details on the scenarios available in OPNFV and the specific composition of components refer to the OPNFV User
Guide & Configuration Guide
Operating Systems
OPNFV currently uses Linux on all target machines, this can include Ubuntu, Centos or SUSE linux. The specific
version of Linux used for any deployment is documented in the installation guide.
Networking Technologies
SDN Controllers
OPNFV, as an NFV focused project, has a significant investment on networking technologies and provides a broad variety of integrated open source reference solutions. The diversity of controllers able to be used in OPNFV is supported
by a similarly diverse set of forwarding technologies.
There are many SDN controllers available today relevant to virtual environments where the OPNFV community supports and contributes to a number of these. The controllers being worked on by the community during this release of
OPNFV include:
• Neutron: an OpenStack project to provide “network connectivity as a service” between interface devices (e.g.,
vNICs) managed by other OpenStack services (e.g., nova).
• OpenDaylight: addresses multivendor, traditional and greenfield networks, establishing the industry’s de facto
SDN platform and providing the foundation for networks of the future.
• ONOS: a carrier-grade SDN network operating system designed for high availability, performance, scale-out.
Data Plane
OPNFV extends Linux virtual networking capabilities by using virtual switching and routing components. The OPNFV community proactively engages with these source communities to address performance, scale and resiliency
needs apparent in carrier networks.
1.1. Platform overview
5
OPNFV Documentation, Release Danube
• FD.io (Fast data - Input/Output): a collection of several projects and libraries to amplify the transformation that
began with Data Plane Development Kit (DPDK) to support flexible, programmable and composable services
on a generic hardware platform.
• Open vSwitch: a production quality, multilayer virtual switch designed to enable massive network automation
through programmatic extension, while still supporting standard management interfaces and protocols.
Deployment Architecture
A typical OPNFV deployment starts with three controller nodes running in a high availability configuration including
control plane components from OpenStack, SDN, etc. and a minimum of two compute nodes for deployment of
workloads (VNFs). A detailed description of the hardware requirements required to support the 5 node configuration
can be found in pharos specification: Pharos Project
In addition to the deployment on a highly available physical infrastructure, OPNFV can be deployed for development
and lab purposes in a virtual environment. In this case each of the hosts is provided by a virtual machine and allows
control and workload placement using nested virtualization.
The initial deployment is done using a staging server, referred to as the “jumphost”. This server-either physical
or virtual-is first installed with the installation program that then installs OpenStack and other components on the
controller nodes and compute nodes. See the OPNFV User Guide & Configuration Guide for more details.
The OPNFV Testing Ecosystem
The OPNFV community has set out to address the needs of virtualization in the carrier network and as such platform
validation and measurements are a cornerstone to the iterative releases and objectives.
To simplify the complex task of feature, component and platform validation and characterization the testing community has established a fully automated method for addressing all key areas of platform validation. This required the
integration of a variety of testing frameworks in our CI systems, real time and automated analysis of results, storage
and publication of key facts for each run as shown in the following diagram.
Release Verification
The OPNFV community relies on its testing community to establish release criteria for each OPNFV release. Each
release cycle the testing criteria become more stringent and better representative of our feature and resiliency requirements.
As each OPNFV release establishes a set of deployment scenarios to validate, the testing infrastructure and test suites
need to accommodate these features and capabilities. It’s not only in the validation of the scenarios themselves where
6
Chapter 1. Release
OPNFV Documentation, Release Danube
complexity increases, there are test cases that require multiple datacenters to execute when evaluating features, including multisite and distributed datacenter solutions.
The release criteria as established by the testing teams include passing a set of test cases derived from the functional
testing project ‘functest,’ a set of test cases derived from our platform system and performance test project ‘yardstick,’
and a selection of test cases for feature capabilities derived from other test projects such as bottlenecks, vsperf, cperf
and storperf. The scenario needs to be able to be deployed, pass these tests, and be removed from the infrastructure
iteratively (no less that 4 times) in order to fulfil the release criteria.
Functest
Functest provides a functional testing framework incorporating a number of test suites and test cases that test and
verify OPNFV platform functionality. The scope of Functest and relevant test cases can be found in the Functest User
Guide
Functest provides both feature project and component test suite integration, leveraging OpenStack and SDN controllers
testing frameworks to verify the key components of the OPNFV platform are running successfully.
Yardstick
Yardstick is a testing project for verifying the infrastructure compliance when running VNF applications. Yardstick benchmarks a number of characteristics and performance vectors on the infrastructure making it a valuable
pre-deployment NFVI testing tools.
Yardstick provides a flexible testing framework for launching other OPNFV testing projects.
There are two types of test cases in Yardstick:
• Yardstick generic test cases and OPNFV feature test cases; including basic characteristics benchmarking in
compute/storage/network area.
• OPNFV feature test cases include basic telecom feature testing from OPNFV projects; for example nfv-kvm,
sfc, ipv6, Parser, Availability and SDN VPN
System Evaluation and compliance testing
The OPNFV community is developing a set of test suites intended to evaluate a set of reference behaviors and capabilities for NFV systems developed externally from the OPNFV ecosystem to evaluate and measure their ability to
provide the features and capabilities developed in the OPNFV ecosystem.
The Dovetail project will provide a test framework and methodology able to be used on any NFV platform, including an
agreed set of test cases establishing an evaluation criteria for exercising an OPNFV compatible system. The Dovetail
project has begun establishing the test framework and will provide a preliminary methodology for the Danube release.
Work will continue to develop these test cases to establish a stand alone compliance evaluation solution in future
releases.
Additional Testing
Besides the test suites and cases for release verification, additional testing is performed to validate specific features
or characteristics of the OPNFV platform. These testing framework and test cases may include some specific needs;
such as extended measurements, additional testing stimuli, or tests simulating environmental disturbances or failures.
These additional testing activities provide a more complete evaluation of the OPNFV platform. Some of the projects
focused on these testing areas include:
1.1. Platform overview
7
OPNFV Documentation, Release Danube
VSPERF
VSPERF provides an automated test-framework and comprehensive test suite for measuring data-plane performance
of the NFVI including switching technology, physical and virtual network interfaces. The provided test cases with
network topologies can be customized while also allowing individual versions of Operating System, vSwitch and
hypervisor to be specified.
Bottlenecks
Bottlenecks provides a framework to find system limitations and bottlenecks, providing root cause isolation capabilities
to facilitate system evaluation.
Installation
Abstract
This document provides an overview of the installation of the Danube release of OPNFV.
The Danube release can be installed making use of any of the installer projects in OPNFV: Apex, Compass4Nfv, Fuel
or JOID. Each installer provides the ability to install a common OPNFV platform as well as integrating additional
features delivered through a variety of scenarios by the OPNFV community.
Introduction
The OPNFV platform is comprised of a variety of upstream components that may be deployed on your infrastructure.
A composition of components, tools and configurations is identified in OPNFV as a deployment scenario.
The various OPNFV scenarios provide unique features and capabilities that you may want to leverage, and it is important to understand your required target platform capabilities before installing and configuring your scenarios.
An OPNFV installation requires either a physical infrastructure environment as defined in the Pharos specification,
or a virtual one. When configuring a physical infrastructure it is strongly advised to follow the Pharos configuration
guidelines.
Scenarios
OPNFV scenarios are designed to host virtualised network functions (VNF’s) in a variety of deployment architectures
and locations. Each scenario provides specific capabilities and/or components aimed at solving specific problems for
the deployment of VNF’s.
A scenario may, for instance, include components such as OpenStack, OpenDaylight, OVS, KVM etc., where each
scenario will include different source components or configurations.
To learn more about the scenarios supported in the Danube release refer to the scenario description documents provided:
• os-nosdn-kvm-ha
• os-nosdn-kvm_ovs_dpdk-noha
• os-nosdn-kvm_ovs_dpdk_bar-noha
• os-odl_l3-fdio-noha
• os-odl_l2-fdio-ha
8
Chapter 1. Release
OPNFV Documentation, Release Danube
• os-odl_l2-fdio-noha
• os-nosdn-fdio-noha
• os-odl_l2-bgpvpn-noha
• os-odl_l2-bgpvpn-ha
• os-odl-gluon-noha
• os-nosdn-openo-ha
• os-odl_l2-sfc-ha
• os-odl_l2-sfc-noha
• os-nosdn-lxd-ha
• os-nosdn-lxd-noha
• k8-nosdn-nofeature-noha
• k8-nosdn-lb-noha
• os-nosdn-ovs-ha
• os-nosdn-ovs-noha
• os-nosdn-ovs
• os-odl_l3-ovs-ha
• os-odl_l3-ovs-noha
• os-odl_l3-fdio-ha
Installation Procedure
Detailed step by step instructions for working with an installation toolchain and installing the required scenario are
provided by the installation projects. The four projects providing installation support for the OPNFV Danube release
are: Apex, Compass4nfv, Fuel and JOID.
The instructions for each toolchain can be found in these links:
• Apex installation instruction
• Compass4nfv installation instruction
• Fuel installation instruction
• JOID installation instruction
OPNFV Test Frameworks
If you have elected to install the OPNFV platform using the deployment toolchain provided by OPNFV your system
will have been validated once the installation is completed. The basic deployment validation only addresses a small
part of capabilities provided in the platform and you may want to execute more exhaustive tests. Some investigation
will be required to select the right test suites to run on your platform.
Many of the OPNFV test project provide user-guide documentation and installation instructions in this document
1.2. Installation
9
OPNFV Documentation, Release Danube
User Guide & Configuration Guide
Abstract
OPNFV is a collaborative project aimed at providing a variety of virtualisation deployments intended to host applications serving the networking and carrier industries. This document provides guidance and instructions for using
platform features designed to support these applications, made available in the OPNFV Danube release.
This document is not intended to replace or replicate documentation from other upstream open source projects such
as KVM, OpenDaylight, or OpenStack, but to highlight the features and capabilities delivered through the OPNFV
project.
Introduction
OPNFV provides a suite of scenarios, infrastructure deployment options, which are able to be installed to host virtualised network functions (VNFs). This Guide intends to help users of the platform leverage the features and capabilities
delivered by the OPNFV project.
OPNFVs’ Continuous Integration builds, deploys and tests combinations of virtual infrastructure components in what
are defined as scenarios. A scenario may include components such as KVM, OpenDaylight, OpenStack, OVS, etc.,
where each scenario will include different source components or configurations. Scenarios are designed to enable
specific features and capabilities in the platform that can be leveraged by the OPNFV User community.
Feature Overview
The following links outline the feature deliverables from participating OPNFV projects in the Danube release. Each
of the participating projects provides detailed descriptions about the delivered features including use cases, implementation and configuration specifics.
The following Configuration Guides and User Guides assume that the reader already has some information about a
given project’s specifics and deliverables. These Guides are intended to be used following the installation with an
OPNFV installer to allow users to deploy and implement feature delivered by OPNFV.
If you are unsure about the specifics of a given project, please refer to the OPNFV wiki page at http://wiki.opnfv.org,
for more details.
Feature Configuration Guides
• Copper Configuration Guide
• Doctor Configuration Guide
• IPv6 Configuration Guide
• KVMforNFV Configuration Guide
• Netready Configuration Guide
• ONOSFW Configuration Guide
• Parser Configuration Guide
• Promise Configuration Guide
• SDNVPN Configuration Guide
• SFC Configuration Guide
10
Chapter 1. Release
OPNFV Documentation, Release Danube
Feature User Guides
• Copper User Guide
• Doctor User Guide
• Domino User Guide
• IPv6 User Guide
• KVMforNFV User Guide
• Netready User Guide
• ONOSFW User Guide
• Parser User Guide
• Promise User Guide
• SDNVPN User Guide
• SFC User Guide
Release Notes
Release notes as provided by the OPNFV participating documents are captured in this section. These include details
of software versions used, known limitations and outstanding trouble reports.
Project release notes:
Apex Release Notes
Armband Release Notes
Bottlenecks Release Notes
Compass4nfv Release Notes
Copper Release Notes
Doctor Release Notes
FDS Release Notes
Fuel Release Notes
Functest Release Notes
IPV6 Release Notes
Joid Release Notes
KVMforNFV Release Notes
Netready Release Notes
Opera Release Notes
Parser Release Notes
QTIP Release Notes
SDNVPN Release Notes
1.4. Release Notes
11
OPNFV Documentation, Release Danube
SFC Release Notes
VSPERF Release Notes
Yardstick Release Notes
12
Chapter 1. Release
CHAPTER 2
Test Frameworks
Test Framework Overview
OPNFV testing
Introduction
Testing is one of the key activities in OPNFV and includes unit, feature, component, system level testing for development, automated deployment, performance characterization or stress testing.
Test projects are dedicated to provide frameworks, tooling and test-cases categorized as functional, performance or
compliance testing. Test projects fulfill different roles such as verifying VIM functionality, benchmarking components
and platforms or analysis of measured KPIs for the scenarios released in OPNFV.
Feature projects also provide their own test suites that either run independently or within a test project.
This document details the OPNFV testing ecosystem, describes common test components used by individual OPNFV
projects and provides links to project specific documentation.
OPNFV testing ecosystem
The testing projects
The OPNFV testing projects may be summarized as follows:
The major testing projects are described in the table below:
13
OPNFV Documentation, Release Danube
Project
Bottlenecks
Description
This project aims to find system bottlenecks by testing and verifying OPNFV infrastructure in a staging
environment before committing it to a production environment. Instead of debugging a deployment in
production environment, an automatic method for executing benchmarks which plans to validate the
deployment during staging is adopted. This project forms a staging framework to find bottlenecks and
to do analysis of the OPNFV infrastructure.
CPerf
SDN Controller benchmarks and performance testing, applicable to controllers in general.
Collaboration of upstream controller testing experts, external test tool developers and the standards
community. Primarily contribute to upstream/external tooling, then add jobs to run those tools on
OPNFV’s infrastructure.
DoveThis project intends to define and provide a set of OPNFV related validation criteria that will provide
tail
input for the evaluation of the use of OPNFV trademarks. The dovetail project is executed with the
guidance and oversight of the Compliance and Certification committee and work to secure the goals of
the C&C committee for each release. The project intends to incrementally define qualification criteria
that establish the foundations of how we are able to measure the ability to utilize the OPNFV platform,
how the platform itself should behave, and how applications may be deployed on the platform.
Functest This project deals with the functional testing of the VIM and NFVI. It leverages several upstream test
suites (OpenStack, ODL, ONOS, etc.) and can be used by feature project to launch feature test suites in
CI/CD. The project is used for scenario validation.
Qtip
QTIP as the project for “Platform Performance Benchmarking” in OPNFV aims to provide user a
simple indicator for performance, supported by comprehensive testing data and transparent calculation
formula. It provides a platform with common services for performance benchmarking which helps
users to build indicators by themselves with ease.
StorThe purpose of this project is to provide a tool to measure block and object storage performance in an
perf
NFVI. When complemented with a characterization of typical VF storage performance requirements, it
can provide pass/fail thresholds for test, staging, and production NFVI environments.
VSperf This project provides a framework for automation of NFV data-plane performance testing and
benchmarking. The NFVI fast-path includes switch technology and network with physical and virtual
interfaces. VSperf can be used to evaluate the suitability of different Switch implementations and
features, quantify data-path performance and optimize platform configurations.
YardThe goal of the Project is to verify the infrastructure compliance when running VNF applications. NFV
stick
Use Cases described in ETSI GS NFV 001 show a large variety of applications, each defining specific
requirements and complex configuration on the underlying infrastructure and test tools.The Yardstick
concept decomposes typical VNF work-load performance metrics into a number of
characteristics/performance vectors, which each of them can be represented by distinct test-cases.
The testing working group resources
The assets
Overall Architecture
The Test result management can be summarized as follows:
+-------------+
+-------------+
+-------------+
|
|
|
|
|
|
|
Test
|
|
Test
|
|
Test
|
| Project #1 |
| Project #2 |
| Project #N |
|
|
|
|
|
|
+-------------+
+-------------+
+-------------+
|
|
|
V
V
V
+---------------------------------------------+
14
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
|
|
|
Test Rest API front end
|
|
http://testresults.opnfv.org/test
|
|
|
+---------------------------------------------+
^
|
^
|
V
|
|
+-------------------------+
|
|
|
|
|
|
|
Test Results DB
|
|
|
|
Mongo DB
|
|
|
|
|
|
|
+-------------------------+
|
|
|
|
|
+----------------------+
+----------------------+
|
|
|
|
| Testing Dashboards
|
|
Landing page
|
|
|
|
|
+----------------------+
+----------------------+
The testing databases
A Mongo DB Database has been introduced for the Brahmaputra release. The following collections are declared in
this database:
• pods: the list of pods used for production CI
• projects: the list of projects providing test cases
• testcases: the test cases related to a given project
• results: the results of the test cases
• scenarios: the OPNFV scenarios tested in CI
This database can be used by any project through the testapi. Please note that projects may also use additional
databases. This database is mainly use to colelct CI results and scenario trust indicators.
This database is also cloned for OPNFV Plugfest.
The test API
The Test API is used to declare pods, projects, test cases and test results. Pods correspond to the cluster of machines
(3 controller and 2 compute nodes in HA mode) used to run the tests and defined in Pharos project. The results pushed
in the database are related to pods, projects and cases. If you try to push results of test done on non referenced pod,
the API will return an error message.
An additional method dashboard has been added to post-process the raw results in the Brahmaputra release (deprecated
in Colorado release).
The data model is very basic, 5 objects are available:
• Pods
• Projects
• Testcases
• Results
2.1. Test Framework Overview
15
OPNFV Documentation, Release Danube
• Scenarios
For detailed information, please go to http://artifacts.opnfv.org/releng/docs/testapi.html
The reporting
The reporting page for the test projects is http://testresults.opnfv.org/reporting/
This page provides a reporting per OPNFV release and per testing project.
An evolution of this page is planned. It was decided to unify the reporting by creating a landing page that should give
the scenario status in one glance (it was previously consolidated manually on a wiki page).
The landing page (planned for Danube 2.0) will be displayed per scenario:
• the status of the deployment
• the score of the test projectS
• a trust indicator
Additional filters (version, installer, test collection time window,... ) are included.
The test case catalog
Until the Colorado release, each testing project was managing the list of its test cases. It was very hard to have a global
view of the available test cases among the different test projects. A common view was possible through the API but it
was not very user friendly. In fact you may know all the cases per project calling:
http://testresults.opnfv.org/test/api/v1/projects/<project_name>/cases
with project_name: bottlenecks, functest, qtip, storperf, vsperf, yardstick
It was decided to build a web site providing a consistent view of the test cases per project and allow any scenario
owner to build his/her custom list of tests (Danube 2.0).
16
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
2.1. Test Framework Overview
17
OPNFV Documentation, Release Danube
Other resources
wiki: https://wiki.opnfv.org/testing
mailing list: [email protected]
IRC chan: #opnfv-testperf
weekly meeting (https://wiki.opnfv.org/display/meetings/TestPerf):
• Usual time: Every Thursday 15:00-16:00 UTC / 7:00-8:00 PST
• APAC time: 2nd Wednesday of the month 8:00-9:00 UTC
Reference documentation
Project
Bottlenecks
CPerf
Dovetail
Functest
Qtip
Storperf
VSperf
Yardstick
Documentation links
https://wiki.opnfv.org/display/bottlenecks/Bottlenecks
https://wiki.opnfv.org/display/cperf
https://wiki.opnfv.org/display/dovetail
https://wiki.opnfv.org/display/functest/
https://wiki.opnfv.org/display/qtip
https://wiki.opnfv.org/display/storperf/Storperf
https://wiki.opnfv.org/display/vsperf
https://wiki.opnfv.org/display/yardstick/Yardstick
Testing User Guides
Bottlenecks
Bottlenecks - User Guide
POSCA Stress (Factor) Test of Perfomance Life-Cycle
Bottlenecks POSCA Stress Test Ping
test case name
description
Test Case
configuration
test result
posca_posca_ping
Stress test regarding life-cycle while using ping to validate the VM pairs constructions
config file: /testsuite/posca/testcase_cfg/posca_posca_ping.yam
stack number: 5, 10, 20, 50 ...
PKT loss rate, success rate, test time, latency
Configuration
load_manager:
scenarios:
tool: ping
test_times: 100
package_size:
num_stack: 5, 10, 20
package_loss: 10%
contexts:
18
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
stack_create: yardstick
flavor:
yardstick_test_ip:
yardstick_test_dir: "samples"
yardstick_testcase: "ping_bottlenecks"
dashboard:
dashboard: "y"
dashboard_ip:
POSCA Stress (Factor) Test of System bandwidth
Bottlenecks POSCA Stress Test Traffic
test case name
description
Test Case
posca_factor_system_bandwith
Stress test regarding baseline of the system for a single
user, i.e., a VM pair while increasing the package size
configuration
config file:
/testsuite/posca/testcase_cfg/
posca_factor_system_bandwith.yaml
stack number: 1
PKT loss rate, latency, throupht, cpu usage
test result
Configration
test_config:
tool: netperf
protocol: tcp
test_time: 20
tx_pkt_sizes: 64, 256, 1024, 4096, 8192, 16384, 32768, 65536
rx_pkt_sizes: 64, 256, 1024, 4096, 8192, 16384, 32768, 65536
cpu_load: 0.9
latency: 100000
runner_config:
dashboard: "y"
dashboard_ip:
stack_create: yardstick
yardstick_test_ip:
yardstick_test_dir: "samples"
yardstick_testcase: "netperf_bottlenecks"
Bottlenecks - Deprecated Test Cases
Bottlenecks Rubbos Test Case Description Basic
Bottlenecks Rubbos Basic
test case name
description
configuration
test result
2.2. Testing User Guides
opnfv_bottlenecks_rub
Rubbos platform for 1
config file: /testsuite/r
client number: 1
throughput
19
OPNFV Documentation, Release Danube
Bottlenecks Rubbos Test Case Description TC1101
Bottlenecks Rubbos TC1101
test case name
description
configuration
opnfv_bottlenecks_
Rubbos platform fo
config file: /testsui
0-1.yaml
client number: 5
throughput
test result
Bottlenecks Rubbos Test Case Description TC1201
Bottlenecks Rubbos TC1201
test case name
description
configuration
opnfv_bottlenecks_
Rubbos platform fo
config file: /testsui
0-1.yaml
client number: 5
throughput
test result
Bottlenecks Rubbos Test Case Description TC1301
Bottlenecks Rubbos TC1301
test case name
description
configuration
opnfv_bottlenecks_
Rubbos platform fo
config file: /testsui
0-1.yaml
client number: 5
throughput
test result
Bottlenecks Rubbos Test Case Description TC1401
Bottlenecks Rubbos TC1401
test case name
description
configuration
opnfv_bottlenecks_
Rubbos platform fo
config file: /testsui
0-1.yaml
client number: 5
throughput
test result
Bottlenecks Rubbos Test Case Description Heavy TC1101
Bottlenecks Rubbos TC Heavy1101
test case name
description
configuration
opnfv_bottle
Rubbos platf
config file:
heavy
client numbe
throughput
test result
Bottlenecks
20
vSwitch
Test
Framework(VSTF)
Test
Case
Description
Ti1
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
Bottlenecks VSTF Ti1
test case name
description
configuration
opnfv_bottlenecks_vstf_Ti1
vSwitch test Ti1.
test result
throughput & latency
Bottlenecks
vSwitch
Bottlenecks VSTF Ti2
test case name
description
configuration
config file: /testsuite/vstf/testcase_cfg/vstf_Ti1.yaml
Test
Test
test result
2.2. Testing User Guides
Ti2
Framework(VSTF)
Test
Case
Description
Ti3
opnfv_bottlenecks_vstf_Ti3
vSwitch test Ti3.
config file: /testsuite/vstf/testcase_cfg/vstf_Ti3.yaml
throughput & latency
Test
Framework(VSTF)
Test
Case
Description
Tn1
opnfv_bottlenecks_vstf_Tn1
vSwitch test Tn1.
config file: /testsuite/vstf/testcase_cfg/vstf_Tn1.yaml
throughput & latency
Test
Framework(VSTF)
Test
Case
Description
Tn2
opnfv_bottlenecks_vstf_Tn2
vSwitch test Tn2.
config file: /testsuite/vstf/testcase_cfg/vstf_Tn2.yaml
test result
Bottlenecks
vSwitch
Bottlenecks VSTF Tu1
test case name
description
configuration
Description
throughput & latency
test result
Bottlenecks
vSwitch
Bottlenecks VSTF Tn2
test case name
description
configuration
Case
config file: /testsuite/vstf/testcase_cfg/vstf_Ti2.yaml
test result
Bottlenecks
vSwitch
Bottlenecks VSTF Tn1
test case name
description
configuration
Test
opnfv_bottlenecks_vstf_Ti2
vSwitch test Ti2.
test result
Bottlenecks
vSwitch
Bottlenecks VSTF Ti3
test case name
description
configuration
Framework(VSTF)
throughput & latency
Test
Framework(VSTF)
Test
Case
Description
Tu1
opnfv_bottlenecks_vstf_Tu1
vSwitch test Tu1.
config file: /testsuite/vstf/testcase_cfg/vstf_Tu1.yaml
throughput & latency
21
OPNFV Documentation, Release Danube
Bottlenecks
vSwitch
Bottlenecks VSTF Tu2
test case name
description
configuration
Test
Framework(VSTF)
Test
Case
Description
Tu2
opnfv_bottlenecks_vstf_Tu2
vSwitch test Tu2.
config file: /testsuite/vstf/testcase_cfg/vstf_Tu2.yaml
test result
throughput & latency
Bottlenecks
vSwitch
Bottlenecks VSTF Tu3
test case name
description
configuration
Test
Framework(VSTF)
test result
Test
Case
Description
Tu3
opnfv_bottlenecks_vstf_Tu3
vSwitch test Tu3.
config file: /testsuite/vstf/testcase_cfg/vstf_Tu3.yaml
throughput & latency
Functest
OPNFV FUNCTEST Configuration Guide
Version history
Date
2016-0817
2017-0119
Ver. Author
1.0.0 Juha Haapavirta Column
Gaynor
1.0.1 Morgan Richomme
Comment
Colorado release
Adaptations for Danube * update testcase list * update docker
command
Introduction
This document describes how to install and configure Functest in OPNFV. The Functest CLI is used during the Functest
environment preparation phase. The given example commands should work in both virtual and bare metal cases alike.
High level architecture The high level architecture of Functest within OPNFV can be described as follows:
CIMC/Lights+out management
Admin Mgmt/API
PXE
+
+
+
|
|
|
|
+----------------------------+
|
|
|
|
|
|
|
+-----+
Jumphost
|
|
|
|
|
+--------+
|
|
|
|
|
|
|
|
+--------------------+
|
|
|
|
|
|
|
|
|
|
|
|
| Tools
|
+----------------+
|
|
| - Rally
|
|
|
|
|
|
| - Robot
|
|
|
|
|
|
| - TestON
|
|
|
|
22
Public
+
|
|
|
|
|
|
|
|
|
|
|
|
Storage Private
+
|
|
|
|
|
|
|
|
|
|
|
|
+
|
|
|
|
|
|
|
|
|
|
|
|
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
|
|
| - RefStack
|
|
|
|
|
|
|
|
|
|
|
|-------------------------+
|
|
|
|
| Testcases
|
|
|
|
|
|
|
|
|
| - VIM
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| - SDN Controller
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| - Features
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| - VNF
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
+--------------------+
|
|
|
|
|
|
|
|
Functest Docker
+
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
+----------------------------+
|
|
|
|
|
|
|
|
|
|
|
|
+----------------+
|
|
|
|
|
|
|
1 |
|
|
|
|
|
+----+ +--------------+-+
|
|
|
|
|
|
| |
2 |
|
|
|
|
|
|
| | +--------------+-+
|
|
|
|
|
|
| | |
3 |
|
|
|
|
|
|
| | | +--------------+-+
|
|
|
|
|
|
| | | |
4 |
|
|
|
|
|
|
+-+ | | +--------------+-+
|
|
|
|
|
|
| | | |
5 +-------------+
|
|
|
|
|
+-+ | | nodes for
|
|
|
|
|
|
|
| | | deploying
+---------------------+
|
|
|
|
+-+ | OPNFV
|
|
|
|
|
|
|
| |
+------------------------------+
|
|
|
+-+
SUT
|
|
|
|
|
|
|
|
+--------------------------------------+
|
|
|
|
|
|
|
|
|
|
|
+----------------------------------------------+
|
+----------------+
|
|
|
|
|
|
|
|
|
|
|
+
+
+
+
+
+
SUT = System Under Test
All the libraries and dependencies needed by all of the Functest tools are pre-installed into the Docker image. This
allows running Functest on any platform on any Operating System.
The automated mechanisms inside the Functest Docker container will:
• Retrieve OpenStack credentials
• Prepare the environment according to the System Under Test (SUT)
• Perform the appropriate functional tests
• Push the test results into the OPNFV test result database
This Docker image can be integrated into CI or deployed independently.
Please note that the Functest Docker container has been designed for OPNFV, however, it would be possible to adapt
it to any OpenStack based VIM + controller environment, since most of the test cases are integrated from upstream
communities.
The functional test cases are described in the Functest User Guide [2]
2.2. Testing User Guides
23
OPNFV Documentation, Release Danube
Prerequisites
The OPNFV deployment is out of the scope of this document but it can be found in http://docs.opnfv.org. The OPNFV
platform is considered as the SUT in this document.
Several prerequisites are needed for Functest:
1. A Jumphost to run Functest on
2. A Docker daemon shall be installed on the Jumphost
3. A public/external network created on the SUT
4. An admin/management network created on the SUT
5. Connectivity from the Jumphost to the SUT public/external network
6. Connectivity from the Jumphost to the SUT admin/management network
WARNING: Connectivity from Jumphost is essential and it is of paramount importance to make sure it is working
before even considering to install and run Functest. Make also sure you understand how your networking is designed
to work.
NOTE: Jumphost refers to any server which meets the previous requirements. Normally it is the same server from
where the OPNFV deployment has been triggered previously.
NOTE: If your Jumphost is operating behind a company http proxy and/or firewall, please consult first the section
Proxy Support, towards the end of this document. The section details some tips/tricks which may be of help in a
proxified environment.
Docker installation
Jumphost.
Docker installation and configuration is only needed to be done once through the life cycle of
If your Jumphost is based on Ubuntu, SUSE, RHEL or CentOS linux, please consult the references below for more
detailed instructions. The commands below are offered as a short reference.
Tip: For running docker containers behind the proxy, you need first some extra configuration which is described in
section Docker Installation on CentOS behind http proxy. You should follow that section before installing the docker
engine.
Docker installation needs to be done as root user. You may use other userid’s to create and run the actual containers
later if so desired. Log on to your Jumphost as root user and install the Docker Engine (e.g. for CentOS family):
curl -sSL https://get.docker.com/ | sh
systemctl start docker
*Tip:* If you are working through proxy, please set the https_proxy
environment variable first before executing the curl command.
Add your user to docker group to be able to run commands without sudo:
sudo usermod -aG docker <your_user>
A reconnection is needed. There are 2 ways for this:
1. Re-login to your account
2. su - <username>
References - Installing Docker Engine on different Linux Operating Systems:
• Ubuntu
24
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
• RHEL
• CentOS
• SUSE
Public/External network on SUT Some of the tests against the VIM (Virtual Infrastructure Manager) need connectivity through an existing public/external network in order to succeed. This is needed, for example, to create floating
IPs to access VM instances through the public/external network (i.e. from the Docker container).
By default, the four OPNFV installers provide a fresh installation with a public/external network created along with a
router. Make sure that the public/external subnet is reachable from the Jumphost.
Hint: For the given OPNFV Installer in use, the IP sub-net address used for the public/external network is usually a
planning item and should thus be known. Consult the OPNFV Configuration guide [4], and ensure you can reach each
node in the SUT, from the Jumphost using the ‘ping’ command using the respective IP address on the public/external
network for each node in the SUT. The details of how to determine the needed IP addresses for each node in the SUT
may vary according to the used installer and are therefore ommitted here.
Connectivity to Admin/Management network on SUT Some of the Functest tools need to have access to the
OpenStack admin/management network of the controllers [1].
For this reason, check the connectivity from the Jumphost to all the controllers in cluster in the OpenStack admin/management network range.
Installation and configuration
Pulling the Docker image Pull the Functest Docker image (‘opnfv/functest’) from the public dockerhub registry
under the OPNFV account: [dockerhub], with the following docker command:
docker pull opnfv/functest:<TagIdentifier>
where <TagIdentifier> identifies a release of the Functest docker container image in the public Dockerhub registry.
There are many tags created automatically by the CI mechanisms, and you must ensure you pull an image with the
correct tag to match the OPNFV software release installed in your environment. All available tagged images can
be seen from location [FunctestDockerTags]. For example, when running on the first official release of the OPNFV
Danube system platform, tag “danube.1.0” is needed. For the second and third releases, the tag “danube.2.0” and
“danube.3.0” can be used respectively. Pulling other tags might cause some problems while running the tests. Docker
images pulled without a tag specifier bear the implicitly assigned label “latest”. If you need to specifically pull the
latest Functest docker image, then omit the tag argument:
docker pull opnfv/functest
After pulling the Docker image, check that it is available with the following docker command:
[[email protected] ~]$ docker images
REPOSITORY
TAG
IMAGE ID
opnfv/functest latest
8cd6683c32ae
opnfv/functest danube.2.0
d2c174a91911
opnfv/functest danube.1.0
13fa54a1b238
CREATED
2 weeks ago
7 minutes ago
4 weeks ago
SIZE
1.321 GB
1.471 GB
1.29 GB
The Functest docker container environment can -in principle- be also used with non-OPNFV official installers (e.g.
‘devstack’), with the disclaimer that support for such environments is outside of the scope and responsibility of the
OPNFV project.
2.2. Testing User Guides
25
OPNFV Documentation, Release Danube
Accessing the Openstack credentials OpenStack credentials are mandatory and can be retrieved in different ways.
From inside the running Functest docker container the “functest env prepare” command will automatically look for the
Openstack credentials file “/home/opnfv/functest/conf/openstack.creds” and retrieve it unless the file already exists.
This Functest environment preparation step is described later in this document.
WARNING: When the installer type is “joid” you have to have the credentials file inside the running container before
initiating the functest environment preparation. For that reason you have to choose either one of the options below,
since the automated copying does not work for “joid”.
You can also specifically pass in the needed file prior to running the environment preparation either:
• by using the -v option when creating the Docker container. This is referred to in docker documentation as “Bind
Mounting”. See the usage of this parameter in the following chapter.
• or creating a local file ‘/home/opnfv/functest/conf/openstack.creds’ inside the running container with the credentials in it. Consult your installer guide for further details. This is however not instructed in this document.
NOTE: When the installer type is “fuel” and virtualized deployment is used, there you have to explicitly fetch the
credentials file executing the following sequence
1. Create a container as described in next chapter but do not “Bind Mount” the credentials
2. Log in to container and execute the following command. Replace the IP with installer address after the “-a”
parameter:
$REPOS_DIR/releng/utils/fetch_os_creds.sh \
-d /home/opnfv/functest/conf/openstack.creds \
-i fuel \
-a 10.20.0.2 \
-v
( -d specifies the full path to the Openstack credential file
-i specifies the INSTALLER_TYPE
-a specifies the INSTALLER_IP
-v indicates a virtualized environment and takes no arguments )
3. Continue with your testing, initiate functest environment preparation, run tests etc.
In proxified environment you may need to change the credentials file. There are some tips in chapter: Proxy support
Functest Docker parameters This chapter explains how to run a container for executing functest test suites. Numbered list below explains some details of the recommended parameters for invoking docker container
1. It is a good practice to assign a precise container name through the –name option.
2. Assign parameter for installer type:
-e "INSTALLER_TYPE=<type>"
# Use one of following apex, compass, fuel or joid
3. Functest needs to know the IP of some installers:
-e "INSTALLER_IP=<Specific IP Address>"
This IP is needed to fetch RC file from deployment, fetch logs, ...
If not provided, there is no way to fetch the RC file. It must be
provided manually as a volume
4. Credentials for accessing the Openstack. Most convenient way of passing them to container is by having a local
copy of the credentials file in Jumphost and then using the -v option. In the example we have local file by the
name of “overcloudrc” and we are using that as an argument:
26
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
-v ~/overcloudrc:/home/opnfv/functest/conf/openstack.creds
The credentials file needs to exist in the Docker container
under the path: '/home/opnfv/functest/conf/openstack.creds'.
WARNING: If you are using the Joid installer, you must pass the credentials using the -v option: -v
/var/lib/jenkins/admin-openrc:/home/opnfv/functest/conf/openstack.creds. See the section Accessing the Openstack credentials above.
5. Passing deployment scenario When running Functest against any of the supported OPNFV scenarios, it is recommended to include also the environment variable DEPLOY_SCENARIO. The DEPLOY_SCENARIO environment variable is passed with the format:
-e "DEPLOY_SCENARIO=os-<controller>-<nfv_feature>-<ha_mode>"
where:
os = OpenStack (No other VIM choices currently available)
controller is one of ( nosdn | odl_l2 | odl_l3 | onos | ocl)
nfv_feature is one or more of ( ovs | kvm | sfc | bgpvpn | nofeature )
If several features are pertinent then use the underscore
character '_' to separate each feature (e.g. ovs_kvm)
'nofeature' indicates no NFV feature is deployed
ha_mode (high availability) is one of ( ha | noha )
NOTE: Not all possible combinations of “DEPLOY_SCENARIO” are supported. The name passed in to the
Functest Docker container must match the scenario used when the actual OPNFV platform was deployed. See
release note to see the list of supported scenarios.
NOTE: The scenario name is mainly used to automatically detect if a test suite is runnable or not (e.g. it will
prevent ONOS test suite to be run on ODL scenarios). If not set, Functest will try to run the default test cases
that might not include SDN controller or a specific feature
NOTE: A HA scenario means that 3 OpenStack controller nodes are deployed. It does not necessarily mean
that the whole system is HA. See installer release notes for details.
Putting all above together, when using installer ‘fuel’ and an invented INSTALLER_IP of ‘10.20.0.2’, the recommended command to create the Functest Docker container is as follows:
docker run --name "FunctestContainer" -it \
-e "INSTALLER_IP=10.20.0.2" \
-e "INSTALLER_TYPE=fuel" \
-e "DEPLOY_SCENARIO=os-odl_l2-ovs_kvm-ha" \
-v ~/overcloudrc:/home/opnfv/functest/conf/openstack.creds \
opnfv/functest /bin/bash
After the run command, a new prompt appears which means that we are inside the container and ready to move to the
next step.
For tips on how to set up container with installer Apex, see chapter Apex Installer Tips.
Finally, three additional environment variables can also be passed in to the Functest Docker Container, using the -e
“<EnvironmentVariable>=<Value>” mechanism. The first two of these are only relevant to Jenkins CI invoked testing
and should not be used when performing manual test scenarios:
-e "NODE_NAME=<Test POD Name>" \
-e "BUILD_TAG=<Jenkins Build Tag>" \
-e "CI_DEBUG=<DebugTraceValue>"
where:
<Test POD Name> = Symbolic name of the POD where the tests are run.
Visible in test results files, which are stored
to the database. This option is only used when
2.2. Testing User Guides
27
OPNFV Documentation, Release Danube
tests are activated under Jenkins CI control.
It indicates the POD/hardware where the test has
been run. If not specified, then the POD name is
defined as "Unknown" by default.
DO NOT USE THIS OPTION IN MANUAL TEST SCENARIOS.
<Jenkins Build tag> = Symbolic name of the Jenkins Build Job.
Visible in test results files, which are stored
to the database. This option is only set when
tests are activated under Jenkins CI control.
It enables the correlation of test results,
which
are independently pushed to the results database
from different Jenkins jobs.
DO NOT USE THIS OPTION IN MANUAL TEST SCENARIOS.
<DebugTraceValue> = "true" or "false"
Default = "false", if not specified
If "true" is specified, then additional debug trace
text can be sent to the test results file / log files
and also to the standard console output.
Apex Installer Tips
section.
Some specific tips are useful for the Apex Installer case. If not using Apex Installer; ignore this
In case of Triple-O based installer (like Apex) the docker container needs to connect to the installer VM, so it is then
required that some known SSH keys are present in docker container. Since the Jumphost root SSH keys are already
known, easiest way is to use those using the ‘Bind mount’ method. See below for sample parameter:
-v /root/.ssh/id_rsa:/root/.ssh/id_rsa
NOTE: You need the "sudo" when creating the container to access root
users ssh credentials even the docker command itself might not
require that.
HINT! In case of Triple-O installers you can find value for the INSTALLER_IP parameter by executing command and
note the returned IP address:
inst=$(sudo virsh list | grep -iEo "undercloud|instack")
sudo virsh domifaddr ${inst}
NOTE: In releases prior to Colorado, the name 'instack' was
used. Currently the name 'undercloud' is used.
You can copy the credentials file from the “stack” users home directory in installer VM to Jumphost. Please check the
correct IP from the command above. In the example below we are using invented IP address “192.168.122.89”:
scp [email protected]:overcloudrc .
Here is an example of the full docker command invocation for an Apex installed system, using latest Functest docker
container, for illustration purposes:
sudo docker run -it --name "ApexFuncTestODL" \
-e "INSTALLER_IP=192.168.122.89" \
-e "INSTALLER_TYPE=apex" \
-e "DEPLOY_SCENARIO=os-odl_l2-nofeature-ha" \
-v /root/.ssh/id_rsa:/root/.ssh/id_rsa \
-v ~/overcloudrc:/home/opnfv/functest/conf/openstack.creds \
opnfv/functest /bin/bash
28
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
Compass installer local development env usage Tips In the compass-functest local test case check and development environment, in order to get openstack service inside the functest container, some parameters should be configured during container creation, which are hard to guess for freshman. This section will provide the guideline, the
parameters values are defaults here, which should be adjusted according to the settings, the complete steps are given
here so as not to appear too abruptly.
1, Pull Functest docker image from public dockerhub:
docker pull opnfv/functest:<Tag>
<Tag> here can be “brahmaputra.1.0”, “colorado.1.0”, etc. Tag omitted means the latest docker image:
docker pull opnfv/functest
2, Functest Docker container creation
To make a file used for the environment, such as ‘functest-docker-env’:
OS_AUTH_URL=http://172.16.1.222:35357/v2.0
OS_USERNAME=admin
OS_PASSWORD=console
OS_TENANT_NAME=admin
OS_VOLUME_API_VERSION=2
OS_PROJECT_NAME=admin
INSTALLER_TYPE=compass
INSTALLER_IP=192.168.200.2
EXTERNAL_NETWORK=ext-net
Note: please adjust the content according to the environment, such as ‘TENANT_ID’ maybe used for some special
cases.
Then to create the Functest docker:
docker run --privileged=true --rm -t \
--env-file functest-docker-env \
--name <Functest_Container_Name> \
opnfv/functest:<Tag> /bin/bash
3, To attach Functest container
Before trying to attach the Functest container, the status can be checked by:
docker ps -a
to attach the ‘Up’ status Functest container and start bash mode:
docker exec -it <Functest_Container_Name> bash
4, Functest environemnt preparation and check
To see the Section below Preparing the Functest environment.
Functest docker container directory structure Inside the Functest docker container, the following directory structure should now be in place:
`-- home
`-- opnfv
|-- functest
|
|-- conf
|
|-- data
|
`-- results
2.2. Testing User Guides
29
OPNFV Documentation, Release Danube
`-- repos
|-- bgpvpn
|-- copper
|-- doctor
|-- domino
|-- functest
|-- kingbird
|-- odl_test
|-- onos
|-- parser
|-- promise
|-- rally
|-- refstack-client
|-- releng
|-- sdnvpn
|-- securityscanning
|-- sfc
|-- tempest
|-- vims_test
`-- vnfs
Underneath the ‘/home/opnfv/’ directory, the Functest docker container includes two main directories:
• The functest directory stores configuration files (e.g.
the OpenStack creds are stored in path
‘/home/opnfv/functest/conf/openstack.creds’), the data directory stores a ‘cirros’ test image used in some functional tests and the results directory stores some temporary result log files
• The repos directory holds various repositories. The directory ‘/home/opnfv/repos/functest’ is used to prepare the
needed Functest environment and to run the tests. The other repository directories are used for the installation
of the needed tooling (e.g. rally) or for the retrieval of feature projects scenarios (e.g. promise)
The structure under the functest repository can be described as follows:
. |-|-|-|-|-|-|-|
|
|
|-|
|
|
|-|
|
|
|
|
|
|
|
|--
30
INFO
LICENSE
requirements.txt
run_unit_tests.sh
setup.py
test-requirements.txt
commons
|-- ims
|-- mobile
`--traffic-profile-guidelines.rst
docker
|-- Dockerfile
|-- config_install_env.sh
`-- docker_remote_api
docs
|-- com
|-- configguide
|-- devguide
|-- images
|-- internship
|-- release-notes
|-- results
`--userguide
functest
|-- __init__.py
|-- ci
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
|
|
|
|
|
|
|
|
|
|
|-|
|
|
|
|
|-|
|
|
|
|
|-|
|
|
|
|
|
|
|-|
|
`--
|-- __init__.py
|-- check_os.sh
|-- config_functest.yaml
|-- config_patch.yaml
|-- generate_report.py
|-- prepare_env.py
|-- run_tests.py
|-- testcases.yaml
|-- tier_builder.py
`-- tier_handler.py
cli
|-- __init__.py
|-- cli_base.py
|-- commands
|-- functest-complete.sh
`-- setup.py
core
|-- __init__.py
|-- feature_base.py
|-- pytest_suite_runner.py
|-- testcase.py
|-- vnf_base.py
opnfv_tests
|-- __init__.py
|-- features
|-- mano
|-- openstack
|-- sdn
|-- security_scan
`-- vnf
tests
|-- __init__.py
`-- unit
utils
|-- __init__.py
|-- config.py
|-- constants.py
|-- env.py
|-- functest_logger.py
|-- functest_utils.py
|-- openstack
|-- openstack_clean.py
|-- openstack_snapshot.py
|-- openstack_tacker.py
`-- openstack_utils.py
(Note: All *.pyc files removed from above list for brevity...)
We may distinguish several directories, the first level has 4 directories:
• commons: This directory is dedicated for storage of traffic profile or any other test inputs that could be reused
by any test project.
• docker: This directory includes the needed files and tools to build the Funtest Docker image.
• docs: This directory includes documentation: Release Notes, User Guide, Configuration Guide and Developer
Guide.
• functest: This directory contains all the code needed to run functest internal cases and OPNFV onboarded
2.2. Testing User Guides
31
OPNFV Documentation, Release Danube
feature or VNF test cases.
Functest directory has 6 directories:
• ci: This directory contains test structure definition files (e.g <filename>.yaml) and bash shell/python scripts
used to configure and execute Functional tests. The test execution script can be executed under the control
of Jenkins CI jobs.
• cli: This directory holds the python based Functest CLI utility source code, which is based on the Python
‘click’ framework.
• core: This directory holds the python based Functest core source code. Three abstraction classes have
been created to ease the integration of internal, feature or vnf cases.
• opnfv_tests: This directory includes the scripts required by Functest internal test cases and other feature
projects test cases.
• tests: This directory includes the functest unit tests
• utils: this directory holds Python source code for some general purpose helper utilities, which testers can
also re-use in their own test code. See for an example the Openstack helper utility: ‘openstack_utils.py’.
Useful Docker commands When typing exit in the container prompt, this will cause exiting the container and
probably stopping it. When stopping a running Docker container all the changes will be lost, there is a keyboard
shortcut to quit the container without stopping it: <CTRL>-P + <CTRL>-Q. To reconnect to the running container
DO NOT use the run command again (since it will create a new container), use the exec or attach command instead:
docker ps # <check the container ID from the output>
docker exec -ti <CONTAINER_ID> /bin/bash
There are other useful Docker commands that might be needed to manage possible issues with the containers.
List the running containers:
docker ps
List all the containers including the stopped ones:
docker ps -a
Start a stopped container named “FunTest”:
docker start FunTest
Attach to a running container named “StrikeTwo”:
docker attach StrikeTwo
It is useful sometimes to remove a container if there are some problems:
docker rm <CONTAINER_ID>
Use the -f option if the container is still running, it will force to destroy it:
docker rm -f <CONTAINER_ID>
Check the Docker documentation dockerdocs for more information.
32
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
Preparing the Functest environment Once the Functest docker container is up and running, the required Functest
environment needs to be prepared. A custom built functest CLI utility is available to perform the needed environment preparation action. Once the environment is prepared, the functest CLI utility can be used to run different
functional tests. The usage of the functest CLI utility to run tests is described further in the Functest User Guide
OPNFV_FuncTestUserGuide
Prior to commencing the Functest environment preparation, we can check the initial status of the environment. Issue
the functest env status command at the prompt:
functest env status
Functest environment is not installed.
Note: When the Functest environment is prepared, the command will
return the status: "Functest environment ready to run tests."
To prepare the Functest docker container for test case execution, issue the functest env prepare command at the
prompt:
functest env prepare
This script will make sure that the requirements to run the tests are met and will install the needed libraries and tools
by all Functest test cases. It should be run only once every time the Functest docker container is started from scratch.
If you try to run this command, on an already prepared enviroment, you will be prompted whether you really want to
continue or not:
functest env prepare
It seems that the environment has been already prepared.
Do you want to do it again? [y|n]
(Type 'n' to abort the request, or 'y' to repeat the
environment preparation)
To list some basic information about an already prepared Functest docker container environment, issue the functest
env show at the prompt:
functest env show
+======================================================+
| Functest Environment info
|
+======================================================+
| INSTALLER: apex, 192.168.122.89
|
|
SCENARIO: os-odl_l2-nofeature-ha
|
|
POD: localhost
|
| GIT BRANCH: master
|
|
GIT HASH: 5bf1647dec6860464eeb082b2875798f0759aa91 |
| DEBUG FLAG: false
|
+------------------------------------------------------+
|
STATUS: ready
|
+------------------------------------------------------+
Where:
INSTALLER:
SCENARIO:
POD:
Displays the INSTALLER_TYPE value
- here = "apex"
and the INSTALLER_IP value
- here = "192.168.122.89"
Displays the DEPLOY_SCENARIO value
- here = "os-odl_l2-nofeature-ha"
Displays the value passed in NODE_NAME
- here = "localhost"
2.2. Testing User Guides
33
OPNFV Documentation, Release Danube
GIT BRANCH: Displays the git branch of the OPNFV Functest
project repository included in the Functest
Docker Container.
- here = "master"
(In first official colorado release
would be "colorado.1.0")
GIT HASH:
Displays the git hash of the OPNFV Functest
project repository included in the Functest
Docker Container.
- here = "5bf1647dec6860464eeb082b2875798f0759aa91"
DEBUG FLAG: Displays the CI_DEBUG value
- here = "false"
NOTE: In Jenkins CI runs, an additional item "BUILD TAG"
would also be listed. The value is set by Jenkins CI.
Finally, the functest CLI has a –help options:
Some examples:
functest --help Usage: functest [OPTIONS] COMMAND [ARGS]...
Options:
--version Show the version and exit.
-h, --help Show this message and exit.
Commands:
env
openstack
testcase
tier
functest env --help
Usage: functest env [OPTIONS] COMMAND [ARGS]...
Options:
-h, --help Show this message and exit.
Commands:
prepare
show
status
Prepares the Functest environment.
Shows information about the current...
Checks if the Functest environment is ready...
Checking Openstack and credentials
credentials are working as expected.
It is recommended and fairly straightforward to check that Openstack and
Once the credentials are there inside the container, they should be sourced before running any Openstack commands:
source /home/opnfv/functest/conf/openstack.creds
After this, try to run any OpenStack command to see if you get any output, for instance:
openstack user list
This will return a list of the actual users in the OpenStack deployment. In any other case, check that the credentials
are sourced:
34
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
env|grep OS_
This command must show a set of environment variables starting with OS_, for example:
OS_REGION_NAME=RegionOne
OS_DEFAULT_DOMAIN=default
OS_PROJECT_NAME=admin
OS_PASSWORD=admin
OS_AUTH_STRATEGY=keystone
OS_AUTH_URL=http://172.30.10.3:5000/v2.0
OS_USERNAME=admin
OS_TENANT_NAME=admin
OS_ENDPOINT_TYPE=internalURL
OS_NO_CACHE=true
If the OpenStack command still does not show anything or complains about connectivity issues, it could be due to an
incorrect url given to the OS_AUTH_URL environment variable. Check the deployment settings.
SSL Support If you need to connect to a server that is TLS-enabled (the auth URL begins with “https”) and it uses
a certificate from a private CA or a self-signed certificate, then you will need to specify the path to an appropriate CA
certificate to use, to validate the server certificate with the environment variable OS_CACERT:
echo $OS_CACERT
/etc/ssl/certs/ca.crt
However, this certificate does not exist in the container by default. It has to be copied manually from the OpenStack
deployment. This can be done in 2 ways:
1. Create manually that file and copy the contents from the OpenStack controller.
2. (Recommended) Add the file using a Docker volume when starting the container:
-v <path_to_your_cert_file>:/etc/ssl/certs/ca.cert
You might need to export OS_CACERT environment variable inside the container:
export OS_CACERT=/etc/ssl/certs/ca.crt
Certificate verification can be turned off using OS_INSECURE=true. For example, Fuel uses self-signed cacerts by
default, so an pre step would be:
export OS_INSECURE=true
Proxy support If your Jumphost node is operating behind a http proxy, then there are 2 places where some special
actions may be needed to make operations succeed:
1. Initial installation of docker engine First, try following the official Docker documentation for Proxy settings.
Some issues were experienced on CentOS 7 based Jumphost. Some tips are documented in section: Docker
Installation on CentOS behind http proxy below.
2. Execution of the Functest environment preparation inside the created docker container Functest needs internet
access to download some resources for some test cases. This might not work properly if the Jumphost is
connecting to internet through a http Proxy.
If that is the case, make sure the resolv.conf and the needed http_proxy and https_proxy environment variables, as well
as the ‘no_proxy’ environment variable are set correctly:
2.2. Testing User Guides
35
OPNFV Documentation, Release Danube
#
#
#
#
#
Make double sure that the 'no_proxy=...' line in the
'openstack.creds' file is commented out first. Otherwise, the
values set into the 'no_proxy' environment variable below will
be ovewrwritten, each time the command
'source ~/functest/conf/openstack.creds' is issued.
cd ~/functest/conf/
sed -i 's/export no_proxy/#export no_proxy/' openstack.creds
source ./openstack.creds
# Next calculate some IP addresses for which http_proxy
# usage should be excluded:
publicURL_IP=$(echo $OS_AUTH_URL | grep -Eo "([0-9]+\.){3}[0-9]+")
adminURL_IP=$(openstack catalog show identity | \
grep adminURL | grep -Eo "([0-9]+\.){3}[0-9]+")
export http_proxy="<your http proxy settings>"
export https_proxy="<your https proxy settings>"
export no_proxy="127.0.0.1,localhost,$publicURL_IP,$adminURL_IP"
# Ensure that "git" uses the http_proxy
# This may be needed if your firewall forbids SSL based git fetch
git config --global http.sslVerify True
git config --global http.proxy <Your http proxy settings>
Validation check: Before running ‘functest env prepare’ CLI command, make sure you can reach http and https sites
from inside the Functest docker container.
For example, try to use the nc command from inside the functest docker container:
nc -v opnfv.org 80
Connection to opnfv.org 80 port [tcp/http] succeeded!
nc -v opnfv.org 443
Connection to opnfv.org 443 port [tcp/https] succeeded!
Note: In a Jumphost node based on the CentOS family OS, the nc commands might not work. You can use the curl
command instead.
curl http://www.opnfv.org:80
</BODY></HTML>
curl https://www.opnfv.org:443
</BODY></HTML>
<HTML><HEAD><meta
<HTML><HEAD><meta
http-equiv=”content-type”
http-equiv=”content-type”
.
.
.
.
(Ignore the content. If command returns a valid HTML page, it proves the connection.)
Docker Installation on CentOS behind http proxy This section is applicable for CentOS family OS on Jumphost
which itself is behind a proxy server. In that case, the instructions below should be followed before installing the
docker engine:
1) # Make a directory '/etc/systemd/system/docker.service.d'
# if it does not exist
sudo mkdir /etc/systemd/system/docker.service.d
2) # Create a file called 'env.conf' in that directory with
# the following contents:
36
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
[Service]
EnvironmentFile=-/etc/sysconfig/docker
3) # Set up a file called 'docker' in directory '/etc/sysconfig'
# with the following contents:
HTTP_PROXY="<Your http proxy settings>"
HTTPS_PROXY="<Your https proxy settings>"
http_proxy="${HTTP_PROXY}"
https_proxy="${HTTPS_PROXY}"
4) # Reload the daemon
systemctl daemon-reload
5) # Sanity check - check the following docker settings:
systemctl show docker | grep -i env
Expected result:
---------------EnvironmentFile=/etc/sysconfig/docker (ignore_errors=yes)
DropInPaths=/etc/systemd/system/docker.service.d/env.conf
Now follow the instructions in [InstallDockerCentOS] to download and install the docker-engine. The instructions
conclude with a “test pull” of a sample “Hello World” docker container. This should now work with the above prerequisite actions.
Integration in CI
In CI we use the Docker image and execute the appropriate commands within the container from Jenkins.
Docker creation in set-functest-env builder [3]:
envs="-e INSTALLER_TYPE=${INSTALLER_TYPE} -e INSTALLER_IP=${INSTALLER_IP} -e NODE_NAME=${NODE_NAME}"
[...]
docker pull opnfv/functest:$DOCKER_TAG >/dev/null
cmd="sudo docker run -id ${envs} ${volumes} ${custom_params} ${TESTCASE_OPTIONS} opnfv/functest:${DOC
echo "Functest: Running docker run command: ${cmd}"
${cmd} >${redirect}
sleep 5
container_id=$(docker ps | grep "opnfv/functest:${DOCKER_TAG}" | awk '{print $1}' | head -1)
echo "Container ID=${container_id}"
if [ -z ${container_id} ]; then
echo "Cannot find opnfv/functest container ID ${container_id}. Please check if it is existing."
docker ps -a
exit 1
fi
echo "Starting the container: docker start ${container_id}"
docker start ${container_id}
sleep 5
docker ps >${redirect}
if [ $(docker ps | grep "opnfv/functest:${DOCKER_TAG}" | wc -l) == 0 ]; then
echo "The container opnfv/functest with ID=${container_id} has not been properly started. Exiting
exit 1
fi
cmd="python ${FUNCTEST_REPO_DIR}/functest/ci/prepare_env.py start"
echo "Executing command inside the docker: ${cmd}"
docker exec ${container_id} ${cmd}
2.2. Testing User Guides
37
OPNFV Documentation, Release Danube
Test execution in functest-all builder [3]:
branch=${GIT_BRANCH##*/}
echo "Functest: run $FUNCTEST_SUITE_NAME on branch ${branch}"
cmd="functest testcase run $FUNCTEST_SUITE_NAME"
fi
container_id=$(docker ps -a | grep opnfv/functest | awk '{print $1}' | head -1)
docker exec $container_id $cmd
ret_value=$?
exit $ret_value
Docker clean in functest-cleanup builder [3] calling docker rm and docker rmi
References
OPNFV main site
Functest page
IRC support channel: #opnfv-functest
OPNFV FUNCTEST user guide
Version history
Date
2016-08-17
2017-01-23
Ver.
1.0.0
1.0.1
Author
Juha Haapavirta Column Gaynor
Morgan Richomme
Comment
Colorado release
Adaptations for Danube
Introduction
The goal of this document is to describe the OPNFV Functest test cases and to provide a procedure to execute them.
In the OPNFV Danube system release, a Functest CLI utility is introduced for an easier execution of test procedures.
IMPORTANT: It is assumed here that the Functest Docker container is already properly deployed and that all instructions described in this guide are to be performed from inside the deployed Functest Docker container.
Overview of the Functest suites
Functest is the OPNFV project primarily targeting function testing. In the Continuous Integration pipeline, it is
launched after an OPNFV fresh installation to validate and verify the basic functions of the infrastructure.
The current list of test suites can be distributed over 5 main domains: VIM (Virtualised Infrastructure Manager),
Controllers (i.e. SDN Controllers), Features, VNF (Virtual Network Functions) and MANO stacks.
Functest test suites are also distributed in the OPNFV testing categories: healthcheck, smoke, features, components,
performance, VNF, Stress tests.
All the Healthcheck and smoke tests of a given scenario must be succesful to validate the scenario for the release.
38
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
Domain
Tier
Test case
connection _check
healthcheck
api_check
snaps_health _check
VIM
vping_ssh
smoke
vping_userdata
tempest_smoke _serial
rally_sanity
snaps_smoke
refstack _defcore
tempest_full _parallel
components
rally_full
tempest_custom
2.2. Testing User Guides
Controllers
odl
smoke
Comments
Check OpenStack connectivity through SNAPS
framework
Check OpenStack API
through SNAPS framework
basic instance creation,
check DHCP
NFV “Hello World” using
an SSH connection to a destination VM over a created
floating IP address on the
SUT Public / External network. Using the SSH connection a test script is then
copied to the destination
VM and then executed via
SSH. The script will ping
another VM on a specified
IP address over the SUT
Private Tenant network.
Uses Ping with given userdata to test intra-VM connectivity over the SUT Private Tenant network. The
correct operation of the
NOVA Metadata service is
also verified in this test.
Generate and run a relevant Tempest Test Suite in
smoke mode. The generated test set is dependent on
the OpenStack deployment
environment.
Run a subset of the OpenStack Rally Test Suite in
smoke mode
Run a subset of the OpenStack Rally Test Suite in
smoke mode
Reference RefStack suite
tempest selection for NFV
Generate and run a full set
of the OpenStack Tempest
Test Suite. See the OpenStack reference test suite
[2]. The generated test set
is dependent on the OpenStack deployment environment.
Run the OpenStack testing
tool benchmarking OpenStack modules See the
Rally documents [3].
Allow to run a customized
list of Tempest cases39
Opendaylight Test suite
Limited test suite to check
the basic neutron (Layer
OPNFV Documentation, Release Danube
As shown in the above table, Functest is structured into different ‘domains’, ‘tiers’ and ‘test cases’. Each ‘test case’
usually represents an actual ‘Test Suite’ comprised -in turn- of several test cases internally.
Test cases also have an implicit execution order. For example, if the early ‘healthcheck’ Tier testcase fails, or if there
are any failures in the ‘smoke’ Tier testcases, there is little point to launch a full testcase execution round.
In Danube, we merged smoke and sdn controller tiers in smoke tier.
An overview of the Functest Structural Concept is depicted graphically below:
Some of the test cases are developed by Functest team members, whereas others are integrated from upstream communities or other OPNFV projects. For example, Tempest is the OpenStack integration test suite and Functest is in
charge of the selection, integration and automation of those tests that fit suitably to OPNFV.
The Tempest test suite is the default OpenStack smoke test suite but no new test cases have been created in OPNFV
Functest.
The results produced by the tests run from CI are pushed and collected into a NoSQL database. The goal is to populate
the database with results from different sources and scenarios and to show them on a Functest Dashboard. A screenshot
of a live Functest Dashboard is shown below:
Basic components (VIM, SDN controllers) are tested through their own suites. Feature projects also provide their own
test suites with different ways of running their tests.
The notion of domain has been introduced in the description of the test cases stored in the Database. This parameters
as well as possible tags can be used for the Test case catalog.
vIMS test case was integrated to demonstrate the capability to deploy a relatively complex NFV scenario on top of the
OPNFV infrastructure.
40
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
2.2. Testing User Guides
41
OPNFV Documentation, Release Danube
Functest considers OPNFV as a black box. As of Danube release the OPNFV offers a lot of potential combinations:
• 3 controllers (OpenDaylight, ONOS, OpenContrail)
• 4 installers (Apex, Compass, Fuel, Joid)
Most of the tests are runnable by any combination, but some tests might have restrictions imposed by the utilized
installers or due to the available deployed features. The system uses the environment variables (INSTALLER_IP and
DEPLOY_SCENARIO) to automatically determine the valid test cases, for each given environment.
A convenience Functest CLI utility is also available to simplify setting up the Functest evironment, management of
the OpenStack environment (e.g. resource clean-up) and for executing tests. The Functest CLI organised the testcase
into logical Tiers, which contain in turn one or more testcases. The CLI allows execution of a single specified testcase,
all test cases in a specified Tier, or the special case of execution of ALL testcases. The Functest CLI is introduced in
more detail in the section Executing the functest suites of this document.
The different test cases are described in the remaining sections of this document.
VIM (Virtualized Infrastructure Manager)
Healthcheck tests
project.
In Danube, healthcheck tests have been refactored and rely on SNAPS, a OPNFV middleware
SNAPS stands for “SDN/NFV Application development Platform and Stack”. SNAPS is an object-oriented OpenStack
library packaged with tests that exercise OpenStack. More information on SNAPS can be found in [13]
Three tests are declared as healthcheck tests and can be used for gating by the installer, they cover functionally the
tests previously done by healthcheck test case.
The tests are:
• connection_check
• api_check
• snaps_health_check
Connection_check consists in 9 test cases (test duration < 5s) checking the connectivity with Glance, Keystone, Neutron, Nova and the external network.
Api_check verifies the retrieval of OpenStack clients: Keystone, Glance, Neutron and Nova and may perform some
simple queries. When the config value of snaps.use_keystone is True, functest must have access to the cloud’s private
network. This suite consists in 49 tests (test duration < 2 minutes).
snaps_health_check creates instance, allocate floating IP, connect to the VM. This test replaced the previous Colorado
healthcheck test.
Self-obviously, successful completion of the ‘healthcheck’ testcase is a necessary pre-requisite for the execution of all
other test Tiers.
vPing_ssh Given the script ping.sh:
#!/bin/sh
while true; do
ping -c 1 $1 2>&1 >/dev/null
RES=$?
if [ "Z$RES" = "Z0" ] ; then
echo 'vPing OK'
break
else
42
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
echo 'vPing KO'
fi
sleep 1
done
The goal of this test is to establish an SSH connection using a floating IP on the Public/External network and verify
that 2 instances can talk over a Private Tenant network:
vPing_ssh test case
+-------------+
+-------------+
|
|
|
|
|
| Boot VM1 with IP1 |
|
|
+------------------->|
|
|
Tester
|
|
System
|
|
| Boot VM2
|
Under
|
|
+------------------->|
Test
|
|
|
|
|
|
| Create floating IP |
|
|
+------------------->|
|
|
|
|
|
|
| Assign floating IP |
|
|
| to VM2
|
|
|
+------------------->|
|
|
|
|
|
|
| Establish SSH
|
|
|
| connection to VM2 |
|
|
| through floating IP|
|
|
+------------------->|
|
|
|
|
|
|
| SCP ping.sh to VM2 |
|
|
+------------------->|
|
|
|
|
|
|
| VM2 executes
|
|
|
| ping.sh to VM1
|
|
|
+------------------->|
|
|
|
|
|
|
|
If ping:
|
|
|
|
exit OK
|
|
|
|
else (timeout): |
|
|
|
exit Failed
|
|
|
|
|
|
+-------------+
+-------------+
This test can be considered as an “Hello World” example. It is the first basic use case which must work on any
deployment.
vPing_userdata This test case is similar to vPing_ssh but without the use of Floating IPs and the Public/External
network to transfer the ping script. Instead, it uses Nova metadata service to pass it to the instance at booting time. As
vPing_ssh, it checks that 2 instances can talk to each other on a Private Tenant network:
vPing_userdata test case
+-------------+
+-------------+
|
|
|
|
|
| Boot VM1 with IP1 |
|
|
+------------------->|
|
|
|
|
|
|
| Boot VM2 with
|
|
|
| ping.sh as userdata|
|
2.2. Testing User Guides
43
OPNFV Documentation, Release Danube
|
| with IP1 as $1.
|
|
|
+------------------->|
|
|
Tester
|
|
System
|
|
| VM2 exeutes ping.sh|
Under
|
|
| (ping IP1)
|
Test
|
|
+------------------->|
|
|
|
|
|
|
| Monitor nova
|
|
|
| console-log VM 2 |
|
|
|
If ping:
|
|
|
|
exit OK
|
|
|
|
else (timeout) |
|
|
|
exit Failed
|
|
|
|
|
|
+-------------+
+-------------+
When the second VM boots it will execute the script passed as userdata automatically. The ping will be detected by
periodically capturing the output in the console-log of the second VM.
Tempest Tempest [2] is the reference OpenStack Integration test suite. It is a set of integration tests to be run against
a live OpenStack cluster. Tempest has suites of tests for:
• OpenStack API validation
• Scenarios
• Other specific tests useful in validating an OpenStack deployment
Functest uses Rally [3] to run the Tempest suite. Rally generates automatically the Tempest configuration file tempest.conf. Before running the actual test cases, Functest creates the needed resources (user, tenant) and updates the
appropriate parameters into the configuration file.
When the Tempest suite is executed, each test duration is measured and the full console output is stored to a log file
for further analysis.
The Tempest testcases are distributed accross two Tiers:
• Smoke Tier - Test Case ‘tempest_smoke_serial’
• Components Tier - Test case ‘tempest_full_parallel’
NOTE: Test case ‘tempest_smoke_serial’ executes a defined set of tempest smoke tests with a single thread (i.e.
serial mode). Test case ‘tempest_full_parallel’ executes all defined Tempest tests using several concurrent threads (i.e.
parallel mode). The number of threads activated corresponds to the number of available logical CPUs.
The goal of the Tempest test suite is to check the basic functionalities of the different OpenStack components on an
OPNFV fresh installation, using the corresponding REST API interfaces.
Rally bench test suites Rally [3] is a benchmarking tool that answers the question:
How does OpenStack work at scale?
The goal of this test suite is to benchmark all the different OpenStack modules and get significant figures that could
help to define Telco Cloud KPIs.
The OPNFV Rally scenarios are based on the collection of the actual Rally scenarios:
• authenticate
• cinder
• glance
44
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
• heat
• keystone
• neutron
• nova
• quotas
• requests
A basic SLA (stop test on errors) has been implemented.
The Rally testcases are distributed accross two Tiers:
• Smoke Tier - Test Case ‘rally_sanity’
• Components Tier - Test case ‘rally_full’
NOTE: Test case ‘rally_sanity’ executes a limited number of Rally smoke test cases. Test case ‘rally_full’ executes
the full defined set of Rally tests.
Refstack-client to run Defcore testcases Refstack-client [8] is a command line utility that allows you to execute
Tempest test runs based on configurations you specify. It is the official tool to run Defcore [9] testcases, which focuses
on testing interoperability between OpenStack clouds.
Refstack-client is integrated in Functest, consumed by Dovetail, which intends to define and provide a set of OPNFV
related validation criteria that will provide input for the evaluation of the use of OPNFV trademarks. This progress is
under the guideline of Compliance Verification Program(CVP).
Defcore testcases Danube Release
Set of DefCore tempest test cases not flagged and required. According to [10], some tests are still flagged
due to outstanding bugs in the Tempest library, particularly tests that require SSH. Refstack developers are
working on correcting these bugs upstream. Please note that although some tests are flagged because of
bugs, there is still an expectation that the capabilities covered by the tests are available. It only contains
Openstack core compute (no object storage). The approved guidelines (2016.08) are valid for Kilo, Liberty,
Mitaka and Newton releases of OpenStack. The list can be generated using the Rest API from RefStack project:
https://refstack.openstack.org/api/v1/guidelines/2016.08/tests?target=compute&type=required&alias=true&flag=false
Running methods Two running methods are provided after refstack-client integrated into Functest, Functest command line and manually, respectively.
By default, for Defcore test cases run by Functest command line, are run followed with automatically generated
configuration file, i.e., refstack_tempest.conf. In some circumstances, the automatic configuration file may not quite
satisfied with the SUT, Functest also inherits the refstack-client command line and provides a way for users to set its
configuration file according to its own SUT manually.
command line
Inside the Functest container, first to prepare Functest environment:
cd /home/opnfv/repos/functest
pip install -e .
functest env prepare
then to run default defcore testcases by using refstack-client:
2.2. Testing User Guides
45
OPNFV Documentation, Release Danube
functest testcase run refstack_defcore
In OPNFV Continuous Integration(CI) system, the command line method is used.
manually
Inside the Functest container, first to prepare the refstack virtualenv:
cd /home/opnfv/repos/refstack-client
source .venv/bin/activate
then prepare the tempest configuration file and the testcases want to run with the SUT, run the testcases with:
./refstack-client test -c <Path of the tempest configuration file to use> -v --test-list <Path or URL
using help for more information:
./refstack-client --help
./refstack-client test --help
Reference tempest configuration command line method
When command line method is used, the default tempest configuration file is generated by Rally.
manually
When running manually is used, recommended way to generate tempest configuration file is:
cd /home/opnfv/repos/functest/functest/opnfv_tests/openstack/refstack_client
python tempest_conf.py
a file called tempest.conf is stored in the current path by default, users can do some adjustment according to the SUT:
vim refstack_tempest.conf
a reference article can be used [15].
snaps_smoke This test case contains tests that setup and destroy environments with VMs with and without Floating
IPs with a newly created user and project. Set the config value snaps.use_floating_ips (True|False) to toggle this
functionality. When the config value of snaps.use_keystone is True, Functest must have access the cloud’s private
network. This suite consists in 38 tests (test duration < 10 minutes)
SDN Controllers There are currently 3 available controllers:
• OpenDaylight (ODL)
• ONOS
• OpenContrail (OCL)
OpenDaylight The OpenDaylight (ODL) test suite consists of a set of basic tests inherited from the ODL project
using the Robot [11] framework. The suite verifies creation and deletion of networks, subnets and ports with OpenDaylight and Neutron.
The list of tests can be described as follows:
• Basic Restconf test cases * Connect to Restconf URL * Check the HTTP code status
• Neutron Reachability test cases * Get the complete list of neutron resources (networks, subnets, ports)
46
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
• Neutron Network test cases * Check OpenStack networks * Check OpenDaylight networks * Create a new
network via OpenStack and check the HTTP status code returned by Neutron * Check that the network has also
been successfully created in OpenDaylight
• Neutron Subnet test cases * Check OpenStack subnets * Check OpenDaylight subnets * Create a new subnet
via OpenStack and check the HTTP status code returned by Neutron * Check that the subnet has also been
successfully created in OpenDaylight
• Neutron Port test cases * Check OpenStack Neutron for known ports * Check OpenDaylight ports * Create a
new port via OpenStack and check the HTTP status code returned by Neutron * Check that the new port has
also been successfully created in OpenDaylight
• Delete operations * Delete the port previously created via OpenStack * Check that the port has been also succesfully deleted in OpenDaylight * Delete previously subnet created via OpenStack * Check that the subnet has
also been successfully deleted in OpenDaylight * Delete the network created via OpenStack * Check that the
network has also been succesfully deleted in OpenDaylight
Note: the checks in OpenDaylight are based on the returned HTTP status code returned by OpenDaylight.
ONOS TestON Framework is used to test the ONOS SDN controller functions. The test cases deal with L2 and L3
functions. The ONOS test suite can be run on any ONOS compliant scenario.
The test cases are described as follows:
• onosfunctest: The main executable file contains the initialization of the docker environment and functions called
by FUNCvirNetNB and FUNCvirNetNBL3
• FUNCvirNetNB
– Create Network: Post Network data and check it in ONOS
– Update Network: Update the Network and compare it in ONOS
– Delete Network: Delete the Network and check if it’s NULL in ONOS or not
– Create Subnet: Post Subnet data and check it in ONOS
– Update Subnet: Update the Subnet and compare it in ONOS
– Delete Subnet: Delete the Subnet and check if it’s NULL in ONOS or not
– Create Port: Post Port data and check it in ONOS
– Update Port: Update the Port and compare it in ONOS
– Delete Port: Delete the Port and check if it’s NULL in ONOS or not
• FUNCvirNetNBL3
– Create Router: Post data for create Router and check it in ONOS
– Update Router: Update the Router and compare it in ONOS
– Delete Router: Delete the Router data and check it in ONOS
– Create RouterInterface: Post Router Interface data to an existing Router and check it in ONOS
– Delete RouterInterface: Delete the RouterInterface and check the Router
– Create FloatingIp: Post data for create FloatingIp and check it in ONOS
– Update FloatingIp: Update the FloatingIp and compare it in ONOS
– Delete FloatingIp: Delete the FloatingIp and check that it is ‘NULL’ in ONOS
2.2. Testing User Guides
47
OPNFV Documentation, Release Danube
– Create External Gateway: Post data to create an External Gateway for an existing Router and check it in
ONOS
– Update External Gateway: Update the External Gateway and compare the change
– Delete External Gateway: Delete the External Gateway and check that it is ‘NULL’ in ONOS
Features In Danube, Functest supports the integration of:
• barometer
• bgpvpn
• doctor
• domino
• fds
• multisite
• netready
• odl-sfc
• promise
• security_scan
Note: copper is not supported in Danube.
Please refer to the dedicated feature user guides for details.
VNF
cloudify_ims The IP Multimedia Subsystem or IP Multimedia Core Network Subsystem (IMS) is an architectural
framework for delivering IP multimedia services.
vIMS has been integrated in Functest to demonstrate the capability to deploy a relatively complex NFV scenario on
the OPNFV platform. The deployment of a complete functional VNF allows the test of most of the essential functions
needed for a NFV platform.
The goal of this test suite consists of:
• deploy a VNF orchestrator (Cloudify)
• deploy a Clearwater vIMS (IP Multimedia Subsystem) VNF from this orchestrator based on a TOSCA blueprint
defined in [5]
• run suite of signaling tests on top of this VNF
The Clearwater architecture is described as follows:
orchestra_ims Orchestra test case deals with the deployment of OpenIMS with OpenBaton orchestrator.
parser See parser user guide for details: [12]
48
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
vyos-vrouter This test case deals with the deployment and the test of vyos vrouter with Cloudify orchestrator. The
test case can do testing for interchangeability of BGP Protocol using vyos.
The Workflow is as follows:
• Deploy Deploy VNF Testing topology by Cloudify using blueprint.
• Configuration Setting configuration to Target VNF and reference VNF using ssh
• Run Execution of test command for test item written YAML format file. Check VNF status and behavior.
• Reporting Output of report based on result using JSON format.
The vyos-vrouter architecture is described in [14]
Executing the functest suites
Manual testing
This section assumes the following:
• The Functest Docker container is running
• The docker prompt is shown
• The Functest environment is ready (Functest CLI command ‘functest env prepare’ has been executed)
If any of the above steps are missing please refer to the Functest Config Guide as they are a prerequisite and all the
commands explained in this section must be performed inside the container.
The Functest CLI offers two commands (functest tier ...) and (functest testcase ... ) for the execution of Test Tiers or
Test Cases:
2.2. Testing User Guides
49
OPNFV Documentation, Release Danube
[email protected]:~/repos/functest/ci# functest tier --help
Usage: functest tier [OPTIONS] COMMAND [ARGS]...
Options:
-h, --help
Show this message and exit.
Commands:
get-tests Prints the tests in a tier.
list
Lists the available tiers.
run
Executes all the tests within a tier.
show
Shows information about a tier.
[email protected]:~/repos/functest/ci# functest testcase --help
Usage: functest testcase [OPTIONS] COMMAND [ARGS]...
Options:
-h, --help
Show this message and exit.
Commands:
list Lists the available testcases.
run
Executes a test case.
show Shows information about a test case.
More details on the existing Tiers and Test Cases can be seen with the ‘list’ command:
[email protected]:~/repos/functest/ci# functest tier list
- 0. healthcheck:
['connection_check', 'api_check', 'snaps_health_check',]
- 1. smoke:
['vping_ssh', 'vping_userdata', 'tempest_smoke_serial', 'odl', 'rally_sanity', 'refstack_d
- 2. features:
['doctor', 'domino', 'promise', security_scan']
- 3. components:
['tempest_full_parallel', 'rally_full']
- 4. vnf:
['cloudify_ims', 'orchestra_ims', 'vyos_vrouter']
and
[email protected]:~/repos/functest/ci# functest testcase list
api_check
connection_check
snaps_health_check
vping_ssh
vping_userdata
snaps_smoke
refstack_defcore
tempest_smoke_serial
rally_sanity
odl
tempest_full_parallel
rally_full
vyos_vrouter
Note the list of test cases depend on the installer and the scenario.
More specific details on specific Tiers or Test Cases can be seen wih the ‘show’ command:
50
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
[email protected]:~/repos/functest/ci# functest tier show smoke
+======================================================================+
| Tier: smoke
|
+======================================================================+
| Order: 1
|
| CI Loop: (daily)|(weekly)
|
| Description:
|
|
Set of basic Functional tests to validate the OpenStack
|
|
deployment.
|
| Test cases:
|
|
- vping_ssh
|
|
- vping_userdata
|
|
- tempest_smoke_serial
|
|
- rally_sanity
|
|
|
+----------------------------------------------------------------------+
and
[email protected]:~/repos/functest/ci# functest testcase show tempest_smoke_serial
+======================================================================+
| Testcase: tempest_smoke_serial
|
+======================================================================+
| Description:
|
|
This test case runs the smoke subset of the OpenStack Tempest
|
|
suite. The list of test cases is generated by Tempest
|
|
automatically and depends on the parameters of the OpenStack
|
|
deplopyment.
|
| Dependencies:
|
|
- Installer:
|
|
- Scenario :
|
|
|
+----------------------------------------------------------------------+
To execute a Test Tier or Test Case, the ‘run’ command is used:
[email protected]:~/repos/functest/ci# functest tier run healthcheck
2017-03-21 13:34:21,400 - run_tests - INFO - ############################################
2017-03-21 13:34:21,400 - run_tests - INFO - Running tier 'healthcheck'
2017-03-21 13:34:21,400 - run_tests - INFO - ############################################
2017-03-21 13:34:21,401 - run_tests - INFO -
2017-03-21 13:34:21,401 - run_tests - INFO - ============================================
2017-03-21 13:34:21,401 - run_tests - INFO - Running test case 'connection_check'...
2017-03-21 13:34:21,401 - run_tests - INFO - ============================================
test_glance_connect_fail (snaps.openstack.utils.tests.glance_utils_tests.GlanceSmokeTests) ... ok
test_glance_connect_success (snaps.openstack.utils.tests.glance_utils_tests.GlanceSmokeTests) ... ok
test_keystone_connect_fail (snaps.openstack.utils.tests.keystone_utils_tests.KeystoneSmokeTests) ...
test_keystone_connect_success (snaps.openstack.utils.tests.keystone_utils_tests.KeystoneSmokeTests) .
test_neutron_connect_fail (snaps.openstack.utils.tests.neutron_utils_tests.NeutronSmokeTests) ... ok
test_neutron_connect_success (snaps.openstack.utils.tests.neutron_utils_tests.NeutronSmokeTests) ...
test_retrieve_ext_network_name (snaps.openstack.utils.tests.neutron_utils_tests.NeutronSmokeTests) ..
test_nova_connect_fail (snaps.openstack.utils.tests.nova_utils_tests.NovaSmokeTests) ... ok
test_nova_connect_success (snaps.openstack.utils.tests.nova_utils_tests.NovaSmokeTests) ... ok
---------------------------------------------------------------------Ran 9 tests in 3.768s
OK
2.2. Testing User Guides
51
OPNFV Documentation, Release Danube
2017-03-21
2017-03-21
2017-03-21
2017-03-21
13:34:26,570
13:34:26,918
13:34:26,918
13:34:26,918
-
functest.core.testcase_base - INFO - connection_check OK
functest.core.testcase_base - INFO - The results were successfully pushed t
run_tests - INFO - Test execution time: 00:05
run_tests - INFO -
2017-03-21 13:34:26,918 - run_tests - INFO - ============================================
2017-03-21 13:34:26,918 - run_tests - INFO - Running test case 'api_check'...
2017-03-21 13:34:26,919 - run_tests - INFO - ============================================
test_create_project_minimal (snaps.openstack.utils.tests.keystone_utils_tests.KeystoneUtilsTests) ...
test_create_user_minimal (snaps.openstack.utils.tests.keystone_utils_tests.KeystoneUtilsTests) ... ok
test_create_delete_user (snaps.openstack.tests.create_user_tests.CreateUserSuccessTests) ... ok
test_create_user (snaps.openstack.tests.create_user_tests.CreateUserSuccessTests) ... ok
test_create_user_2x (snaps.openstack.tests.create_user_tests.CreateUserSuccessTests) ...
2017-03-21 13:34:32,684 - create_user - INFO - Found user with name - CreateUserSuccessTests-7e741e11
test_create_delete_project (snaps.openstack.tests.create_project_tests.CreateProjectSuccessTests) ...
test_create_project (snaps.openstack.tests.create_project_tests.CreateProjectSuccessTests) ... ok
test_create_project_2x (snaps.openstack.tests.create_project_tests.CreateProjectSuccessTests) ...
2017-03-21 13:34:35,922 - create_image - INFO - Found project with name - CreateProjectSuccessTests-b
test_create_project_sec_grp_one_user (snaps.openstack.tests.create_project_tests.CreateProjectUserTes
2017-03-21 13:34:37,907 - OpenStackSecurityGroup - INFO - Creating security group CreateProjectUserTe
2017-03-21 13:34:37,907 - neutron_utils - INFO - Retrieving security group with name - CreateProjectU
2017-03-21 13:34:38,376 - neutron_utils - INFO - Creating security group with name - CreateProjectUse
2017-03-21 13:34:38,716 - neutron_utils - INFO - Retrieving security group rules associate with the s
2017-03-21 13:34:38,762 - neutron_utils - INFO - Retrieving security group with ID - 821419cb-c54c-41
2017-03-21 13:34:38,886 - neutron_utils - INFO - Retrieving security group with ID - 821419cb-c54c-41
2017-03-21 13:34:39,000 - neutron_utils - INFO - Retrieving security group with name - CreateProjectU
2017-03-21 13:34:39,307 - neutron_utils - INFO - Deleting security group rule with ID - d85fafc0-9649
2017-03-21 13:34:39,531 - neutron_utils - INFO - Deleting security group rule with ID - 69d79c09-bc3b
2017-03-21 13:34:39,762 - neutron_utils - INFO - Deleting security group with name - CreateProjectUse
test_create_project_sec_grp_two_users (snaps.openstack.tests.create_project_tests.CreateProjectUserTe
2017-03-21 13:34:43,511 - OpenStackSecurityGroup - INFO - Creating security group CreateProjectUserTe
2017-03-21 13:34:43,511 - neutron_utils - INFO - Retrieving security group with name - CreateProjectU
2017-03-21 13:34:44,090 - neutron_utils - INFO - Creating security group with name - CreateProjectUse
2017-03-21 13:34:44,784 - neutron_utils - INFO - Retrieving security group rules associate with the s
2017-03-21 13:34:44,864 - neutron_utils - INFO - Retrieving security group with ID - 780193e4-9bd2-4f
2017-03-21 13:34:45,233 - neutron_utils - INFO - Retrieving security group with ID - 780193e4-9bd2-4f
2017-03-21 13:34:45,332 - neutron_utils - INFO - Retrieving security group with name - CreateProjectU
2017-03-21 13:34:45,779 - OpenStackSecurityGroup - INFO - Creating security group CreateProjectUserTe
2017-03-21 13:34:45,779 - neutron_utils - INFO - Retrieving security group with name - CreateProjectU
2017-03-21 13:34:46,112 - neutron_utils - INFO - Retrieving security group rules associate with the s
2017-03-21 13:34:46,184 - neutron_utils - INFO - Retrieving security group with ID - 780193e4-9bd2-4f
2017-03-21 13:34:46,296 - neutron_utils - INFO - Retrieving security group with ID - 780193e4-9bd2-4f
2017-03-21 13:34:46,387 - neutron_utils - INFO - Deleting security group rule with ID - 2320a573-ec56
2017-03-21 13:34:46,636 - neutron_utils - INFO - Deleting security group rule with ID - 6186282b-db37
2017-03-21 13:34:46,780 - neutron_utils - INFO - Deleting security group with name - CreateProjectUse
2017-03-21 13:34:47,006 - neutron_utils - INFO - Deleting security group rule with ID - 2320a573-ec56
2017-03-21 13:34:47,072 - OpenStackSecurityGroup - WARNING - Rule not found, cannot delete - Security
Neutron server returns request_ids: ['req-d74eb2e2-b26f-4236-87dc-7255866141d9']
2017-03-21 13:34:47,072 - neutron_utils - INFO - Deleting security group rule with ID - 6186282b-db37
2017-03-21 13:34:47,118 - OpenStackSecurityGroup - WARNING - Rule not found, cannot delete - Security
Neutron server returns request_ids: ['req-8c0a5a24-be90-4844-a9ed-2a85cc6f59a5']
2017-03-21 13:34:47,118 - neutron_utils - INFO - Deleting security group with name - CreateProjectUse
2017-03-21 13:34:47,172 - OpenStackSecurityGroup - WARNING - Security Group not found, cannot delete
Neutron server returns request_ids: ['req-c6e1a6b5-43e0-4d46-bb68-c2e1672d4d21'] ok
test_create_image_minimal_file (snaps.openstack.utils.tests.glance_utils_tests.GlanceUtilsTests) ...
test_create_image_minimal_url (snaps.openstack.utils.tests.glance_utils_tests.GlanceUtilsTests) ... o
test_create_network (snaps.openstack.utils.tests.neutron_utils_tests.NeutronUtilsNetworkTests) ...
2017-03-21 13:35:22,275 - neutron_utils - INFO - Creating network with name NeutronUtilsNetworkTests-
52
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
2017-03-21 13:35:23,965 - neutron_utils - INFO - Deleting network with name NeutronUtilsNetworkTeststest_create_network_empty_name (snaps.openstack.utils.tests.neutron_utils_tests.NeutronUtilsNetworkTe
test_create_network_null_name (snaps.openstack.utils.tests.neutron_utils_tests.NeutronUtilsNetworkTes
test_create_subnet (snaps.openstack.utils.tests.neutron_utils_tests.NeutronUtilsSubnetTests) ...
2017-03-21 13:35:25,495 - neutron_utils - INFO - Creating network with name NeutronUtilsSubnetTests-4
2017-03-21 13:35:26,841 - neutron_utils - INFO - Creating subnet with name NeutronUtilsSubnetTests-4f
2017-03-21 13:35:28,311 - neutron_utils - INFO - Deleting subnet with name NeutronUtilsSubnetTests-4f
2017-03-21 13:35:29,585 - neutron_utils - INFO - Deleting network with name NeutronUtilsSubnetTests-4
test_create_subnet_empty_cidr (snaps.openstack.utils.tests.neutron_utils_tests.NeutronUtilsSubnetTest
2017-03-21 13:35:31,013 - neutron_utils - INFO - Creating network with name NeutronUtilsSubnetTests-4
2017-03-21 13:35:31,652 - neutron_utils - INFO - Deleting network with name NeutronUtilsSubnetTests-4
test_create_subnet_empty_name (snaps.openstack.utils.tests.neutron_utils_tests.NeutronUtilsSubnetTest
2017-03-21 13:35:32,379 - neutron_utils - INFO - Creating network with name NeutronUtilsSubnetTests-1
2017-03-21 13:35:33,516 - neutron_utils - INFO - Creating subnet with name NeutronUtilsSubnetTests-10
2017-03-21 13:35:34,160 - neutron_utils - INFO - Deleting network with name NeutronUtilsSubnetTests-1
test_create_subnet_null_cidr (snaps.openstack.utils.tests.neutron_utils_tests.NeutronUtilsSubnetTests
2017-03-21 13:35:35,784 - neutron_utils - INFO - Creating network with name NeutronUtilsSubnetTests-1
2017-03-21 13:35:36,367 - neutron_utils - INFO - Deleting network with name NeutronUtilsSubnetTests-1
test_create_subnet_null_name (snaps.openstack.utils.tests.neutron_utils_tests.NeutronUtilsSubnetTests
2017-03-21 13:35:37,055 - neutron_utils - INFO - Creating network with name NeutronUtilsSubnetTests-0
2017-03-21 13:35:37,691 - neutron_utils - INFO - Deleting network with name NeutronUtilsSubnetTests-0
test_add_interface_router (snaps.openstack.utils.tests.neutron_utils_tests.NeutronUtilsRouterTests) .
2017-03-21 13:35:38,994 - neutron_utils - INFO - Creating network with name NeutronUtilsRouterTests-4
2017-03-21 13:35:40,311 - neutron_utils - INFO - Creating subnet with name NeutronUtilsRouterTests-43
2017-03-21 13:35:41,713 - neutron_utils - INFO - Creating router with name - NeutronUtilsRouterTests2017-03-21 13:35:44,131 - neutron_utils - INFO - Adding interface to router with name NeutronUtilsRou
2017-03-21 13:35:45,725 - neutron_utils - INFO - Removing router interface from router named NeutronU
2017-03-21 13:35:47,464 - neutron_utils - INFO - Deleting router with name - NeutronUtilsRouterTests2017-03-21 13:35:48,670 - neutron_utils - INFO - Deleting subnet with name NeutronUtilsRouterTests-43
2017-03-21 13:35:50,921 - neutron_utils - INFO - Deleting network with name NeutronUtilsRouterTests-4
test_add_interface_router_null_router (snaps.openstack.utils.tests.neutron_utils_tests.NeutronUtilsRo
2017-03-21 13:35:52,230 - neutron_utils - INFO - Creating network with name NeutronUtilsRouterTests-1
2017-03-21 13:35:53,662 - neutron_utils - INFO - Creating subnet with name NeutronUtilsRouterTests-1f
2017-03-21 13:35:55,203 - neutron_utils - INFO - Deleting subnet with name NeutronUtilsRouterTests-1f
2017-03-21 13:35:55,694 - neutron_utils - INFO - Deleting network with name NeutronUtilsRouterTests-1
test_add_interface_router_null_subnet (snaps.openstack.utils.tests.neutron_utils_tests.NeutronUtilsRo
2017-03-21 13:35:57,392 - neutron_utils - INFO - Creating network with name NeutronUtilsRouterTests-2
2017-03-21 13:35:58,215 - neutron_utils - INFO - Creating router with name - NeutronUtilsRouterTests2017-03-21 13:36:00,369 - neutron_utils - INFO - Adding interface to router with name NeutronUtilsRou
2017-03-21 13:36:00,369 - neutron_utils - INFO - Deleting router with name - NeutronUtilsRouterTests2017-03-21 13:36:02,742 - neutron_utils - INFO - Deleting network with name NeutronUtilsRouterTests-2
test_create_port (snaps.openstack.utils.tests.neutron_utils_tests.NeutronUtilsRouterTests) ...
2017-03-21 13:36:05,010 - neutron_utils - INFO - Creating network with name NeutronUtilsRouterTests-d
2017-03-21 13:36:05,996 - neutron_utils - INFO - Creating subnet with name NeutronUtilsRouterTests-dd
2017-03-21 13:36:09,103 - neutron_utils - INFO - Creating port for network with name - NeutronUtilsRo
2017-03-21 13:36:10,312 - neutron_utils - INFO - Deleting port with name NeutronUtilsRouterTests-dde0
2017-03-21 13:36:11,045 - neutron_utils - INFO - Deleting subnet with name NeutronUtilsRouterTests-dd
2017-03-21 13:36:14,265 - neutron_utils - INFO - Deleting network with name NeutronUtilsRouterTests-d
test_create_port_empty_name (snaps.openstack.utils.tests.neutron_utils_tests.NeutronUtilsRouterTests)
2017-03-21 13:36:16,250 - neutron_utils - INFO - Creating network with name NeutronUtilsRouterTests-b
2017-03-21 13:36:16,950 - neutron_utils - INFO - Creating subnet with name NeutronUtilsRouterTests-b9
2017-03-21 13:36:17,798 - neutron_utils - INFO - Creating port for network with name - NeutronUtilsRo
2017-03-21 13:36:18,544 - neutron_utils - INFO - Deleting port with name NeutronUtilsRouterTests-b986
2017-03-21 13:36:19,582 - neutron_utils - INFO - Deleting subnet with name NeutronUtilsRouterTests-b9
2017-03-21 13:36:21,606 - neutron_utils - INFO - Deleting network with name NeutronUtilsRouterTests-b
test_create_port_invalid_ip (snaps.openstack.utils.tests.neutron_utils_tests.NeutronUtilsRouterTests)
2017-03-21 13:36:23,779 - neutron_utils - INFO - Creating network with name NeutronUtilsRouterTests-7
2017-03-21 13:36:25,201 - neutron_utils - INFO - Creating subnet with name NeutronUtilsRouterTests-7a
2.2. Testing User Guides
53
OPNFV Documentation, Release Danube
2017-03-21 13:36:25,599 - neutron_utils - INFO - Deleting subnet with name NeutronUtilsRouterTests-7a
2017-03-21 13:36:26,220 - neutron_utils - INFO - Deleting network with name NeutronUtilsRouterTests-7
test_create_port_invalid_ip_to_subnet (snaps.openstack.utils.tests.neutron_utils_tests.NeutronUtilsRo
2017-03-21 13:36:27,112 - neutron_utils - INFO - Creating network with name NeutronUtilsRouterTests-c
2017-03-21 13:36:28,720 - neutron_utils - INFO - Creating subnet with name NeutronUtilsRouterTests-c0
2017-03-21 13:36:29,457 - neutron_utils - INFO - Deleting subnet with name NeutronUtilsRouterTests-c0
2017-03-21 13:36:29,909 - neutron_utils - INFO - Deleting network with name NeutronUtilsRouterTests-c
test_create_port_null_ip (snaps.openstack.utils.tests.neutron_utils_tests.NeutronUtilsRouterTests) ..
2017-03-21 13:36:31,037 - neutron_utils - INFO - Creating network with name NeutronUtilsRouterTests-9
2017-03-21 13:36:31,695 - neutron_utils - INFO - Creating subnet with name NeutronUtilsRouterTests-9a
2017-03-21 13:36:32,305 - neutron_utils - INFO - Deleting subnet with name NeutronUtilsRouterTests-9a
2017-03-21 13:36:33,553 - neutron_utils - INFO - Deleting network with name NeutronUtilsRouterTests-9
test_create_port_null_name (snaps.openstack.utils.tests.neutron_utils_tests.NeutronUtilsRouterTests)
2017-03-21 13:36:34,593 - neutron_utils - INFO - Creating network with name NeutronUtilsRouterTests-4
2017-03-21 13:36:35,217 - neutron_utils - INFO - Creating subnet with name NeutronUtilsRouterTests-42
2017-03-21 13:36:36,648 - neutron_utils - INFO - Deleting subnet with name NeutronUtilsRouterTests-42
2017-03-21 13:36:37,251 - neutron_utils - INFO - Deleting network with name NeutronUtilsRouterTests-4
test_create_port_null_network_object (snaps.openstack.utils.tests.neutron_utils_tests.NeutronUtilsRou
2017-03-21 13:36:37,885 - neutron_utils - INFO - Creating network with name NeutronUtilsRouterTests-6
2017-03-21 13:36:38,468 - neutron_utils - INFO - Creating subnet with name NeutronUtilsRouterTests-61
2017-03-21 13:36:40,005 - neutron_utils - INFO - Deleting subnet with name NeutronUtilsRouterTests-61
2017-03-21 13:36:41,637 - neutron_utils - INFO - Deleting network with name NeutronUtilsRouterTests-6
test_create_router_empty_name (snaps.openstack.utils.tests.neutron_utils_tests.NeutronUtilsRouterTest
test_create_router_null_name (snaps.openstack.utils.tests.neutron_utils_tests.NeutronUtilsRouterTests
test_create_router_simple (snaps.openstack.utils.tests.neutron_utils_tests.NeutronUtilsRouterTests) .
2017-03-21 13:36:43,424 - neutron_utils - INFO - Creating router with name - NeutronUtilsRouterTests2017-03-21 13:36:45,013 - neutron_utils - INFO - Deleting router with name - NeutronUtilsRouterTeststest_create_router_with_public_interface (snaps.openstack.utils.tests.neutron_utils_tests.NeutronUtil
2017-03-21 13:36:47,829 - neutron_utils - INFO - Creating router with name - NeutronUtilsRouterTests2017-03-21 13:36:49,448 - neutron_utils - INFO - Deleting router with name - NeutronUtilsRouterTeststest_create_delete_simple_sec_grp (snaps.openstack.utils.tests.neutron_utils_tests.NeutronUtilsSecuri
2017-03-21 13:36:51,067 - neutron_utils - INFO - Creating security group with name - NeutronUtilsSecu
2017-03-21 13:36:51,493 - neutron_utils - INFO - Retrieving security group with name - NeutronUtilsSe
2017-03-21 13:36:51,568 - neutron_utils - INFO - Deleting security group with name - NeutronUtilsSecu
2017-03-21 13:36:51,772 - neutron_utils - INFO - Retrieving security group with name - NeutronUtilsSe
test_create_sec_grp_no_name (snaps.openstack.utils.tests.neutron_utils_tests.NeutronUtilsSecurityGrou
test_create_sec_grp_no_rules (snaps.openstack.utils.tests.neutron_utils_tests.NeutronUtilsSecurityGro
2017-03-21 13:36:52,253 - neutron_utils - INFO - Creating security group with name - NeutronUtilsSecu
2017-03-21 13:36:52,634 - neutron_utils - INFO - Retrieving security group with name - NeutronUtilsSe
2017-03-21 13:36:52,718 - neutron_utils - INFO - Deleting security group with name - NeutronUtilsSecu
test_create_sec_grp_one_rule (snaps.openstack.utils.tests.neutron_utils_tests.NeutronUtilsSecurityGro
2017-03-21 13:36:53,082 - neutron_utils - INFO - Creating security group with name - NeutronUtilsSecu
2017-03-21 13:36:53,483 - neutron_utils - INFO - Retrieving security group rules associate with the s
2017-03-21 13:36:53,548 - neutron_utils - INFO - Creating security group to security group - NeutronU
2017-03-21 13:36:53,548 - neutron_utils - INFO - Retrieving security group with name - NeutronUtilsSe
2017-03-21 13:36:53,871 - neutron_utils - INFO - Retrieving security group with name - NeutronUtilsSe
2017-03-21 13:36:53,944 - neutron_utils - INFO - Retrieving security group rules associate with the s
2017-03-21 13:36:53,991 - neutron_utils - INFO - Retrieving security group with name - NeutronUtilsSe
2017-03-21 13:36:54,069 - neutron_utils - INFO - Deleting security group rule with ID - 7f76046c-d043
2017-03-21 13:36:54,185 - neutron_utils - INFO - Deleting security group rule with ID - f18a9ed1-466f
2017-03-21 13:36:54,338 - neutron_utils - INFO - Deleting security group rule with ID - fe34a3d0-948e
2017-03-21 13:36:54,444 - neutron_utils - INFO - Deleting security group with name - NeutronUtilsSecu
test_create_delete_keypair (snaps.openstack.utils.tests.nova_utils_tests.NovaUtilsKeypairTests) ...
2017-03-21 13:36:54,637 - nova_utils - INFO - Creating keypair with name - NovaUtilsKeypairTests-5ce6
test_create_key_from_file (snaps.openstack.utils.tests.nova_utils_tests.NovaUtilsKeypairTests) ...
2017-03-21 13:36:58,989 - nova_utils - INFO - Saved public key to - tmp/NovaUtilsKeypairTests-df3e848
2017-03-21 13:36:58,990 - nova_utils - INFO - Saved private key to - tmp/NovaUtilsKeypairTests-df3e84
2017-03-21 13:36:58,990 - nova_utils - INFO - Saving keypair to - tmp/NovaUtilsKeypairTests-df3e848d-
54
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
2017-03-21 13:36:58,990 - nova_utils - INFO - Creating keypair with name - NovaUtilsKeypairTests-df3e
test_create_keypair (snaps.openstack.utils.tests.nova_utils_tests.NovaUtilsKeypairTests) ...
2017-03-21 13:36:59,807 - nova_utils - INFO - Creating keypair with name - NovaUtilsKeypairTests-fc7f
test_floating_ips (snaps.openstack.utils.tests.nova_utils_tests.NovaUtilsKeypairTests) ...
2017-03-21 13:37:02,765 - nova_utils - INFO - Creating floating ip to external network - admin_floati
test_create_delete_flavor (snaps.openstack.utils.tests.nova_utils_tests.NovaUtilsFlavorTests) ... ok
test_create_flavor (snaps.openstack.utils.tests.nova_utils_tests.NovaUtilsFlavorTests) ... ok
test_create_clean_flavor (snaps.openstack.tests.create_flavor_tests.CreateFlavorTests) ... ok
test_create_delete_flavor (snaps.openstack.tests.create_flavor_tests.CreateFlavorTests) ... ok
test_create_flavor (snaps.openstack.tests.create_flavor_tests.CreateFlavorTests) ... ok
test_create_flavor_existing (snaps.openstack.tests.create_flavor_tests.CreateFlavorTests) ...
2017-03-21 13:37:18,545 - create_image - INFO - Found flavor with name - CreateFlavorTests-3befc152-4
---------------------------------------------------------------------Ran 48 tests in 171.000s
OK
2017-03-21
2017-03-21
2017-03-21
2017-03-21
13:37:18,620
13:37:18,977
13:37:18,977
13:37:18,981
-
functest.core.testcase_base - INFO - api_check OK
functest.core.testcase_base - INFO - The results were successfully pushed t
run_tests - INFO - Test execution time: 02:52
run_tests - INFO -
2017-03-21 13:37:18,981 - run_tests - INFO - ============================================
2017-03-21 13:37:18,981 - run_tests - INFO - Running test case 'snaps_health_check'...
2017-03-21 13:37:18,981 - run_tests - INFO - ============================================
2017-03-21 13:37:19,098 - file_utils - INFO - Attempting to read OS environment file - /home/opnfv/fu
2017-03-21 13:37:19,099 - openstack_tests - INFO - OS Credentials = OSCreds - username=admin, passwor
2017-03-21 13:37:19,434 - file_utils - INFO - Attempting to read OS environment file - /home/opnfv/fu
2017-03-21 13:37:19,435 - openstack_tests - INFO - OS Credentials = OSCreds - username=admin, passwor
test_check_vm_ip_dhcp (snaps.openstack.tests.create_instance_tests.SimpleHealthCheck) ...
2017-03-21 13:37:26,082 - create_image - INFO - Creating image
2017-03-21 13:37:28,793 - create_image - INFO - Image is active with name - SimpleHealthCheck-2324472
2017-03-21 13:37:28,793 - create_image - INFO - Image is now active with name - SimpleHealthCheck-232
2017-03-21 13:37:28,794 - OpenStackNetwork - INFO - Creating neutron network SimpleHealthCheck-232447
2017-03-21 13:37:29,308 - neutron_utils - INFO - Creating network with name SimpleHealthCheck-2324472
2017-03-21 13:37:30,771 - neutron_utils - INFO - Creating subnet with name SimpleHealthCheck-23244728
2017-03-21 13:37:36,974 - neutron_utils - INFO - Creating port for network with name - SimpleHealthCh
2017-03-21 13:37:38,188 - create_instance - INFO - Creating VM with name - SimpleHealthCheck-23244728
2017-03-21 13:37:41,538 - create_instance - INFO - Created instance with name - SimpleHealthCheck-232
2017-03-21 13:37:59,577 - create_instance - INFO - VM is - ACTIVE
2017-03-21 13:37:59,577 - create_instance_tests - INFO - Looking for expression Lease of.*obtained in
2017-03-21 13:37:59,830 - create_instance_tests - INFO - DHCP lease obtained logged in console
2017-03-21 13:37:59,830 - create_instance_tests - INFO - With correct IP address
2017-03-21 13:37:59,830 - create_instance - INFO - Deleting Port - SimpleHealthCheck-23244728-5a5a-45
2017-03-21 13:37:59,830 - neutron_utils - INFO - Deleting port with name SimpleHealthCheck-23244728-5
2017-03-21 13:38:00,705 - create_instance - INFO - Deleting VM instance - SimpleHealthCheck-232447282017-03-21 13:38:01,412 - create_instance - INFO - Checking deletion status
2017-03-21 13:38:04,938 - create_instance - INFO - VM has been properly deleted VM with name - Simple
ok
---------------------------------------------------------------------Ran 1 test in 46.982s
OK
2017-03-21
2017-03-21
2017-03-21
2017-03-21
13:38:06,417
13:38:06,778
13:38:06,779
13:38:06,779
2.2. Testing User Guides
-
functest.core.testcase_base - INFO - snaps_health_check OK
functest.core.testcase_base - INFO - The results were successfully pushed t
run_tests - INFO - Test execution time: 00:47
run_tests - INFO -
55
OPNFV Documentation, Release Danube
and
[email protected]:~/repos/functest/ci# functest testcase run vping_ssh
Executing command: 'python /home/opnfv/repos/functest/ci/run_tests.py -t vping_ssh'
2016-06-30 11:50:31,861 - run_tests - INFO - Sourcing the OpenStack RC file...
2016-06-30 11:50:31,865 - run_tests - INFO - ============================================
2016-06-30 11:50:31,865 - run_tests - INFO - Running test case 'vping_ssh'...
2016-06-30 11:50:31,865 - run_tests - INFO - ============================================
2016-06-30 11:50:32,977 - vping_ssh - INFO - Creating image 'functest-vping' from '/home/opnfv/functe
2016-06-30 11:50:45,470 - vping_ssh - INFO - Creating neutron network vping-net...
2016-06-30 11:50:47,645 - vping_ssh - INFO - Creating security group 'vPing-sg'...
2016-06-30 11:50:48,843 - vping_ssh - INFO - Using existing Flavor 'm1.small'...
2016-06-30 11:50:48,927 - vping_ssh - INFO - vPing Start Time:'2016-06-30 11:50:48'
2016-06-30 11:50:48,927 - vping_ssh - INFO - Creating instance 'opnfv-vping-1'...
2016-06-30 11:51:34,664 - vping_ssh - INFO - Instance 'opnfv-vping-1' is ACTIVE.
2016-06-30 11:51:34,818 - vping_ssh - INFO - Adding 'opnfv-vping-1' to security group 'vPing-sg'...
2016-06-30 11:51:35,209 - vping_ssh - INFO - Creating instance 'opnfv-vping-2'...
2016-06-30 11:52:01,439 - vping_ssh - INFO - Instance 'opnfv-vping-2' is ACTIVE.
2016-06-30 11:52:01,439 - vping_ssh - INFO - Adding 'opnfv-vping-2' to security group 'vPing-sg'...
2016-06-30 11:52:01,754 - vping_ssh - INFO - Creating floating IP for VM 'opnfv-vping-2'...
2016-06-30 11:52:01,969 - vping_ssh - INFO - Floating IP created: '10.17.94.140'
2016-06-30 11:52:01,969 - vping_ssh - INFO - Associating floating ip: '10.17.94.140' to VM 'opnfv-vpi
2016-06-30 11:52:02,792 - vping_ssh - INFO - Trying to establish SSH connection to 10.17.94.140...
2016-06-30 11:52:19,915 - vping_ssh - INFO - Waiting for ping...
2016-06-30 11:52:21,108 - vping_ssh - INFO - vPing detected!
2016-06-30 11:52:21,108 - vping_ssh - INFO - vPing duration:'92.2' s.
2016-06-30 11:52:21,109 - vping_ssh - INFO - vPing OK
2016-06-30 11:52:21,153 - clean_openstack - INFO - +++++++++++++++++++++++++++++++
2016-06-30 11:52:21,153 - clean_openstack - INFO - Cleaning OpenStack resources...
2016-06-30 11:52:21,153 - clean_openstack - INFO - +++++++++++++++++++++++++++++++
Version 1 is deprecated, use alternative version 2 instead.
:
:
etc.
To list the test cases which are part of a specific Test Tier, the ‘get-tests’ command is used with ‘functest tier’:
[email protected]:~/repos/functest/ci# functest tier get-tests healthcheck
Test cases in tier 'healthcheck':
['connection_check', 'api_check', 'snaps_health_check']
Please note that for some scenarios some test cases might not be launched. For example, the last example displayed
only the ‘odl’ testcase for the given environment. In this particular system the deployment does not support the ‘ocl’
SDN Controller Test Case; for example.
Important If you use the command ‘functest tier run <tier_name>’, then the Functest CLI utility will call all valid
Test Cases, which belong to the specified Test Tier, as relevant to scenarios deployed to the SUT environment. Thus,
the Functest CLI utility calculates automatically which tests can be executed and which cannot, given the environment
variable DEPLOY_SCENARIO, which is passed in to the Functest docker container.
Currently, the Functest CLI command ‘functest testcase run <testcase_name>’, supports two possibilities:
*
*
Run a single Test Case, specified by a valid choice of <testcase_name>
Run ALL test Test Cases (for all Tiers) by specifying <testcase_name> = 'all'
Functest includes a cleaning mechanism in order to remove all the OpenStack resources except those present before
running any test. The script $REPOS_DIR/functest/functest/utils/openstack_snapshot.py is called once when setting
up the Functest environment (i.e. CLI command ‘functest env prepare’) to snapshot all the OpenStack resources
(images, networks, volumes, security groups, tenants, users) so that an eventual cleanup does not remove any of these
56
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
defaults.
It is also called before running a test except if it is disabled by configuration in the testcases.yaml file (clean_flag=false).
This flag has been added as some upstream tests already include their own cleaning mechanism (e.g. Rally).
The script openstack_clean.py which is located in $REPOS_DIR/functest/functest/utils/ is normally called after a
test execution. It is in charge of cleaning the OpenStack resources that are not specified in the defaults file generated
previously which is stored in /home/opnfv/functest/conf/openstack_snapshot.yaml in the Functest docker container.
It is important to mention that if there are new OpenStack resources created manually after the snapshot done before
running the tests, they will be removed, unless you use the special method of invoking the test case with specific
suppression of clean up. (See the Troubleshooting section).
The reason to include this cleanup meachanism in Functest is because some test suites create a lot of resources (users,
tenants, networks, volumes etc.) that are not always properly cleaned, so this function has been set to keep the system
as clean as it was before a full Functest execution.
Although the Functest CLI provides an easy way to run any test, it is possible to do a direct call to the desired test
script. For example:
python $REPOS_DIR/functest/functest/opnfv_tests/openstack/vping/vping_ssh.py
Automated testing As mentioned previously, the Functest Docker container preparation as well as invocation of
Test Cases can be called within the container from the Jenkins CI system. There are 3 jobs that automate the whole
process. The first job runs all the tests referenced in the daily loop (i.e. that must been run daily), the second job
runs the tests referenced in the weekly loop (usually long duration tests run once a week maximum) and the third job
allows testing test suite by test suite specifying the test suite name. The user may also use either of these Jenkins jobs
to execute the desired test suites.
One of the most challenging task in the Danube release consists in dealing with lots of scenarios and installers. Thus,
when the tests are automatically started from CI, a basic algorithm has been created in order to detect whether a given
test is runnable or not on the given scenario. Some Functest test suites cannot be systematically run (e.g. ODL suite
can not be run on an ONOS scenario). The daily/weekly notion has been introduces in Colorado in order to save CI
time and avoid running systematically long duration tests. It was not used in Colorado due to CI resource shortage.
The mechanism remains however as part of the CI evolution.
CI provides some useful information passed to the container as environment variables:
• Installer (apex|compass|fuel|joid), stored in INSTALLER_TYPE
• Installer IP of the engine or VM running the actual deployment, stored in INSTALLER_IP
• The scenario [controller]-[feature]-[mode], stored in DEPLOY_SCENARIO with
– controller = (odl|ocl|nosdn|onos)
– feature = (ovs(dpdk)|kvm|sfc|bgpvpn|multisites|netready|ovs_dpdk_bar)
– mode = (ha|noha)
The
constraints
per
test
case
are
/home/opnfv/repos/functest/functest/ci/testcases.yaml:
defined
in
the
Functest
configuration
file
tiers:
name: smoke
order: 1
ci_loop: '(daily)|(weekly)'
description : >Set of basic Functional tests to validate the OpenStack deployment.
testcases:
-
2.2. Testing User Guides
57
OPNFV Documentation, Release Danube
name: vping_ssh
criteria: 'status == "PASS"'
blocking: true
description: >This test case verifies: 1) SSH to an instance using floating
IPs over the public network. 2) Connectivity between 2 instances
over a private network.
dependencies:
installer: ''
scenario: '^((?!bgpvpn|odl_l3).)*$'
run:
module: 'functest.opnfv_tests.openstack.vping.vping_ssh'
class: 'VPingSSH'
....
We may distinguish 2 levels in the test case description:
• Tier level
• Test case level
At the tier level, we define the following parameters:
• ci_loop: indicate if in automated mode, the test case must be run in dail and/or weekly jobs
• description: a high level view of the test case
For a given test case we defined:
• the name of the test case
• the criteria (experimental): a criteria used to declare the test case as PASS or FAIL
• blocking: if set to true, if the test is failed, the execution of the following tests is canceled
• clean_flag: shall the functect internal mechanism be invoked after the test
• the description of the test case
• the dependencies: a combination of 2 regex on the scenario and the installer name
• run: In Danube we introduced the notion of abstract class in order to harmonize the way to run internal,
feature or vnf tests
For further details on abstraction classes, see developper guide.
Additional parameters have been added in the desription in the Database. The target is to use the configuration stored
in the Database and consider the local file as backup if the Database is not reachable. The additional fields related to a
test case are:
• trust: we introduced this notion to put in place a mechanism of scenario promotion.
• Version: it indicates since which version you can run this test
• domains: the main domain covered by the test suite
• tags: a list of tags related to the test suite
The order of execution is the one defined in the file if all test cases are selected.
In CI daily job the tests are executed in the following order:
1. healthcheck (blocking)
2. smoke: both vPings are blocking
3. Feature project tests cases
58
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
In CI weekly job we add 2 tiers:
4. VNFs (vIMS)
5. Components (Rally and Tempest long duration suites)
As explained before, at the end of an automated execution, the OpenStack resources might be eventually removed.
Please note that a system snapshot is taken before any test case execution.
This testcase.yaml file is used for CI, for the CLI and for the automatic reporting.
Test results
Manual testing In manual mode test results are displayed in the console and result files are put in
/home/opnfv/functest/results.
Automated testing In automated mode, test results are displayed in jenkins logs, a summary is provided at the end
of the job and can be described as follow:
+====================================================================================================
|
FUNCTEST REPORT
+====================================================================================================
|
| Deployment description:
|
INSTALLER: fuel
|
SCENARIO: os-odl_l2-nofeature-ha
|
BUILD TAG: jenkins-functest-fuel-baremetal-daily-master-324
|
CI LOOP:
daily
|
+=========================+===============+============+===============+=============================
| TEST CASE
| TIER
| DURATION
| RESULT
| URL
+=========================+===============+============+===============+=============================
| connection_check
| healthcheck
| 00:02
| PASS
| http://testresults.opnfv.org
+-------------------------+---------------+------------+---------------+----------------------------| api_check
| healthcheck
| 01:15
| PASS
| http://testresults.opnfv.org
+-------------------------+---------------+------------+---------------+----------------------------| snaps_health_check
| healthcheck
| 00:50
| PASS
| http://testresults.opnfv.org
+-------------------------+---------------+------------+---------------+----------------------------| vping_ssh
| smoke
| 01:10
| PASS
| http://testresults.opnfv.org
+-------------------------+---------------+------------+---------------+----------------------------| vping_userdata
| smoke
| 00:59
| PASS
| http://testresults.opnfv.org
+-------------------------+---------------+------------+---------------+----------------------------| tempest_smoke_serial
| smoke
| 12:57
| PASS
| http://testresults.opnfv.org
+-------------------------+---------------+------------+---------------+----------------------------| rally_sanity
| smoke
| 10:22
| PASS
| http://testresults.opnfv.org
+-------------------------+---------------+------------+---------------+----------------------------| refstack_defcore
| smoke
| 12:28
| PASS
| http://testresults.opnfv.org
+-------------------------+---------------+------------+---------------+----------------------------| snaps_smoke
| smoke
| 12:04
| PASS
| http://testresults.opnfv.org
+-------------------------+---------------+------------+---------------+----------------------------| domino
| features
| 00:29
| PASS
| http://testresults.opnfv.org
+-------------------------+---------------+------------+---------------+-----------------------------
Results are automatically pushed to the test results database, some additional result files are pushed to OPNFV artifact
web sites.
Based on the results stored in the result database, a Functest reporting portal is also automatically updated. This portal
provides information on:
2.2. Testing User Guides
59
OPNFV Documentation, Release Danube
• The overall status per scenario and per installer
• Tempest: Tempest test case including reported errors per scenario and installer
• vIMS: vIMS details per scenario and installer
Troubleshooting
This section gives some guidelines about how to troubleshoot the test cases owned by Functest.
IMPORTANT: As in the previous section, the steps defined below must be executed inside the Functest Docker
container and after sourcing the OpenStack credentials:
. $creds
or:
source /home/opnfv/functest/conf/openstack.creds
60
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
VIM This section covers the test cases related to the VIM (healthcheck, vping_ssh, vping_userdata, tempest_smoke_serial, tempest_full_parallel, rally_sanity, rally_full).
vPing common For both vPing test cases (vPing_ssh, and vPing_userdata), the first steps are similar:
• Create Glance image
• Create Network
• Create Security Group
• Create Instances
After these actions, the test cases differ and will be explained in their respective section.
These test cases can be run inside the container, using new Functest CLI as follows:
$ functest testcase run vping_ssh
$ functest testcase run vping_userdata
The Functest CLI is designed to route a call to the corresponding internal python scripts, located in paths:
$REPOS_DIR/functest/functest/opnfv_tests/openstack/vping/vping_ssh.py and $REPOS_DIR/functest/functest/opnfv_tests/openstack/vping/vping_userdata.py
Notes:
1. There is one difference, between the Functest CLI based test case execution compared to the earlier used Bash
shell script, which is relevant to point out in troubleshooting scenarios:
The Functest CLI does not yet support the option to suppress clean-up of the generated OpenStack
resources, following the execution of a test case.
Explanation: After finishing the test execution, the corresponding script will remove, by default, all created
resources in OpenStack (image, instances, network and security group). When troubleshooting, it is advisable
sometimes to keep those resources in case the test fails and a manual testing is needed.
It is actually still possible to invoke test execution, with suppression of OpenStack resource cleanup, however
this requires invocation of a specific Python script: ‘/home/opnfv/repos/functest/ci/run_test.py’. The OPNFV
Functest Developer Guide provides guidance on the use of that Python script in such troubleshooting cases.
Some of the common errors that can appear in this test case are:
vPing_ssh- ERROR - There has been a problem when creating the neutron network....
This means that there has been some problems with Neutron, even before creating the instances. Try to create manually
a Neutron network and a Subnet to see if that works. The debug messages will also help to see when it failed (subnet
and router creation). Example of Neutron commands (using 10.6.0.0/24 range for example):
neutron net-create net-test
neutron subnet-create --name subnet-test --allocation-pool start=10.6.0.2,end=10.6.0.100 \
--gateway 10.6.0.254 net-test 10.6.0.0/24
neutron router-create test_router
neutron router-interface-add <ROUTER_ID> test_subnet
neutron router-gateway-set <ROUTER_ID> <EXT_NET_NAME>
Another related error can occur while creating the Security Groups for the instances:
vPing_ssh- ERROR - Failed to create the security group...
In this case, proceed to create it manually. These are some hints:
2.2. Testing User Guides
61
OPNFV Documentation, Release Danube
neutron security-group-create sg-test
neutron security-group-rule-create sg-test --direction
--remote-ip-prefix 0.0.0.0/0
neutron security-group-rule-create sg-test --direction
--protocol tcp --port-range-min 80 --port-range-max 80
neutron security-group-rule-create sg-test --direction
--protocol tcp --port-range-min 80 --port-range-max 80
ingress --protocol icmp \
ingress --ethertype IPv4 \
--remote-ip-prefix 0.0.0.0/0
egress --ethertype IPv4 \
--remote-ip-prefix 0.0.0.0/0
The next step is to create the instances. The image used is located in /home/opnfv/functest/data/cirros-0.3.5-x86_64disk.img and a Glance image is created with the name functest-vping. If booting the instances fails (i.e. the status is
not ACTIVE), you can check why it failed by doing:
nova list
nova show <INSTANCE_ID>
It might show some messages about the booting failure. To try that manually:
nova boot --flavor m1.small --image functest-vping --nic net-id=<NET_ID> nova-test
This will spawn a VM using the network created previously manually. In all the OPNFV tested scenarios from CI, it
never has been a problem with the previous actions. Further possible problems are explained in the following sections.
vPing_SSH This test case creates a floating IP on the external network and assigns it to the second instance opnfv-vping-2.
The purpose of this is to establish a SSH connection to that instance
and SCP a script that will ping the first instance. This script is located in the repository under $REPOS_DIR/functest/functest/opnfv_tests/openstack/vping/ping.sh and takes an IP as a parameter. When the SCP is
completed, the test will do an SSH call to that script inside the second instance. Some problems can happen here:
vPing_ssh- ERROR - Cannot establish connection to IP xxx.xxx.xxx.xxx. Aborting
If this is displayed, stop the test or wait for it to finish, if you have used the special method of test invocation with
specific supression of OpenStack resource clean-up, as explained earler. It means that the Container can not reach the
Public/External IP assigned to the instance opnfv-vping-2. There are many possible reasons, and they really depend on
the chosen scenario. For most of the ODL-L3 and ONOS scenarios this has been noticed and it is a known limitation.
First, make sure that the instance opnfv-vping-2 succeeded to get an IP from the DHCP agent. It can be checked by
doing:
nova console-log opnfv-vping-2
If the message Sending discover and No lease, failing is shown, it probably means that the Neutron dhcp-agent failed
to assign an IP or even that it was not responding. At this point it does not make sense to try to ping the floating IP.
If the instance got an IP properly, try to ping manually the VM from the container:
nova list
<grab the public IP>
ping <public IP>
If the ping does not return anything, try to ping from the Host where the Docker container is running. If that solves
the problem, check the iptable rules because there might be some rules rejecting ICMP or TCP traffic coming/going
from/to the container.
At this point, if the ping does not work either, try to reproduce the test manually with the steps described above in the
vPing common section with the addition:
neutron floatingip-create <EXT_NET_NAME>
nova floating-ip-associate nova-test <FLOATING_IP>
62
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
Further troubleshooting is out of scope of this document, as it might be due to problems with the SDN controller.
Contact the installer team members or send an email to the corresponding OPNFV mailing list for more information.
vPing_userdata This test case does not create any floating IP neither establishes an SSH connection. Instead, it uses
nova-metadata service when creating an instance to pass the same script as before (ping.sh) but as 1-line text. This
script will be executed automatically when the second instance opnfv-vping-2 is booted.
The only known problem here for this test to fail is mainly the lack of support of cloud-init (nova-metadata service).
Check the console of the instance:
nova console-log opnfv-vping-2
If this text or similar is shown:
checking http://169.254.169.254/2009-04-04/instance-id
failed 1/20: up 1.13. request failed
failed 2/20: up 13.18. request failed
failed 3/20: up 25.20. request failed
failed 4/20: up 37.23. request failed
failed 5/20: up 49.25. request failed
failed 6/20: up 61.27. request failed
failed 7/20: up 73.29. request failed
failed 8/20: up 85.32. request failed
failed 9/20: up 97.34. request failed
failed 10/20: up 109.36. request failed
failed 11/20: up 121.38. request failed
failed 12/20: up 133.40. request failed
failed 13/20: up 145.43. request failed
failed 14/20: up 157.45. request failed
failed 15/20: up 169.48. request failed
failed 16/20: up 181.50. request failed
failed 17/20: up 193.52. request failed
failed 18/20: up 205.54. request failed
failed 19/20: up 217.56. request failed
failed 20/20: up 229.58. request failed
failed to read iid from metadata. tried 20
it means that the instance failed to read from the metadata service. Contact the Functest or installer teams for more
information.
NOTE: Cloud-init in not supported on scenarios dealing with ONOS and the tests have been excluded from CI in those
scenarios.
Tempest In the upstream OpenStack CI all the Tempest test cases are supposed to pass. If some test cases fail in an
OPNFV deployment, the reason is very probably one of the following
2.2. Testing User Guides
63
OPNFV Documentation, Release Danube
Error
Resources required for test case
execution are missing
OpenStack components or
services are missing or not
configured properly
Some resources required for
execution test cases are missing
Details
Such resources could be e.g. an external network and access to the
management subnet (adminURL) from the Functest docker container.
Check running services in the controller and compute nodes (e.g. with
“systemctl” or “service” commands). Configuration parameters can be
verified from the related .conf files located under ‘/etc/<component>’
directories.
The tempest.conf file, automatically generated by Rally in Functest, does not
contain all the needed parameters or some parameters are not set properly.
The tempest.conf file is located in directory
‘/home/opnfv/.rally/verification/verifier-<UUID> /for-deployment-<UUID>’
in the Functest Docker container. Use the “rally deployment list” command
in order to check the UUID the UUID of the current deployment.
When some Tempest test case fails, captured traceback and possibly also the related REST API requests/responses are
output to the console. More detailed debug information can be found from tempest.log file stored into related Rally
deployment folder.
Rally The same error causes which were mentioned above for Tempest test cases, may also lead to errors in Rally
as well.
Possible scenarios are:
• authenticate
• glance
• cinder
• heat
• keystone
• neutron
• nova
• quotas
• requests
• vm
To know more about what those scenarios are doing, they are defined in directory:
$REPOS_DIR/functest/functest/opnfv_tests/openstack/rally/scenario For more info about Rally scenario definition
please refer to the Rally official documentation. [3]
To check any possible problems with Rally, the logs are stored under /home/opnfv/functest/results/rally/ in the Functest
Docker container.
Controllers
Opendaylight If the Basic Restconf test suite fails, check that the ODL controller is reachable and its Restconf
module has been installed.
If the Neutron Reachability test fails, verify that the modules implementing Neutron requirements have been properly
installed.
If any of the other test cases fails, check that Neutron and ODL have been correctly configured to work together. Check
Neutron configuration files, accounts, IP addresses etc.).
64
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
ONOS Please refer to the ONOS documentation. ONOSFW User Guide .
Features Please refer to the dedicated feature user guides for details.
VNF
cloudify_ims vIMS deployment may fail for several reasons, the most frequent ones are described in the following
table:
Error
Keystone admin API not reachable
Impossible to retrieve admin role id
Error when uploading image from OpenStack to glance
Cinder quota cannot be updated
Impossible to create a volume
SSH connection issue between the Test Docker container
and the VM
No Internet access from the VM
No access to OpenStack API from the VM
Comments
Impossible to create vIMS user and tenant
Impossible to create vIMS user and tenant
impossible to deploy VNF
Default quotas not sufficient, they are adapted in the
script
VNF cannot be deployed
if vPing test fails, vIMS test will fail...
the VMs of the VNF must have an external access to
Internet
Orchestrator can be installed but the vIMS VNF
installation fails
References
OPNFV main site
Functest page
IRC support chan: #opnfv-functest
QTIP
QTIP Installation & Configuration
Configuration
QTIP currently supports by using a Docker image. Detailed steps about setting up QTIP can be found below.
To use QTIP you should have access to an OpenStack environment, with at least Nova, Neutron, Glance, Keystone
and Heat installed. Add a brief introduction to configure OPNFV with this specific installer
Installing QTIP using Docker
QTIP docker image
hub:
QTIP has a Docker images on the docker hub. Pulling opnfv/qtip docker image from docker
docker pull opnfv/qtip:stable
Verify that opnfv/qtip has been downloaded. It should be listed as an image by running the following command.
2.2. Testing User Guides
65
OPNFV Documentation, Release Danube
docker images
Run and enter the docker instance 1. If you want to run benchmarks:
envs="INSTALLER_TYPE={INSTALLER_TYPE} -e INSTALLER_IP={INSTALLER_IP}"
docker run --name qtip -id -e $envs opnfv/qtip
docker exec -i -t qtip /bin/bash
INSTALLER_TYPE should be one of OPNFV installer, e.g. apex, compass, daisy, fuel and joid. Currenty, QTIP only
supports installer fuel.
INSTALLER_IP is the ip address of the installer that can be accessed by QTIP.
2. If you do not want to run any benchmarks:
docker run --name qtip -id opnfv/qtip
docker exec -i -t qtip /bin/bash
Now you are in the container and QTIP can be found in the /repos/qtip and can be navigated to using the
following command.
cd repos/qtip
Environment configuration
Hardware configuration QTIP does not have specific hardware requriements, and it can runs over any OPNFV
installer.
Jumphost configuration Installer Docker on Jumphost, which is used for running QTIP image.
You can refer to these links:
Ubuntu: https://docs.docker.com/engine/installation/linux/ubuntu/
Centos: https://docs.docker.com/engine/installation/linux/centos/
Platform components configuration Describe the configuration of each component in the installer.
QTIP User Guide
Overview
QTIP is the project for Platform Performance Benchmarking in OPNFV. It aims to provide user a simple indicator
for performance, simple but supported by comprehensive testing data and transparent calculation formula.
QTIP introduces a concept called QPI, a.k.a. QTIP Performance Index, which aims to be a TRUE indicator of
performance. TRUE reflects the core value of QPI in four aspects
• Transparent: being an open source project, user can inspect all details behind QPI, e.g. formulas, metrics, raw
data
• Reliable: the integrity of QPI will be guaranteed by traceability in each step back to raw test result
• Understandable: QPI is broke down into section scores, and workload scores in report to help user to understand
• Extensible: users may create their own QPI by composing the existed metrics in QTIP or extend new metrics
66
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
Benchmarks The builtin benchmarks of QTIP are located in <package_root>/benchmarks folder
• QPI: specifications about how an QPI is calculated and sources of metrics
• metric: performance metrics referred in QPI, currently it is categorized by performance testing tools
• plan: executable benchmarking plan which collects metrics and calculate QPI
CLI User Manual
QTIP consists of a number of benchmarking tools or metrics, grouped under QPI’s. QPI’s map to the different components of a NFVI ecosystem, such as compute, network and storage. Depending on the type of application, a user
may group them under plans.
QTIP CLI provides interface to all of the above the components. A help page provides a list of all the commands along
with a short description.
qtip [-h|--help]
Typically a complete plan is executed at the target environment. QTIP defaults to a number of sample plans. A list of
all the available plans can be viewed
qtip plan list
In order to view the details about a specific plan.
qtip plan show <plan_name>
where plan_name is one of those listed from the previous command.
To execute a complete plan
qtip plan run <plan_name> -p <path_to_result_directory>
QTIP does not limit result storage at a specific directory. Instead a user may specify his own result storage as above.
An important thing to remember is to provide absolute path of result directory.
mkdir result
qtip plan run <plan_name> -p $PWD/result
Similarly, the same commands can be used for the other two components making up the plans, i.e QPI’s and metrics.
For example, in order to run a single metric
qtip metric run <metric_name> -p $PWD/result
The same can be applied for a QPI.
QTIP also provides the utility to view benchmarking results on the console. One just need to provide to where the
results are stored. Extending the example above
qtip report show <metric_name> -p $PWD/result
Debug option helps identify the error by providing a detailed traceback. It can be enabled as
qtip [-d|--debug] plan run <plan_name>
2.2. Testing User Guides
67
OPNFV Documentation, Release Danube
API User Manual
QTIP consists of a number of benchmarking tools or metrics, grouped under QPI’s. QPI’s map to the different components of an NFVI ecosystem, such as compute, network and storage. Depending on the type of application, a user
may group them under plans.
QTIP API provides a RESTful interface to all of the above components. User can retrieve list of plans, QPIs and
metrics and their individual information.
Running After installing QTIP. API server can be run using command qtip-api on the local machine.
All the resources and their corresponding operation details can be seen at /v1.0/ui, on hosting
server(0.0.0.0:5000 for the local machine).
The whole API specification in json format can be seen at /v1.0/swagger.json.
The data models are given below:
• Plan
• Metric
• QPI
Plan:
{
"name": <plan name>,
"description": <plan profile>,
"info": <{plan info}>,
"config": <{plan configuration}>,
"QPIs": <[list of qpis]>,
},
Metric:
{
"name": <metric name>,
"description": <metric description>,
"links": <[links with metric information]>,
"workloads": <[cpu workloads(single_cpu, multi_cpu]>,
},
QPI:
{
"name": <qpi name>,
"description": <qpi description>,
"formula": <formula>,
"sections": <[list of sections with different metrics and formulaes]>,
}
The API can be described as follows
Plans:
Method
GET
GET
Path
/v1.0/plans
/v1.0/plans/{name}
Description
Get the list of of all plans
Get details of the specified plan
Metrics:
68
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
Method
GET
GET
Path
/v1.0/metrics
/v1.0/metrics/{name}
Method
GET
GET
Path
/v1.0/qpis
/v1.0/qpis/{name}
Description
Get the list of all metrics
Get details of specified metric
QPIs:
Description
Get the list of all QPIs
Get details of specified QPI
Note: running API with connexion cli does not require base path (/v1.0/) in url
Compute Performance Benchmarking
The compute QPI aims to benchmark the compute components of an OPNFV platform. Such components include, the
CPU performance, the memory performance.
The compute QPI consists of both synthetic and application specific benchmarks to test compute components.
All the compute benchmarks could be run in the scenario: On Baremetal Machines provisioned by an OPNFV installer
(Host machines)
Note: The Compute benchmank constains relatively old benchmarks such as dhrystone and whetstone. The suite
would be updated for better benchmarks such as Linbench for the OPNFV E release.
Getting started Notice: All descriptions are based on QTIP container.
Inventory File QTIP uses Ansible to trigger benchmark test. Ansible uses an inventory file to determine what hosts
to work against. QTIP can automatically generate a inventory file via OPNFV installer. Users also can write their
own inventory infomation into /home/opnfv/qtip/hosts. This file is just a text file containing a list of host IP
addresses. For example:
[hosts]
10.20.0.11
10.20.0.12
QTIP key Pair QTIP use a SSH key pair to connect to remote hosts. When users execute compute QPI, QTIP will
generate a key pair named QtipKey under /home/opnfv/qtip/ and pass public key to remote hosts.
If environment variable CI_DEBUG is set to true, users should delete it by manual. If CI_DEBUG is not set or set to
false, QTIP will delete the key from remote hosts before the execution ends. Please make sure the key deleted from
remote hosts or it can introduce a security flaw.
Commands In a QTIP container, you can run compute QPI by using QTIP CLI:
mkdir result
qtip plan run <plan_name> -p $PWD/result
QTIP generates results in the $PWD/result directory are listed down under the timestamp name.
you can get more details from userguide/cli.rst.
Metrics The benchmarks include:
2.2. Testing User Guides
69
OPNFV Documentation, Release Danube
Dhrystone 2.1 Dhrystone is a synthetic benchmark for measuring CPU performance. It uses integer calculations to
evaluate CPU capabilities. Both Single CPU performance is measured along multi-cpu performance.
Dhrystone, however, is a dated benchmark and has some short comings. Written in C, it is a small program that doesn’t
test the CPU memory subsystem. Additionally, dhrystone results could be modified by optimizing the compiler and
insome cases hardware configuration.
References: http://www.eembc.org/techlit/datasheets/dhrystone_wp.pdf
Whetstone Whetstone is a synthetic benchmark to measure CPU floating point operation performance. Both Single
CPU performance is measured along multi-cpu performance.
Like Dhrystone, Whetstone is a dated benchmark and has short comings.
References:
http://www.netlib.org/benchmark/whetstone.c
OpenSSL Speed OpenSSL Speed can be used to benchmark compute performance of a machine. In QTIP, two
OpenSSL Speed benchmarks are incorporated:
1. RSA signatunes/sec signed by a machine
2. AES 128-bit encryption throughput for a machine for cipher block sizes
References:
https://www.openssl.org/docs/manmaster/apps/speed.html
RAMSpeed RAMSpeed is used to measure a machine’s memory perfomace. The problem(array)size is large enough
to ensure Cache Misses so that the main machine memory is used.
INTmem and FLOATmem benchmarks are executed in 4 different scenarios:
1. Copy: a(i)=b(i)
2. Add: a(i)=b(i)+c(i)
3. Scale: a(i)=b(i)*d
4. Tniad: a(i)=b(i)+c(i)*d
INTmem uses integers in these four benchmarks whereas FLOATmem uses floating points for these benchmarks.
References:
http://alasir.com/software/ramspeed/
https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/W51a7ffcf4dfd_4b40_9d82_446ebc23c550/page/Untan
DPI nDPI is a modified variant of OpenDPI, Open source Deep packet Inspection, that is maintained by ntop. An
example application called pcapreader has been developed and is available for use along nDPI.
A sample .pcap file is passed to the pcapreader application. nDPI classifies traffic in the pcap file into different
categories based on string matching. The pcapreader application provides a throughput number for the rate at which
traffic was classified, indicating a machine’s computational performance. The results are run 10 times and an average
is taken for the obtained number.
nDPI may provide non consistent results and was added to Brahmaputra for experimental purposes
References:
70
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
http://www.ntop.org/products/deep-packet-inspection/ndpi/
http://www.ntop.org/wp-content/uploads/2013/12/nDPI_QuickStartGuide.pdf
Storperf
StorPerf User Guide
StorPerf Container Execution Guide
Planning There are some ports that the container can expose:
• 22 for SSHD. Username and password are root/storperf. This is used for CLI access only
• 5000 for StorPerf ReST API.
• 8000 for StorPerf’s Graphite Web Server
OpenStack Credentials You must have your OpenStack Controller environment variables defined and passed to the
StorPerf container. The easiest way to do this is to put the rc file contents into a clean file the looks similar to this for
V2 authentication:
OS_AUTH_URL=http://10.13.182.243:5000/v2.0
OS_TENANT_ID=e8e64985506a4a508957f931d1800aa9
OS_TENANT_NAME=admin
OS_PROJECT_NAME=admin
OS_USERNAME=admin
OS_PASSWORD=admin
OS_REGION_NAME=RegionOne
For V3 authentication, use the following:
OS_AUTH_URL=http://10.13.182.243:5000/v3
OS_PROJECT_ID=32ae78a844bc4f108b359dd7320463e5
OS_PROJECT_NAME=admin
OS_USER_DOMAIN_NAME=Default
OS_USERNAME=admin
OS_PASSWORD=admin
OS_REGION_NAME=RegionOne
OS_INTERFACE=public
OS_IDENTITY_API_VERSION=3
Additionally, if you want your results published to the common OPNFV Test Results DB, add the following:
TEST_DB_URL=http://testresults.opnfv.org/testapi
Running StorPerf Container You might want to have the local disk used for storage as the default size of the docker
container is only 10g. This is done with the -v option, mounting under /opt/graphite/storage/whisper
mkdir -p ~/carbon
sudo chown 33:33 ~/carbon
The recommended method of running StorPerf is to expose only the ReST and Graphite ports. The command line
below shows how to run the container with local disk for the carbon database.
docker run -t --env-file admin-rc -p 5000:5000 -p 8000:8000 -v ~/carbon:/opt/graphite/storage/whisper
2.2. Testing User Guides
71
OPNFV Documentation, Release Danube
Docker Exec Instead of exposing port 5022 externally, you can use the exec method in docker. This provides a
slightly more secure method of running StorPerf container without having to expose port 22.
If needed, the container can be entered with docker exec. This is not normally required.
docker exec -it storperf bash
Container with SSH Running the StorPerf Container with all ports open and a local disk for the result storage. This
is not recommended as the SSH port is open.
docker run -t --env-file admin-rc -p 5022:22 -p 5000:5000 -p 8000:8000 -v ~/carbon:/opt/graphite/stor
This will then permit ssh to localhost port 5022 for CLI access.
StorPerf Installation Guide
OpenStack Prerequisites If you do not have an Ubuntu 16.04 image in Glance, you will need to add one. There are
scripts in storperf/ci directory to assist, or you can use the follow code snippets:
# Put an Ubuntu Image in glance
wget -q https://cloud-images.ubuntu.com/releases/16.04/release/ubuntu-16.04-server-cloudimg-amd64-dis
openstack image create "Ubuntu 16.04 x86_64" --disk-format qcow2 --public \
--container-format bare --file ubuntu-16.04-server-cloudimg-amd64-disk1.img
# Create StorPerf flavor
openstack flavor create storperf \
--id auto \
--ram 8192 \
--disk 4 \
--vcpus 2
Planning StorPerf is delivered as a Docker container. There are two possible methods for installation in your
environment:
1. Run container on Jump Host
2. Run container in a VM
Running StorPerf on Jump Host Requirements:
• Docker must be installed
• Jump Host must have access to the OpenStack Controller API
• Jump Host must have internet connectivity for downloading docker image
• Enough floating IPs must be available to match your agent count
Running StorPerf in a VM Requirements:
• VM has docker installed
• VM has OpenStack Controller credentials and can communicate with the Controller API
• VM has internet connectivity for downloading the docker image
• Enough floating IPs must be available to match your agent count
72
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
VM Creation The following procedure will create the VM in your environment
# Create the StorPerf VM itself. Here we use the network ID generated by OPNFV FUEL.
ADMIN_NET_ID=`neutron net-list | grep 'admin_internal_net ' | awk '{print $2}'`
nova boot --nic net-id=$ADMIN_NET_ID --flavor m1.small --key-name=StorPerf --image 'Ubuntu 14.04' 'St
At this point, you may associate a floating IP with the StorPerf master VM.
VM Docker Installation The following procedure will install Docker on Ubuntu 14.04.
sudo apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070AD
cat << EOF | sudo tee /etc/apt/sources.list.d/docker.list
deb https://apt.dockerproject.org/repo ubuntu-trusty main
EOF
sudo apt-get update
sudo apt-get install -y docker-engine
sudo usermod -aG docker ubuntu
Pulling StorPerf Container
Danube The tag for the latest stable Danube will be:
docker pull opnfv/storperf:danube.1.0
Colorado The tag for the latest stable Colorado release is:
docker pull opnfv/storperf:colorado.0.1
Brahmaputra The tag for the latest stable Brahmaputra release is:
docker pull opnfv/storperf:brahmaputra.1.2
Development The tag for the latest development version is:
docker pull opnfv/storperf:master
StorPerf Test Execution Guide
Prerequisites This guide requires StorPerf to be running and have its ReST API accessible. If the ReST API is not
running on port 5000, adjust the commands provided here as needed.
Interacting With StorPerf Once the StorPerf container has been started and the ReST API exposed, you can interact
directly with it using the ReST API. StorPerf comes with a Swagger interface that is accessible through the exposed
port at:
http://StorPerf:5000/swagger/index.html
The typical test execution follows this pattern:
1. Configure the environment
2.2. Testing User Guides
73
OPNFV Documentation, Release Danube
2. Initialize the cinder volumes
3. Execute one or more performance runs
4. Delete the environment
Configure The Environment The following pieces of information are required to prepare the environment:
• The number of VMs/Cinder volumes to create
• The Glance image that holds the VM operating system to use. StorPerf has only been tested with Ubuntu 16.04
• The name of the public network that agents will use
• The size, in gigabytes, of the Cinder volumes to create
The ReST API is a POST to http://StorPerf:5000/api/v1.0/configurations and takes a JSON payload as follows.
{
"agent_count": int,
"agent_image": string,
"public_network": string,
"volume_size": int
}
This call will block until the stack is created, at which point it will return the OpenStack heat stack id.
Initialize the Cinder Volumes Before executing a test run for the purpose of measuring performance, it is necessary
to fill the Cinder volume with random data. Failure to execute this step can result in meaningless numbers, especially
for read performance. Most Cinder drivers are smart enough to know what blocks contain data, and which do not.
Uninitialized blocks return “0” immediately without actually reading from the volume.
Initiating the data fill looks the same as a regular performance test, but uses the special workload called “_warm_up”.
StorPerf will never push _warm_up data to the OPNFV Test Results DB, nor will it terminate the run on steady state.
It is guaranteed to run to completion, which fills 100% of the volume with random data.
The ReST API is a POST to http://StorPerf:5000/api/v1.0/jobs and takes a JSON payload as follows.
{
"workload": "_warm_up"
}
This will return a job ID as follows.
{
"job_id": "edafa97e-457e-4d3d-9db4-1d6c0fc03f98"
}
This job ID can be used to query the state to determine when it has completed. See the section on querying jobs for
more information.
Execute a Performance Run Performance runs can execute either a single workload, or iterate over a matrix of
workload types, block sizes and queue depths.
Workload Types
rr Read, Random. 100% read of random blocks
rs Read, Sequential. 100% read of sequential blocks of data
74
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
rw Read / Write Mix, Random. 70% random read, 30% random write
wr Write, Random. 100% write of random blocks
ws Write, Sequential. 100% write of sequential blocks.
Block Sizes A comma delimited list of the different block sizes to use when reading and writing data. Note: Some
Cinder drivers (such as Ceph) cannot support block sizes larger than 16k (16384).
Queue Depths A comma delimited list of the different queue depths to use when reading and writing data. The
queue depth parameter causes FIO to keep this many I/O requests outstanding at one time. It is used to simulate traffic
patterns on the system. For example, a queue depth of 4 would simulate 4 processes constantly creating I/O requests.
Deadline The deadline is the maximum amount of time in minutes for a workload to run. If steady state has not
been reached by the deadline, the workload will terminate and that particular run will be marked as not having reached
steady state. Any remaining workloads will continue to execute in order.
{
"block_sizes": "2048,16384,
"deadline": 20,
"queue_depths": "2,4",
"workload": "wr,rr,rw",
}
Metadata A job can have metadata associated with it for tagging. The following metadata is required in order to
push results to the OPNFV Test Results DB:
"metadata": {
"disk_type": "HDD or SDD",
"pod_name": "OPNFV Pod Name",
"scenario_name": string,
"storage_node_count": int,
"version": string,
"build_tag": string,
"test_case": "snia_steady_state"
}
Query Jobs Information By issuing a GET to the job API http://StorPerf:5000/api/v1.0/jobs?job_id=<ID>, you
can fetch information about the job as follows:
• &type=status: to report on the status of the job.
• &type=metrics: to report on the collected metrics.
• &type=metadata: to report back any metadata sent with the job ReST API
Status The Status field can be: - Running to indicate the job is still in progress, or - Completed to indicate the job is
done. This could be either normal completion
or manually terminated via HTTP DELETE call.
Workloads can have a value of: - Pending to indicate the workload has not yet started, - Running to indicate this is the
active workload, or - Completed to indicate this workload has completed.
This is an example of a type=status call.
2.2. Testing User Guides
75
OPNFV Documentation, Release Danube
{
"Status": "Running",
"TestResultURL": null,
"Workloads": {
"eeb2e587-5274-4d2f-ad95-5c85102d055e.ws.queue-depth.1.block-size.16384": "Pending",
"eeb2e587-5274-4d2f-ad95-5c85102d055e.ws.queue-depth.1.block-size.4096": "Pending",
"eeb2e587-5274-4d2f-ad95-5c85102d055e.ws.queue-depth.1.block-size.512": "Pending",
"eeb2e587-5274-4d2f-ad95-5c85102d055e.ws.queue-depth.4.block-size.16384": "Running",
"eeb2e587-5274-4d2f-ad95-5c85102d055e.ws.queue-depth.4.block-size.4096": "Pending",
"eeb2e587-5274-4d2f-ad95-5c85102d055e.ws.queue-depth.4.block-size.512": "Pending",
"eeb2e587-5274-4d2f-ad95-5c85102d055e.ws.queue-depth.8.block-size.16384": "Completed",
"eeb2e587-5274-4d2f-ad95-5c85102d055e.ws.queue-depth.8.block-size.4096": "Pending",
"eeb2e587-5274-4d2f-ad95-5c85102d055e.ws.queue-depth.8.block-size.512": "Pending"
}
}
Metrics Metrics can be queried at any time during or after the completion of a run. Note that the metrics show up
only after the first interval has passed, and are subject to change until the job completes.
This is a sample of a type=metrics call.
{
"rw.queue-depth.1.block-size.512.read.bw": 52.8,
"rw.queue-depth.1.block-size.512.read.iops": 106.76199999999999,
"rw.queue-depth.1.block-size.512.read.lat.mean": 93.176,
"rw.queue-depth.1.block-size.512.write.bw": 22.5,
"rw.queue-depth.1.block-size.512.write.iops": 45.760000000000005,
"rw.queue-depth.1.block-size.512.write.lat.mean": 21764.184999999998
}
Abort a Job Issuing an HTTP DELETE to the job api http://StorPerf:5000/api/v1.0/jobs will force the termination
of the whole job, regardless of how many workloads remain to be executed.
curl -X DELETE --header 'Accept: application/json' http://StorPerf:5000/api/v1.0/jobs
Delete the Environment After you are done testing, you can have StorPerf delete the Heat stack by issuing an HTTP
DELETE to the configurations API.
curl -X DELETE --header 'Accept: application/json' http://StorPerf:5000/api/v1.0/configurations
You may also want to delete an environment, and then create a new one with a different number of VMs/Cinder
volumes to test the impact of the number of VMs in your environment.
VSPERF
VSPERF
VSPERF is an OPNFV testing project.
VSPERF provides an automated test-framework and comprehensive test suite based on industry standards for measuring data-plane performance of Telco NFV switching technologies as well as physical and virtual network interfaces
(NFVI). The VSPERF architecture is switch and traffic generator agnostic and provides full control of software component versions and configurations as well as test-case customization.
76
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
The Danube release of VSPERF includes improvements in documentation and capabilities. This includes additional
test-cases such as RFC 5481 Latency test and RFC-2889 address-learning-rate test. Hardware traffic generator support
is now provided for Spirent and Xena in addition to Ixia. The Moongen software traffic generator is also now fully
supported. VSPERF can be used in a variety of modes for configuration and setup of the network and/or for control of
the test-generator and test execution.
• Wiki: https://wiki.opnfv.org/characterize_vswitch_performance_for_telco_nfv_use_cases
• Repository: https://git.opnfv.org/vswitchperf
• Artifacts: https://artifacts.opnfv.org/vswitchperf.html
• Continuous Integration status: https://build.opnfv.org/ci/view/vswitchperf/
VSPERF User Guide
Installing vswitchperf
Downloading vswitchperf The vswitchperf can be downloaded from its official git repository, which is hosted by
OPNFV. It is necessary to install a git at your DUT before downloading vswitchperf. Installation of git is specific
to the packaging system used by Linux OS installed at DUT.
Example of installation of GIT package and its dependencies:
• in case of OS based on RedHat Linux:
sudo yum install git
• in case of Ubuntu or Debian:
sudo apt-get install git
After the git is successfully installed at DUT, then vswitchperf can be downloaded as follows:
git clone http://git.opnfv.org/vswitchperf
The last command will create a directory vswitchperf with a local copy of vswitchperf repository.
Supported Operating Systems
• CentOS 7.3
• Fedora 24 (kernel 4.8 requires DPDK 16.11 and newer)
• Fedora 25 (kernel 4.9 requires DPDK 16.11 and newer)
• openSUSE 42.2
• RedHat 7.2 Enterprise Linux
• RedHat 7.3 Enterprise Linux
• Ubuntu 14.04
• Ubuntu 16.04
• Ubuntu 16.10 (kernel 4.8 requires DPDK 16.11 and newer)
2.2. Testing User Guides
77
OPNFV Documentation, Release Danube
Supported vSwitches The vSwitch must support Open Flow 1.3 or greater.
• Open vSwitch
• Open vSwitch with DPDK support
• TestPMD application from DPDK (supports p2p and pvp scenarios)
Supported Hypervisors
• Qemu version 2.3 or greater (version 2.5.0 is recommended)
Supported VNFs In theory, it is possible to use any VNF image, which is compatible with supported hypervisor.
However such VNF must ensure, that appropriate number of network interfaces is configured and that traffic is properly
forwarded among them. For new vswitchperf users it is recommended to start with official vloop-vnf image, which is
maintained by vswitchperf community.
vloop-vnf The official VM image is called vloop-vnf and it is available for free download from OPNFV artifactory.
This image is based on Linux Ubuntu distribution and it supports following applications for traffic forwarding:
• DPDK testpmd
• Linux Bridge
• Custom l2fwd module
The vloop-vnf can be downloaded to DUT, for example by wget:
wget http://artifacts.opnfv.org/vswitchperf/vnf/vloop-vnf-ubuntu-14.04_20160823.qcow2
NOTE: In case that wget is not installed at your DUT, you could install it at RPM based system by sudo yum
install wget or at DEB based system by sudo apt-get install wget.
Changelog of vloop-vnf:
• vloop-vnf-ubuntu-14.04_20160823
– ethtool installed
– only 1 NIC is configured by default to speed up boot with 1 NIC setup
– security updates applied
• vloop-vnf-ubuntu-14.04_20160804
– Linux kernel 4.4.0 installed
– libnuma-dev installed
– security updates applied
• vloop-vnf-ubuntu-14.04_20160303
– snmpd service is disabled by default to avoid error messages during VM boot
– security updates applied
• vloop-vnf-ubuntu-14.04_20151216
– version with development tools required for build of DPDK and l2fwd
78
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
Installation The test suite requires Python 3.3 or newer and relies on a number of other system and python packages.
These need to be installed for the test suite to function.
Installation of required packages, preparation of Python 3 virtual environment and compilation of OVS, DPDK and
QEMU is performed by script systems/build_base_machine.sh. It should be executed under user account, which will
be used for vsperf execution.
NOTE: Password-less sudo access must be configured for given user account before script is executed.
$ cd systems
$ ./build_base_machine.sh
NOTE: you don’t need to go into any of the systems subdirectories, simply run the top level build_base_machine.sh,
your OS will be detected automatically.
Script build_base_machine.sh will install all the vsperf dependencies in terms of system packages, Python 3.x and required Python modules. In case of CentOS 7 or RHEL it will install Python 3.3 from an additional repository provided
by Software Collections (a link). Installation script will also use virtualenv to create a vsperf virtual environment,
which is isolated from the default Python environment. This environment will reside in a directory called vsperfenv
in $HOME. It will ensure, that system wide Python installation is not modified or broken by VSPERF installation.
The complete list of Python packages installed inside virtualenv can be found at file requirements.txt, which is
located at vswitchperf repository.
NOTE: For RHEL 7.3 Enterprise and CentOS 7.3 OVS Vanilla is not built from upstream source due to kernel
incompatibilities. Please see the instructions in the vswitchperf_design document for details on configuring OVS
Vanilla for binary package usage.
Using vswitchperf You will need to activate the virtual environment every time you start a new shell session. Its
activation is specific to your OS:
• CentOS 7 and RHEL
$ scl enable python33 bash
$ source $HOME/vsperfenv/bin/activate
• Fedora and Ubuntu
$ source $HOME/vsperfenv/bin/activate
After the virtual environment is configued, then VSPERF can be used. For example:
(vsperfenv) $ cd vswitchperf
(vsperfenv) $ ./vsperf --help
Gotcha In case you will see following error during environment activation:
$ source $HOME/vsperfenv/bin/activate
Badly placed ()'s.
then check what type of shell you are using:
$ echo $SHELL
/bin/tcsh
See what scripts are available in $HOME/vsperfenv/bin
$ ls $HOME/vsperfenv/bin/
activate
activate.csh
2.2. Testing User Guides
activate.fish
activate_this.py
79
OPNFV Documentation, Release Danube
source the appropriate script
$ source bin/activate.csh
Working Behind a Proxy If you’re behind a proxy, you’ll likely want to configure this before running any of the
above. For example:
export http_proxy=proxy.mycompany.com:123
export https_proxy=proxy.mycompany.com:123
Hugepage Configuration Systems running vsperf with either dpdk and/or tests with guests must configure hugepage
amounts to support running these configurations. It is recommended to configure 1GB hugepages as the pagesize.
The amount of hugepages needed depends on your configuration files in vsperf. Each guest image requires 2048 MB
by default according to the default settings in the 04_vnf.conf file.
GUEST_MEMORY = ['2048']
The dpdk startup parameters also require an amount of hugepages depending on your configuration in the
02_vswitch.conf file.
VSWITCHD_DPDK_ARGS = ['-c', '0x4', '-n', '4', '--socket-mem 1024,1024']
VSWITCHD_DPDK_CONFIG = {
'dpdk-init' : 'true',
'dpdk-lcore-mask' : '0x4',
'dpdk-socket-mem' : '1024,1024',
}
NOTE: Option VSWITCHD_DPDK_ARGS is used for vswitchd, which supports --dpdk parameter. In recent
vswitchd versions, option VSWITCHD_DPDK_CONFIG is used to configure vswitchd via ovs-vsctl calls.
With the --socket-mem argument set to use 1 hugepage on the specified sockets as seen above, the configuration
will need 10 hugepages total to run all tests within vsperf if the pagesize is set correctly to 1GB.
VSPerf will verify hugepage amounts are free before executing test environments. In case of hugepage amounts not
being free, test initialization will fail and testing will stop.
NOTE: In some instances on a test failure dpdk resources may not release hugepages used in dpdk configuration.
It is recommended to configure a few extra hugepages to prevent a false detection by VSPerf that not enough free
hugepages are available to execute the test environment. Normally dpdk would use previously allocated hugepages
upon initialization.
Depending on your OS selection configuration of hugepages may vary. Please refer to your OS documentation to set
hugepages correctly. It is recommended to set the required amount of hugepages to be allocated by default on reboots.
Information on hugepage requirements for dpdk can be found at http://dpdk.org/doc/guides/linux_gsg/sys_reqs.html
You can review your hugepage amounts by executing the following command
cat /proc/meminfo | grep Huge
If no hugepages are available vsperf will try to automatically allocate some. Allocation is controlled by
HUGEPAGE_RAM_ALLOCATION configuration parameter in 02_vswitch.conf file. Default is 2GB, resulting
in either 2 1GB hugepages or 1024 2MB hugepages.
Upgrading vswitchperf
80
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
Generic In case, that VSPERF is cloned from git repository, then it is easy to upgrade it to the newest stable version
or to the development version.
You could get a list of stable releases by git command. It is necessary to update local git repository first.
NOTE: Git commands must be executed from directory, where VSPERF repository was cloned, e.g. vswitchperf.
Update of local git repository:
$ git pull
List of stable releases:
$ git tag
brahmaputra.1.0
colorado.1.0
colorado.2.0
colorado.3.0
danube.1.0
You could select which stable release should be used. For example, select danube.1.0:
$ git checkout danube.1.0
Development version of VSPERF can be selected by:
$ git checkout master
Colorado to Danube upgrade notes
Obsoleted features Support of vHost Cuse interface has been removed in Danube release. It means, that it is not
possible to select QemuDpdkVhostCuse as a VNF anymore. Option QemuDpdkVhostUser should be used
instead. Please check you configuration files and definition of your testcases for any occurrence of:
VNF = "QemuDpdkVhostCuse"
or
"VNF" : "QemuDpdkVhostCuse"
In case that QemuDpdkVhostCuse is found, it must be modified to QemuDpdkVhostUser.
NOTE: In case that execution of VSPERF is automated by scripts (e.g. for CI purposes), then these scripts must be
checked and updated too. It means, that any occurrence of:
./vsperf --vnf QemuDpdkVhostCuse
must be updated to:
./vsperf --vnf QemuDpdkVhostUser
Configuration Several configuration changes were introduced during Danube release. The most important changes
are discussed below.
2.2. Testing User Guides
81
OPNFV Documentation, Release Danube
Paths to DPDK, OVS and QEMU VSPERF uses external tools for proper testcase execution. Thus it is important
to properly configure paths to these tools. In case that tools are installed by installation scripts and are located inside
./src directory inside VSPERF home, then no changes are needed. On the other hand, if path settings was changed
by custom configuration file, then it is required to update configuration accordingly. Please check your configuration
files for following configuration options:
OVS_DIR
OVS_DIR_VANILLA
OVS_DIR_USER
OVS_DIR_CUSE
RTE_SDK_USER
RTE_SDK_CUSE
QEMU_DIR
QEMU_DIR_USER
QEMU_DIR_CUSE
QEMU_BIN
In case that any of these options is defined, then configuration must be updated. All paths to the tools are now stored
inside PATHS dictionary. Please refer to the Configuration of PATHS dictionary and update your configuration where
necessary.
Configuration change via CLI In previous releases it was possible to modify selected configuration options (mostly
VNF specific) via command line interface, i.e. by --test-params argument. This concept has been generalized
in Danube release and it is possible to modify any configuration parameter via CLI or via Parameters section of the
testcase definition. Old configuration options were obsoleted and it is required to specify configuration parameter
name in the same form as it is defined inside configuration file, i.e. in uppercase. Please refer to the Overriding values
defined in configuration files for additional details.
NOTE: In case that execution of VSPERF is automated by scripts (e.g. for CI purposes), then these scripts must be
checked and updated too. It means, that any occurrence of
guest_loopback
vanilla_tgen_port1_ip
vanilla_tgen_port1_mac
vanilla_tgen_port2_ip
vanilla_tgen_port2_mac
tunnel_type
shall be changed to the uppercase form and data type of entered values must match to data types of original values
from configuration files.
In case that guest_nic1_name or guest_nic2_name is changed, then new dictionary GUEST_NICS must be
modified accordingly. Please see Configuration of GUEST options and conf/04_vnf.conf for additional details.
Traffic configuration via CLI In previous releases it was possible to modify selected attributes of generated traffic
via command line interface. This concept has been enhanced in Danube release and it is now possible to modify all
traffic specific options via CLI or by TRAFFIC dictionary in configuration file. Detailed description is available at
Configuration of TRAFFIC dictionary section of documentation.
Please check your automated scripts for VSPERF execution for following CLI parameters and update them according
to the documentation:
bidir
duration
frame_rate
82
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
iload
lossrate
multistream
pkt_sizes
pre-installed_flows
rfc2544_tests
stream_type
traffic_type
‘vsperf’ Traffic Gen Guide
Overview VSPERF supports the following traffic generators:
• Dummy (DEFAULT)
• Ixia
• Spirent TestCenter
• Xena Networks
• MoonGen
To see the list of traffic gens from the cli:
$ ./vsperf --list-trafficgens
This guide provides the details of how to install and configure the various traffic generators.
Background Information
as follows:
The traffic default configuration can be found in conf/03_traffic.conf, and is configured
TRAFFIC = {
'traffic_type' : 'rfc2544_throughput',
'frame_rate' : 100,
'bidir' : 'True', # will be passed as string in title format to tgen
'multistream' : 0,
'stream_type' : 'L4',
'pre_installed_flows' : 'No',
# used by vswitch implementation
'flow_type' : 'port',
# used by vswitch implementation
'l2': {
'framesize': 64,
'srcmac': '00:00:00:00:00:00',
'dstmac': '00:00:00:00:00:00',
},
'l3': {
'proto': 'udp',
'srcip': '1.1.1.1',
'dstip': '90.90.90.90',
},
'l4': {
'srcport': 3000,
'dstport': 3001,
},
'vlan': {
'enabled': False,
'id': 0,
2.2. Testing User Guides
83
OPNFV Documentation, Release Danube
'priority': 0,
'cfi': 0,
},
}
The framesize parameter can be overridden from the configuration files by adding the following to your custom configuration file 10_custom.conf:
TRAFFICGEN_PKT_SIZES = (64, 128,)
OR from the commandline:
$ ./vsperf --test-params "TRAFFICGEN_PKT_SIZES=(x,y)" $TESTNAME
You can also modify the traffic transmission duration and the number of tests run by the traffic generator by extending
the example commandline above to:
$ ./vsperf --test-params "TRAFFICGEN_PKT_SIZES=(x,y);TRAFFICGEN_DURATION=10;" \
"TRAFFICGEN_RFC2544_TESTS=1" $TESTNAME
Dummy The Dummy traffic generator can be used to test VSPERF installation or to demonstrate VSPERF functionality at DUT without connection to a real traffic generator.
You could also use the Dummy generator in case, that your external traffic generator is not supported by VSPERF. In
such case you could use VSPERF to setup your test scenario and then transmit the traffic. After the transmission is
completed you could specify values for all collected metrics and VSPERF will use them to generate final reports.
Setup To select the Dummy generator please add the following to your custom configuration file
10_custom.conf.
TRAFFICGEN = 'Dummy'
OR run vsperf with the --trafficgen argument
$ ./vsperf --trafficgen Dummy $TESTNAME
Where $TESTNAME is the name of the vsperf test you would like to run. This will setup the vSwitch and the VNF
(if one is part of your test) print the traffic configuration and prompt you to transmit traffic when the setup is complete.
Please send 'continuous' traffic with the following stream config:
30mS, 90mpps, multistream False
and the following flow config:
{
"flow_type": "port",
"l3": {
"srcip": "1.1.1.1",
"proto": "tcp",
"dstip": "90.90.90.90"
},
"traffic_type": "rfc2544_continuous",
"multistream": 0,
"bidir": "True",
"vlan": {
"cfi": 0,
"priority": 0,
"id": 0,
"enabled": false
},
84
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
"frame_rate": 90,
"l2": {
"dstport": 3001,
"srcport": 3000,
"dstmac": "00:00:00:00:00:00",
"srcmac": "00:00:00:00:00:00",
"framesize": 64
}
}
What was the result for 'frames tx'?
When your traffic generator has completed traffic transmission and provided the results please input these at the
VSPERF prompt. VSPERF will try to verify the input:
Is '$input_value' correct?
Please answer with y OR n.
VSPERF will ask you to provide a value for every of collected metrics. The list of metrics can be found at traffic-typemetrics. Finally vsperf will print out the results for your test and generate the appropriate logs and report files.
Metrics collected for supported traffic types Below you could find a list of metrics collected by VSPERF for each
of supported traffic types.
RFC2544 Throughput and Continuous:
• frames tx
• frames rx
• min latency
• max latency
• avg latency
• frameloss
RFC2544 Back2back:
• b2b frames
• b2b frame loss %
Dummy result pre-configuration In case of a Dummy traffic generator it is possible to pre-configure the test results.
This is useful for creation of demo testcases, which do not require a real traffic generator. Such testcase can be run by
any user and it will still generate all reports and result files.
Result values can be specified within TRAFFICGEN_DUMMY_RESULTS dictionary, where every of collected metrics
must be properly defined. Please check the list of traffic-type-metrics.
Dictionary with dummy results can be passed by CLI argument --test-params or specified in Parameters
section of testcase definition.
Example of testcase execution with dummy results defined by CLI argument:
$ ./vsperf back2back --trafficgen Dummy --test-params \
"TRAFFICGEN_DUMMY_RESULTS={'b2b frames':'3000','b2b frame loss %':'0.0'}"
Example of testcase definition with pre-configured dummy results:
2.2. Testing User Guides
85
OPNFV Documentation, Release Danube
{
"Name": "back2back",
"Traffic Type": "rfc2544_back2back",
"Deployment": "p2p",
"biDirectional": "True",
"Description": "LTD.Throughput.RFC2544.BackToBackFrames",
"Parameters" : {
'TRAFFICGEN_DUMMY_RESULTS' : {'b2b frames':'3000','b2b frame loss %':'0.0'}
},
},
NOTE: Pre-configured results for the Dummy traffic generator will be used only in case, that the Dummy traffic
generator is used. Otherwise the option TRAFFICGEN_DUMMY_RESULTS will be ignored.
Ixia VSPERF can use both IxNetwork and IxExplorer TCL servers to control Ixia chassis. However usage of IxNetwork TCL server is a preferred option. Following sections will describe installation and configuration of IxNetwork
components used by VSPERF.
Installation On the system under the test you need to install IxNetworkTclClient$(VER_NUM)Linux.bin.tgz.
On the IXIA client software system you need to install IxNetwork TCL server. After its installation you should
configure it as follows:
1. Find the IxNetwork TCL server app (start -> All Programs -> IXIA -> IxNetwork -> IxNetwork_$(VER_NUM)
-> IxNetwork TCL Server)
2. Right click on IxNetwork TCL Server, select properties - Under shortcut tab in the Target dialogue box make
sure there is the argument “-tclport xxxx” where xxxx is your port number (take note of this port number as you
will need it for the 10_custom.conf file).
86
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
3. Hit Ok and start the TCL server application
VSPERF configuration There are several configuration options specific to the IxNetwork traffic generator from
IXIA. It is essential to set them correctly, before the VSPERF is executed for the first time.
Detailed description of options follows:
• TRAFFICGEN_IXNET_MACHINE - IP address of server, where IxNetwork TCL Server is running
• TRAFFICGEN_IXNET_PORT - PORT, where IxNetwork TCL Server is accepting connections from TCL
clients
• TRAFFICGEN_IXNET_USER - username, which will be used during communication with IxNetwork TCL
Server and IXIA chassis
• TRAFFICGEN_IXIA_HOST - IP address of IXIA traffic generator chassis
• TRAFFICGEN_IXIA_CARD - identification of card with dedicated ports at IXIA chassis
• TRAFFICGEN_IXIA_PORT1 - identification of the first dedicated port at TRAFFICGEN_IXIA_CARD at
IXIA chassis; VSPERF uses two separated ports for traffic generation. In case of unidirectional traffic, it
is essential to correctly connect 1st IXIA port to the 1st NIC at DUT, i.e. to the first PCI handle from
WHITELIST_NICS list. Otherwise traffic may not be able to pass through the vSwitch.
2.2. Testing User Guides
87
OPNFV Documentation, Release Danube
• TRAFFICGEN_IXIA_PORT2 - identification of the second dedicated port at TRAFFICGEN_IXIA_CARD
at IXIA chassis; VSPERF uses two separated ports for traffic generation. In case of unidirectional traffic, it
is essential to correctly connect 2nd IXIA port to the 2nd NIC at DUT, i.e. to the second PCI handle from
WHITELIST_NICS list. Otherwise traffic may not be able to pass through the vSwitch.
• TRAFFICGEN_IXNET_LIB_PATH - path to the DUT specific installation of IxNetwork TCL API
• TRAFFICGEN_IXNET_TCL_SCRIPT - name of the TCL script, which VSPERF will use for communication
with IXIA TCL server
• TRAFFICGEN_IXNET_TESTER_RESULT_DIR - folder accessible from IxNetwork TCL server, where test
results are stored, e.g. c:/ixia_results; see test-results-share
• TRAFFICGEN_IXNET_DUT_RESULT_DIR - directory accessible from the DUT, where test results from
IxNetwork TCL server are stored, e.g. /mnt/ixia_results; see test-results-share
Test results share VSPERF is not able to retrieve test results via TCL API directly. Instead, all test results are stored
at IxNetwork TCL server. Results are stored at folder defined by TRAFFICGEN_IXNET_TESTER_RESULT_DIR
configuration parameter. Content of this folder must be shared (e.g. via samba protocol) between TCL Server and
DUT, where VSPERF is executed. VSPERF expects, that test results will be available at directory configured by
TRAFFICGEN_IXNET_DUT_RESULT_DIR configuration parameter.
Example of sharing configuration:
• Create a new folder at IxNetwork TCL server machine, e.g. c:\ixia_results
• Modify sharing options of ixia_results folder to share it with everybody
• Create a new directory at DUT, where shared directory with results will be mounted,
/mnt/ixia_results
e.g.
• Update your custom VSPERF configuration file as follows:
TRAFFICGEN_IXNET_TESTER_RESULT_DIR = 'c:/ixia_results'
TRAFFICGEN_IXNET_DUT_RESULT_DIR = '/mnt/ixia_results'
NOTE:
It
is
essential
to
use
slashes
‘/’
TRAFFICGEN_IXNET_TESTER_RESULT_DIR parameter.
also
in
path
configured
by
• Install cifs-utils package.
e.g. at rpm based Linux distribution:
yum install cifs-utils
• Mount shared directory, so VSPERF can access test results.
e.g. by adding new record into /etc/fstab
mount -t cifs //_TCL_SERVER_IP_OR_FQDN_/ixia_results /mnt/ixia_results
-o file_mode=0777,dir_mode=0777,nounix
It is recommended to verify, that any new file inserted into c:/ixia_results folder is visible at DUT inside
/mnt/ixia_results directory.
Spirent Setup Spirent installation files and instructions are available on the Spirent support website at:
http://support.spirent.com
Select a version of Spirent TestCenter software to utilize. This example will use Spirent TestCenter v4.57 as an
example. Substitute the appropriate version in place of ‘v4.57’ in the examples, below.
88
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
On the CentOS 7 System Download and install the following:
Spirent TestCenter Application, v4.57 for 64-bit Linux Client
Spirent Virtual Deployment Service (VDS) Spirent VDS is required for both TestCenter hardware and virtual
chassis in the vsperf environment. For installation, select the version that matches the Spirent TestCenter Application
version. For v4.57, the matching VDS version is 1.0.55. Download either the ova (VMware) or qcow2 (QEMU) image
and create a VM with it. Initialize the VM according to Spirent installation instructions.
Using Spirent TestCenter Virtual (STCv) STCv is available in both ova (VMware) and qcow2 (QEMU) formats.
For VMware, download:
Spirent TestCenter Virtual Machine for VMware, v4.57 for Hypervisor - VMware ESX.ESXi
Virtual test port performance is affected by the hypervisor configuration. For best practice results in deploying STCv,
the following is suggested:
• Create a single VM with two test ports rather than two VMs with one port each
• Set STCv in DPDK mode
• Give STCv 2*n + 1 cores, where n = the number of ports. For vsperf, cores = 5.
• Turning off hyperthreading and pinning these cores will improve performance
• Give STCv 2 GB of RAM
To get the highest performance and accuracy, Spirent TestCenter hardware is recommended. vsperf can run with either
stype test ports.
Using STC REST Client The stcrestclient package provides the stchttp.py ReST API wrapper module. This allows
simple function calls, nearly identical to those provided by StcPython.py, to be used to access TestCenter server
sessions via the STC ReST API. Basic ReST functionality is provided by the resthttp module, and may be used for
writing ReST clients independent of STC.
• Project page: <https://github.com/Spirent/py-stcrestclient>
• Package download: <http://pypi.python.org/pypi/stcrestclient>
To use REST interface, follow the instructions in the Project page to install the package. Once installed, the scripts
named with ‘rest’ keyword can be used. For example: testcenter-rfc2544-rest.py can be used to run RFC 2544 tests
using the REST interface.
Configuration:
1. The Labserver and license server addresses. These parameters applies to all the tests, and are mandatory for all
tests.
TRAFFICGEN_STC_LAB_SERVER_ADDR = " "
TRAFFICGEN_STC_LICENSE_SERVER_ADDR = " "
TRAFFICGEN_STC_PYTHON2_PATH = " "
TRAFFICGEN_STC_TESTCENTER_PATH = " "
TRAFFICGEN_STC_TEST_SESSION_NAME = " "
TRAFFICGEN_STC_CSV_RESULTS_FILE_PREFIX = " "
2. For RFC2544 tests, the following parameters are mandatory
2.2. Testing User Guides
89
OPNFV Documentation, Release Danube
TRAFFICGEN_STC_EAST_CHASSIS_ADDR = " "
TRAFFICGEN_STC_EAST_SLOT_NUM = " "
TRAFFICGEN_STC_EAST_PORT_NUM = " "
TRAFFICGEN_STC_EAST_INTF_ADDR = " "
TRAFFICGEN_STC_EAST_INTF_GATEWAY_ADDR = " "
TRAFFICGEN_STC_WEST_CHASSIS_ADDR = ""
TRAFFICGEN_STC_WEST_SLOT_NUM = " "
TRAFFICGEN_STC_WEST_PORT_NUM = " "
TRAFFICGEN_STC_WEST_INTF_ADDR = " "
TRAFFICGEN_STC_WEST_INTF_GATEWAY_ADDR = " "
TRAFFICGEN_STC_RFC2544_TPUT_TEST_FILE_NAME
3. RFC2889 tests: Currently, the forwarding, address-caching, and address-learning-rate tests of RFC2889 are
supported. The testcenter-rfc2889-rest.py script implements the rfc2889 tests. The configuration for RFC2889
involves test-case definition, and parameter definition, as described below. New results-constants, as shown
below, are added to support these tests.
Example of testcase definition for RFC2889 tests:
{
"Name": "phy2phy_forwarding",
"Deployment": "p2p",
"Description": "LTD.Forwarding.RFC2889.MaxForwardingRate",
"Parameters" : {
"TRAFFIC" : {
"traffic_type" : "rfc2889_forwarding",
},
},
}
For RFC2889 tests, specifying the locations for the monitoring ports is mandatory. Necessary parameters are:
Other Configurations are :
TRAFFICGEN_STC_RFC2889_MIN_LR = 1488
TRAFFICGEN_STC_RFC2889_MAX_LR = 14880
TRAFFICGEN_STC_RFC2889_MIN_ADDRS = 1000
TRAFFICGEN_STC_RFC2889_MAX_ADDRS = 65536
TRAFFICGEN_STC_RFC2889_AC_LR = 1000
The first 2 values are for address-learning test where as other 3 values are for the Address caching capacity test. LR:
Learning Rate. AC: Address Caching. Maximum value for address is 16777216. Whereas, maximum for LR is
4294967295.
Results for RFC2889 Tests: Forwarding tests outputs following values:
TX_RATE_FPS : "Transmission Rate in Frames/sec"
THROUGHPUT_RX_FPS: "Received Throughput Frames/sec"
TX_RATE_MBPS : " Transmission rate in MBPS"
THROUGHPUT_RX_MBPS: "Received Throughput in MBPS"
TX_RATE_PERCENT: "Transmission Rate in Percentage"
FRAME_LOSS_PERCENT: "Frame loss in Percentage"
FORWARDING_RATE_FPS: " Maximum Forwarding Rate in FPS"
Whereas, the address caching test outputs following values,
CACHING_CAPACITY_ADDRS = 'Number of address it can cache'
ADDR_LEARNED_PERCENT = 'Percentage of address successfully learned'
and address learning test outputs just a single value:
90
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
OPTIMAL_LEARNING_RATE_FPS = 'Optimal learning rate in fps'
Note that ‘FORWARDING_RATE_FPS’, ‘CACHING_CAPACITY_ADDRS’, ‘ADDR_LEARNED_PERCENT’ and
‘OPTIMAL_LEARNING_RATE_FPS’ are the new result-constants added to support RFC2889 tests.
Xena Networks
Installation Xena Networks traffic generator requires specific files and packages to be installed. It is assumed
the user has access to the Xena2544.exe file which must be placed in VSPerf installation location under the
tools/pkt_gen/xena folder. Contact Xena Networks for the latest version of this file. The user can also visit
www.xenanetworks/downloads to obtain the file with a valid support contract.
Note VSPerf has been fully tested with version v2.43 of Xena2544.exe
To execute the Xena2544.exe file under Linux distributions the mono-complete package must be installed. To
install this package follow the instructions below. Further information can be obtained from http://www.monoproject.com/docs/getting-started/install/linux/
rpm --import "http://keyserver.ubuntu.com/pks/lookup?op=get&search=0x3FA7E0328081BFF6A14DA29AA6A19B38
yum-config-manager --add-repo http://download.mono-project.com/repo/centos/
yum -y install mono-complete
To prevent gpg errors on future yum installation of packages the mono-project repo should be disabled once installed.
yum-config-manager --disable download.mono-project.com_repo_centos_
Configuration Connection information for your Xena Chassis must be supplied inside the 10_custom.conf or
03_custom.conf file. The following parameters must be set to allow for proper connections to the chassis.
TRAFFICGEN_XENA_IP = ''
TRAFFICGEN_XENA_PORT1 = ''
TRAFFICGEN_XENA_PORT2 = ''
TRAFFICGEN_XENA_USER = ''
TRAFFICGEN_XENA_PASSWORD = ''
TRAFFICGEN_XENA_MODULE1 = ''
TRAFFICGEN_XENA_MODULE2 = ''
RFC2544 Throughput Testing Xena traffic generator testing for rfc2544 throughput can be modified for different
behaviors if needed. The default options for the following are optimized for best results.
TRAFFICGEN_XENA_2544_TPUT_INIT_VALUE = '10.0'
TRAFFICGEN_XENA_2544_TPUT_MIN_VALUE = '0.1'
TRAFFICGEN_XENA_2544_TPUT_MAX_VALUE = '100.0'
TRAFFICGEN_XENA_2544_TPUT_VALUE_RESOLUTION = '0.5'
TRAFFICGEN_XENA_2544_TPUT_USEPASS_THRESHHOLD = 'false'
TRAFFICGEN_XENA_2544_TPUT_PASS_THRESHHOLD = '0.0'
Each value modifies the behavior of rfc 2544 throughput testing. Refer to your Xena documentation to understand the
behavior changes in modifying these values.
Continuous Traffic Testing Xena continuous traffic by default does a 3 second learning preemption to allow the
DUT to receive learning packets before a continuous test is performed. If a custom test case requires this learning be
disabled, you can disable the option or modify the length of the learning by modifying the following settings.
2.2. Testing User Guides
91
OPNFV Documentation, Release Danube
TRAFFICGEN_XENA_CONT_PORT_LEARNING_ENABLED = False
TRAFFICGEN_XENA_CONT_PORT_LEARNING_DURATION = 3
MoonGen
Installation MoonGen architecture overview and general installation instructions can be found here:
https://github.com/emmericp/MoonGen
• Note: Today, MoonGen with VSPERF only supports 10Gbps line speeds.
For VSPERF use, MoonGen should be cloned from here (as opposed to the previously mentioned GitHub):
git clone https://github.com/atheurer/lua-trafficgen
and use the master branch:
git checkout master
VSPERF uses a particular Lua script with the MoonGen project:
trafficgen.lua
Follow MoonGen set up and execution instructions here:
https://github.com/atheurer/lua-trafficgen/blob/master/README.md
Note one will need to set up ssh login to not use passwords between the server running MoonGen and the device under
test (running the VSPERF test infrastructure). This is because VSPERF on one server uses ‘ssh’ to configure and run
MoonGen upon the other server.
One can set up this ssh access by doing the following on both servers:
ssh-keygen -b 2048 -t rsa
ssh-copy-id <other server>
Configuration Connection information for MoonGen must be supplied inside the 10_custom.conf or
03_custom.conf file. The following parameters must be set to allow for proper connections to the host with
MoonGen.
TRAFFICGEN_MOONGEN_HOST_IP_ADDR = ""
TRAFFICGEN_MOONGEN_USER = ""
TRAFFICGEN_MOONGEN_BASE_DIR = ""
TRAFFICGEN_MOONGEN_PORTS = ""
TRAFFICGEN_MOONGEN_LINE_SPEED_GBPS = ""
vSwitchPerf test suites userguide
General VSPERF requires a traffic generators to run tests, automated traffic gen support in VSPERF includes:
• IXIA traffic generator (IxNetwork hardware) and a machine that runs the IXIA client software.
• Spirent traffic generator (TestCenter hardware chassis or TestCenter virtual in a VM) and a VM to run the Spirent
Virtual Deployment Service image, formerly known as “Spirent LabServer”.
• Xena Network traffic generator (Xena hardware chassis) that houses the Xena Traffic generator modules.
• Moongen software traffic generator. Requires a separate machine running moongen to execute packet generation.
92
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
If you want to use another traffic generator, please select the Dummy generator.
VSPERF Installation To see the supported Operating Systems, vSwitches and system requirements, please follow
the installation instructions <vsperf-installation>.
Traffic Generator Setup Follow the Traffic generator instructions <trafficgen-installation> to install and configure
a suitable traffic generator.
Cloning and building src dependencies In order to run VSPERF, you will need to download DPDK and OVS. You
can do this manually and build them in a preferred location, OR you could use vswitchperf/src. The vswitchperf/src
directory contains makefiles that will allow you to clone and build the libraries that VSPERF depends on, such as
DPDK and OVS. To clone and build simply:
$ cd src
$ make
VSPERF can be used with stock OVS (without DPDK support). When build is finished, the libraries are stored in
src_vanilla directory.
The ‘make’ builds all options in src:
• Vanilla OVS
• OVS with vhost_user as the guest access method (with DPDK support)
The vhost_user build will reside in src/ovs/ The Vanilla OVS build will reside in vswitchperf/src_vanilla
To delete a src subdirectory and its contents to allow you to re-clone simply use:
$ make clobber
Configure the ./conf/10_custom.conf file The 10_custom.conf file is the configuration file that overrides default configurations in all the other configuration files in ./conf The supplied 10_custom.conf file
MUST be modified, as it contains configuration items for which there are no reasonable default values.
The configuration items that can be added is not limited to the initial contents. Any configuration item mentioned in
any .conf file in ./conf directory can be added and that item will be overridden by the custom configuration value.
Further details about configuration files evaluation and special behaviour of options with GUEST_ prefix could be
found at design document.
Using a custom settings file If your 10_custom.conf doesn’t reside in the ./conf directory of if you want to
use an alternative configuration file, the file can be passed to vsperf via the --conf-file argument.
$ ./vsperf --conf-file <path_to_custom_conf> ...
Note that configuration passed in via the environment (--load-env) or via another command line argument will
override both the default and your custom configuration files. This “priority hierarchy” can be described like so (1 =
max priority):
1. Testcase definition section Parameters
2. Command line arguments
3. Environment variables
4. Configuration file(s)
2.2. Testing User Guides
93
OPNFV Documentation, Release Danube
Further details about configuration files evaluation and special behaviour of options with GUEST_ prefix could be
found at design document.
Overriding values defined in configuration files The configuration items can be overridden by command line
argument --test-params. In this case, the configuration items and their values should be passed in form of
item=value and separated by semicolon.
Example:
$ ./vsperf --test-params "TRAFFICGEN_DURATION=10;TRAFFICGEN_PKT_SIZES=(128,);" \
"GUEST_LOOPBACK=['testpmd','l2fwd']" pvvp_tput
The second option is to override configuration items by Parameters section of the test case definition. The configuration items can be added into Parameters dictionary with their new values. These values will override values
defined in configuration files or specified by --test-params command line argument.
Example:
"Parameters" : {'TRAFFICGEN_PKT_SIZES' : (128,),
'TRAFFICGEN_DURATION' : 10,
'GUEST_LOOPBACK' : ['testpmd','l2fwd'],
}
NOTE: In both cases, configuration item names and their values must be specified in the same form as they are
defined inside configuration files. Parameter names must be specified in uppercase and data types of original and new
value must match. Python syntax rules related to data types and structures must be followed. For example, parameter
TRAFFICGEN_PKT_SIZES above is defined as a tuple with a single value 128. In this case trailing comma is
mandatory, otherwise value can be wrongly interpreted as a number instead of a tuple and vsperf execution would
fail. Please check configuration files for default values and their types and use them as a basis for any customized
values. In case of any doubt, please check official python documentation related to data structures like tuples, lists and
dictionaries.
NOTE: Vsperf execution will terminate with runtime error in case, that unknown parameter name is passed via
--test-params CLI argument or defined in Parameters section of test case definition. It is also forbidden
to redefine a value of TEST_PARAMS configuration item via CLI or Parameters section.
vloop_vnf VSPERF uses a VM image called vloop_vnf for looping traffic in the deployment scenarios involving
VMs. The image can be downloaded from http://artifacts.opnfv.org/.
Please see the installation instructions for information on vloop-vnf images.
l2fwd Kernel Module A Kernel Module that provides OSI Layer 2 Ipv4 termination or forwarding with support
for Destination Network Address Translation (DNAT) for both the MAC and IP addresses. l2fwd can be found in
<vswitchperf_dir>/src/l2fwd
Executing tests All examples inside these docs assume, that user is inside the VSPERF directory. VSPERF can be
executed from any directory.
Before running any tests make sure you have root permissions by adding the following line to /etc/sudoers:
username ALL=(ALL)
NOPASSWD: ALL
username in the example above should be replaced with a real username.
To list the available tests:
94
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
$ ./vsperf --list
To run a single test:
$ ./vsperf $TESTNAME
Where $TESTNAME is the name of the vsperf test you would like to run.
To run a group of tests, for example all tests with a name containing ‘RFC2544’:
$ ./vsperf --conf-file=<path_to_custom_conf>/10_custom.conf --tests="RFC2544"
To run all tests:
$ ./vsperf --conf-file=<path_to_custom_conf>/10_custom.conf
Some tests allow for configurable parameters, including test duration (in seconds) as well as packet sizes (in bytes).
$ ./vsperf --conf-file user_settings.py \
--tests RFC2544Tput \
--test-params "TRAFFICGEN_DURATION=10;TRAFFICGEN_PKT_SIZES=(128,)"
For all available options, check out the help dialog:
$ ./vsperf --help
Executing Vanilla OVS tests
1. If needed, recompile src for all OVS variants
$ cd src
$ make distclean
$ make
2. Update your 10_custom.conf file to use Vanilla OVS:
VSWITCH = 'OvsVanilla'
3. Run test:
$ ./vsperf --conf-file=<path_to_custom_conf>
Please note if you don’t want to configure Vanilla OVS through the configuration file, you can pass it as a CLI
argument.
$ ./vsperf --vswitch OvsVanilla
Executing tests with VMs To run tests using vhost-user as guest access method:
1. Set VHOST_METHOD and VNF of your settings file to:
VSWITCH = 'OvsDpdkVhost'
VNF = 'QemuDpdkVhost'
2. If needed, recompile src for all OVS variants
$ cd src
$ make distclean
$ make
2.2. Testing User Guides
95
OPNFV Documentation, Release Danube
3. Run test:
$ ./vsperf --conf-file=<path_to_custom_conf>/10_custom.conf
Executing tests with VMs using Vanilla OVS To run tests using Vanilla OVS:
1. Set the following variables:
VSWITCH = 'OvsVanilla'
VNF = 'QemuVirtioNet'
VANILLA_TGEN_PORT1_IP = n.n.n.n
VANILLA_TGEN_PORT1_MAC = nn:nn:nn:nn:nn:nn
VANILLA_TGEN_PORT2_IP = n.n.n.n
VANILLA_TGEN_PORT2_MAC = nn:nn:nn:nn:nn:nn
VANILLA_BRIDGE_IP = n.n.n.n
or use --test-params option
$ ./vsperf --conf-file=<path_to_custom_conf>/10_custom.conf \
--test-params "VANILLA_TGEN_PORT1_IP=n.n.n.n;" \
"VANILLA_TGEN_PORT1_MAC=nn:nn:nn:nn:nn:nn;" \
"VANILLA_TGEN_PORT2_IP=n.n.n.n;" \
"VANILLA_TGEN_PORT2_MAC=nn:nn:nn:nn:nn:nn"
2. If needed, recompile src for all OVS variants
$ cd src
$ make distclean
$ make
3. Run test:
$ ./vsperf --conf-file<path_to_custom_conf>/10_custom.conf
Using vfio_pci with DPDK To use vfio with DPDK instead of igb_uio add into your custom configuration file the
following parameter:
PATHS['dpdk']['src']['modules'] = ['uio', 'vfio-pci']
NOTE:
In
case,
that
DPDK
is
installed
PATHS[’dpdk’][’bin’][’modules’] instead.
from
binary
package,
then
please
set
NOTE: Please ensure that Intel VT-d is enabled in BIOS.
NOTE: Please ensure your boot/grub parameters include the following:
iommu=pt intel_iommu=on
To check that IOMMU is enabled on your platform:
$ dmesg | grep
[
0.000000]
[
0.139882]
[
0.139888]
[
0.139893]
[
0.139894]
96
IOMMU
Intel-IOMMU: enabled
dmar: IOMMU 0: reg_base_addr
dmar: IOMMU 1: reg_base_addr
IOAPIC id 2 under DRHD base
IOAPIC id 0 under DRHD base
fbffe000 ver 1:0
ebffc000 ver 1:0
0xfbffe000 IOMMU
0xebffc000 IOMMU
cap d2078c106f0466 ecap f020de
cap d2078c106f0466 ecap f020de
0
1
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
[
0.139895] IOAPIC id 1 under DRHD base 0xebffc000 IOMMU 1
[
3.335744] IOMMU: dmar0 using Queued invalidation
[
3.335746] IOMMU: dmar1 using Queued invalidation
....
Using SRIOV support To use virtual functions of NIC with SRIOV support, use extended form of NIC PCI slot
definition:
WHITELIST_NICS = ['0000:05:00.0|vf0', '0000:05:00.1|vf3']
Where ‘vf’ is an indication of virtual function usage and following number defines a VF to be used. In case that VF
usage is detected, then vswitchperf will enable SRIOV support for given card and it will detect PCI slot numbers of
selected VFs.
So in example above, one VF will be configured for NIC ‘0000:05:00.0’ and four VFs will be configured for NIC
‘0000:05:00.1’. Vswitchperf will detect PCI addresses of selected VFs and it will use them during test execution.
At the end of vswitchperf execution, SRIOV support will be disabled.
SRIOV support is generic and it can be used in different testing scenarios. For example:
• vSwitch tests with DPDK or without DPDK support to verify impact of VF usage on vSwitch performance
• tests without vSwitch, where traffic is forwared directly between VF interfaces by packet forwarder (e.g. testpmd
application)
• tests without vSwitch, where VM accesses VF interfaces directly by PCI-passthrough to measure raw VM
throughput performance.
Using QEMU with PCI passthrough support Raw virtual machine throughput performance can be measured by
execution of PVP test with direct access to NICs by PCI passthrough. To execute VM with direct access to PCI
devices, enable vfio-pci. In order to use virtual functions, SRIOV-support must be enabled.
Execution of test with PCI passthrough with vswitch disabled:
$ ./vsperf --conf-file=<path_to_custom_conf>/10_custom.conf \
--vswitch none --vnf QemuPciPassthrough pvp_tput
Any of supported guest-loopback-application can be used inside VM with PCI passthrough support.
Note: Qemu with PCI passthrough support can be used only with PVP test deployment.
Selection of loopback application for tests with VMs To select the loopback applications which will forward
packets inside VMs, the following parameter should be configured:
GUEST_LOOPBACK = ['testpmd']
or use --test-params CLI argument:
$ ./vsperf --conf-file=<path_to_custom_conf>/10_custom.conf \
--test-params "GUEST_LOOPBACK=['testpmd']"
Supported loopback applications are:
'testpmd'
'l2fwd'
'linux_bridge'
'buildin'
-
testpmd from dpdk will be built and used
l2fwd module provided by Huawei will be built and used
linux bridge will be configured
nothing will be configured by vsperf; VM image must
ensure traffic forwarding between its interfaces
2.2. Testing User Guides
97
OPNFV Documentation, Release Danube
Guest loopback application must be configured, otherwise traffic will not be forwarded by VM and testcases with VM
related deployments will fail. Guest loopback application is set to ‘testpmd’ by default.
NOTE: In case that only 1 or more than 2 NICs are configured for VM, then ‘testpmd’ should be used. As it is able to
forward traffic between multiple VM NIC pairs.
NOTE: In case of linux_bridge, all guest NICs are connected to the same bridge inside the guest.
Mergable Buffers Options with QEMU Mergable buffers can be disabled with VSPerf within QEMU. This option can increase performance significantly when not using jumbo frame sized packets. By default VSPerf disables
mergable buffers. If you wish to enable it you can modify the setting in the a custom conf file.
GUEST_NIC_MERGE_BUFFERS_DISABLE = [False]
Then execute using the custom conf file.
$ ./vsperf --conf-file=<path_to_custom_conf>/10_custom.conf
Alternatively you can just pass the param during execution.
$ ./vsperf --test-params "GUEST_NIC_MERGE_BUFFERS_DISABLE=[False]"
Selection of dpdk binding driver for tests with VMs To select dpdk binding driver, which will specify which
driver the vm NICs will use for dpdk bind, the following configuration parameter should be configured:
GUEST_DPDK_BIND_DRIVER = ['igb_uio_from_src']
The supported dpdk guest bind drivers are:
'uio_pci_generic'
'igb_uio_from_src'
'vfio_no_iommu'
- Use uio_pci_generic driver
- Build and use the igb_uio driver from the dpdk src
files
- Use vfio with no iommu option. This requires custom
guest images that support this option. The default
vloop image does not support this driver.
Note: uio_pci_generic does not support sr-iov testcases with guests attached. This is because uio_pci_generic only
supports legacy interrupts. In case uio_pci_generic is selected with the vnf as QemuPciPassthrough it will be modified
to use igb_uio_from_src instead.
Note: vfio_no_iommu requires kernels equal to or greater than 4.5 and dpdk 16.04 or greater. Using this option will
also taint the kernel.
Please refer to the dpdk documents at http://dpdk.org/doc/guides for more information on these drivers.
Multi-Queue Configuration VSPerf currently supports multi-queue with the following limitations:
1. Requires QEMU 2.5 or greater and any OVS version higher than 2.5. The default upstream package versions
installed by VSPerf satisfies this requirement.
2. Guest image must have ethtool utility installed if using l2fwd or linux bridge inside guest for loopback.
3. If using OVS versions 2.5.0 or less enable old style multi-queue as shown in the “02_vswitch.conf” file.
OVS_OLD_STYLE_MQ = True
To enable multi-queue for dpdk modify the “02_vswitch.conf” file.
98
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
VSWITCH_DPDK_MULTI_QUEUES = 2
NOTE: you should consider using the switch affinity to set a pmd cpu mask that can optimize your performance.
Consider the numa of the NIC in use if this applies by checking /sys/class/net/<eth_name>/device/numa_node and
setting an appropriate mask to create PMD threads on the same numa node.
When multi-queue is enabled, each dpdk or dpdkvhostuser port that is created on the switch will set the option for
multiple queues. If old style multi queue has been enabled a global option for multi queue will be used instead of the
port by port option.
To enable multi-queue on the guest modify the “04_vnf.conf” file.
GUEST_NIC_QUEUES = [2]
Enabling multi-queue at the guest will add multiple queues to each NIC port when qemu launches the guest.
In case of Vanilla OVS, multi-queue is enabled on the tuntap ports and nic queues will be enabled inside the guest with
ethtool. Simply enabling the multi-queue on the guest is sufficient for Vanilla OVS multi-queue.
Testpmd should be configured to take advantage of multi-queue on the guest if using DPDKVhostUser. This can be
done by modifying the “04_vnf.conf” file.
GUEST_TESTPMD_PARAMS = ['-l 0,1,2,3,4 -n 4 --socket-mem 512 -- '
'--burst=64 -i --txqflags=0xf00 '
'--nb-cores=4 --rxq=2 --txq=2 '
'--disable-hw-vlan']
NOTE: The guest SMP cores must be configured to allow for testpmd to use the optimal number of cores to take
advantage of the multiple guest queues.
In case of using Vanilla OVS and qemu virtio-net you can increase performance by binding vhost-net threads to cpus.
This can be done by enabling the affinity in the “04_vnf.conf” file. This can be done to non multi-queue enabled
configurations as well as there will be 2 vhost-net threads.
VSWITCH_VHOST_NET_AFFINITIZATION = True
VSWITCH_VHOST_CPU_MAP = [4,5,8,11]
NOTE: This method of binding would require a custom script in a real environment.
NOTE: For optimal performance guest SMPs and/or vhost-net threads should be on the same numa as the NIC in use
if possible/applicable. Testpmd should be assigned at least (nb_cores +1) total cores with the cpu mask.
Executing Packet Forwarding tests To select the applications which will forward packets, the following parameters
should be configured:
VSWITCH = 'none'
PKTFWD = 'TestPMD'
or use --vswitch and --fwdapp CLI arguments:
$ ./vsperf phy2phy_cont --conf-file user_settings.py \
--vswitch none \
--fwdapp TestPMD
Supported Packet Forwarding applications are:
'testpmd'
- testpmd from dpdk
1. Update your “10_custom.conf” file to use the appropriate variables for selected Packet Forwarder:
2.2. Testing User Guides
99
OPNFV Documentation, Release Danube
# testpmd configuration
TESTPMD_ARGS = []
# packet forwarding mode supported by testpmd; Please see DPDK documentation
# for comprehensive list of modes supported by your version.
# e.g. io|mac|mac_retry|macswap|flowgen|rxonly|txonly|csum|icmpecho|...
# Note: Option "mac_retry" has been changed to "mac retry" since DPDK v16.07
TESTPMD_FWD_MODE = 'csum'
# checksum calculation layer: ip|udp|tcp|sctp|outer-ip
TESTPMD_CSUM_LAYER = 'ip'
# checksum calculation place: hw (hardware) | sw (software)
TESTPMD_CSUM_CALC = 'sw'
# recognize tunnel headers: on|off
TESTPMD_CSUM_PARSE_TUNNEL = 'off'
2. Run test:
$ ./vsperf phy2phy_tput --conf-file <path_to_settings_py>
Executing Packet Forwarding tests with one guest TestPMD with DPDK 16.11 or greater can be used to forward
packets as a switch to a single guest using TestPMD vdev option. To set this configuration the following parameters
should be used.
VSWITCH = 'none'
PKTFWD = 'TestPMD'
or use --vswitch and --fwdapp CLI arguments:
$ ./vsperf pvp_tput --conf-file user_settings.py \
--vswitch none \
--fwdapp TestPMD
Guest forwarding application only supports TestPMD in this configuration.
GUEST_LOOPBACK = ['testpmd']
For optimal performance one cpu per port +1 should be used for TestPMD. Also set additional params for packet
forwarding application to use the correct number of nb-cores.
VSWITCHD_DPDK_ARGS = ['-l', '46,44,42,40,38', '-n', '4', '--socket-mem 1024,0']
TESTPMD_ARGS = ['--nb-cores=4', '--txq=1', '--rxq=1']
For guest TestPMD 3 VCpus should be assigned with the following TestPMD params.
GUEST_TESTPMD_PARAMS = ['-l 0,1,2 -n 4 --socket-mem 1024 -- '
'--burst=64 -i --txqflags=0xf00 '
'--disable-hw-vlan --nb-cores=2 --txq=1 --rxq=1']
Execution of TestPMD can be run with the following command line
./vsperf pvp_tput --vswitch=none --fwdapp=TestPMD --conf-file <path_to_settings_py>
NOTE: To achieve the best 0% loss numbers with rfc2544 throughput testing, other tunings should be applied to host
and guest such as tuned profiles and CPU tunings to prevent possible interrupts to worker threads.
VSPERF modes of operation VSPERF can be run in different modes. By default it will configure vSwitch, traffic
generator and VNF. However it can be used just for configuration and execution of traffic generator. Another option is
execution of all components except traffic generator itself.
100
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
Mode of operation is driven by configuration parameter -m or –mode
-m MODE, --mode MODE vsperf mode of operation;
Values:
"normal" - execute vSwitch, VNF and traffic generator
"trafficgen" - execute only traffic generator
"trafficgen-off" - execute vSwitch and VNF
"trafficgen-pause" - execute vSwitch and VNF but wait before traffic transmission
In case, that VSPERF is executed in “trafficgen” mode, then configuration of traffic generator can be modified through
TRAFFIC dictionary passed to the --test-params option. It is not needed to specify all values of TRAFFIC dictionary. It is sufficient to specify only values, which should be changed. Detailed description of TRAFFIC dictionary
can be found at Configuration of TRAFFIC dictionary.
Example of execution of VSPERF in “trafficgen” mode:
$ ./vsperf -m trafficgen --trafficgen IxNet --conf-file vsperf.conf \
--test-params "TRAFFIC={'traffic_type':'rfc2544_continuous','bidir':'False','framerate':60}"
Code change verification by pylint Every developer participating in VSPERF project should run pylint before his
python code is submitted for review. Project specific configuration for pylint is available at ‘pylint.rc’.
Example of manual pylint invocation:
$ pylint --rcfile ./pylintrc ./vsperf
GOTCHAs:
Custom image fails to boot Using custom VM images may not boot within VSPerf pxp testing because of the drive
boot and shared type which could be caused by a missing scsi driver inside the image. In case of issues you can try
changing the drive boot type to ide.
GUEST_BOOT_DRIVE_TYPE = ['ide']
GUEST_SHARED_DRIVE_TYPE = ['ide']
OVS with DPDK and QEMU If you encounter the following error: “before (last 100 chars): ‘path=/dev/hugepages,share=on: unable to map backing store for hugepages: Cannot allocate memoryrnrn” during
qemu initialization, check the amount of hugepages on your system:
$ cat /proc/meminfo | grep HugePages
By default the vswitchd is launched with 1Gb of memory, to change this, modify –socket-mem parameter in
conf/02_vswitch.conf to allocate an appropriate amount of memory:
VSWITCHD_DPDK_ARGS = ['-c', '0x4', '-n', '4', '--socket-mem 1024,0']
VSWITCHD_DPDK_CONFIG = {
'dpdk-init' : 'true',
'dpdk-lcore-mask' : '0x4',
'dpdk-socket-mem' : '1024,0',
}
Note: Option VSWITCHD_DPDK_ARGS is used for vswitchd, which supports –dpdk parameter. In recent vswitchd
versions, option VSWITCHD_DPDK_CONFIG will be used to configure vswitchd via ovs-vsctl calls.
More information For more information and details refer to the rest of vSwitchPerfuser documentation.
2.2. Testing User Guides
101
OPNFV Documentation, Release Danube
Step driven tests In general, test scenarios are defined by a deployment used in the particular test case definition.
The chosen deployment scenario will take care of the vSwitch configuration, deployment of VNFs and it can also affect
configuration of a traffic generator. In order to allow a more flexible way of testcase scripting, VSPERF supports a
detailed step driven testcase definition. It can be used to configure and program vSwitch, deploy and terminate VNFs,
execute a traffic generator, modify a VSPERF configuration, execute external commands, etc.
Execution of step driven tests is done on a step by step work flow starting with step 0 as defined inside the test case.
Each step of the test increments the step number by one which is indicated in the log.
(testcases.integration) - Step 0 'vswitch add_vport ['br0']' start
Step driven tests can be used for both performance and integration testing. In case of integration test, each step in the
test case is validated. If a step does not pass validation the test will fail and terminate. The test will continue until a
failure is detected or all steps pass. A csv report file is generated after a test completes with an OK or FAIL result.
In case of performance test, the validation of steps is not performed and standard output files with results from traffic
generator and underlying OS details are generated by vsperf.
Step driven testcases can be used in two different ways:
# description of full testcase - in this case clean deployment is used to indicate that vsperf should
neither configure vSwitch nor deploy any VNF. Test shall perform all required vSwitch configuration and programming and deploy required number of VNFs.
# modification of existing deployment - in this case, any of supported deployments can be used to
perform initial vSwitch configuration and deployment of VNFs. Additional actions defined by TestSteps can be used to alter vSwitch configuration or deploy additional VNFs. After the last step is
processed, the test execution will continue with traffic execution.
Test objects and their functions Every test step can call a function of one of the supported test objects. The list of
supported objects and their most common functions follows:
• vswitch - provides functions for vSwitch configuration
List of supported functions:
– add_switch br_name - creates a new switch (bridge) with given br_name
– del_switch br_name - deletes switch (bridge) with given br_name
– add_phy_port br_name - adds a physical port into bridge specified by br_name
– add_vport br_name - adds a virtual port into bridge specified by br_name
– del_port br_name port_name - removes physical or virtual port specified by
port_name from bridge br_name
– add_flow br_name flow - adds flow specified by flow dictionary into the bridge
br_name; Content of flow dictionary will be passed to the vSwitch. In case of Open vSwitch it
will be passed to the ovs-ofctl add-flow command. Please see Open vSwitch documentation for the list of supported flow parameters.
– del_flow br_name [flow] - deletes flow specified by flow dictionary from bridge
br_name; In case that optional parameter flow is not specified or set to an empty dictionary
{}, then all flows from bridge br_name will be deleted.
– dump_flows br_name - dumps all flows from bridge specified by br_name
– enable_stp br_name - enables Spanning Tree Protocol for bridge br_name
– disable_stp br_name - disables Spanning Tree Protocol for bridge br_name
– enable_rstp br_name - enables Rapid Spanning Tree Protocol for bridge br_name
102
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
– disable_rstp br_name - disables Rapid Spanning Tree Protocol for bridge br_name
Examples:
['vswitch', 'add_switch', 'int_br0']
['vswitch', 'del_switch', 'int_br0']
['vswitch', 'add_phy_port', 'int_br0']
['vswitch', 'del_port', 'int_br0', '#STEP[2][0]']
['vswitch', 'add_flow', 'int_br0', {'in_port': '1', 'actions': ['output:2'],
'idle_timeout': '0'}],
['vswitch', 'enable_rstp', 'int_br0']
• vnf[ID] - provides functions for deployment and termination of VNFs; Optional alfanumerical ID is used for
VNF identification in case that testcase deploys multiple VNFs.
List of supported functions:
– start - starts a VNF based on VSPERF configuration
– stop - gracefully terminates given VNF
Examples:
['vnf1',
['vnf2',
['vnf2',
['vnf1',
'start']
'start']
'stop']
'stop']
• trafficgen - triggers traffic generation
List of supported functions:
– send_traffic traffic - starts a traffic based on the vsperf configuration and given
traffic dictionary. More details about traffic dictionary and its possible values are available at Traffic Generator Integration Guide
Examples:
['trafficgen', 'send_traffic', {'traffic_type' : 'rfc2544_throughput'}]
['trafficgen', 'send_traffic', {'traffic_type' : 'rfc2544_back2back', 'bidir' : 'True'}]
• settings - reads or modifies VSPERF configuration
List of supported functions:
– getValue param - returns value of given param
– setValue param value - sets value of param to given value
Examples:
['settings', 'getValue', 'TOOLS']
['settings', 'setValue', 'GUEST_USERNAME', ['root']]
• namespace - creates or modifies network namespaces
List of supported functions:
2.2. Testing User Guides
103
OPNFV Documentation, Release Danube
– create_namespace name - creates new namespace with given name
– delete_namespace name - deletes namespace specified by its name
– assign_port_to_namespace port name [port_up] - assigns NIC specified by
port into given namespace name; If optional parameter port_up is set to True, then port
will be brought up.
– add_ip_to_namespace_eth port name addr cidr - assigns an IP address
addr/cidr to the NIC specified by port within namespace name
– reset_port_to_root port name - returns given port from namespace name back to
the root namespace
Examples:
['namespace', 'create_namespace', 'testns']
['namespace', 'assign_port_to_namespace', 'eth0', 'testns']
• veth - manipulates with eth and veth devices
List of supported functions:
– add_veth_port port peer_port - adds a pair of veth ports named port and
peer_port
– del_veth_port port peer_port - deletes a veth port pair specified by port and
peer_port
– bring_up_eth_port eth_port [namespace] - brings up eth_port in (optional)
namespace
Examples:
['veth', 'add_veth_port', 'veth', 'veth1']
['veth', 'bring_up_eth_port', 'eth1']
• tools - provides a set of helper functions
List of supported functions:
– Assert condition - evaluates given condition and raises AssertionError in case
that condition is not True
– Eval expression - evaluates given expression as a python code and returns its result
– Exec command [regex] - executes a shell command and filters its output by (optional)
regular expression
Examples:
['tools', 'exec', 'numactl -H', 'available: ([0-9]+)']
['tools', 'assert', '#STEP[-1][0]>1']
• wait - is used for test case interruption. This object doesn’t have any functions. Once reached, vsperf will
pause test execution and waits for press of Enter key. It can be used during testcase design for debugging
purposes.
Examples:
['wait']
104
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
Test Macros Test profiles can include macros as part of the test step. Each step in the profile may return a value
such as a port name. Recall macros use #STEP to indicate the recalled value inside the return structure. If the method
the test step calls returns a value it can be later recalled, for example:
{
"Name": "vswitch_add_del_vport",
"Deployment": "clean",
"Description": "vSwitch - add and delete virtual port",
"TestSteps": [
['vswitch', 'add_switch', 'int_br0'],
['vswitch', 'add_vport', 'int_br0'],
['vswitch', 'del_port', 'int_br0', '#STEP[1][0]'],
['vswitch', 'del_switch', 'int_br0'],
]
#
#
#
#
STEP
STEP
STEP
STEP
0
1
2
3
}
This test profile uses the vswitch add_vport method which returns a string value of the port added. This is later called
by the del_port method using the name from step 1.
It is also possible to use negative indexes in step macros. In that case #STEP[-1] will refer to the result from previous
step, #STEP[-2] will refer to result of step called before previous step, etc. It means, that you could change STEP
2 from previous example to achieve the same functionality:
['vswitch', 'del_port', 'int_br0', '#STEP[-1][0]'],
# STEP 2
Also commonly used steps can be created as a separate profile.
STEP_VSWITCH_PVP_INIT = [
['vswitch', 'add_switch', 'int_br0'],
['vswitch', 'add_phy_port', 'int_br0'],
['vswitch', 'add_phy_port', 'int_br0'],
['vswitch', 'add_vport', 'int_br0'],
['vswitch', 'add_vport', 'int_br0'],
]
#
#
#
#
#
STEP
STEP
STEP
STEP
STEP
0
1
2
3
4
This profile can then be used inside other testcases
{
"Name": "vswitch_pvp",
"Deployment": "clean",
"Description": "vSwitch - configure switch and one vnf",
"TestSteps": STEP_VSWITCH_PVP_INIT +
[
['vnf', 'start'],
['vnf', 'stop'],
] +
STEP_VSWITCH_PVP_FINIT
}
HelloWorld and other basic Testcases The following examples are for demonstration purposes. You can run them
by copying and pasting into the conf/integration/01_testcases.conf file. A command-line instruction is shown at the
end of each example.
HelloWorld The first example is a HelloWorld testcase. It simply creates a bridge with 2 physical ports, then sets
up a flow to drop incoming packets from the port that was instantiated at the STEP #1. There’s no interaction with
the traffic generator. Then the flow, the 2 ports and the bridge are deleted. ‘add_phy_port’ method creates a ‘dpdk’
type interface that will manage the physical port. The string value returned is the port name that will be referred by
‘del_port’ later on.
2.2. Testing User Guides
105
OPNFV Documentation, Release Danube
{
"Name": "HelloWorld",
"Description": "My first testcase",
"Deployment": "clean",
"TestSteps": [
['vswitch', 'add_switch', 'int_br0'],
# STEP 0
['vswitch', 'add_phy_port', 'int_br0'], # STEP 1
['vswitch', 'add_phy_port', 'int_br0'], # STEP 2
['vswitch', 'add_flow', 'int_br0', {'in_port': '#STEP[1][1]', \
'actions': ['drop'], 'idle_timeout': '0'}],
['vswitch', 'del_flow', 'int_br0'],
['vswitch', 'del_port', 'int_br0', '#STEP[1][0]'],
['vswitch', 'del_port', 'int_br0', '#STEP[2][0]'],
['vswitch', 'del_switch', 'int_br0'],
]
},
To run HelloWorld test:
./vsperf --conf-file user_settings.py --integration HelloWorld
Specify a Flow by the IP address The next example shows how to explicitly set up a flow by specifying a destination
IP address. All packets received from the port created at STEP #1 that have a destination IP address = 90.90.90.90
will be forwarded to the port created at the STEP #2.
{
"Name": "p2p_rule_l3da",
"Description": "Phy2Phy with rule on L3 Dest Addr",
"Deployment": "clean",
"biDirectional": "False",
"TestSteps": [
['vswitch', 'add_switch', 'int_br0'],
# STEP 0
['vswitch', 'add_phy_port', 'int_br0'], # STEP 1
['vswitch', 'add_phy_port', 'int_br0'], # STEP 2
['vswitch', 'add_flow', 'int_br0', {'in_port': '#STEP[1][1]', \
'dl_type': '0x0800', 'nw_dst': '90.90.90.90', \
'actions': ['output:#STEP[2][1]'], 'idle_timeout': '0'}],
['trafficgen', 'send_traffic', \
{'traffic_type' : 'rfc2544_continuous'}],
['vswitch', 'dump_flows', 'int_br0'],
# STEP 5
['vswitch', 'del_flow', 'int_br0'],
# STEP 7 == del-flows
['vswitch', 'del_port', 'int_br0', '#STEP[1][0]'],
['vswitch', 'del_port', 'int_br0', '#STEP[2][0]'],
['vswitch', 'del_switch', 'int_br0'],
]
},
To run the test:
./vsperf --conf-file user_settings.py --integration p2p_rule_l3da
Multistream feature The next testcase uses the multistream feature. The traffic generator will send packets with
different UDP ports. That is accomplished by using “Stream Type” and “MultiStream” keywords. 4 different flows
are set to forward all incoming packets.
106
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
{
"Name": "multistream_l4",
"Description": "Multistream on UDP ports",
"Deployment": "clean",
"Parameters": {
'TRAFFIC' : {
"multistream": 4,
"stream_type": "L4",
},
},
"TestSteps": [
['vswitch', 'add_switch', 'int_br0'],
# STEP 0
['vswitch', 'add_phy_port', 'int_br0'], # STEP 1
['vswitch', 'add_phy_port', 'int_br0'], # STEP 2
# Setup Flows
['vswitch', 'add_flow', 'int_br0', {'in_port': '#STEP[1][1]',
'dl_type': '0x0800', 'nw_proto': '17', 'udp_dst': '0', \
'actions': ['output:#STEP[2][1]'], 'idle_timeout': '0'}],
['vswitch', 'add_flow', 'int_br0', {'in_port': '#STEP[1][1]',
'dl_type': '0x0800', 'nw_proto': '17', 'udp_dst': '1', \
'actions': ['output:#STEP[2][1]'], 'idle_timeout': '0'}],
['vswitch', 'add_flow', 'int_br0', {'in_port': '#STEP[1][1]',
'dl_type': '0x0800', 'nw_proto': '17', 'udp_dst': '2', \
'actions': ['output:#STEP[2][1]'], 'idle_timeout': '0'}],
['vswitch', 'add_flow', 'int_br0', {'in_port': '#STEP[1][1]',
'dl_type': '0x0800', 'nw_proto': '17', 'udp_dst': '3', \
'actions': ['output:#STEP[2][1]'], 'idle_timeout': '0'}],
# Send mono-dir traffic
['trafficgen', 'send_traffic', \
{'traffic_type' : 'rfc2544_continuous', \
'bidir' : 'False'}],
# Clean up
['vswitch', 'del_flow', 'int_br0'],
['vswitch', 'del_port', 'int_br0', '#STEP[1][0]'],
['vswitch', 'del_port', 'int_br0', '#STEP[2][0]'],
['vswitch', 'del_switch', 'int_br0'],
]
\
\
\
\
},
To run the test:
./vsperf --conf-file user_settings.py --integration multistream_l4
PVP with a VM Replacement This example launches a 1st VM in a PVP topology, then the VM is replaced by
another VM. When VNF setup parameter in ./conf/04_vnf.conf is “QemuDpdkVhostUser” ‘add_vport’ method creates
a ‘dpdkvhostuser’ type port to connect a VM.
{
"Name": "ex_replace_vm",
"Description": "PVP with VM replacement",
"Deployment": "clean",
"TestSteps": [
['vswitch', 'add_switch', 'int_br0'],
['vswitch', 'add_phy_port', 'int_br0'],
['vswitch', 'add_phy_port', 'int_br0'],
['vswitch', 'add_vport', 'int_br0'],
['vswitch', 'add_vport', 'int_br0'],
2.2. Testing User Guides
#
#
#
#
#
STEP
STEP
STEP
STEP
STEP
0
1
2
3
4
vm1
107
OPNFV Documentation, Release Danube
# Setup Flows
['vswitch', 'add_flow', 'int_br0', {'in_port': '#STEP[1][1]',
'actions': ['output:#STEP[3][1]'], 'idle_timeout': '0'}],
['vswitch', 'add_flow', 'int_br0', {'in_port': '#STEP[4][1]',
'actions': ['output:#STEP[2][1]'], 'idle_timeout': '0'}],
['vswitch', 'add_flow', 'int_br0', {'in_port': '#STEP[2][1]',
'actions': ['output:#STEP[4][1]'], 'idle_timeout': '0'}],
['vswitch', 'add_flow', 'int_br0', {'in_port': '#STEP[3][1]',
'actions': ['output:#STEP[1][1]'], 'idle_timeout': '0'}],
\
\
\
\
# Start VM 1
['vnf1', 'start'],
# Now we want to replace VM 1 with another VM
['vnf1', 'stop'],
['vswitch', 'add_vport', 'int_br0'],
# STEP 11
vm2
['vswitch', 'add_vport', 'int_br0'],
# STEP 12
['vswitch', 'del_flow', 'int_br0'],
['vswitch', 'add_flow', 'int_br0', {'in_port': '#STEP[1][1]', \
'actions': ['output:#STEP[11][1]'], 'idle_timeout': '0'}],
['vswitch', 'add_flow', 'int_br0', {'in_port': '#STEP[12][1]', \
'actions': ['output:#STEP[2][1]'], 'idle_timeout': '0'}],
# Start VM 2
['vnf2', 'start'],
['vnf2', 'stop'],
['vswitch', 'dump_flows', 'int_br0'],
# Clean up
['vswitch',
['vswitch',
['vswitch',
['vswitch',
['vswitch',
['vswitch',
['vswitch',
['vswitch',
'del_flow', 'int_br0'],
'del_port', 'int_br0', '#STEP[1][0]'],
'del_port', 'int_br0', '#STEP[2][0]'],
'del_port', 'int_br0', '#STEP[3][0]'],
'del_port', 'int_br0', '#STEP[4][0]'],
'del_port', 'int_br0', '#STEP[11][0]'],
'del_port', 'int_br0', '#STEP[12][0]'],
'del_switch', 'int_br0'],
# vm1
# vm2
]
},
To run the test:
./vsperf --conf-file user_settings.py --integration ex_replace_vm
VM with a Linux bridge This example setups a PVP topology and routes traffic to the VM based on the destination
IP address. A command-line parameter is used to select a Linux bridge as a guest loopback application. It is also
possible to select a guest loopback application by a configuration option GUEST_LOOPBACK.
{
"Name": "ex_pvp_rule_l3da",
"Description": "PVP with flow on L3 Dest Addr",
"Deployment": "clean",
"TestSteps": [
['vswitch', 'add_switch', 'int_br0'],
['vswitch', 'add_phy_port', 'int_br0'],
['vswitch', 'add_phy_port', 'int_br0'],
['vswitch', 'add_vport', 'int_br0'],
108
#
#
#
#
STEP
STEP
STEP
STEP
0
1
2
3
vm1
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
['vswitch', 'add_vport', 'int_br0'],
# STEP 4
# Setup Flows
['vswitch', 'add_flow', 'int_br0', {'in_port': '#STEP[1][1]', \
'dl_type': '0x0800', 'nw_dst': '90.90.90.90', \
'actions': ['output:#STEP[3][1]'], 'idle_timeout': '0'}],
# Each pkt from the VM is forwarded to the 2nd dpdk port
['vswitch', 'add_flow', 'int_br0', {'in_port': '#STEP[4][1]', \
'actions': ['output:#STEP[2][1]'], 'idle_timeout': '0'}],
# Start VMs
['vnf1', 'start'],
['trafficgen', 'send_traffic', \
{'traffic_type' : 'rfc2544_continuous', \
'bidir' : 'False'}],
['vnf1', 'stop'],
# Clean up
['vswitch', 'dump_flows', 'int_br0'],
# STEP 10
['vswitch', 'del_flow', 'int_br0'],
# STEP 11
['vswitch', 'del_port', 'int_br0', '#STEP[1][0]'],
['vswitch', 'del_port', 'int_br0', '#STEP[2][0]'],
['vswitch', 'del_port', 'int_br0', '#STEP[3][0]'], # vm1 ports
['vswitch', 'del_port', 'int_br0', '#STEP[4][0]'],
['vswitch', 'del_switch', 'int_br0'],
]
},
To run the test:
./vsperf --conf-file user_settings.py --test-params \
"GUEST_LOOPBACK=['linux_bridge']" --integration ex_pvp_rule_l3da
Forward packets based on UDP port This examples launches 2 VMs connected in parallel. Incoming packets will
be forwarded to one specific VM depending on the destination UDP port.
{
"Name": "ex_2pvp_rule_l4dp",
"Description": "2 PVP with flows on L4 Dest Port",
"Deployment": "clean",
"Parameters": {
'TRAFFIC' : {
"multistream": 2,
"stream_type": "L4",
},
},
"TestSteps": [
['vswitch', 'add_switch', 'int_br0'],
# STEP 0
['vswitch', 'add_phy_port', 'int_br0'],
# STEP 1
['vswitch', 'add_phy_port', 'int_br0'],
# STEP 2
['vswitch', 'add_vport', 'int_br0'],
# STEP 3
vm1
['vswitch', 'add_vport', 'int_br0'],
# STEP 4
['vswitch', 'add_vport', 'int_br0'],
# STEP 5
vm2
['vswitch', 'add_vport', 'int_br0'],
# STEP 6
# Setup Flows to reply ICMPv6 and similar packets, so to
# avoid flooding internal port with their re-transmissions
['vswitch', 'add_flow', 'int_br0', \
{'priority': '1', 'dl_src': '00:00:00:00:00:01', \
'actions': ['output:#STEP[3][1]'], 'idle_timeout': '0'}],
['vswitch', 'add_flow', 'int_br0', \
{'priority': '1', 'dl_src': '00:00:00:00:00:02', \
2.2. Testing User Guides
109
OPNFV Documentation, Release Danube
'actions': ['output:#STEP[4][1]'], 'idle_timeout': '0'}],
['vswitch', 'add_flow', 'int_br0', \
{'priority': '1', 'dl_src': '00:00:00:00:00:03', \
'actions': ['output:#STEP[5][1]'], 'idle_timeout': '0'}],
['vswitch', 'add_flow', 'int_br0', \
{'priority': '1', 'dl_src': '00:00:00:00:00:04', \
'actions': ['output:#STEP[6][1]'], 'idle_timeout': '0'}],
# Forward UDP packets depending on dest port
['vswitch', 'add_flow', 'int_br0', {'in_port': '#STEP[1][1]', \
'dl_type': '0x0800', 'nw_proto': '17', 'udp_dst': '0', \
'actions': ['output:#STEP[3][1]'], 'idle_timeout': '0'}],
['vswitch', 'add_flow', 'int_br0', {'in_port': '#STEP[1][1]', \
'dl_type': '0x0800', 'nw_proto': '17', 'udp_dst': '1', \
'actions': ['output:#STEP[5][1]'], 'idle_timeout': '0'}],
# Send VM output to phy port #2
['vswitch', 'add_flow', 'int_br0', {'in_port': '#STEP[4][1]', \
'actions': ['output:#STEP[2][1]'], 'idle_timeout': '0'}],
['vswitch', 'add_flow', 'int_br0', {'in_port': '#STEP[6][1]', \
'actions': ['output:#STEP[2][1]'], 'idle_timeout': '0'}],
# Start VMs
['vnf1', 'start'],
# STEP 16
['vnf2', 'start'],
# STEP 17
['trafficgen', 'send_traffic', \
{'traffic_type' : 'rfc2544_continuous', \
'bidir' : 'False'}],
['vnf1', 'stop'],
['vnf2', 'stop'],
['vswitch', 'dump_flows', 'int_br0'],
# Clean up
['vswitch', 'del_flow', 'int_br0'],
['vswitch', 'del_port', 'int_br0', '#STEP[1][0]'],
['vswitch', 'del_port', 'int_br0', '#STEP[2][0]'],
['vswitch', 'del_port', 'int_br0', '#STEP[3][0]'], # vm1 ports
['vswitch', 'del_port', 'int_br0', '#STEP[4][0]'],
['vswitch', 'del_port', 'int_br0', '#STEP[5][0]'], # vm2 ports
['vswitch', 'del_port', 'int_br0', '#STEP[6][0]'],
['vswitch', 'del_switch', 'int_br0'],
]
},
To run the test:
./vsperf --conf-file user_settings.py --integration ex_2pvp_rule_l4dp
Modification of existing PVVP deployment This is an example of modification of a standard deployment scenario
with additional TestSteps. Standard PVVP scenario is used to configure a vSwitch and to deploy two VNFs connected
in series. Additional TestSteps will deploy a 3rd VNF and connect it in parallel to already configured VNFs. Traffic
generator is instructed (by Multistream feature) to send two separate traffic streams. One stream will be sent to the
standalone VNF and second to two chained VNFs.
In case, that test is defined as a performance test, then traffic results will be collected and available in both csv and rst
report files.
{
"Name": "pvvp_pvp_cont",
"Deployment": "pvvp",
"Description": "PVVP and PVP in parallel with Continuous Stream",
"Parameters" : {
110
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
"TRAFFIC" : {
"traffic_type" : "rfc2544_continuous",
"multistream": 2,
},
},
"TestSteps": [
['vswitch', 'add_vport', 'br0'],
['vswitch', 'add_vport', 'br0'],
# priority must be higher than default 32768, otherwise flows won't match
['vswitch', 'add_flow', 'br0',
{'in_port': '1', 'actions': ['output:#STEP[-2][1]'], 'idle_timeout': '0', 'dl_ty
'nw_proto':'17', 'tp_dst':'0', 'priority': '33000'}
['vswitch', 'add_flow', 'br0',
{'in_port': '2', 'actions': ['output:#STEP[-2][1]'], 'idle_timeout': '0', 'dl_ty
'nw_proto':'17', 'tp_dst':'0', 'priority': '33000'}
['vswitch', 'add_flow', 'br0', {'in_port': '#STEP[-4][1]', 'actions': ['output:1'
'idle_timeout': '0'}],
['vswitch', 'add_flow', 'br0', {'in_port': '#STEP[-4][1]', 'actions': ['output:2'
'idle_timeout': '0'}],
['vswitch', 'dump_flows', 'br0'],
['vnf1', 'start'],
]
},
To run the test:
./vsperf --conf-file user_settings.py pvvp_pvp_cont
Integration tests VSPERF includes a set of integration tests defined in conf/integration. These tests can be run by
specifying –integration as a parameter to vsperf. Current tests in conf/integration include switch functionality and
Overlay tests.
Tests in the conf/integration can be used to test scaling of different switch configurations by adding steps into the test
case.
For the overlay tests VSPERF supports VXLAN, GRE and GENEVE tunneling protocols. Testing of these protocols
is limited to unidirectional traffic and P2P (Physical to Physical scenarios).
NOTE: The configuration for overlay tests provided in this guide is for unidirectional traffic only.
Executing Integration Tests To execute integration tests VSPERF is run with the integration parameter. To view
the current test list simply execute the following command:
./vsperf --integration --list
The standard tests included are defined inside the conf/integration/01_testcases.conf file.
Executing Tunnel encapsulation tests The VXLAN OVS DPDK encapsulation tests requires IPs, MAC addresses,
bridge names and WHITELIST_NICS for DPDK.
NOTE: Only Ixia traffic generators currently support the execution of the tunnel encapsulation tests. Support for other
traffic generators may come in a future release.
Default values are already provided. To customize for your environment, override the following variables in you
user_settings.py file:
2.2. Testing User Guides
111
OPNFV Documentation, Release Danube
# Variables defined in conf/integration/02_vswitch.conf
# Tunnel endpoint for Overlay P2P deployment scenario
# used for br0
VTEP_IP1 = '192.168.0.1/24'
# Used as remote_ip in adding OVS tunnel port and
# to set ARP entry in OVS (e.g. tnl/arp/set br-ext 192.168.240.10 02:00:00:00:00:02
VTEP_IP2 = '192.168.240.10'
# Network to use when adding a route for inner frame data
VTEP_IP2_SUBNET = '192.168.240.0/24'
# Bridge names
TUNNEL_INTEGRATION_BRIDGE = 'br0'
TUNNEL_EXTERNAL_BRIDGE = 'br-ext'
# IP of br-ext
TUNNEL_EXTERNAL_BRIDGE_IP = '192.168.240.1/24'
# vxlan|gre|geneve
TUNNEL_TYPE = 'vxlan'
# Variables defined conf/integration/03_traffic.conf
# For OP2P deployment scenario
TRAFFICGEN_PORT1_MAC = '02:00:00:00:00:01'
TRAFFICGEN_PORT2_MAC = '02:00:00:00:00:02'
TRAFFICGEN_PORT1_IP = '1.1.1.1'
TRAFFICGEN_PORT2_IP = '192.168.240.10'
To run VXLAN encapsulation tests:
./vsperf --conf-file user_settings.py --integration \
--test-params 'TUNNEL_TYPE=vxlan' overlay_p2p_tput
To run GRE encapsulation tests:
./vsperf --conf-file user_settings.py --integration \
--test-params 'TUNNEL_TYPE=gre' overlay_p2p_tput
To run GENEVE encapsulation tests:
./vsperf --conf-file user_settings.py --integration \
--test-params 'TUNNEL_TYPE=geneve' overlay_p2p_tput
To run OVS NATIVE tunnel tests (VXLAN/GRE/GENEVE):
1. Install the OVS kernel modules
cd src/ovs/ovs
sudo -E make modules_install
2. Set the following variables:
VSWITCH = 'OvsVanilla'
# Specify vport_* kernel module to test.
PATHS['vswitch']['OvsVanilla']['src']['modules'] = [
'vport_vxlan',
'vport_gre',
'vport_geneve',
112
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
'datapath/linux/openvswitch.ko',
]
NOTE: In case, that Vanilla OVS is installed from binary package,
PATHS[’vswitch’][’OvsVanilla’][’bin’][’modules’] instead.
then please set
3. Run tests:
./vsperf --conf-file user_settings.py --integration \
--test-params 'TUNNEL_TYPE=vxlan' overlay_p2p_tput
Executing VXLAN decapsulation tests To run VXLAN decapsulation tests:
1. Set the variables used in “Executing Tunnel encapsulation tests”
2. Set dstmac of DUT_NIC2_MAC to the MAC adddress of the 2nd NIC of your DUT
DUT_NIC2_MAC = '<DUT NIC2 MAC>'
3. Run test:
./vsperf --conf-file user_settings.py --integration overlay_p2p_decap_cont
If you want to use different values for your VXLAN frame, you may set:
VXLAN_FRAME_L3 = {'proto': 'udp',
'packetsize': 64,
'srcip': TRAFFICGEN_PORT1_IP,
'dstip': '192.168.240.1',
}
VXLAN_FRAME_L4 = {'srcport': 4789,
'dstport': 4789,
'vni': VXLAN_VNI,
'inner_srcmac': '01:02:03:04:05:06',
'inner_dstmac': '06:05:04:03:02:01',
'inner_srcip': '192.168.0.10',
'inner_dstip': '192.168.240.9',
'inner_proto': 'udp',
'inner_srcport': 3000,
'inner_dstport': 3001,
}
Executing GRE decapsulation tests To run GRE decapsulation tests:
1. Set the variables used in “Executing Tunnel encapsulation tests”
2. Set dstmac of DUT_NIC2_MAC to the MAC adddress of the 2nd NIC of your DUT
DUT_NIC2_MAC = '<DUT NIC2 MAC>'
3. Run test:
./vsperf --conf-file user_settings.py --test-params 'TUNNEL_TYPE=gre' \
--integration overlay_p2p_decap_cont
If you want to use different values for your GRE frame, you may set:
2.2. Testing User Guides
113
OPNFV Documentation, Release Danube
GRE_FRAME_L3 = {'proto': 'gre',
'packetsize': 64,
'srcip': TRAFFICGEN_PORT1_IP,
'dstip': '192.168.240.1',
}
GRE_FRAME_L4 = {'srcport': 0,
'dstport': 0
'inner_srcmac': '01:02:03:04:05:06',
'inner_dstmac': '06:05:04:03:02:01',
'inner_srcip': '192.168.0.10',
'inner_dstip': '192.168.240.9',
'inner_proto': 'udp',
'inner_srcport': 3000,
'inner_dstport': 3001,
}
Executing GENEVE decapsulation tests IxNet 7.3X does not have native support of GENEVE protocol. The
template, GeneveIxNetTemplate.xml_ClearText.xml, should be imported into IxNET for this testcase to work.
To import the template do:
1. Run the IxNetwork TCL Server
2. Click on the Traffic menu
3. Click on the Traffic actions and click Edit Packet Templates
4. On the Template editor window,
click Import.
Select the template
3rd_party/ixia/GeneveIxNetTemplate.xml_ClearText.xml and click import.
located
at
5. Restart the TCL Server.
To run GENEVE decapsulation tests:
1. Set the variables used in “Executing Tunnel encapsulation tests”
2. Set dstmac of DUT_NIC2_MAC to the MAC adddress of the 2nd NIC of your DUT
DUT_NIC2_MAC = '<DUT NIC2 MAC>'
3. Run test:
./vsperf --conf-file user_settings.py --test-params 'tunnel_type=geneve' \
--integration overlay_p2p_decap_cont
If you want to use different values for your GENEVE frame, you may set:
GENEVE_FRAME_L3 = {'proto': 'udp',
'packetsize': 64,
'srcip': TRAFFICGEN_PORT1_IP,
'dstip': '192.168.240.1',
}
GENEVE_FRAME_L4 = {'srcport': 6081,
'dstport': 6081,
'geneve_vni': 0,
'inner_srcmac': '01:02:03:04:05:06',
'inner_dstmac': '06:05:04:03:02:01',
'inner_srcip': '192.168.0.10',
'inner_dstip': '192.168.240.9',
114
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
'inner_proto': 'udp',
'inner_srcport': 3000,
'inner_dstport': 3001,
}
Executing Native/Vanilla OVS VXLAN decapsulation tests To run VXLAN decapsulation tests:
1. Set the following variables in your user_settings.py file:
PATHS['vswitch']['OvsVanilla']['src']['modules'] = [
'vport_vxlan',
'datapath/linux/openvswitch.ko',
]
DUT_NIC1_MAC = '<DUT NIC1 MAC ADDRESS>'
TRAFFICGEN_PORT1_IP = '172.16.1.2'
TRAFFICGEN_PORT2_IP = '192.168.1.11'
VTEP_IP1 = '172.16.1.2/24'
VTEP_IP2 = '192.168.1.1'
VTEP_IP2_SUBNET = '192.168.1.0/24'
TUNNEL_EXTERNAL_BRIDGE_IP = '172.16.1.1/24'
TUNNEL_INT_BRIDGE_IP = '192.168.1.1'
VXLAN_FRAME_L2 = {'srcmac':
'01:02:03:04:05:06',
'dstmac': DUT_NIC1_MAC
}
VXLAN_FRAME_L3 = {'proto': 'udp',
'packetsize': 64,
'srcip': TRAFFICGEN_PORT1_IP,
'dstip': '172.16.1.1',
}
VXLAN_FRAME_L4 = {
'srcport': 4789,
'dstport': 4789,
'protocolpad': 'true',
'vni': 99,
'inner_srcmac': '01:02:03:04:05:06',
'inner_dstmac': '06:05:04:03:02:01',
'inner_srcip': '192.168.1.2',
'inner_dstip': TRAFFICGEN_PORT2_IP,
'inner_proto': 'udp',
'inner_srcport': 3000,
'inner_dstport': 3001,
}
NOTE: In case, that Vanilla OVS is installed from binary package,
PATHS[’vswitch’][’OvsVanilla’][’bin’][’modules’] instead.
then please set
2. Run test:
./vsperf --conf-file user_settings.py --integration \
--test-params 'tunnel_type=vxlan' overlay_p2p_decap_cont
2.2. Testing User Guides
115
OPNFV Documentation, Release Danube
Executing Native/Vanilla OVS GRE decapsulation tests To run GRE decapsulation tests:
1. Set the following variables in your user_settings.py file:
PATHS['vswitch']['OvsVanilla']['src']['modules'] = [
'vport_gre',
'datapath/linux/openvswitch.ko',
]
DUT_NIC1_MAC = '<DUT NIC1 MAC ADDRESS>'
TRAFFICGEN_PORT1_IP = '172.16.1.2'
TRAFFICGEN_PORT2_IP = '192.168.1.11'
VTEP_IP1 = '172.16.1.2/24'
VTEP_IP2 = '192.168.1.1'
VTEP_IP2_SUBNET = '192.168.1.0/24'
TUNNEL_EXTERNAL_BRIDGE_IP = '172.16.1.1/24'
TUNNEL_INT_BRIDGE_IP = '192.168.1.1'
GRE_FRAME_L2 = {'srcmac':
'01:02:03:04:05:06',
'dstmac': DUT_NIC1_MAC
}
GRE_FRAME_L3 = {'proto': 'udp',
'packetsize': 64,
'srcip': TRAFFICGEN_PORT1_IP,
'dstip': '172.16.1.1',
}
GRE_FRAME_L4 = {
'srcport': 4789,
'dstport': 4789,
'protocolpad': 'true',
'inner_srcmac': '01:02:03:04:05:06',
'inner_dstmac': '06:05:04:03:02:01',
'inner_srcip': '192.168.1.2',
'inner_dstip': TRAFFICGEN_PORT2_IP,
'inner_proto': 'udp',
'inner_srcport': 3000,
'inner_dstport': 3001,
}
NOTE: In case, that Vanilla OVS is installed from binary package,
PATHS[’vswitch’][’OvsVanilla’][’bin’][’modules’] instead.
then please set
2. Run test:
./vsperf --conf-file user_settings.py --integration \
--test-params 'tunnel_type=gre' overlay_p2p_decap_cont
Executing Native/Vanilla OVS GENEVE decapsulation tests To run GENEVE decapsulation tests:
1. Set the following variables in your user_settings.py file:
PATHS['vswitch']['OvsVanilla']['src']['modules'] = [
'vport_geneve',
'datapath/linux/openvswitch.ko',
116
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
]
DUT_NIC1_MAC = '<DUT NIC1 MAC ADDRESS>'
TRAFFICGEN_PORT1_IP = '172.16.1.2'
TRAFFICGEN_PORT2_IP = '192.168.1.11'
VTEP_IP1 = '172.16.1.2/24'
VTEP_IP2 = '192.168.1.1'
VTEP_IP2_SUBNET = '192.168.1.0/24'
TUNNEL_EXTERNAL_BRIDGE_IP = '172.16.1.1/24'
TUNNEL_INT_BRIDGE_IP = '192.168.1.1'
GENEVE_FRAME_L2 = {'srcmac':
'01:02:03:04:05:06',
'dstmac': DUT_NIC1_MAC
}
GENEVE_FRAME_L3 = {'proto': 'udp',
'packetsize': 64,
'srcip': TRAFFICGEN_PORT1_IP,
'dstip': '172.16.1.1',
}
GENEVE_FRAME_L4 = {'srcport': 6081,
'dstport': 6081,
'protocolpad': 'true',
'geneve_vni': 0,
'inner_srcmac': '01:02:03:04:05:06',
'inner_dstmac': '06:05:04:03:02:01',
'inner_srcip': '192.168.1.2',
'inner_dstip': TRAFFICGEN_PORT2_IP,
'inner_proto': 'udp',
'inner_srcport': 3000,
'inner_dstport': 3001,
}
NOTE: In case, that Vanilla OVS is installed from binary package,
PATHS[’vswitch’][’OvsVanilla’][’bin’][’modules’] instead.
then please set
2. Run test:
./vsperf --conf-file user_settings.py --integration \
--test-params 'tunnel_type=geneve' overlay_p2p_decap_cont
Executing Tunnel encapsulation+decapsulation tests The OVS DPDK encapsulation_decapsulation tests requires
IPs, MAC addresses, bridge names and WHITELIST_NICS for DPDK.
The test cases can test the tunneling encap and decap without using any ingress overlay traffic as compared to above
test cases. To achieve this the OVS is configured to perform encap and decap in a series on the same traffic stream as
given below.
TRAFFIC-IN –> [ENCAP] –> [MOD-PKT] –> [DECAP] –> TRAFFIC-OUT
Default values are already provided. To customize for your environment, override the following variables in you
user_settings.py file:
2.2. Testing User Guides
117
OPNFV Documentation, Release Danube
# Variables defined in conf/integration/02_vswitch.conf
# Bridge names
TUNNEL_EXTERNAL_BRIDGE1
TUNNEL_EXTERNAL_BRIDGE2
TUNNEL_MODIFY_BRIDGE1 =
TUNNEL_MODIFY_BRIDGE2 =
= 'br-phy1'
= 'br-phy2'
'br-mod1'
'br-mod2'
# IP of br-mod1
TUNNEL_MODIFY_BRIDGE_IP1 = '10.0.0.1/24'
# Mac of br-mod1
TUNNEL_MODIFY_BRIDGE_MAC1 = '00:00:10:00:00:01'
# IP of br-mod2
TUNNEL_MODIFY_BRIDGE_IP2 = '20.0.0.1/24'
#Mac of br-mod2
TUNNEL_MODIFY_BRIDGE_MAC2 = '00:00:20:00:00:01'
# vxlan|gre|geneve, Only VXLAN is supported for now.
TUNNEL_TYPE = 'vxlan'
To run VXLAN encapsulation+decapsulation tests:
./vsperf --conf-file user_settings.py --integration \
overlay_p2p_mod_tput
Execution of vswitchperf testcases by Yardstick
General Yardstick is a generic framework for a test execution, which is used for validation of installation of OPNFV
platform. In the future, Yardstick will support two options of vswitchperf testcase execution:
• plugin mode, which will execute native vswitchperf testcases; Tests will be executed natively by vsperf, and test
results will be processed and reported by yardstick.
• traffic generator mode, which will run vswitchperf in trafficgen mode only; Yardstick framework will be used
to launch VNFs and to configure flows to ensure, that traffic is properly routed. This mode will allow to test
OVS performance in real world scenarios.
In Colorado release only the traffic generator mode is supported.
Yardstick Installation In order to run Yardstick testcases, you will need to prepare your test environment. Please
follow the installation instructions to install the yardstick.
Please note, that yardstick uses OpenStack for execution of testcases. OpenStack must be installed with Heat and
Neutron services. Otherwise vswitchperf testcases cannot be executed.
VM image with vswitchperf A special VM image is required for execution of vswitchperf specific testcases by
yardstick. It is possible to use a sample VM image available at OPNFV artifactory or to build customized image.
Sample VM image with vswitchperf Sample VM image is available at vswitchperf section of OPNFV artifactory
for free download:
118
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
$ wget http://artifacts.opnfv.org/vswitchperf/vnf/vsperf-yardstick-image.qcow2
This image can be used for execution of sample testcases with dummy traffic generator.
NOTE: Traffic generators might require an installation of client software. This software is not included in the sample
image and must be installed by user.
NOTE: This image will be updated only in case, that new features related to yardstick integration will be added to the
vswitchperf.
Preparation of custom VM image In general, any Linux distribution supported by vswitchperf can be used as a
base image for vswitchperf. One of the possibilities is to modify vloop-vnf image, which can be downloaded from
http://artifacts.opnfv.org/vswitchperf.html/ (see vloop-vnf ).
Please follow the Installing vswitchperf to install vswitchperf inside vloop-vnf image. As vswitchperf will be run
in trafficgen mode, it is possible to skip installation and compilation of OVS, QEMU and DPDK to keep image size
smaller.
In case, that selected traffic generator requires installation of additional client software, please follow appropriate
documentation. For example in case of IXIA, you would need to install IxOS and IxNetowrk TCL API.
VM image usage Image with vswitchperf must be uploaded into the glance service and vswitchperf specific flavor
configured, e.g.:
$ glance --os-username admin --os-image-api-version 1 image-create --name \
vsperf --is-public true --disk-format qcow2 --container-format bare --file \
vsperf-yardstick-image.qcow2
$ nova --os-username admin flavor-create vsperf-flavor 100 2048 25 1
Testcase execution After installation, yardstick is available as python package within yardstick specific virtual environment. It means, that yardstick environment must be enabled before the test execution, e.g.:
source ~/yardstick_venv/bin/activate
Next step is configuration of OpenStack environment, e.g. in case of devstack:
source /opt/openstack/devstack/openrc
export EXTERNAL_NETWORK=public
Vswitchperf testcases executable by yardstick are located at vswitchperf repository inside yardstick/tests directory. Example of their download and execution follows:
git clone https://gerrit.opnfv.org/gerrit/vswitchperf
cd vswitchperf
yardstick -d task start yardstick/tests/rfc2544_throughput_dummy.yaml
NOTE: Optional argument -d shows debug output.
Testcase customization Yardstick testcases are described by YAML files. vswitchperf specific testcases are part of
the vswitchperf repository and their yaml files can be found at yardstick/tests directory. For detailed description of yaml file structure, please see yardstick documentation and testcase samples. Only vswitchperf specific parts
will be discussed here.
Example of yaml file:
2.2. Testing User Guides
119
OPNFV Documentation, Release Danube
...
scenarios:
type: Vsperf
options:
testname: 'p2p_rfc2544_throughput'
trafficgen_port1: 'eth1'
trafficgen_port2: 'eth3'
external_bridge: 'br-ex'
test_params: 'TRAFFICGEN_DURATION=30;TRAFFIC={'traffic_type':'rfc2544_throughput}'
conf_file: '~/vsperf-yardstick.conf'
host: vsperf.demo
runner:
type: Sequence
scenario_option_name: frame_size
sequence:
- 64
- 128
- 512
- 1024
- 1518
sla:
metrics: 'throughput_rx_fps'
throughput_rx_fps: 500000
action: monitor
context:
...
Section option Section option defines details of vswitchperf test scenario. Lot of options are identical to the vswitchperf parameters passed through --test-params argument. Following options are supported:
• frame_size - a packet size for which test should be executed; Multiple packet sizes can be tested by modification
of Sequence runner section inside YAML definition. Default: ‘64’
• conf_file - sets path to the vswitchperf configuration file, which will be uploaded to VM; Default: ‘~/vsperfyardstick.conf’
• setup_script - sets path to the setup script, which will be executed during setup and teardown phases
• trafficgen_port1 - specifies device name of 1st interface connected to the trafficgen
• trafficgen_port2 - specifies device name of 2nd interface connected to the trafficgen
• external_bridge - specifies name of external bridge configured in OVS; Default: ‘br-ex’
• test_params - specifies a string with a list of vsperf configuration parameters, which will be passed to the
--test-params CLI argument; Parameters should be stated in the form of param=value and separated
by a semicolon. Configuration of traffic generator is driven by TRAFFIC dictionary, which can be also updated
by values defined by test_params. Please check VSPERF documentation for details about available configuration parameters and their data types. In case that both test_params and conf_file are specified, then values
from test_params will override values defined in the configuration file.
In case that trafficgen_port1 and/or trafficgen_port2 are defined, then these interfaces will be inserted into the
external_bridge of OVS. It is expected, that OVS runs at the same node, where the testcase is executed. In case of
more complex OpenStack installation or a need of additional OVS configuration, setup_script can be used.
120
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
NOTE It is essential to specify a configuration for selected traffic generator. In case, that standalone testcase is
created, then traffic generator can be selected and configured directly in YAML file by test_params. On the other
hand, if multiple testcases should be executed with the same traffic generator settings, then a customized configuration
file should be prepared and its name passed by conf_file option.
Section runner Yardstick supports several runner types. In case of vswitchperf specific TCs, Sequence runner type
can be used to execute the testcase for given list of frame sizes.
Section sla In case that sla section is not defined, then testcase will be always considered as successful. On the other
hand, it is possible to define a set of test metrics and their minimal values to evaluate test success. Any numeric value,
reported by vswitchperf inside CSV result file, can be used. Multiple metrics can be defined as a coma separated list
of items. Minimal value must be set separately for each metric.
e.g.:
sla:
metrics: 'throughput_rx_fps,throughput_rx_mbps'
throughput_rx_fps: 500000
throughput_rx_mbps: 1000
In case that any of defined metrics will be lower than defined value, then testcase will be marked as failed. Based on
action policy, yardstick will either stop test execution (value assert) or it will run next test (value monitor).
NOTE The throughput SLA (or any other SLA) cannot be set to a meaningful value without knowledge of the server
and networking environment, possibly including prior testing in that environment to establish a baseline SLA level
under well-understood circumstances.
Indices
• search
Yardstick
Performance Testing User Guide (Yardstick)
Introduction
Welcome to Yardstick’s documentation !
Yardstick is an OPNFV Project.
The project’s goal is to verify infrastructure compliance, from the perspective of a Virtual Network Function (VNF).
The Project’s scope is the development of a test framework, Yardstick, test cases and test stimuli to enable Network
Function Virtualization Infrastructure (NFVI) verification. The Project also includes a sample VNF, the Virtual Traffic
Classifier (VTC) and its experimental framework, ApexLake !
Yardstick is used in OPNFV for verifying the OPNFV infrastructure and some of the OPNFV features. The Yardstick
framework is deployed in several OPNFV community labs. It is installer, infrastructure and application independent.
See also:
Pharos for information on OPNFV community labs and this Presentation for an overview of Yardstick
2.2. Testing User Guides
121
OPNFV Documentation, Release Danube
About This Document This document consists of the following chapters:
• Chapter Introduction provides a brief introduction to Yardstick project’s background and describes the structure
of this document.
• Chapter Methodology describes the methodology implemented by the Yardstick Project for NFVI verification.
• Chapter Architecture provides information on the software architecture of Yardstick.
• Chapter Yardstick Installation provides instructions to install Yardstick.
• Chapter Installing a plug-in into Yardstick provides information on how to integrate other OPNFV testing
projects into Yardstick.
• Chapter Store Other Project’s Test Results in InfluxDB provides inforamtion on how to run plug-in test cases
and store test results into community’s InfluxDB.
• Chapter Grafana dashboard provides inforamtion on Yardstick grafana dashboard and how to add a dashboard
into Yardstick grafana dashboard.
• Chapter Yardstick Restful API provides inforamtion on Yardstick ReST API and how to use Yardstick API.
• Chapter Yardstick User Interface provides inforamtion on how to use yardstick report CLI to view the test result
in table format and also values pinned on to a graph
• Chapter Virtual Traffic Classifier provides information on the VTC.
• Chapter Apexlake Installation Guide provides instructions to install the experimental framework ApexLake
• Chapter Apexlake API Interface Definition explains how this framework is integrated in Yardstick.
• Chapter Network Services Benchmarking (NSB) describes the methodology implemented by the Yardstick Network service benchmarking to test real world usecase for a given VNF.
• Chapter Yardstick - NSB Testing -Installation provides instructions to install Yardstick - Network service benchmarking testing.
• Chapter Yardstick Test Cases includes a list of available Yardstick test cases.
Contact Yardstick Feedback? Contact us
Methodology
Abstract This chapter describes the methodology implemented by the Yardstick project for verifying the NFVI from
the perspective of a VNF.
ETSI-NFV The document ETSI GS NFV-TST001, “Pre-deployment Testing; Report on Validation of NFV Environments and Services”, recommends methods for pre-deployment testing of the functional components of an NFV
environment.
The Yardstick project implements the methodology described in chapter 6, “Pre- deployment validation of NFV infrastructure”.
The methodology consists in decomposing the typical VNF work-load performance metrics into a number of characteristics/performance vectors, which each can be represented by distinct test-cases.
The methodology includes five steps:
• Step1: Define Infrastruture - the Hardware, Software and corresponding configuration target for validation; the OPNFV infrastructure, in OPNFV community labs.
122
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
• Step2: Identify VNF type - the application for which the infrastructure is to be validated, and its requirements on the underlying infrastructure.
• Step3: Select test cases - depending on the workload that represents the application for which the infrastruture is to be validated, the relevant test cases amongst the list of available Yardstick test cases.
• Step4: Execute tests - define the duration and number of iterations for the selected test cases, tests runs
are automated via OPNFV Jenkins Jobs.
• Step5: Collect results - using the common API for result collection.
See also:
Yardsticktst for material on alignment ETSI TST001 and Yardstick.
Metrics The metrics, as defined by ETSI GS NFV-TST001, are shown in Table1, Table2 and Table3.
In OPNFV Colorado release, generic test cases covering aspects of the listed metrics are available; further OPNFV
releases will provide extended testing of these metrics. The view of available Yardstick test cases cross ETSI definitions in Table1, Table2 and Table3 is shown in Table4. It shall be noticed that the Yardstick test cases are examples,
the test duration and number of iterations are configurable, as are the System Under Test (SUT) and the attributes (or,
in Yardstick nomemclature, the scenario options). Table 1 - Performance/Speed Metrics
Category
Compute
Network
Storage
Performance/Speed
•
•
•
•
Latency for random memory access
Latency for cache read/write operations
Processing speed (instructions per second)
Throughput for random memory access (bytes per
second)
• Throughput per NFVI node (frames/byte per second)
• Throughput provided to a VM (frames/byte per
second)
• Latency per traffic flow
• Latency between VMs
• Latency between NFVI nodes
• Packet delay variation (jitter) between VMs
• Packet delay variation (jitter) between NFVI
nodes
•
•
•
•
Sequential read/write IOPS
Random read/write IOPS
Latency for storage read/write operations
Throughput for storage read/write operations
Table 2 - Capacity/Scale Metrics
2.2. Testing User Guides
123
OPNFV Documentation, Release Danube
Category
Compute
Network
Storage
Capacity/Scale
• Number of cores and threads- Available memory
size
• Cache size
• Processor utilization (max, average, standard deviation)
• Memory utilization (max, average, standard deviation)
• Cache utilization (max, average, standard deviation)
• Number of connections
• Number of frames sent/received
• Maximum throughput between VMs (frames/byte
per second)
• Maximum throughput between NFVI nodes
(frames/byte per second)
• Network utilization (max, average, standard deviation)
• Number of traffic flows
•
•
•
•
•
•
Storage/Disk size
Capacity allocation (block-based, object-based)
Block size
Maximum sequential read/write IOPS
Maximum random read/write IOPS
Disk utilization (max, average, standard deviation)
Table 3 - Availability/Reliability Metrics
Category
Compute
Network
Storage
Availability/Reliability
•
•
•
•
•
Processor availability (Error free processing time)
Memory availability (Error free memory time)
Processor mean-time-to-failure
Memory mean-time-to-failure
Number of processing faults per second
•
•
•
•
•
NIC availability (Error free connection time)
Link availability (Error free transmission time)
NIC mean-time-to-failure
Network timeout duration due to link failure
Frame loss rate
• Disk availability (Error free disk access time)
• Disk mean-time-to-failure
• Number of failed storage read/write operations
per second
Table 4 - Yardstick Generic Test Cases
124
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
Category
Compute
Network
Storage
Performance/Speed
Capacity/Scale
TC003 1 TC004 TC010 TC012 TC014 TC069
TC003 1 TC004
TC024 TC055
TC044 TC073 TC075
TC001 TC002 TC009 TC011 TC042 TC043
Availability/Reliability
TC013 1 TC015
1
TC016 1 TC018
1
TC005
TC063
TC017 1
Note: The description in this OPNFV document is intended as a reference for users to understand the scope of the
Yardstick Project and the deliverables of the Yardstick framework. For complete description of the methodology,
please refer to the ETSI document.
Architecture
Abstract This chapter describes the yardstick framework software architecture. we will introduce it from Use-Case
View, Logical View, Process View and Deployment View. More technical details will be introduced in this chapter.
Overview
Architecture overview Yardstick is mainly written in Python, and test configurations are made in YAML. Documentation is written in reStructuredText format, i.e. .rst files. Yardstick is inspired by Rally. Yardstick is intended to
run on a computer with access and credentials to a cloud. The test case is described in a configuration file given as an
argument.
How it works: the benchmark task configuration file is parsed and converted into an internal model. The context part
of the model is converted into a Heat template and deployed into a stack. Each scenario is run using a runner, either
serially or in parallel. Each runner runs in its own subprocess executing commands in a VM using SSH. The output
of each scenario is written as json records to a file or influxdb or http server, we use influxdb as the backend, the test
result will be shown with grafana.
Concept
Benchmark - assess the relative performance of something
Benchmark configuration file - describes a single test case in yaml format
Context - The set of Cloud resources used by a scenario, such as user names, image names, affinity rules and network
configurations. A context is converted into a simplified Heat template, which is used to deploy onto the Openstack
environment.
Data - Output produced by running a benchmark, written to a file in json format
Runner - Logic that determines how a test scenario is run and reported, for example the number of test iterations,
input value stepping and test duration. Predefined runner types exist for re-usage, see Runner types.
Scenario - Type/class of measurement for example Ping, Pktgen, (Iperf, LmBench, ...)
SLA - Relates to what result boundary a test case must meet to pass. For example a latency limit, amount or ratio of
lost packets and so on. Action based on SLA can be configured, either just to log (monitor) or to stop further testing
(assert). The SLA criteria is set in the benchmark configuration file and evaluated by the runner.
1 To
be included in future deliveries.
2.2. Testing User Guides
125
OPNFV Documentation, Release Danube
Runner types There exists several predefined runner types to choose between when designing a test scenario:
Arithmetic: Every test run arithmetically steps the specified input value(s) in the test scenario, adding a value to the
previous input value. It is also possible to combine several input values for the same test case in different combinations.
Snippet of an Arithmetic runner configuration:
runner:
type: Arithmetic
iterators:
name: stride
start: 64
stop: 128
step: 64
Duration: The test runs for a specific period of time before completed.
Snippet of a Duration runner configuration:
runner:
type: Duration
duration: 30
Sequence: The test changes a specified input value to the scenario. The input values to the sequence are specified in
a list in the benchmark configuration file.
Snippet of a Sequence runner configuration:
runner:
type: Sequence
scenario_option_name: packetsize
sequence:
- 100
- 200
- 250
Iteration: Tests are run a specified number of times before completed.
Snippet of an Iteration runner configuration:
runner:
type: Iteration
iterations: 2
Use-Case View Yardstick Use-Case View shows two kinds of users. One is the Tester who will do testing in cloud,
the other is the User who is more concerned with test result and result analyses.
For testers, they will run a single test case or test case suite to verify infrastructure compliance or bencnmark their own
infrastructure performance. Test result will be stored by dispatcher module, three kinds of store method (file, influxdb
and http) can be configured. The detail information of scenarios and runners can be queried with CLI by testers.
For users, they would check test result with four ways.
If dispatcher module is configured as file(default), there are two ways to check test result. One is to get result from
yardstick.out ( default path: /tmp/yardstick.out), the other is to get plot of test result, it will be shown if users execute
command “yardstick-plot”.
If dispatcher module is configured as influxdb, users will check test result on Grafana which is most commonly used
for visualizing time series data.
If dispatcher module is configured as http, users will check test result on OPNFV testing dashboard which use MongoDB as backend.
126
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
Logical View Yardstick Logical View describes the most important classes, their organization, and the most important use-case realizations.
Main classes:
TaskCommands - “yardstick task” subcommand handler.
HeatContext - Do test yaml file context section model convert to HOT, deploy and undeploy Openstack heat stack.
Runner - Logic that determines how a test scenario is run and reported.
TestScenario - Type/class of measurement for example Ping, Pktgen, (Iperf, LmBench, ...)
Dispatcher - Choose user defined way to store test results.
TaskCommands is the “yardstick task” subcommand’s main entry. It takes yaml file (e.g. test.yaml) as input, and uses
HeatContext to convert the yaml file’s context section to HOT. After Openstack heat stack is deployed by HeatContext
with the converted HOT, TaskCommands use Runner to run specified TestScenario. During first runner initialization, it
will create output process. The output process use Dispatcher to push test results. The Runner will also create a process
2.2. Testing User Guides
127
OPNFV Documentation, Release Danube
to execute TestScenario. And there is a multiprocessing queue between each runner process and output process, so
the runner process can push the real-time test results to the storage media. TestScenario is commonly connected with
VMs by using ssh. It sets up VMs and run test measurement scripts through the ssh tunnel. After all TestScenaio is
finished, TaskCommands will undeploy the heat stack. Then the whole test is finished.
Process View (Test execution flow) Yardstick process view shows how yardstick runs a test case. Below is the
sequence graph about the test execution flow using heat context, and each object represents one module in yardstick:
128
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
A user wants to do a test with yardstick. He can use the CLI to input the command to start a task. “TaskCommands”
will receive the command and ask “HeatContext” to parse the context. “HeatContext” will then ask “Model” to
convert the model. After the model is generated, “HeatContext” will inform “Openstack” to deploy the heat stack by
heat template. After “Openstack” deploys the stack, “HeatContext” will inform “Runner” to run the specific test case.
Firstly, “Runner” would ask “TestScenario” to process the specific scenario. Then “TestScenario” will start to log
on the openstack by ssh protocal and execute the test case on the specified VMs. After the script execution finishes, “TestScenario” will send a message to inform “Runner”. When the testing job is done, “Runner” will inform
“Dispatcher” to output the test result via file, influxdb or http. After the result is output, “HeatContext” will call
“Openstack” to undeploy the heat stack. Once the stack is undepoyed, the whole test ends.
Deployment View Yardstick deployment view shows how the yardstick tool can be deployed into the underlying
platform. Generally, yardstick tool is installed on JumpServer(see 07-installation for detail installation steps), and
JumpServer is connected with other control/compute servers by networking. Based on this deployment, yardstick can
run the test cases on these hosts, and get the test result for better showing.
2.2. Testing User Guides
129
OPNFV Documentation, Release Danube
Yardstick Directory structure
yardstick/ - Yardstick main directory.
tests/ci/ - Used for continuous integration of Yardstick at different PODs and with support for different installers.
docs/ - All documentation is stored here, such as configuration guides, user guides and Yardstick descriptions.
etc/ - Used for test cases requiring specific POD configurations.
samples/ - test case samples are stored here, most of all scenario and feature’s samples are shown in this directory.
tests/ - Here both Yardstick internal tests (functional/ and unit/) as well as the test cases run to verify the NFVI
(opnfv/ ) are stored. Also configurations of what to run daily and weekly at the different PODs is located here.
tools/ - Currently contains tools to build image for VMs which are deployed by Heat. Currently contains how to
build the yardstick-trusty-server image with the different tools that are needed from within the image.
plugin/ - Plug-in configuration files are stored here.
vTC/ - Contains the files for running the virtual Traffic Classifier tests.
yardstick/ - Contains the internals of Yardstick: Runners, Scenario, Contexts, CLI parsing, keys, plotting tools,
dispatcher, plugin install/remove scripts and so on.
Yardstick Installation
Abstract Yardstick supports installation by Docker or directly in Ubuntu. The installation procedure for Docker and
direct installation are detailed in the sections below.
To use Yardstick you should have access to an OpenStack environment, with at least Nova, Neutron, Glance, Keystone
and Heat installed.
The steps needed to run Yardstick are:
1. Install Yardstick.
2. Load OpenStack environment variables.
130
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
3. Create Yardstick flavor.
4. Build a guest image and load it into the OpenStack environment.
5. Create the test configuration .yaml file and run the test case/suite.
Prerequisites The OPNFV deployment is out of the scope of this document and can be found here. The OPNFV
platform is considered as the System Under Test (SUT) in this document.
Several prerequisites are needed for Yardstick:
1. A Jumphost to run Yardstick on
2. A Docker daemon or a virtual environment installed on the Jumphost
3. A public/external network created on the SUT
4. Connectivity from the Jumphost to the SUT public/external network
NOTE: Jumphost refers to any server which meets the previous requirements. Normally it is the same server from
where the OPNFV deployment has been triggered.
WARNING: Connectivity from Jumphost is essential and it is of paramount importance to make sure it is working
before even considering to install and run Yardstick. Make also sure you understand how your networking is designed
to work.
NOTE: If your Jumphost is operating behind a company http proxy and/or Firewall, please consult first the section
‘Proxy Support (**Todo**)‘_, towards the end of this document. That section details some tips/tricks which may be
of help in a proxified environment.
Install Yardstick using Docker (recommended) Yardstick has a Docker image. It is recommended to use this
Docker image to run Yardstick test.
Prepare the Yardstick container Install docker on your guest system with the following command, if not done yet:
wget -qO- https://get.docker.com/ | sh
Pull the Yardstick Docker image (opnfv/yardstick) from the public dockerhub registry under the OPNFV account: dockerhub, with the following docker command:
docker pull opnfv/yardstick:stable
After pulling the Docker image, check that it is available with the following docker command:
[[email protected] ~]$ docker images
REPOSITORY
TAG
IMAGE ID
opnfv/yardstick
stable
a4501714757a
CREATED
1 day ago
SIZE
915.4 MB
Run the Docker image to get a Yardstick container:
docker run -itd --privileged -v /var/run/docker.sock:/var/run/docker.sock -p 8888:5000 -e INSTALLER_I
Note:
2.2. Testing User Guides
131
OPNFV Documentation, Release Danube
parameters
-itd
Detail
-i: interactive, Keep STDIN open even if not attached. -t: allocate a
pseudo-TTY. -d: run container in detached mode, in the background.
–privileged
If you want to build yardstick-image in Yardstick container, this
parameter is needed.
-e INIf you want to use yardstick env prepare command(or related API) to load the
STALLER_IP=192.168.200.2
images that Yardstick needs, these parameters should be provided. The
-e
INSTALLER_IP and INSTALLER_TYPE are depending on your OpenStack
INSTALLER_TYPE=compass
installer. Currently Apex, Compass, Fuel and Joid are supported. If you use
other installers, such as devstack, these parameters can be ignores.
-p 8888:5000
If you want to call Yardstick API out of Yardstick container, this parameter is
needed.
-v
If you want to use yardstick env grafana/influxdb to create a grafana/influxdb
/var/run/docker.sock:/var/run/docker.sock
container out of Yardstick container, this parameter is needed.
–name yardstick
The name for this container, not needed and can be defined by the user.
Configure the Yardstick container environment There are three ways to configure environments for running Yardstick, which will be shown in the following sections. Before that, enter the Yardstick container:
docker exec -it yardstick /bin/bash
and then configure Yardstick environments in the Yardstick container.
The first way (recommended) In the Yardstick container, the Yardstick repository is located in the
/home/opnfv/repos directory. Yardstick provides a CLI to prepare OpenStack environment variables and create
Yardstick flavor and guest images automatically:
yardstick env prepare
NOTE: The above command works for four OPNFV installers – Apex, Compass, Fuel and Joid. For Non-OPNFV installer OpenStack environment, the above command can also be used to configure the environment. But before running
the above command in a Non-OPNFV installer environment, it is necessary to create the /etc/yardstick/openstack.creds
file and save OpenStack environment variables in it. For details of the required OpenStack environment variables
please refer to section Export OpenStack environment variables
The env prepare command may take up to 6-8 minutes to finish building yardstick-image and other environment
preparation. Meanwhile if you wish to monitor the env prepare process, you can enter the Yardstick container in a new
terminal window and execute the following command:
tail -f /var/log/yardstick/uwsgi.log
The second way
Export OpenStack environment variables Before running Yardstick it is necessary to export OpenStack environment variables:
source openrc
Environment variables in the openrc file have to include at least:
• OS_AUTH_URL
• OS_USERNAME
• OS_PASSWORD
132
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
• OS_TENANT_NAME
• EXTERNAL_NETWORK
A sample openrc file may look like this:
export
export
export
export
export
export
OS_PASSWORD=console
OS_TENANT_NAME=admin
OS_AUTH_URL=http://172.16.1.222:35357/v2.0
OS_USERNAME=admin
OS_VOLUME_API_VERSION=2
EXTERNAL_NETWORK=net04_ext
Manually create Yardstick falvor and guest images Before executing Yardstick test cases, make sure that Yardstick flavor and guest image are available in OpenStack. Detailed steps about creating the Yardstick flavor and building
the Yardstick guest image can be found below.
Most of the sample test cases in Yardstick are using an OpenStack flavor called yardstick-flavor which deviates
from the OpenStack standard m1.tiny flavor by the disk size - instead of 1GB it has 3GB. Other parameters are the
same as in m1.tiny.
Create yardstick-flavor:
nova flavor-create yardstick-flavor 100 512 3 1
Most of the sample test cases in Yardstick are using a guest image called yardstick-image which deviates from
an Ubuntu Cloud Server image containing all the required tools to run test cases supported by Yardstick. Yardstick has
a tool for building this custom image. It is necessary to have sudo rights to use this tool.
Also you may need install several additional packages to use this tool, by follwing the commands below:
sudo apt-get update && sudo apt-get install -y qemu-utils kpartx
This image can be built using the following command in the directory where Yardstick is installed:
export YARD_IMG_ARCH='amd64'
sudo echo "Defaults env_keep += \'YARD_IMG_ARCH\'" >> /etc/sudoers
sudo tools/yardstick-img-modify tools/ubuntu-server-cloudimg-modify.sh
Warning: Before building the guest image inside the Yardstick container, make sure the container is granted with
privilege. The script will create files by default in /tmp/workspace/yardstick and the files will be owned by
root!
The created image can be added to OpenStack using the glance image-create or via the OpenStack Dashboard.
Example command is:
glance --os-image-api-version 1 image-create \
--name yardstick-image --is-public true \
--disk-format qcow2 --container-format bare \
--file /tmp/workspace/yardstick/yardstick-image.img
Some Yardstick test cases use a Cirros 0.3.5 image and/or a Ubuntu 16.04 image. Add Cirros and Ubuntu images to
OpenStack:
openstack image create \
--disk-format qcow2 \
--container-format bare \
--file $cirros_image_file \
cirros-0.3.5
2.2. Testing User Guides
133
OPNFV Documentation, Release Danube
openstack image create \
--disk-format qcow2 \
--container-format bare \
--file $ubuntu_image_file \
Ubuntu-16.04
The third way Similar to the second way, the first step is also to Export OpenStack environment variables. Then the
following steps should be done.
Automatically create Yardstcik flavor and guest images Yardstick has a script for automatically creating Yardstick
flavor and building Yardstick guest images. This script is mainly used for CI and can be also used in the local
environment:
source $YARDSTICK_REPO_DIR/tests/ci/load_images.sh
Delete the Yardstick container If you want to uninstall Yardstick, just delete the Yardstick container:
docker stop yardstick && docker rm yardstick
Install Yardstick directly in Ubuntu Alternatively you can install Yardstick framework directly in Ubuntu or in
an Ubuntu Docker image. No matter which way you choose to install Yardstick, the following installation steps are
identical.
If you choose to use the Ubuntu Docker image, you can pull the Ubuntu Docker image from Docker hub:
docker pull ubuntu:16.04
Install Yardstick Prerequisite preparation:
apt-get update && apt-get install -y git python-setuptools python-pip
easy_install -U setuptools==30.0.0
pip install appdirs==1.4.0
pip install virtualenv
Create a virtual environment:
virtualenv ~/yardstick_venv
export YARDSTICK_VENV=~/yardstick_venv
source ~/yardstick_venv/bin/activate
Download the source code and install Yardstick from it:
git clone https://gerrit.opnfv.org/gerrit/yardstick
export YARDSTICK_REPO_DIR=~/yardstick
cd yardstick
./install.sh
Configure the Yardstick environment (Todo) For installing Yardstick directly in Ubuntu, the yardstick env
command is not available. You need to prepare OpenStack environment variables and create Yardstick flavor and guest
images manually.
134
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
Uninstall Yardstick For unistalling Yardstick, just delete the virtual environment:
rm -rf ~/yardstick_venv
Verify the installation It is recommended to verify that Yardstick was installed successfully by executing some
simple commands and test samples. Before executing Yardstick test cases make sure yardstick-flavor and
yardstick-image can be found in OpenStack and the openrc file is sourced. Below is an example invocation
of Yardstick help command and ping.py test sample:
yardstick -h
yardstick task start samples/ping.yaml
NOTE: The above commands could be run in both the Yardstick container and the Ubuntu directly.
Each testing tool supported by Yardstick has a sample configuration file. These configuration files can be found in the
samples directory.
Default location for the output is /tmp/yardstick.out.
Deploy InfluxDB and Grafana using Docker Without InfluxDB, Yardstick stores results for runnning test case in
the file /tmp/yardstick.out. However, it’s unconvenient to retrieve and display test results. So we will show
how to use InfluxDB to store data and use Grafana to display data in the following sections.
Automatically deploy InfluxDB and Grafana containers (recommended) Firstly, enter the Yardstick container:
docker exec -it yardstick /bin/bash
Secondly, create InfluxDB container and configure with the following command:
yardstick env influxdb
Thirdly, create and configure Grafana container:
yardstick env grafana
Then you can run a test case and visit http://host_ip:3000 (admin/admin) to see the results.
NOTE: Executing yardstick env command to deploy InfluxDB and Grafana requires Jumphost’s docker API
version => 1.24. Run the following command to check the docker API version on the Jumphost:
docker version
Manually deploy InfluxDB and Grafana containers You could also deploy influxDB and Grafana containers
manually on the Jumphost. The following sections show how to do.
Pull docker images
docker pull tutum/influxdb
docker pull grafana/grafana
Run and configure influxDB Run influxDB:
docker run -d --name influxdb \
-p 8083:8083 -p 8086:8086 --expose 8090 --expose 8099 \
tutum/influxdb
docker exec -it influxdb bash
2.2. Testing User Guides
135
OPNFV Documentation, Release Danube
Configure influxDB:
influx
>CREATE USER root WITH PASSWORD 'root' WITH ALL PRIVILEGES
>CREATE DATABASE yardstick;
>use yardstick;
>show MEASUREMENTS;
Run and configure Grafana Run Grafana:
docker run -d --name grafana -p 3000:3000 grafana/grafana
Log on http://{YOUR_IP_HERE}:3000 using admin/admin and configure database resource to be
{YOUR_IP_HERE}:8086.
Configure yardstick.conf
docker exec -it yardstick /bin/bash
cp etc/yardstick/yardstick.conf.sample /etc/yardstick/yardstick.conf
vi /etc/yardstick/yardstick.conf
Modify yardstick.conf:
[DEFAULT]
debug = True
dispatcher = influxdb
[dispatcher_influxdb]
timeout = 5
target = http://{YOUR_IP_HERE}:8086
136
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
db_name = yardstick
username = root
password = root
Now you can run Yardstick test cases and store the results in influxDB.
Deploy InfluxDB and Grafana directly in Ubuntu (Todo)
Run Yardstick in a local environment We also have a guide about how to run Yardstick in a local environment.
This work is contributed by Tapio Tallgren. You can find this guide at here.
Create a test suite for Yardstick A test suite in yardstick is a yaml file which include one or more test cases.
Yardstick is able to support running test suite task, so you can customize your own test suite and run it in one task.
tests/opnfv/test_suites is the folder where Yardstick puts CI test suite. A typical test suite is like below
(the fuel_test_suite.yaml example):
--# Fuel integration test task suite
schema: "yardstick:suite:0.1"
name: "fuel_test_suite"
test_cases_dir: "samples/"
test_cases:
file_name: ping.yaml
file_name: iperf3.yaml
As you can see, there are two test cases in the fuel_test_suite.yaml. The schema and the name must be
specified. The test cases should be listed via the tag test_cases and their relative path is also marked via the tag
test_cases_dir.
Yardstick test suite also supports constraints and task args for each test case. Here is another sample (the
os-nosdn-nofeature-ha.yaml example) to show this, which is digested from one big test suite:
--schema: "yardstick:suite:0.1"
name: "os-nosdn-nofeature-ha"
test_cases_dir: "tests/opnfv/test_cases/"
test_cases:
file_name: opnfv_yardstick_tc002.yaml
file_name: opnfv_yardstick_tc005.yaml
file_name: opnfv_yardstick_tc043.yaml
constraint:
installer: compass
pod: huawei-pod1
task_args:
huawei-pod1: '{"pod_info": "etc/yardstick/.../pod.yaml",
"host": "node4.LF","target": "node5.LF"}'
2.2. Testing User Guides
137
OPNFV Documentation, Release Danube
As you can see in test case opnfv_yardstick_tc043.yaml, there are two tags, constraint and
task_args. constraint is to specify which installer or pod it can be run in the CI environment. task_args is
to specify the task arguments for each pod.
All in all, to create a test suite in Yardstick, you just need to create a yaml file and add test cases, constraint or task
arguments if necessary.
Proxy Support (Todo)
Installing a plug-in into Yardstick
Abstract Yardstick provides a plugin CLI command to support integration with other OPNFV testing projects.
Below is an example invocation of Yardstick plugin command and Storperf plug-in sample.
Installing
Storperf
into
Yardstick Storperf
https://hub.docker.com/r/opnfv/storperf/tags/.
is
delivered
as
a
Docker
container
from
There are two possible methods for installation in your environment:
• Run container on Jump Host
• Run container in a VM
In this introduction we will install Storperf on Jump Host.
Step 0: Environment preparation Running Storperf on Jump Host Requirements:
• Docker must be installed
• Jump Host must have access to the OpenStack Controller API
• Jump Host must have internet connectivity for downloading docker image
• Enough floating IPs must be available to match your agent count
Before installing Storperf into yardstick you need to check your openstack environment and other dependencies:
1. Make sure docker is installed.
2. Make sure Keystone, Nova, Neutron, Glance, Heat are installed correctly.
3. Make sure Jump Host have access to the OpenStack Controller API.
4. Make sure Jump Host must have internet connectivity for downloading docker image.
5. You need to know where to get basic openstack Keystone authorization info, such as OS_PASSWORD,
OS_TENANT_NAME, OS_AUTH_URL, OS_USERNAME.
6. To run a Storperf container, you need to have OpenStack Controller environment variables defined and passed
to Storperf container. The best way to do this is to put environment variables in a “storperf_admin-rc” file. The
storperf_admin-rc should include credential environment variables at least:
• OS_AUTH_URL
• OS_USERNAME
• OS_PASSWORD
• OS_TENANT_ID
• OS_TENANT_NAME
138
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
• OS_PROJECT_NAME
• OS_PROJECT_ID
• OS_USER_DOMAIN_ID
Yardstick has a “prepare_storperf_admin-rc.sh” script which can be used to generate the “storperf_admin-rc” file, this
script is located at test/ci/prepare_storperf_admin-rc.sh
#!/bin/bash
# Prepare storperf_admin-rc for StorPerf.
AUTH_URL=${OS_AUTH_URL}
USERNAME=${OS_USERNAME:-admin}
PASSWORD=${OS_PASSWORD:-console}
TENANT_NAME=${OS_TENANT_NAME:-admin}
TENANT_ID=`openstack project show admin|grep '\bid\b' |awk -F '|' '{print $3}'|sed -e 's/^[[:space:]]
PROJECT_NAME=${OS_PROJECT_NAME:-$TENANT_NAME}
PROJECT_ID=`openstack project show admin|grep '\bid\b' |awk -F '|' '{print $3}'|sed -e 's/^[[:space:]
USER_DOMAIN_ID=${OS_USER_DOMAIN_ID:-default}
rm -f ~/storperf_admin-rc
touch ~/storperf_admin-rc
echo
echo
echo
echo
echo
echo
echo
echo
"OS_AUTH_URL="$AUTH_URL >> ~/storperf_admin-rc
"OS_USERNAME="$USERNAME >> ~/storperf_admin-rc
"OS_PASSWORD="$PASSWORD >> ~/storperf_admin-rc
"OS_PROJECT_NAME="$PROJECT_NAME >> ~/storperf_admin-rc
"OS_PROJECT_ID="$PROJECT_ID >> ~/storperf_admin-rc
"OS_TENANT_NAME="$TENANT_NAME >> ~/storperf_admin-rc
"OS_TENANT_ID="$TENANT_ID >> ~/storperf_admin-rc
"OS_USER_DOMAIN_ID="$USER_DOMAIN_ID >> ~/storperf_admin-rc
The generated “storperf_admin-rc” file will be stored in the root directory. If you installed Yardstick using Docker,
this file will be located in the container. You may need to copy it to the root directory of the Storperf deployed host.
Step 1: Plug-in configuration file preparation To install a plug-in, first you need to prepare a plug-in configuration
file in YAML format and store it in the “plugin” directory. The plugin configration file work as the input of yardstick
“plugin” command. Below is the Storperf plug-in configuration file sample:
--# StorPerf plugin configuration file
# Used for integration StorPerf into Yardstick as a plugin
schema: "yardstick:plugin:0.1"
plugins:
name: storperf
deployment:
ip: 192.168.23.2
user: root
password: root
In the plug-in configuration file, you need to specify the plug-in name and the plug-in deployment info, including node
ip, node login username and password. Here the Storperf will be installed on IP 192.168.23.2 which is the Jump Host
in my local environment.
Step 2: Plug-in install/remove scripts preparation In “yardstick/resource/scripts” directory, there are two folders:
a “install” folder and a “remove” folder. You need to store the plug-in install/remove scripts in these two folders
respectively.
2.2. Testing User Guides
139
OPNFV Documentation, Release Danube
The detailed installation or remove operation should de defined in these two scripts. The name of both install and
remove scripts should match the plugin-in name that you specified in the plug-in configuration file.
For example, the install and remove scripts for Storperf are both named to “storperf.bash”.
Step 3: Install and remove Storperf To install Storperf, simply execute the following command:
# Install Storperf
yardstick plugin install plugin/storperf.yaml
removing Storperf from yardstick To remove Storperf, simply execute the following command:
# Remove Storperf
yardstick plugin remove plugin/storperf.yaml
What yardstick plugin command does is using the username and password to log into the deployment target and then
execute the corresponding install or remove script.
Store Other Project’s Test Results in InfluxDB
Abstract This chapter illustrates how to run plug-in test cases and store test results into community’s InfluxDB. The
framework is shown in Framework.
Store Storperf Test Results into Community’s InfluxDB As shown in Framework, there are two ways to store
Storperf test results into community’s InfluxDB:
1. Yardstick executes Storperf test case (TC074), posting test job to Storperf container via ReST API. After the test
job is completed, Yardstick reads test results via ReST API from Storperf and posts test data to the influxDB.
2. Additionally, Storperf can run tests by itself and post the test result directly to the InfluxDB. The method for
posting data directly to influxDB will be supported in the future.
140
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
Our plan is to support rest-api in D release so that other testing projects can call the rest-api to use yardstick dispatcher
service to push data to yardstick’s influxdb database.
For now, influxdb only support line protocol, and the json protocol is deprecated.
Take ping test case for example, the raw_result is json format like this:
"benchmark": {
"timestamp": 1470315409.868095,
"errors": "",
"data": {
"rtt": {
"ares": 1.125
}
},
"sequence": 1
},
"runner_id": 2625
}
With the help of “influxdb_line_protocol”, the json is transform to like below as a line string:
'ping,deploy_scenario=unknown,host=athena.demo,installer=unknown,pod_name=unknown,
runner_id=2625,scenarios=Ping,target=ares.demo,task_id=77755f38-1f6a-4667-a7f3301c99963656,version=unknown rtt.ares=1.125 1470315409868094976'
So, for data output of json format, you just need to transform json into line format and call influxdb api to post the
data into the database. All this function has been implemented in Influxdb. If you need support on this, please contact
Mingjiang.
curl -i -XPOST 'http://104.197.68.199:8086/write?db=yardstick' -data-binary 'ping,deploy_scenario=unknown,host=athena.demo,installer=unknown, ...'
Grafana will be used for visualizing the collected test data, which is shown in Visual. Grafana can be accessed by
Login.
Grafana dashboard
Abstract This chapter describes the Yardstick grafana dashboard. The Yardstick grafana dashboard can be found
here: http://testresults.opnfv.org/grafana/
2.2. Testing User Guides
141
OPNFV Documentation, Release Danube
Public access Yardstick provids a public account for accessing to the dashboard. The username and password are
both set to ‘opnfv’.
Testcase dashboard For each test case, there is a dedicated dashboard. Shown here is the dashboard of TC002.
For each test case dashboard. On the top left, we have a dashboard selection, you can switch to different test cases
using this pull-down menu.
Underneath, we have a pod and scenario selection. All the pods and scenarios that have ever published test data to the
InfluxDB will be shown here.
You can check multiple pods or scenarios.
For each test case, we have a short description and a link to detailed test case information in Yardstick user guide.
Underneath, it is the result presentation section. You can use the time period selection on the top right corner to zoom
in or zoom out the chart.
Administration access For a user with administration rights it is easy to update and save any dashboard configuration. Saved updates immediately take effect and become live. This may cause issues like:
• Changes and updates made to the live configuration in Grafana can compromise existing Grafana content in an
unwanted, unpredicted or incompatible way. Grafana as such is not version controlled, there exists one single
Grafana configuration per dashboard.
• There is a risk several people can disturb each other when doing updates to the same Grafana dashboard at the
same time.
Any change made by administrator should be careful.
142
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
Add a dashboard into yardstick grafana Due to security concern, users that using the public opnfv account are not
able to edit the yardstick grafana directly.It takes a few more steps for a non-yardstick user to add a custom dashboard
into yardstick grafana.
There are 6 steps to go.
1. You need to build a local influxdb and grafana, so you can do the work locally. You can refer to How to deploy
InfluxDB and Grafana locally wiki page about how to do this.
2. Once step one is done, you can fetch the existing grafana dashboard configuration file from the yardstick repository and import it to your local grafana. After import is done, you grafana dashboard will be ready to use just
like the community’s dashboard.
3. The third step is running some test cases to generate test results and publishing it to your local influxdb.
4. Now you have some data to visualize in your dashboard. In the fourth step, it is time to create your own
dashboard. You can either modify an existing dashboard or try to create a new one from scratch. If you choose
to modify an existing dashboard then in the curtain menu of the existing dashboard do a “Save As...” into a new
dashboard copy instance, and then continue doing all updates and saves within the dashboard copy.
5. When finished with all Grafana configuration changes in this temporary dashboard then chose “export” of the
updated dashboard copy into a JSON file and put it up for review in Gerrit, in file /yardstick/dashboard/YardstickTCxxx-yyyyyyyyyyyyy. For instance a typical default name of the file would be “Yardstick-TC001 Copy1234567891234”.
6. Once you finish your dashboard, the next step is exporting the configuration file and propose a patch into
Yardstick. Yardstick team will review and merge it into Yardstick repository. After approved review Yardstick
team will do an “import” of the JSON file and also a “save dashboard” as soon as possible to replace the old live
dashboard configuration.
Yardstick Restful API
Abstract Yardstick support restful API in danube.
Available API
2.2. Testing User Guides
143
OPNFV Documentation, Release Danube
/yardstick/env/action Description: This API is used to do some work related to environment. For now, we support:
1. Prepare yardstick environment(Including fetch openrc file, get external network and load images)
2. Start a InfluxDB docker container and config yardstick output to InfluxDB.
3. Start a Grafana docker container and config with the InfluxDB.
Which API to call will depend on the Parameters.
Method: POST
Prepare Yardstick Environment Example:
{
'action': 'prepareYardstickEnv'
}
This is an asynchronous API. You need to call /yardstick/asynctask API to get the task result.
Start and Config InfluxDB docker container Example:
{
'action': 'createInfluxDBContainer'
}
This is an asynchronous API. You need to call /yardstick/asynctask API to get the task result.
Start and Config Grafana docker container Example:
{
'action': 'createGrafanaContainer'
}
This is an asynchronous API. You need to call /yardstick/asynctask API to get the task result.
/yardstick/asynctask Description: This API is used to get the status of asynchronous task
Method: GET
Get the status of asynchronous task Example:
http://localhost:8888/yardstick/asynctask?task_id=3f3f5e03-972a-4847-a5f8-154f1b31db8c
The returned status will be 0(running), 1(finished) and 2(failed).
/yardstick/testcases Description: This API is used to list all release test cases now in yardstick.
Method: GET
Get a list of release test cases Example:
http://localhost:8888/yardstick/testcases
/yardstick/testcases/release/action Description: This API is used to run a yardstick release test case.
Method: POST
Run a release test case Example:
144
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
{
'action': 'runTestCase',
'args': {
'opts': {},
'testcase': 'tc002'
}
}
This is an asynchronous API. You need to call /yardstick/results to get the result.
/yardstick/testcases/samples/action Description: This API is used to run a yardstick sample test case.
Method: POST
Run a sample test case Example:
{
'action': 'runTestCase',
'args': {
'opts': {},
'testcase': 'ping'
}
}
This is an asynchronous API. You need to call /yardstick/results to get the result.
/yardstick/testsuites/action Description: This API is used to run a yardstick test suite.
Method: POST
Run a test suite Example:
{
'action': 'runTestSuite',
'args': {
'opts': {},
'testcase': 'smoke'
}
}
This is an asynchronous API. You need to call /yardstick/results to get the result.
/yardstick/results Description: This API is used to get the test results of certain task. If you call /yardstick/testcases/samples/action API, it will return a task id. You can use the returned task id to get the results by
using this API.
Get test results of one task Example:
http://localhost:8888/yardstick/results?task_id=3f3f5e03-972a-4847-a5f8-154f1b31db8c
This API will return a list of test case result
Yardstick User Interface
This interface provides a user to view the test result in table format and also values pinned on to a graph.
2.2. Testing User Guides
145
OPNFV Documentation, Release Danube
Command
yardstick report generate <task-ID> <testcase-filename>
Description 1. When the command is triggered using the task-id and the testcase name provided the respective
values are retrieved from the database (influxdb in this particular case).
2. The values are then formatted and then provided to the html template framed with complete html body using Django
Framework.
3. Then the whole template is written into a html file.
The graph is framed with Timestamp on x-axis and output values (differ from testcase to testcase) on y-axis with the
help of “Highcharts”.
Virtual Traffic Classifier
Abstract This chapter provides an overview of the virtual Traffic Classifier, a contribution to OPNFV Yardstick
from the EU Project TNOVA. Additional documentation is available in TNOVAresults.
Overview The virtual Traffic Classifier (VTC) VNF, comprises of a Virtual Network Function Component (VNFC).
The VNFC contains both the Traffic Inspection module, and the Traffic forwarding module, needed to run the VNF. The
exploitation of Deep Packet Inspection (DPI) methods for traffic classification is built around two basic assumptions:
• third parties unaffiliated with either source or recipient are able to
inspect each IP packet’s payload
• the classifier knows the relevant syntax of each application’s packet
payloads (protocol signatures, data patterns, etc.).
The proposed DPI based approach will only use an indicative, small number of the initial packets from each flow in
order to identify the content and not inspect each packet.
In this respect it follows the Packet Based per Flow State (term:PBFS). This method uses a table to track each session
based on the 5-tuples (src address, dest address, src port,dest port, transport protocol) that is maintained for each flow.
Concepts
• Traffic Inspection: The process of packet analysis and application
identification of network traffic that passes through the VTC.
• Traffic Forwarding: The process of packet forwarding from an incoming
network interface to a pre-defined outgoing network interface.
• Traffic Rule Application: The process of packet tagging, based on a
predefined set of rules. Packet tagging may include e.g. Type of Service (ToS) field modification.
Architecture The Traffic Inspection module is the most computationally intensive component of the VNF. It implements filtering and packet matching algorithms in order to support the enhanced traffic forwarding capability of the
VNF. The component supports a flow table (exploiting hashing algorithms for fast indexing of flows) and an inspection
engine for traffic classification.
146
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
The implementation used for these experiments exploits the nDPI library. The packet capturing mechanism is implemented using libpcap. When the DPI engine identifies a new flow, the flow register is updated with the appropriate
information and transmitted across the Traffic Forwarding module, which then applies any required policy updates.
The Traffic Forwarding moudle is responsible for routing and packet forwarding. It accepts incoming network traffic, consults the flow table for classification information for each incoming flow and then applies pre-defined policies
marking e.g. ToS/Differentiated Services Code Point (DSCP) multimedia traffic for Quality of Service (QoS) enablement on the forwarded traffic. It is assumed that the traffic is forwarded using the default policy until it is identified
and new policies are enforced.
The expected response delay is considered to be negligible, as only a small number of packets are required to identify
each flow.
Graphical Overview
+----------------------------+
|
|
| Virtual Traffic Classifier |
|
|
|
Analysing/Forwarding
|
|
------------>
|
|
ethA
ethB
|
|
|
+----------------------------+
|
^
|
|
v
|
+----------------------------+
|
|
|
Virtual Switch
|
|
|
+----------------------------+
Install run the vTC/build.sh with root privileges
Run
sudo ./pfbridge -a eth1 -b eth2
Note: Virtual Traffic Classifier is not support in OPNFV Danube release.
Development Environment Ubuntu 14.04 Ubuntu 16.04
Apexlake Installation Guide
Abstract ApexLake is a framework that provides automatic execution of experiments and related data collection to
enable a user validate infrastructure from the perspective of a Virtual Network Function (VNF).
In the context of Yardstick, a virtual Traffic Classifier (VTC) network function is utilized.
Framework Hardware Dependencies
for ApexLake.
In order to run the framework there are some hardware related dependencies
The framework needs to be installed on the same physical node where DPDK-pktgen is installed.
2.2. Testing User Guides
147
OPNFV Documentation, Release Danube
The installation requires the physical node hosting the packet generator must have 2 NICs which are DPDK compatible.
The 2 NICs will be connected to the switch where the OpenStack VM network is managed.
The switch used must support multicast traffic and IGMP snooping. Further details about the configuration are provided at the following here.
The corresponding ports to which the cables are connected need to be configured as VLAN trunks using two of the
VLAN IDs available for Neutron. Note the VLAN IDs used as they will be required in later configuration steps.
Framework Software Dependencies Before starting the framework, a number of dependencies must first be installed. The following describes the set of instructions to be executed via the Linux shell in order to install and
configure the required dependencies.
1. Install Dependencies.
To support the framework dependencies the following packages must be installed. The example provided is based on
Ubuntu and needs to be executed in root mode.
apt-get
apt-get
apt-get
apt-get
apt-get
install
install
install
install
install
python-dev
python-pip
python-mock
tcpreplay
libpcap-dev
2. Source OpenStack openrc file.
source openrc
3. Configure Openstack Neutron
In order to support traffic generation and management by the virtual Traffic Classifier, the configuration of the port
security driver extension is required for Neutron.
For further details please follow the following link: PORTSEC This step can be skipped in case the target OpenStack
is Juno or Kilo release, but it is required to support Liberty. It is therefore required to indicate the release version in
the configuration file located in ./yardstick/vTC/apexlake/apexlake.conf
4. Create Two Networks based on VLANs in Neutron.
To enable network communications between the packet generator and the compute node, two networks must be created
via Neutron and mapped to the VLAN IDs that were previously used in the configuration of the physical switch. The
following shows the typical set of commands required to configure Neutron correctly. The physical switches need to
be configured accordingly.
VLAN_1=2032
VLAN_2=2033
PHYSNET=physnet2
neutron net-create apexlake_inbound_network \
--provider:network_type vlan \
--provider:segmentation_id $VLAN_1 \
--provider:physical_network $PHYSNET
neutron subnet-create apexlake_inbound_network \
192.168.0.0/24 --name apexlake_inbound_subnet
neutron net-create apexlake_outbound_network \
--provider:network_type vlan \
--provider:segmentation_id $VLAN_2 \
--provider:physical_network $PHYSNET
148
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
neutron subnet-create apexlake_outbound_network 192.168.1.0/24 \
--name apexlake_outbound_subnet
5. Download Ubuntu Cloud Image and load it on Glance
The virtual Traffic Classifier is supported on top of Ubuntu 14.04 cloud image. The image can be downloaded on the
local machine and loaded on Glance using the following commands:
wget cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-disk1.img
glance image-create \
--name ubuntu1404 \
--is-public true \
--disk-format qcow \
--container-format bare \
--file trusty-server-cloudimg-amd64-disk1.img
6. Configure the Test Cases
The VLAN tags must also be included in the test case Yardstick yaml file as parameters for the following test cases:
• Yardstick Test Case Description TC006
• Yardstick Test Case Description TC007
• Yardstick Test Case Description TC020
• Yardstick Test Case Description TC021
Install and Configure DPDK Pktgen Execution of the framework is based on DPDK Pktgen. If DPDK Pktgen
has not installed, it is necessary to download, install, compile and configure it. The user can create a directory and
download the dpdk packet generator source code:
cd experimental_framework/libraries
mkdir dpdk_pktgen
git clone https://github.com/pktgen/Pktgen-DPDK.git
For instructions on the installation and configuration of DPDK and DPDK Pktgen please follow the official DPDK
Pktgen README file. Once the installation is completed, it is necessary to load the DPDK kernel driver, as follow:
insmod uio
insmod DPDK_DIR/x86_64-native-linuxapp-gcc/kmod/igb_uio.ko
It is necessary to set the configuration file to support the desired Pktgen configuration. A description of the required
configuration parameters and supporting examples is provided in the following:
[PacketGen]
packet_generator = dpdk_pktgen
# This is the directory where the packet generator is installed
# (if the user previously installed dpdk-pktgen,
# it is required to provide the director where it is installed).
pktgen_directory = /home/user/software/dpdk_pktgen/dpdk/examples/pktgen/
# This is the directory where DPDK is installed
dpdk_directory = /home/user/apexlake/experimental_framework/libraries/Pktgen-DPDK/dpdk/
# Name of the dpdk-pktgen program that starts the packet generator
program_name = app/app/x86_64-native-linuxapp-gcc/pktgen
# DPDK coremask (see DPDK-Pktgen readme)
2.2. Testing User Guides
149
OPNFV Documentation, Release Danube
coremask = 1f
# DPDK memory channels (see DPDK-Pktgen readme)
memory_channels = 3
# Name of the interface of the pktgen to be used to send traffic (vlan_sender)
name_if_1 = p1p1
# Name of the interface of the pktgen to be used to receive traffic (vlan_receiver)
name_if_2 = p1p2
# PCI bus address correspondent to if_1
bus_slot_nic_1 = 01:00.0
# PCI bus address correspondent to if_2
bus_slot_nic_2 = 01:00.1
To find the parameters related to names of the NICs and the addresses of the PCI buses the user may find it useful to
run the DPDK tool nic_bind as follows:
DPDK_DIR/tools/dpdk_nic_bind.py --status
Lists the NICs available on the system, and shows the available drivers and bus addresses for each interface. Please
make sure to select NICs which are DPDK compatible.
Installation and Configuration of smcroute The user is required to install smcroute which is used by the framework
to support multicast communications.
The following is the list of commands required to download and install smroute.
cd ~
git clone https://github.com/troglobit/smcroute.git
cd smcroute
git reset --hard c3f5c56
sed -i 's/aclocal-1.11/aclocal/g' ./autogen.sh
sed -i 's/automake-1.11/automake/g' ./autogen.sh
./autogen.sh
./configure
make
sudo make install
cd ..
It is required to do the reset to the specified commit ID. It is also requires the creation a configuration file using the
following command:
SMCROUTE_NIC=(name of the nic)
where name of the nic is the name used previously for the variable “name_if_2”. For example:
SMCROUTE_NIC=p1p2
Then create the smcroute configuration file /etc/smcroute.conf
echo mgroup from $SMCROUTE_NIC group 224.192.16.1 > /etc/smcroute.conf
At the end of this procedure it will be necessary to perform the following actions to add the user to the sudoers:
adduser USERNAME sudo
echo "user ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers
150
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
Experiment using SR-IOV Configuration on the Compute Node To enable SR-IOV interfaces on the physical
NIC of the compute node, a compatible NIC is required. NIC configuration depends on model and vendor. After
proper configuration to support SR-IOV, a proper configuration of OpenStack is required. For further information,
please refer to the SRIOV configuration guide
Finalize installation the framework on the system The installation of the framework on the system requires the
setup of the project. After entering into the apexlake directory, it is sufficient to run the following command.
python setup.py install
Since some elements are copied into the /tmp directory (see configuration file) it could be necessary to repeat this step
after a reboot of the host.
Apexlake API Interface Definition
Abstract The API interface provided by the framework to enable the execution of test cases is defined as follows.
init
static init()
Initializes the Framework
Returns None
execute_framework
static execute_framework (test_cases,
iterations,
heat_template,
heat_template_parameters,
deployment_configuration,
openstack_credentials)
Executes the framework according the specified inputs
Parameters
• test_cases
Test cases to be run with the workload (dict() of dict())
Example: test_case = dict()
test_case[’name’] = ‘module.Class’
test_case[’params’] = dict()
test_case[’params’][’throughput’] = ‘1’
test_case[’params’][’vlan_sender’] = ‘1000’
test_case[’params’][’vlan_receiver’] = ‘1001’
test_cases = [test_case]
• iterations Number of test cycles to be executed (int)
• heat_template (string) File name of the heat template corresponding to the workload
to be deployed. It contains the parameters to be evaluated in the form of #parameter_name. (See heat_templates/vTC.yaml as example).
2.2. Testing User Guides
151
OPNFV Documentation, Release Danube
• heat_template_parameters (dict) Parameters to be provided as input to the heat template. See http://docs.openstack.org/developer/heat/ template_guide/hot_guide.html
section “Template input parameters” for further info.
• deployment_configuration ( dict[string] = list(strings) ) ) Dictionary of parameters representing the deployment configuration of the workload.
The key is a string corresponding to the name of the parameter, the value is a list
of strings representing the value to be assumed by a specific param. The parameters
are user defined: they have to correspond to the place holders (#parameter_name)
specified in the heat template.
Returns dict() containing results
Network Services Benchmarking (NSB)
Abstract This chapter provides an overview of the NSB, a contribution to OPNFV Yardstick from Intel.
Overview GOAL: Extend Yardstick to perform real world VNFs and NFVi Characterization and benchmarking with
repeatable and deterministic methods.
The Network Service Benchmarking (NSB) extends the yardstick framework to do VNF characterization and benchmarking in three different execution environments - bare metal i.e. native Linux environment, standalone virtual
environment and managed virtualized environment (e.g. Open stack etc.). It also brings in the capability to interact
with external traffic generators both hardware & software based for triggering and validating the traffic according to
user defined profiles.
NSB extension includes:
• Generic data models of Network Services, based on ETSI spec (ETSI GS NFV-TST 001) .. _ETSI GS NFV-TST
001: http://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/001/01.01.01_60/gs_nfv-tst001v010101p.pdf
• New Standalone context for VNF testing like SRIOV, OVS, OVS-DPDK etc
• Generic VNF configuration models and metrics implemented with Python classes
• Traffic generator features and traffic profiles
– L1-L3 state-less traffic profiles
– L4-L7 state-full traffic profiles
– Tunneling protocol / network overlay support
• Test case samples
– Ping
– Trex
– vPE,vCGNAT, vFirewall etc - ipv4 throughput, latency etc
• Traffic generators like Trex, ab/nginx, ixia, iperf etc
• KPIs for a given use case:
– System agent support for collecting NFVi KPI. This includes:
* CPU statistic
* Memory BW
* OVS-DPDK Stats
152
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
– Network KPIs, e.g., inpackets, outpackets, thoughput, latency etc
– VNF KPIs, e.g., packet_in, packet_drop, packet_fwd etc
Architecture The Network Service (NS) defines a set of Virtual Network Functions (VNF) connected together using
NFV infrastructure.
The Yardstick NSB extension can support multiple VNFs created by different vendors including traffic generators.
Every VNF being tested has its own data model. The Network service defines a VNF modelling on base of performed
network functionality. The part of the data model is a set of the configuration parameters, number of connection points
used and flavor including core and memory amount.
The ETSI defines a Network Service as a set of configurable VNFs working in some NFV Infrastructure connecting
each other using Virtual Links available through Connection Points. The ETSI MANO specification defines a set
of management entities called Network Service Descriptors (NSD) and VNF Descriptors (VNFD) that define real
Network Service. The picture below makes an example how the real Network Operator use-case can map into ETSI
Network service definition
Network Service framework performs the necessary test steps. It may involve
• Interacting with traffic generator and providing the inputs on traffic type / packet structure to generate the
required traffic as per the test case. Traffic profiles will be used for this.
• Executing the commands required for the test procedure and analyses the command output for confirming
whether the command got executed correctly or not. E.g. As per the test case, run the traffic for the given
time period / wait for the necessary time delay
• Verify the test result.
• Validate the traffic flow from SUT
• Fetch the table / data from SUT and verify the value as per the test case
• Upload the logs from SUT onto the Test Harness server
• Read the KPI’s provided by particular VNF
Components of Network Service
• Models for Network Service benchmarking: The Network Service benchmarking requires the proper modelling
approach. The NSB provides models using Python files and defining of NSDs and VNFDs.
The benchmark control application being a part of OPNFV yardstick can call that python models to instantiate and
configure the VNFs. Depending on infrastructure type (bare-metal or fully virtualized) that calls could be made
directly or using MANO system.
• Traffic generators in NSB: Any benchmark application requires a set of traffic generator and traffic profiles
defining the method in which traffic is generated.
The Network Service benchmarking model extends the Network Service definition with a set of Traffic Generators
(TG) that are treated same way as other VNFs being a part of benchmarked network service. Same as other VNFs the
traffic generator are instantiated and terminated.
Every traffic generator has own configuration defined as a traffic profile and a set of KPIs supported. The python
models for TG is extended by specific calls to listen and generate traffic.
• The stateless TREX traffic generator: The main traffic generator used as Network Service stimulus is open
source TREX tool.
The TREX tool can generate any kind of stateless traffic.
2.2. Testing User Guides
153
OPNFV Documentation, Release Danube
+--------+
+-------+
+--------+
|
|
|
|
|
|
| Trex | ---> | VNF | ---> | Trex |
|
|
|
|
|
|
+--------+
+-------+
+--------+
Supported testcases scenarios:
• Correlated UDP traffic using TREX traffic generator and replay VNF.
– using different IMIX configuration like pure voice, pure video traffic etc
– using different number IP flows like 1 flow, 1K, 16K, 64K, 256K, 1M flows
– Using different number of rules configured like 1 rule, 1K, 10K rules
For UDP correlated traffic following Key Performance Indicators are collected for every combination of test case
parameters:
• RFC2544 throughput for various loss rate defined (1% is a default)
Graphical Overview NSB Testing with yardstick framework facilitate performance testing of various VNFs provided.
+-----------+
|
|
|
vPE
|
| TestCase |
|
|
+-----------+
+-----------+
->|TGen Port 0|
| +-----------+
|
+------------------+
+-------+
|
|
| -- API --> | VNF | <--->
+-----------+
|
Yardstick
|
+-------+
|
| Test Case | --> |
NSB Testing
|
|
+-----------+
|
|
|
|
|
|
|
|
+------------------+
|
+-----------+
| +-----------+
|
Traffic |
->|TGen Port 1|
| patterns |
+-----------+
+-----------+
Figure 1: Network Service - 2 server configuration
Yardstick - NSB Testing -Installation
Abstract The Network Service Benchmarking (NSB) extends the yardstick framework to do VNF characterization
and benchmarking in three different execution environments viz., bare metal i.e. native Linux environment, standalone
virtual environment and managed virtualized environment (e.g. Open stack etc.). It also brings in the capability
to interact with external traffic generators both hardware & software based for triggering and validating the traffic
according to user defined profiles.
The steps needed to run Yardstick with NSB testing are:
• Install Yardstick (NSB Testing).
• Setup pod.yaml describing Test topology
• Create the test configuration yaml file.
• Run the test case.
154
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
Prerequisites Refer chapter Yardstick Instalaltion for more information on yardstick prerequisites
Several prerequisites are needed for Yardstick(VNF testing):
• Python Modules: pyzmq, pika.
• flex
• bison
• build-essential
• automake
• libtool
• librabbitmq-dev
• rabbitmq-server
• collectd
• intel-cmt-cat
Install Yardstick (NSB Testing) Refer chapter Yardstick Installation for more information on installing Yardstick
After Yardstick is installed, executing the “nsb_setup.sh” script to setup NSB testing.
./nsb_setup.sh
It will also automatically download all the packages needed for NSB Testing setup.
System Topology:
+----------+
+----------+
|
|
|
|
|
| (0)----->(0) |
Ping/ |
|
TG1
|
|
vPE/
|
|
|
|
2Trex |
|
| (1)<-----(1) |
|
+----------+
+----------+
trafficgen_1
vnf
OpenStack parameters and credentials
Environment variables Before running Yardstick (NSB Testing) it is necessary to export traffic generator libraries.
source ~/.bash_profile
Config yardstick conf
cp ./etc/yardstick/yardstick.conf.sample /etc/yardstick/yardstick.conf
vi /etc/yardstick/yardstick.conf
Add trex_path and bin_path in ‘nsb’ section.
[DEFAULT]
debug = True
dispatcher = influxdb
2.2. Testing User Guides
155
OPNFV Documentation, Release Danube
[dispatcher_influxdb]
timeout = 5
target = http://{YOUR_IP_HERE}:8086
db_name = yardstick
username = root
password = root
[nsb]
trex_path=/opt/nsb_bin/trex/scripts
bin_path=/opt/nsb_bin
Config pod.yaml describing Topology Before executing Yardstick test cases, make sure that pod.yaml reflects the
topology and update all the required fields.
cp /etc/yardstick/nodes/pod.yaml.nsb.sample /etc/yardstick/nodes/pod.yaml
Config pod.yaml
nodes:
name: trafficgen_1
role: TrafficGen
ip: 1.1.1.1
user: root
password: r00t
interfaces:
xe0: # logical name from topology.yaml and vnfd.yaml
vpci:
"0000:07:00.0"
driver:
i40e # default kernel driver
dpdk_port_num: 0
local_ip: "152.16.100.20"
netmask:
"255.255.255.0"
local_mac: "00:00:00:00:00:01"
xe1: # logical name from topology.yaml and vnfd.yaml
vpci:
"0000:07:00.1"
driver:
i40e # default kernel driver
dpdk_port_num: 1
local_ip: "152.16.40.20"
netmask:
"255.255.255.0"
local_mac: "00:00.00:00:00:02"
name: vnf
role: vnf
ip: 1.1.1.2
user: root
password: r00t
host: 1.1.1.2 #BM - host == ip, virtualized env - Host - compute node
interfaces:
xe0: # logical name from topology.yaml and vnfd.yaml
vpci:
"0000:07:00.0"
driver:
i40e # default kernel driver
dpdk_port_num: 0
local_ip: "152.16.100.19"
netmask:
"255.255.255.0"
local_mac: "00:00:00:00:00:03"
xe1:
156
# logical name from topology.yaml and vnfd.yaml
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
vpci:
"0000:07:00.1"
driver:
i40e # default kernel driver
dpdk_port_num: 1
local_ip: "152.16.40.19"
netmask:
"255.255.255.0"
local_mac: "00:00:00:00:00:04"
routing_table:
- network: "152.16.100.20"
netmask: "255.255.255.0"
gateway: "152.16.100.20"
if: "xe0"
- network: "152.16.40.20"
netmask: "255.255.255.0"
gateway: "152.16.40.20"
if: "xe1"
nd_route_tbl:
- network: "0064:ff9b:0:0:0:0:9810:6414"
netmask: "112"
gateway: "0064:ff9b:0:0:0:0:9810:6414"
if: "xe0"
- network: "0064:ff9b:0:0:0:0:9810:2814"
netmask: "112"
gateway: "0064:ff9b:0:0:0:0:9810:2814"
if: "xe1"
Enable yardstick virtual environment
virtual environment
Before executing yardstick test cases, make sure to activate yardstick python
source /opt/nsb_bin/yardstick_venv/bin/activate
Run Yardstick - Network Service Testcases
NS testing - using NSBperf CLI
source /opt/nsb_setup/yardstick_venv/bin/activate
PYTHONPATH: ". ~/.bash_profile"
cd <yardstick_repo>/yardstick/cmd
Execute command: ./NSPerf.py -h
./NSBperf.py --vnf <selected vnf> --test <rfc test>
eg: ./NSBperf.py --vnf vpe --test tc_baremetal_rfc2544_ipv4_1flow_64B.yaml
NS testing - using yardstick CLI
source /opt/nsb_setup/yardstick_venv/bin/activate
PYTHONPATH: ". ~/.bash_profile"
Go to test case forlder type we want to execute. e.g. <yardstick repo>/samples/vnf_samples/nsut/<vnf>/ run: yardstick –debug task start <test_case.yaml>
Yardstick Test Cases
Abstract This chapter lists available Yardstick test cases. Yardstick test cases are divided in two main categories:
• Generic NFVI Test Cases - Test Cases developed to realize the methodology
2.2. Testing User Guides
157
OPNFV Documentation, Release Danube
described in Methodology
• OPNFV Feature Test Cases - Test Cases developed to verify one or more
aspect of a feature delivered by an OPNFV Project, including the test cases developed for the VTC.
Generic NFVI Test Case Descriptions
158
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
Network Performance
test case id
metric
test purpose
test tool
test description
traffic profile
Yardstick Test Case Description TC001
configuration
applicability
usability
2.2. Testing User Guides
references
pre-test conditions
OPNFV_YARDSTICK_TC001
FORMANCE
Number of flows and throughpu
The purpose of TC001 is to ev
performance with regards to flo
as if and how different amount
throughput between hosts on d
Typically e.g. the performanc
on the number of flows running
mance of other equipment or en
number of flows or the packet s
The purpose is also to be able to
sults, graphs and similar shall b
reasons and product evolution
different OPNFV versions and/
pktgen
Linux packet generator is a too
very high speed in the kernel. p
drive and LAN equipment test n
multi threading. To generate ra
address, port number UDP pac
ple CPU processors in the diffe
bus) with Gigabit Ethernet test
depends on the CPU processin
PCI bus speed hardware parame
can be even larger than 10GB
most card test requirements.
(Pktgen is not always part of a L
it needs to be installed. It is part
image. As an example see the
tory for how to generate a Linu
cluded.)
This test case uses Pktgen to g
tween two hosts for simulating
the SUT.
An IP table is setup on server
packets.
file: opnfv_yardstick_tc001.yam
Packet size is set to 60 bytes.
50, 100, 500 and 1000, where e
The whole sequence is run twi
are distributed on different hard
For SLA max_ppm is set to 10
figured ports map to between 1
respectively.
Test can be configured with diff
• packet sizes;
• amount of flows;
• test duration.
Default values exist.
SLA (optional): max_ppm: Th
million packets sent that are acc
ceived.
This test case is used for ge
throughput to simulate certain
Hence it 159
should work with othe
pktgen
ETSI-NFV-TST001
The test case image needs to b
OPNFV Documentation, Release Danube
160
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
Network Latency
test case id
metric
test purpose
test tool
test topology
configuration
applicability
Yardstick Test Case Description TC002
usability
references
pre-test conditions
2.2. Testing User Guides
test sequence
step 1
step 2
OPNFV_YARDSTICK_TC002
TENCY
RTT (Round Trip Time)
The purpose of TC002 is to do
network latency is within acce
packets travel between hosts lo
ent compute blades.
The purpose is also to be able to
sults, graphs and similar shall b
reasons and product evolution
different OPNFV versions and/
ping
Ping is a computer network adm
ity used to test the reachability
Protocol (IP) network. It meas
for packet sent from the origina
computer that are echoed back t
Ping is normally part of any Lin
doesn’t need to be installed. It
stick Docker image. (For exam
can be downloaded from cirrosPing
packets
(ICMP
p
ECHO_REQUEST datagram) a
target VM(s) to elicit ICMP EC
For one host VM there can be m
VM and target VM(s) can be on
pute blades.
file: opnfv_yardstick_tc002.yam
Packet size 100 bytes. Test du
ping each 10 seconds. Test is i
RTT is set to maximum 10 ms.
This test case can be configured
• packet sizes;
• burst sizes;
• ping intervals;
• test durations;
• test iterations.
Default values exist.
SLA is optional. The SLA in th
example. Considerably lower R
normal to achieve in balanced
ever, to cover most configuratio
fully virtualized ones, this valu
achieve and acceptable for blac
time applications start to suffer
higher than this. Some may suf
RTT, while others may not suff
mise that may have to be tuned
tion purposes.
This test case is one of Yardstic
is runnable on most of the scena
Ping
ETSI-NFV-TST001
The test case image (cirros-ima
into Glance with ping included
No POD 161
specific requirements
description and expected result
Two host VMs are booted, as se
Yardstick is connected with the
OPNFV Documentation, Release Danube
Cache Utilization
test case id
metric
test purpose
test tool
test description
Yardstick Test Case Description TC004
configuration
applicability
usability
references
pre-test conditions
test sequence
step 1
step 2
step 3
step 4
test verdict
162
OPNFV_YARDSTICK_TC004
cache hit, cache miss, hit/miss ra
cache size
The purpose of TC004 is to ev
capability with regards to cache
should be run in parallel with o
and not run as a stand-alone tes
This test case measures cache
ing cache hit, cache miss, hit rat
page cache size, with some wo
frastructure. Both average and m
lected.
The purpose is also to be able to
sults, graphs and similar shall b
reasons and product evolution
different OPNFV versions and/
cachestat
cachestat is a tool using Linu
showing Linux page cache hit/m
(cachestat is not always part o
hence it needs to be installed.
/yardstick/tools/ directory for h
image with cachestat included.)
cachestat test is invoked in a
blade, cachestat test requires so
ning in the host to stimulate wo
File: cachestat.yaml (in the ‘sam
Interval is set 1. Test repeat, p
in-between. Test durarion is set
SLA is not available in this test
Test can be configured with diff
• interval;
• runner Duration.
Default values exist.
This test case is one of Yardstic
is runnable on most of the scena
cachestat
ETSI-NFV-TST001
The test case image needs to b
with cachestat included in the im
No POD specific requirements
description and expected result
A host VM with cachestat insta
Yardstick is connected with the
‘cache_stat’ bash script is copy
the server VM via the ssh tunne
‘cache_stat’ script is invoked.
tics are collected and filtrated.
values are calculated and recor
and stored.
Result: Logs are stored.
The host VM is deleted.
None. Cache utilization results
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
2.2. Testing User Guides
163
OPNFV Documentation, Release Danube
Storage Performance
test case id
metric
test purpose
test tool
test description
configuration
Yardstick Test Case Description TC005
applicability
usability
references
pre-test conditions
164
test sequence
step 1
step 2
OPNFV_YARDSTICK_TC005
MANCE
IOPS (Average IOs performed p
(Average disk read/write bandw
erage disk read/write latency)
The purpose of TC005 is to ev
performance with regards to IO
tency.
The purpose is also to be able to
sults, graphs and similar shall b
reasons and product evolution
different OPNFV versions and/
fio
fio is an I/O tool meant to b
mark and stress/hardware verifi
19 different types of I/O engin
posixaio, SG v3, splice, null, ne
larisaio, and more), I/O prioriti
nels), rate I/O, forked or threade
(fio is not always part of a Li
it needs to be installed. As an
stick/tools/ directory for how to
with fio included.)
fio test is invoked in a host VM
job file as well as parameters a
will start doing what the job file
file: opnfv_yardstick_tc005.yam
IO types is set to read, write, ran
block size is set to 4KB, 64KB
each IO type and IO block size
runs for 30 seconds (10 for ram
For SLA, minimum read/write
mum read/write throughput is se
imum read/write latency is set t
This test case can be configured
• IO types;
• IO block size;
• IO depth;
• ramp time;
• test duration.
Default values exist.
SLA is optional. The SLA in
an example. Considerably high
latency are expected. However
rations, both baremetal and ful
value should be possible to ach
black box testing. Many heavy
suffer badly if the read/write ba
this.
This test case is one of Yardstic
is runnable on most of the scena
fio
ETSI-NFV-TST001
The test case image needs to b
with fio included in it.
Chapter 2. Test Frameworks
No POD specific requirements
description and expected result
A host VM with fio installed is
Yardstick is connected with the
OPNFV Documentation, Release Danube
Yardstick Test Case Description TC008
2.2. Testing User Guides
Packet Loss Extended Test
test case id OPNFV_YARDSTICK_TC008_NW PERF, Packet loss Extended Test
metric
Number of flows, packet size and throughput
test
To evaluate the IaaS network performance with regards to flows and through
purpose
different amounts of packet sizes and flows matter for the throughput betwee
compute blades. Typically e.g. the performance of a vSwitch depends on the
running through it. Also performance of other equipment or entities can dep
flows or the packet sizes used. The purpose is also to be able to spot trends.
similar shall be stored for comparison reasons and product evolution underst
different OPNFV versions and/or configurations.
configurafile: opnfv_yardstick_tc008.yaml
tion
Packet size: 64, 128, 256, 512, 1024, 1280 and 1518 bytes.
Number of ports: 1, 10, 50, 100, 500 and 1000. The amount of configured po
1001000 flows, respectively. Each packet_size/port_amount combination is r
seconds each. Then the next packet_size/port_amount combination is run, an
The client and server are distributed on different HW.
For SLA max_ppm is set to 1000.
test tool
pktgen
(Pktgen is not always part of a Linux distribution, hence it needs to be instal
Yardstick Docker image. As an example see the /yardstick/tools/ directory fo
Linux image with pktgen included.)
references
pktgen
ETSI-NFV-TST001
applicabil- Test can be configured with different packet sizes, amount of flows and test d
ity
exist.
SLA (optional): max_ppm: The number of packets per million packets sent
loose, not received.
pre-test
The test case image needs to be installed into Glance with pktgen included in
conditions No POD specific requirements have been identified.
test
description and expected result
sequence
step 1
The hosts are installed, as server and client. pktgen is invoked and logs are p
Result: Logs are stored.
test verdict Fails only if SLA is not passed, or if there is a test case execution problem.
165
OPNFV Documentation, Release Danube
Packet Loss
test case id
metric
test
purpose
configuration
Yardstick Test Case Description TC009
test tool
references
applicability
pre-test
conditions
test
sequence
step 1
test verdict
166
OPNFV_YARDSTICK_TC009_NW PERF, Packet loss
Number of flows, packets lost and throughput
To evaluate the IaaS network performance with regards to flows and through
different amounts of flows matter for the throughput between VMs on differ
Typically e.g. the performance of a vSwitch depends on the number of flows
Also performance of other equipment or entities can depend on the number o
sizes used. The purpose is also to be able to spot trends. Test results, graphs
stored for comparison reasons and product evolution understanding between
versions and/or configurations.
file: opnfv_yardstick_tc009.yaml
Packet size: 64 bytes
Number of ports: 1, 10, 50, 100, 500 and 1000. The amount of configured po
1001000 flows, respectively. Each port amount is run ten times, for 20 secon
port_amount is run, and so on.
The client and server are distributed on different HW.
For SLA max_ppm is set to 1000.
pktgen
(Pktgen is not always part of a Linux distribution, hence it needs to be instal
Yardstick Docker image. As an example see the /yardstick/tools/ directory fo
Linux image with pktgen included.)
pktgen
ETSI-NFV-TST001
Test can be configured with different packet sizes, amount of flows and test d
exist.
SLA (optional): max_ppm: The number of packets per million packets sent
loose, not received.
The test case image needs to be installed into Glance with pktgen included in
No POD specific requirements have been identified.
description and expected result
The hosts are installed, as server and client. pktgen is invoked and logs are p
Result: logs are stored.
Fails only if SLA is not passed, or if there is a test case execution problem.
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
Memory Latency
test case id
metric
test purpose
test tool
test description
Yardstick Test Case Description TC010
configuration
2.2. Testing User Guides
OPNFV_YARDSTICK_TC010
TENCY
Memory read latency (nanoseco
The purpose of TC010 is to ev
performance with regards to m
measures the memory read late
sizes and strides. Whole memor
The purpose is also to be able to
sults, graphs and similar shall b
reasons and product evolution
different OPNFV versions and/
Lmbench
Lmbench is a suite of operati
marks. This test uses lat_mem
including:
• Context switching
• Networking: connectio
TCP, UDP, and RPC hot p
• File system creates and d
• Process creation
• Signal handling
• System call overhead
• Memory read latency
(LMbench is not always part
hence it needs to be installed.
/yardstick/tools/ directory for h
image with LMbench included.
LMbench lat_mem_rd benchm
read latency for varying memor
The benchmark runs as two nest
is the stride size. The inner loo
each array size, the benchmark
that point backward one stride
done by:
p = (char **)*p;
in a for loop (the over head of
nificant; the loop is an unrolle
The size of the array varies from
eight megabytes. For the small s
an effect, and the loads will be
comes much more apparent wh
Only data accesses are measure
is not measured.
The results are reported in nan
have been verified accurate to w
on an SGI Indy.
File: opnfv_yardstick_tc010.ya
• SLA (max_latency): 30 n
• Stride - 128 bytes
• Stop size - 64 megabytes
• Iterations: 10 - test is run
• Interval: 1 - there is 1 sec
iteration.
SLA is optional. The SLA in th
example. Considerably lower r
However,167
to cover most configu
and fully virtualized ones, this v
to achieve and acceptable for b
heavy IO applications start to
OPNFV Documentation, Release Danube
168
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
Packet delay variation between VMs
test case id
metric
test purpose
test tool
test description
configuration
Yardstick Test Case Description TC011
applicability
Test can be configured with diff
• bandwidth: Test case ca
bandwidth.
• duration: The test duratio
• jitter: SLA is optional. T
serves as an exampl
usability
This test case is one of Yardstic
is runnable on most of the scena
iperf3
ETSI-NFV-TST001
The test 169
case image needs to b
with iperf3 included in the imag
No POD specific requirements
description and expected result
references
2.2. Testing User Guides
OPNFV_YARDSTICK_TC011
VARIATION BETWEEN VMs
jitter: packet delay variation (m
The purpose of TC011 is to ev
performance with regards to ne
lay variation). It measures the
sending the packets from one V
The purpose is also to be able to
sults, graphs and similar shall b
reasons and product evolution
different OPNFV versions and/
iperf3
iPerf3 is a tool for active measur
achievable bandwidth on IP ne
ing of various parameters relate
protocols. The UDP protocols
jitter delay.
(iperf3 is not always part of a L
it needs to be installed. It is part
image. As an example see the
tory for how to generate a Linu
cluded.)
iperf3 test is invoked between
VM.
Jitter calculations are continu
server, as specified by RTP in
records a 64 bit second/micros
packet. The server computes th
(server’s receive time - client’s
and server’s clocks do not need
difference is subtracted outin th
is the smoothed mean of differ
tive transit times.
File: opnfv_yardstick_tc011.ya
• options: protocol: udp #
iperf3 tools bandwidth:
given number of packets
without pausing
• runner: duration: 30 # To
onds.
• SLA (optional): jitter: 1
amount of jitter that is
accepted.
pre-test conditions
test sequence
OPNFV Documentation, Release Danube
170
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
Memory Bandwidth
test case id
metric
test purpose
test tool
test description
configuration
Yardstick Test Case Description TC012
applicability
2.2. Testing User Guides
OPNFV_YARDSTICK_TC012
WIDTH
Memory read/write bandwidth (
The purpose of TC012 is to ev
performance with regards to m
measures the rate at which dat
written to the memory (this inc
ory).
The purpose is also to be able to
sults, graphs and similar shall b
reasons and product evolution
different OPNFV versions and/
LMbench
LMbench is a suite of operati
marks. This test uses bw_mem
cluding:
• Cached file read
• Memory copy (bcopy)
• Memory read
• Memory write
• Pipe
• TCP
(LMbench is not always part
hence it needs to be installed.
/yardstick/tools/ directory for h
image with LMbench included.
LMbench bw_mem benchmark
ified amount of memory, zero
copying of the first half to the s
mark is invoked in a host VM o
sults are reported in megabytes
File: opnfv_yardstick_tc012.ya
• SLA (optional): 15000
minimum amount of me
accepted.
• Size: 10 240 kB - test a
(20 480kB) zeros it and t
takes to copy from one si
• Benchmark: rdwr - measu
into memory and then w
cation.
• Warmup: 0 - the number
before taking actual meas
• Iterations: 10 - test is run
• Interval: 1 - there is 1 sec
iteration.
SLA is optional. The SLA in th
example. Considerably higher
However, to cover most configu
and fully virtualized ones, thi
sible to achieve and acceptabl
Many heavy IO applications st
read/write bandwidths are lowe
Test can be configured with diff
• memory sizes;
171 operations (such
• memory
fwr, fcp, bzero, bcopy);
• number of warmup iterati
• iterations and intervals.
OPNFV Documentation, Release Danube
172
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
Processing speed
test case id
metric
test purpose
test tool
test description
Yardstick Test Case Description TC014
configuration
applicability
usability
references
pre-test conditions
2.2. Testing User Guides
test sequence
step 1
step 2
step 3
OPNFV_YARDSTICK_TC014
SPEED
score of single cpu running, sco
The purpose of TC014 is to ev
performance with regards to CP
measures score of single cpu ru
ning.
The purpose is also to be able to
sults, graphs and similar shall b
reasons and product evolution
different OPNFV versions and/
UnixBench
Unixbench is the most used C
ware tool. It can measure th
scripts, CPUs in multithreadin
It can also measure the perfor
Also, specific disk IO for smal
formed. You can use it to measu
servers and linux vps servers, ru
Ubuntu, Fedora and other distro
(UnixBench is not always part
hence it needs to be installed.
/yardstick/tools/ directory for h
image with UnixBench included
The UnixBench runs system be
on a compute blade, getting info
the system. If the system has m
tests will be run twice – once w
test running at once, and once w
the number of CPUs.
UnixBench will processs a set
test by averaging the individal p
final value.
file: opnfv_yardstick_tc014.yam
run_mode: Run unixbench in
mode test_type: dhry2reg, whet
For SLA with single_score and
be set by user, default is NA.
Test can be configured with diff
• test types;
• dhry2reg;
• whetstone.
Default values exist.
SLA (optional) : min_score: T
score that is accepted.
This test case is one of Yardstic
is runnable on most of the scena
unixbench
ETSI-NFV-TST001
The test case image needs to b
with unixbench included in it.
No POD specific requirements
description and expected result
A host VM with UnixBench ins
Yardstick is connected with t
173
ssh. “unixbench_benchmark” b
Jump Host to the host VM via s
UnixBench is invoked. All the
the “Run” script in the top-level
OPNFV Documentation, Release Danube
CPU Load
test case id
metric
test purpose
configuration
test tool
Yardstick Test Case Description TC024
references
applicability
pre-test conditions
test sequence
step 1
test verdict
174
OPNFV_YARDSTICK_TC024
CPU load
To evaluate the CPU load perfo
test case should be run in parall
cases and not run as a stand-al
minimum and maximun values
pose is also to be able to spot tre
and similar shall be stored for
product evolution understandin
NFV versions and/or configurat
file: cpuload.yaml (in the ‘samp
• interval: 1 - repeat, paus
between.
• count: 10 - display statist
mpstat
(mpstat is not always part of a L
it needs to be installed. It is part
image. However, if mpstat is no
uses /proc/stats as source to pro
man-pages
Test can be configured with diff
• interval;
• count;
• runner Iteration and inter
There are default values for ea
tion. Run in background with o
The test case image needs to b
with mpstat included in it.
No POD specific requirements
description and expected result
The host is installed. The relate
and mpstat logs are produced an
Result: Stored logs
None. CPU load results are fetc
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
Latency, CPU Load, Throughput, Packet Loss
test case id
metric
test purpose
test tool
test description
Yardstick Test Case Description TC037
2.2. Testing User Guides
configuration
OPNFV_YARDSTICK_TC037
LOAD,THROUGHPUT, PACK
Number of flows, latency, throu
utilization percentage, CPU inte
The purpose of TC037 is to ev
capacity and network performan
utilization, packet flows and ne
as if and how different amount
throughput between hosts on d
and the CPU load variation.
Typically e.g. the performanc
on the number of flows running
mance of other equipment or en
number of flows or the packet s
The purpose is also to be able to
sults, graphs and similar shall b
reasons and product evolution
different OPNFV versions and/
Ping, Pktgen, mpstat
Ping is a computer network adm
ity used to test the reachability
Protocol (IP) network. It meas
for packet sent from the origina
computer that are echoed back t
Linux packet generator is a too
very high speed in the kernel. p
drive and LAN equipment test n
multi threading. To generate ra
address, port number UDP pac
ple CPU processors in the diffe
bus) with Gigabit Ethernet test
depends on the CPU processin
PCI bus speed hardware parame
can be even larger than 10GB
most card test requirements.
The mpstat command writes to s
for each available processor, pr
one. Global average activities a
also reported. The mpstat com
on SMP and UP machines, but
average activities will be printe
(Ping is normally part of any L
it doesn’t need to be installed. I
stick Docker image. For exam
can be downloaded from cirrosPktgen and mpstat are not alw
tribution, hence it needs to be i
Yardstick Docker image. As an
stick/tools/ directory for how to
with pktgen and mpstat include
This test case uses Pktgen to g
tween two hosts for simulating
the SUT. Ping packets (ICMP
ECHO_REQUEST datagram) a
to the target VM(s) to elicit ICM
175
meanwhile
CPU activities are m
file: opnfv_yardstick_tc037.yam
Packet size is set to 64 bytes.
50, 100, 300, 500, 750 and 100
OPNFV Documentation, Release Danube
Yardstick Test Case Description TC038
176
Latency, CPU Load, Throughput, Packet Loss (Extended measurements)
test case id OPNFV_YARDSTICK_TC038_Latency,CPU Load,Throughput,Packet Los
metric
Number of flows, latency, throughput, CPU load, packet loss
test
To evaluate the IaaS network performance with regards to flows and through
purpose
different amounts of flows matter for the throughput between hosts on differ
Typically e.g. the performance of a vSwitch depends on the number of flows
Also performance of other equipment or entities can depend on the number o
sizes used. The purpose is also to be able to spot trends. Test results, graphs
stored for comparison reasons and product evolution understanding between
versions and/or configurations.
configurafile: opnfv_yardstick_tc038.yaml
tion
Packet size: 64 bytes Number of ports: 1, 10, 50, 100, 300, 500, 750 and 100
configured ports map from 2 up to 1001000 flows, respectively. Each port am
for 20 seconds each. Then the next port_amount is run, and so on. During th
client and server, and the network latency between the client and server are m
and server are distributed on different HW. For SLA max_ppm is set to 1000
test tool
pktgen
(Pktgen is not always part of a Linux distribution, hence it needs to be instal
Yardstick Glance image. As an example see the /yardstick/tools/ directory fo
Linux image with pktgen included.)
ping
Ping is normally part of any Linux distribution, hence it doesn’t need to be in
the Yardstick Glance image. (For example also a cirros image can be downlo
mpstat
(Mpstat is not always part of a Linux distribution, hence it needs to be instal
Yardstick Glance image.
references
Ping and Mpstat man pages
pktgen
ETSI-NFV-TST001
applicabil- Test can be configured with different packet sizes, amount of flows and test d
ity
exist.
SLA (optional): max_ppm: The number of packets per million packets sent
loose, not received.
pre-test
The test case image needs to be installed into Glance with pktgen included in
conditions No POD specific requirements have been identified.
test
description and expected result
sequence
step 1
The hosts are installed, as server and client. pktgen is invoked and logs are p
Result: Logs are stored.
test verdict Fails only if SLA is not passed, or if there is a test case execution problem.
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
Network Performance
test case id
metric
test purpose
configuration
test tool
references
applicability
Yardstick Test Case Description TC0042
pre-test conditions
test sequence
step 1
step 2
step 3
test verdict
2.2. Testing User Guides
OPNFV_YARDSTICK_TC04
measurements
L2 Network Latency
Measure L2 network latency w
tween hosts on different comp
file: opnfv_yardstick_tc042.ya
• Packet size: 64 bytes
• SLA(max_latency): 100
DPDK Pktgen-dpdk
(DPDK and Pktgen-dpdk are
bution, hence they needs to be
see the /yardstick/tools/ direct
Linux image with DPDK and
DPDK
Pktgen-dpdk
ETSI-NFV-TST001
Test can be configured with d
fault values exist.
The test case image needs to
with DPDK and pktgen-dpdk
The NICs of compute nodes
POD.
And at least compute nodes se
If you want to achievement a
it is recommend to use NUAM
on.
description and expected resul
The hosts are installed on diffe
client. Both server and client h
first one is management such
used by DPDK.
Testpmd is invoked with config
ets from one DPDK port to the
Pktgen-dpdk is invoked with
generator and logs are produce
Result: Logs are stored.
Fails only if SLA is not passed
execution problem.
177
OPNFV Documentation, Release Danube
Network Latency Between NFVI Nodes
test case id
metric
test purpose
test tool
test topology
configuration
applicability
Yardstick Test Case Description TC043
references
pre_test conditions
test sequence
step 1
step 2
test verdict
178
OPNFV_YARDSTICK_TC043
RTT (Round Trip Time)
The purpose of TC043 is to do
network latency is within acce
packets travel between different
The purpose is also to be able to
sults, graphs and similar shall b
reasons and product evolution
different OPNFV versions and/
ping
Ping is a computer network adm
ity used to test the reachability
Protocol (IP) network. It meas
for packet sent from the origina
computer that are echoed back t
Ping
packets
(ICMP
p
ECHO_REQUEST datagram)
node to target node to elicit ICM
file: opnfv_yardstick_tc043.yam
Packet size 100 bytes. Total te
One ping each 10 seconds. SLA
10 ms.
This test case can be configured
• packet sizes;
• burst sizes;
• ping intervals;
• test durations;
• test iterations.
Default values exist.
SLA is optional. The SLA in th
example. Considerably lower R
normal to achieve in balanced
ever, to cover most configuratio
fully virtualized ones, this valu
achieve and acceptable for blac
time applications start to suffer
higher than this. Some may suf
RTT, while others may not suff
mise that may have to be tuned
tion purposes.
Ping
ETSI-NFV-TST001
Each pod node must have ping
description and expected result
Yardstick is connected with the
‘ping_benchmark’ bash script
Host to the NFVI node via the s
Ping is invoked. Ping packets a
to client node. RTT results are
against the SLA. Logs are prod
Result: Logs are stored.
Test should not PASS if any RT
SLA value, or if there is a test c
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
Memory Utilization
test case id
metric
test purpose
configuration
test tool
Yardstick Test Case Description TC044
references
applicability
pre-test conditions
test sequence
step 1
test verdict
2.2. Testing User Guides
OPNFV_YARDSTICK_TC044
Memory utilization
To evaluate the IaaS compute c
memory utilization.This test ca
allel to other Yardstick test case
alone test case. Measure the m
including used memory, free me
shared memory. Both average a
obtained. The purpose is also t
Test results, graphs and similar
parison reasons and product evo
tween different OPNFV version
File: memload.yaml (in the ‘sam
• interval: 1 - repeat, paus
between.
• count: 10 - display statist
free
free provides information about
ory and swap space on any com
another Unix-like operating sy
part of a Linux distribution, he
be installed.
man-pages
ETSI-NFV-TST001
Test can be configured with diff
• interval;
• count;
• runner Iteration and inter
There are default values for ea
tion. Run in background with o
The test case image needs to b
with free included in the image.
No POD specific requirements
description and expected result
The host is installed as client.
is invoked and free logs are pro
Result: logs are stored.
None. Memory utilization resul
179
OPNFV Documentation, Release Danube
Yardstick Test Case Description TC055
180
Compute Capacity
test case id OPNFV_YARDSTICK_TC055_Compute Capacity
metric
Number of cpus, number of cores, number of threads, available memory size
test
To evaluate the IaaS compute capacity with regards to hardware specification
purpose
cpus, number of cores, number of threads, available memory size and total c
graphs and similar shall be stored for comparison reasons and product evolu
between different OPNFV versions and/or configurations.
configurafile: opnfv_yardstick_tc055.yaml
tion
There is are no additional configurations to be set for this TC.
test tool
/proc/cpuinfo
this TC uses /proc/cpuinfo as source to produce compute capacity output.
references
/proc/cpuinfo_
ETSI-NFV-TST001
applicabil- None.
ity
pre-test
No POD specific requirements have been identified.
conditions
test
description and expected result
sequence
step 1
The hosts are installed, TC is invoked and logs are produced and stored.
Result: Logs are stored.
test verdict None. Hardware specification are fetched and stored.
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
Network Utilization
test case id
metric
test purpose
configuration
Yardstick Test Case Description TC061
test tool
references
applicability
pre-test conditions
test sequence
step 1
test verdict
2.2. Testing User Guides
OPNFV_YARDSTICK_TC061
Network utilization
To evaluate the IaaS network ca
network utilization, including T
received per second, Total num
ted per second, Total number o
second, Total number of kiloby
ond, Number of compressed pac
(for cslip etc.), Number of comp
ted per second, Number of mu
per second, Utilization percenta
face. This test case should be
Yardstick test cases and not ru
case. Measure the network usag
work devices Average, minimu
are obtained. The purpose is
trends. Test results, graphs and
for comparison reasons and p
standing between different OPN
figurations.
File: netutilization.yaml (in the
• interval: 1 - repeat, paus
between.
• count: 1 - display statistic
sar
The sar command writes to stan
of selected cumulative activity
ing system. sar is normally part
hence it doesn’t needs to be inst
man-pages
ETSI-NFV-TST001
Test can be configured with diff
• interval;
• count;
• runner Iteration and inter
There are default values for ea
tion. Run in background with o
The test case image needs to b
with sar included in the image.
No POD specific requirements
description and expected result.
The host is installed as client.
is invoked and sar logs are prod
Result: logs are stored.
None. Network utilization resul
181
OPNFV Documentation, Release Danube
Storage Capacity
test case id
metric
test purpose
configuration
test tool
Yardstick Test Case Description TC063
references
applicability
pre-test conditions
test sequence
step 1
test verdict
182
OPNFV_YARDSTICK_TC063
Storage/disk size, block size Di
This test case will check the par
cide several models and each mo
to measure. The test purposes a
block size and disk utilization.
could evaluate the storage capac
file: opnfv_yardstick_tc0
• test_type: “disk_size”
• runner: type: Iteration i
1 time iteratively.
fdisk A command-line utility t
tioning functions
iostat This is a computer syste
collect and show operating syste
put statistics.
iostat fdisk
ETSI-NFV-TST001
Test can be configured with diff
• test_type: “disk size”, “b
tion”
• interval: 1 - how ofter to
type: int unit: secon
• count: 15 - how many ti
type: int unit: na
There are default values for ea
tion. Run in background with o
The test case image needs to be
No POD specific requirements
Output the specific storage capa
as the sequence into file.
The pod is available and the ho
is used and logs are produced a
Result: Logs are stored.
None.
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
Memory Bandwidth
test case id
metric
test purpose
configuration
Yardstick Test Case Description TC069
test tool
references
applicability
pre-test conditions
test sequence
step 1
2.2. Testing User Guides
test verdict
OPNFV_YARDSTICK_TC069
Megabyte per second (MBps)
To evaluate the IaaS compute p
to memory bandwidth. Measu
ble cache and memory perform
writing certain blocks of data
further in power of 2) continuo
FPU respectively. Measure diff
performance via synthetic simu
consists of four performances (C
Test results, graphs and similar
parison reasons and product evo
tween different OPNFV version
File: opnfv_yardstick_tc069.ya
• SLA (optional): 7000 (
The minimum amount of
is accepted.
• type_id: 1 - runs a specifi
number):
1 – INTmark [wri
mark [writing] 2 – I
5 – FLOATmark [
mem 6 – FLOATme
• block_size: 64 Megabyt
size per array.
• load: 32 Gigabytes - the
pass.
• iterations: 5 - test is run 5
• interval: 1 - there is 1 sec
iteration.
RAMspeed
RAMspeed is a free open sour
to measure cache and memory p
systems. RAMspeed is not alw
tribution, hence it needs to be in
RAMspeed
ETSI-NFV-TST001
Test can be configured with diff
• benchmark operations (s
ing], INTmark [reading],
FLOATmark [reading], IN
• block size per array;
• load per pass;
• number of batch run itera
• iterations and intervals.
There are default values for ea
tion.
The test case image needs to b
with RAmspeed included in the
No POD specific requirements
description and expected result
The host is installed as client. R
logs are produced and stored.
Result: logs are stored.
Test fails183
if the measured mem
the SLA value or if there is a t
lem.
OPNFV Documentation, Release Danube
Yardstick Test Case Description TC070
184
Latency, Memory Utilization, Throughput, Packet Loss
test case id OPNFV_YARDSTICK_TC070_Latency, Memory Utilization, Throughput,P
metric
Number of flows, latency, throughput, Memory Utilization, packet loss
test
To evaluate the IaaS network performance with regards to flows and through
purpose
different amounts of flows matter for the throughput between hosts on differ
Typically e.g. the performance of a vSwitch depends on the number of flows
Also performance of other equipment or entities can depend on the number o
sizes used. The purpose is also to be able to spot trends. Test results, graphs
stored for comparison reasons and product evolution understanding between
versions and/or configurations.
configurafile: opnfv_yardstick_tc070.yaml
tion
Packet size: 64 bytes Number of ports: 1, 10, 50, 100, 300, 500, 750 and 100
configured ports map from 2 up to 1001000 flows, respectively. Each port am
for 20 seconds each. Then the next port_amount is run, and so on. During th
Utilization on both client and server, and the network latency between the cl
measured. The client and server are distributed on different HW. For SLA m
test tool
pktgen
Pktgen is not always part of a Linux distribution, hence it needs to be installe
Yardstick Glance image. (As an example see the /yardstick/tools/ directory f
Linux image with pktgen included.)
ping
Ping is normally part of any Linux distribution, hence it doesn’t need to be in
the Yardstick Glance image. (For example also a cirros image can be downlo
free
free provides information about unused and used memory and swap space on
Linux or another Unix-like operating system. free is normally part of a Linu
doesn’t needs to be installed.
references
Ping and free man pages
pktgen
ETSI-NFV-TST001
applicabil- Test can be configured with different packet sizes, amount of flows and test d
ity
exist.
SLA (optional): max_ppm: The number of packets per million packets sent
lose, not received.
pre-test
The test case image needs to be installed into Glance with pktgen included in
conditions No POD specific requirements have been identified.
test
description and expected result
sequence
step 1
The hosts are installed, as server and client. pktgen is invoked and logs are p
Result: Logs are stored.
test verdict Fails only if SLA is not passed, or if there is a test case execution problem.
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
Yardstick Test Case Description TC071
2.2. Testing User Guides
Latency, Cache Utilization, Throughput, Packet Loss
test case id OPNFV_YARDSTICK_TC071_Latency, Cache Utilization, Throughput,Pac
metric
Number of flows, latency, throughput, Cache Utilization, packet loss
test
To evaluate the IaaS network performance with regards to flows and through
purpose
different amounts of flows matter for the throughput between hosts on differ
Typically e.g. the performance of a vSwitch depends on the number of flows
Also performance of other equipment or entities can depend on the number o
sizes used. The purpose is also to be able to spot trends. Test results, graphs
stored for comparison reasons and product evolution understanding between
versions and/or configurations.
configurafile: opnfv_yardstick_tc071.yaml
tion
Packet size: 64 bytes Number of ports: 1, 10, 50, 100, 300, 500, 750 and 100
configured ports map from 2 up to 1001000 flows, respectively. Each port am
for 20 seconds each. Then the next port_amount is run, and so on. During th
on both client and server, and the network latency between the client and ser
client and server are distributed on different HW. For SLA max_ppm is set to
test tool
pktgen
Pktgen is not always part of a Linux distribution, hence it needs to be installe
Yardstick Glance image. (As an example see the /yardstick/tools/ directory f
Linux image with pktgen included.)
ping
Ping is normally part of any Linux distribution, hence it doesn’t need to be in
the Yardstick Glance image. (For example also a cirros image can be downlo
cachestat
cachestat is not always part of a Linux distribution, hence it needs to be insta
references
Ping man pages
pktgen
cachestat
ETSI-NFV-TST001
applicabil- Test can be configured with different packet sizes, amount of flows and test d
ity
exist.
SLA (optional): max_ppm: The number of packets per million packets sent
lose, not received.
pre-test
The test case image needs to be installed into Glance with pktgen included in
conditions No POD specific requirements have been identified.
test
description and expected result
sequence
step 1
The hosts are installed, as server and client. pktgen is invoked and logs are p
Result: Logs are stored.
test verdict Fails only if SLA is not passed, or if there is a test case execution problem.
185
OPNFV Documentation, Release Danube
Yardstick Test Case Description TC072
186
Latency, Network Utilization, Throughput, Packet Loss
test case id OPNFV_YARDSTICK_TC072_Latency, Network Utilization, Throughput,P
metric
Number of flows, latency, throughput, Network Utilization, packet loss
test
To evaluate the IaaS network performance with regards to flows and through
purpose
different amounts of flows matter for the throughput between hosts on differ
Typically e.g. the performance of a vSwitch depends on the number of flows
Also performance of other equipment or entities can depend on the number o
sizes used. The purpose is also to be able to spot trends. Test results, graphs
stored for comparison reasons and product evolution understanding between
versions and/or configurations.
configurafile: opnfv_yardstick_tc072.yaml
tion
Packet size: 64 bytes Number of ports: 1, 10, 50, 100, 300, 500, 750 and 100
configured ports map from 2 up to 1001000 flows, respectively. Each port am
for 20 seconds each. Then the next port_amount is run, and so on. During th
Utilization on both client and server, and the network latency between the cl
measured. The client and server are distributed on different HW. For SLA m
test tool
pktgen
Pktgen is not always part of a Linux distribution, hence it needs to be installe
Yardstick Glance image. (As an example see the /yardstick/tools/ directory f
Linux image with pktgen included.)
ping
Ping is normally part of any Linux distribution, hence it doesn’t need to be in
the Yardstick Glance image. (For example also a cirros image can be downlo
sar
The sar command writes to standard output the contents of selected cumulat
the operating system. sar is normally part of a Linux distribution, hence it do
installed.
references
Ping and sar man pages
pktgen
ETSI-NFV-TST001
applicabil- Test can be configured with different packet sizes, amount of flows and test d
ity
exist.
SLA (optional): max_ppm: The number of packets per million packets sent
lose, not received.
pre-test
The test case image needs to be installed into Glance with pktgen included in
conditions No POD specific requirements have been identified.
test
description and expected result
sequence
step 1
The hosts are installed, as server and client. pktgen is invoked and logs are p
Result: Logs are stored.
test verdict Fails only if SLA is not passed, or if there is a test case execution problem.
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
Yardstick Test Case Description TC073
Throughput per NFVI node test
test case id OPNFV_YARDSTICK_TC073_Network latency and throughput between n
metric
Network latency and throughput
test
To evaluate the IaaS network performance with regards to flows and through
purpose
different amounts of packet sizes and flows matter for the throughput betwee
configurafile: opnfv_yardstick_tc073.yaml
tion
Packet size: default 1024 bytes.
Test length: default 20 seconds.
The client and server are distributed on different nodes.
For SLA max_mean_latency is set to 100.
test tool
netperf Netperf is a software application that provides network bandwidth te
on a network. It supports Unix domain sockets, TCP, SCTP, DLPI and UDP
Netperf provides a number of predefined tests e.g. to measure bulk (unidirec
request response performance. (netperf is not always part of a Linux distribu
be installed.)
references
netperf Man pages ETSI-NFV-TST001
applicabil- Test can be configured with different packet sizes and test duration. Default
ity
SLA (optional): max_mean_latency
pre-test
The POD can be reached by external ip and logged on via ssh
conditions
test
description and expected result
sequence
step 1
Install netperf tool on each specified node, one is as the server, and the other
step 2
Log on to the client node and use the netperf command to execute the netwo
step 3
The throughput results stored.
test verdict Fails only if SLA is not passed, or if there is a test case execution problem.
Yardstick Test Case Description TC075
Network Capacity and Scale Testing
test case id
OPNFV_YARDSTICK_TC075_Network_Capacity_and_Scale_testin
metric
Number of connections, Number of frames sent/received
test purpose
To evaluate the network capacity and scale with regards to connection
configuration
file: opnfv_yardstick_tc075.yaml
There is no additional configuration to be set for this TC.
test tool
netstar
Netstat is normally part of any Linux distribution, hence it doesn’t ne
references
Netstat man page
ETSI-NFV-TST001
applicability
This test case is mainly for evaluating network performance.
pre_test
Each pod node must have netstat included in it.
conditions
test sequence
description and expected result
step 1
The pod is available. Netstat is invoked and logs are produced and sto
Result: Logs are stored.
test verdict
None. Number of connections and frames are fetched and stored.
2.2. Testing User Guides
187
OPNFV Documentation, Release Danube
Yardstick Test Case Description TC076
Monitor Network Metrics
test case id OPNFV_YARDSTICK_TC076_Monitor_Network_Metrics
metric
IP datagram error rate, ICMP message error rate, TCP segment error rate an
rate
test
The purpose of TC076 is to evaluate the IaaS network reliability with regard
purpose
rate, ICMP message error rate, TCP segment error rate and UDP datagram e
TC076 monitors network metrics provided by the Linux kernel in a host and
error rate, ICMP message error rate, TCP segment error rate and UDP datag
The purpose is also to be able to spot the trends. Test results, graphs and sim
comparison reasons and product evolution understanding between different
configurations.
test tool
nstat
nstat is a simple tool to monitor kernel snmp counters and network interface
(nstat is not always part of a Linux distribution, hence it needs to be installed
the iproute2 collection, which is usually also the name of the package in ma
distributions.As an example see the /yardstick/tools/ directory for how to gen
with iproute2 included.)
test
Ping packets (ICMP protocol’s mandatory ECHO_REQUEST datagram) are
description target VM(s) to elicit ICMP ECHO_RESPONSE.
nstat is invoked on the target vm to monitors network metrics provided by th
configurafile: opnfv_yardstick_tc076.yaml
tion
There is no additional configuration to be set for this TC.
references
nstat man page
ETSI-NFV-TST001
applicabil- This test case is mainly for monitoring network metrics.
ity
pre_test
The test case image needs to be installed into Glance with fio included in it.
conditions
No POD specific requirements have been identified.
test
description and expected result
sequence
step 1
Two host VMs are booted, as server and client.
step 2
Yardstick is connected with the server VM by using ssh. ‘ping_benchmark’
from Jump Host to the server VM via the ssh tunnel.
step 3
Ping is invoked. Ping packets are sent from server VM to client VM. RTT re
checked against the SLA. nstat is invoked on the client vm to monitors netw
the Linux kernel. IP datagram error rate, ICMP message error rate, TCP seg
UDP datagram error rate are calculated. Logs are produced and stored.
Result: Logs are stored.
step 4
Two host VMs are deleted.
test verdict None.
OPNFV Feature Test Cases
HA
188
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
Control Node Openstack Service High Availability
test case id
test purpose
test method
attackers
monitors
Yardstick Test Case Description TC019
metrics
test tool
references
configuration
2.2. Testing User Guides
test sequence
OPNFV_YARDSTICK_TC019
Openstack service down
This test case will verify the
service provided by OpenStack
server) on control node.
This test case kills the processes
service on a selected control no
the request of the related Ope
and the killed processes are reco
In this test case, an attacker
needed. This attacker include
fault_type: which is used fo
scripts. It should be always set
test case. 2) process_name: wh
of the specified OpenStack serv
processes use the same name on
killed by this attacker. 3) host:
control node being attacked.
e.g. -fault_type: “kill-process”
api” -host: node1
In this test case, two kinds of m
“openstack-cmd” monitor const
Openstack command, wh
rameters:
1) monitor_type: which is use
tor class and related scritps. It
“openstack-cmd” for this moni
which is the command name us
2. the “process” monitor ch
running on a specific nod
rameters:
1) monitor_type: which used
class and related scritps. It shou
cess” for this monitor. 2) proc
process name for monitor 3) ho
the node runing the process
e.g. monitor1: -monitor_typ
command_name: “openstack s
monitor_type: “process” -proc
host: node1
In this test case, there are
vice_outage_time: which indic
age time (seconds) of the specifi
request. 2)process_recover_tim
maximun time (seconds) from
to recovered
Developed by the project. Pl
stick/benchmark/scenarios/avai
ETSI NFV REL001
This test case needs two configu
file: opnfv_yardstick_tc019.yam
“attackers” discription -waiting
(seconds) from the process bein
itors the monitors -Monitors: se
cription -SLA: see above “metr
189 pod.yaml The PO
2)POD file:
record on pod.yaml first. the “ho
will use the node name in the po
description and expected result
OPNFV Documentation, Release Danube
190
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
Yardstick Test Case Description TC025
2.2. Testing User Guides
OpenStack Controller Node abnormally shutdown High Availability
test case id
OPNFV_YARDSTICK_TC025
troller Node abnormally shutdo
test purpose
This test case will verify the h
troller node. When one of the
mally shutdown, the service p
OK.
test method
This test case shutdowns a speci
some fault injection tools, then
vices provided by the controller
monitor tools.
attackers
In this test case, an attacker
is needed. This attacker inclu
fault_type: which is used fo
scripts. It should be always se
this test case. 2) host: the na
being attacked.
e.g. -fault_type: “host-shutdow
monitors
In this test case, one kind of mo
“openstack-cmd” monitor const
Openstack command, wh
rameters
1) monitor_type: which is use
tor class and related scritps. It
“openstack-cmd” for this moni
which is the command name us
There are four instance of the
itor: monitor1: -monitor_typ
api_name: “nova image-list” m
“openstack-cmd” -api_name: “n
itor3: -monitor_type: “open
“heat stack-list” monitor4: -mo
cmd” -api_name: “cinder list”
metrics
In this test case, there is
vice_outage_time: which indic
age time (seconds) of the specifi
request.
test tool
Developed by the project. Pl
stick/benchmark/scenarios/avai
references
ETSI NFV REL001
configuration
This test case needs two configu
file: opnfv_yardstick_tc019.yam
“attackers” discription -waiting
(seconds) from the process bein
itors the monitors -Monitors: se
cription -SLA: see above “metr
2)POD file: pod.yaml The PO
record on pod.yaml first. the “ho
will use the node name in the po
test sequence
description and expected result
step 1
start monitors: each monitor wil
process
Result: The monitor info will b
step 2
do attacker: connect the host th
ecute shutdown script on the ho
Result: The host will be shutdo
191
step 3
stop monitors after a period of
ing_time”
Result: All monitor result will b
step 4
verify the SLA
OPNFV Documentation, Release Danube
Yardstick Test Case Description TC045
192
Control Node Openstack Service High Availability - Neutron Server
test
OPNFV_YARDSTICK_TC045: Control node Openstack service down - neutro
case id
test
This test case will verify the high availability of the network service provided b
purpose (neutro-server) on control node.
test
This test case kills the processes of neutron-server service on a selected control
method
whether the request of the related Openstack command is OK and the killed pro
attackIn this test case, an attacker called “kill-process” is needed. This attacker includ
ers
fault_type: which is used for finding the attacker’s scripts. It should be always s
this test case. 2) process_name: which is the process name of the specified Ope
are multiple processes use the same name on the host, all of them are killed by
case. This parameter should always set to “neutron- server”. 3) host: which is t
node being attacked.
e.g. -fault_type: “kill-process” -process_name: “neutron-server” -host: node1
moniIn this test case, two kinds of monitor are needed: 1. the “openstack-cmd” mon
tors
specific Openstack command, which needs two parameters: 1) monitor_type: w
the monitor class and related scritps. It should be always set to “openstack-cmd
command_name: which is the command name used for request. In this case, th
should be neutron related commands.
2. the “process” monitor check whether a process is running on a specific node
parameters: 1) monitor_type: which used for finding the monitor class and rela
always set to “process” for this monitor. 2) process_name: which is the process
host: which is the name of the node runing the process
e.g. monitor1: -monitor_type: “openstack-cmd” -command_name: “neutron ag
-monitor_type: “process” -process_name: “neutron-server” -host: node1
metrics
In this test case, there are two metrics: 1)service_outage_time: which indicates
time (seconds) of the specified Openstack command request. 2)process_recove
the maximun time (seconds) from the process being killed to recovered
test tool Developed by the project. Please see folder: “yardstick/benchmark/scenarios/av
referETSI NFV REL001
ences
configThis test case needs two configuration files: 1) test case file: opnfv_yardstick_t
uration
see above “attackers” discription -waiting_time: which is the time (seconds) fro
killed to stoping monitors the monitors -Monitors: see above “monitors” discrip
“metrics” discription
2)POD file: pod.yaml The POD configuration should record on pod.yaml first.
test case will use the node name in the pod.yaml.
test sedescription and expected result
quence
step 1
start monitors: each monitor will run with independently process
Result: The monitor info will be collected.
step 2
do attacker: connect the host through SSH, and then execute the kill process scr
specified by “process_name”
Result: Process will be killed.
step 3
stop monitors after a period of time specified by “waiting_time”
Result: The monitor info will be aggregated.
step 4
verify the SLA
Result: The test case is passed or not.
postIt is the action when the test cases exist. It will check the status of the specified
action
and restart the process if it is not running for next test cases
test
Fails only if SLA is not passed, or if there is a test case execution problem.
verdict
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
Yardstick Test Case Description TC046
2.2. Testing User Guides
Control Node Openstack Service High Availability - Keystone
test
OPNFV_YARDSTICK_TC046: Control node Openstack service down - keysto
case id
test
This test case will verify the high availability of the user service provided by O
purpose control node.
test
This test case kills the processes of keystone service on a selected control node
method
the request of the related Openstack command is OK and the killed processes a
attackIn this test case, an attacker called “kill-process” is needed. This attacker includ
ers
fault_type: which is used for finding the attacker’s scripts. It should be always s
this test case. 2) process_name: which is the process name of the specified Ope
are multiple processes use the same name on the host, all of them are killed by
case. This parameter should always set to “keystone” 3) host: which is the nam
being attacked.
e.g. -fault_type: “kill-process” -process_name: “keystone” -host: node1
moniIn this test case, two kinds of monitor are needed: 1. the “openstack-cmd” mon
tors
specific Openstack command, which needs two parameters: 1) monitor_type: w
the monitor class and related scritps. It should be always set to “openstack-cmd
command_name: which is the command name used for request. In this case, th
should be keystone related commands.
2. the “process” monitor check whether a process is running on a specific node
parameters: 1) monitor_type: which used for finding the monitor class and rela
always set to “process” for this monitor. 2) process_name: which is the process
host: which is the name of the node runing the process
e.g. monitor1: -monitor_type: “openstack-cmd” -command_name: “keystone u
-monitor_type: “process” -process_name: “keystone” -host: node1
metrics
In this test case, there are two metrics: 1)service_outage_time: which indicates
time (seconds) of the specified Openstack command request. 2)process_recove
the maximun time (seconds) from the process being killed to recovered
test tool Developed by the project. Please see folder: “yardstick/benchmark/scenarios/av
referETSI NFV REL001
ences
configThis test case needs two configuration files: 1) test case file: opnfv_yardstick_t
uration
see above “attackers” discription -waiting_time: which is the time (seconds) fro
killed to stoping monitors the monitors -Monitors: see above “monitors” discrip
“metrics” discription
2)POD file: pod.yaml The POD configuration should record on pod.yaml first.
test case will use the node name in the pod.yaml.
test sedescription and expected result
quence
step 1
start monitors: each monitor will run with independently process
Result: The monitor info will be collected.
step 2
do attacker: connect the host through SSH, and then execute the kill process scr
specified by “process_name”
Result: Process will be killed.
step 3
stop monitors after a period of time specified by “waiting_time”
Result: The monitor info will be aggregated.
step 4
verify the SLA
Result: The test case is passed or not.
postIt is the action when the test cases exist. It will check the status of the specified
action
and restart the process if it is not running for next test cases
test
Fails only if SLA is not passed, or if there is a test case execution problem.
verdict
193
OPNFV Documentation, Release Danube
Yardstick Test Case Description TC047
194
Control Node Openstack Service High Availability - Glance Api
test
OPNFV_YARDSTICK_TC047: Control node Openstack service down - glance
case id
test
This test case will verify the high availability of the image service provided by
purpose on control node.
test
This test case kills the processes of glance-api service on a selected control nod
method
the request of the related Openstack command is OK and the killed processes a
attackIn this test case, an attacker called “kill-process” is needed. This attacker includ
ers
fault_type: which is used for finding the attacker’s scripts. It should be always s
this test case. 2) process_name: which is the process name of the specified Ope
are multiple processes use the same name on the host, all of them are killed by
case. This parameter should always set to “glance- api”. 3) host: which is the n
being attacked.
e.g. -fault_type: “kill-process” -process_name: “glance-api” -host: node1
moniIn this test case, two kinds of monitor are needed: 1. the “openstack-cmd” mon
tors
specific Openstack command, which needs two parameters: 1) monitor_type: w
the monitor class and related scritps. It should be always set to “openstack-cmd
command_name: which is the command name used for request. In this case, th
should be glance related commands.
2. the “process” monitor check whether a process is running on a specific node
parameters: 1) monitor_type: which used for finding the monitor class and rela
always set to “process” for this monitor. 2) process_name: which is the process
host: which is the name of the node runing the process
e.g. monitor1: -monitor_type: “openstack-cmd” -command_name: “glance ima
-monitor_type: “process” -process_name: “glance-api” -host: node1
metrics
In this test case, there are two metrics: 1)service_outage_time: which indicates
time (seconds) of the specified Openstack command request. 2)process_recove
the maximun time (seconds) from the process being killed to recovered
test tool Developed by the project. Please see folder: “yardstick/benchmark/scenarios/av
referETSI NFV REL001
ences
configThis test case needs two configuration files: 1) test case file: opnfv_yardstick_t
uration
see above “attackers” discription -waiting_time: which is the time (seconds) fro
killed to stoping monitors the monitors -Monitors: see above “monitors” discrip
“metrics” discription
2)POD file: pod.yaml The POD configuration should record on pod.yaml first.
test case will use the node name in the pod.yaml.
test sedescription and expected result
quence
step 1
start monitors: each monitor will run with independently process
Result: The monitor info will be collected.
step 2
do attacker: connect the host through SSH, and then execute the kill process scr
specified by “process_name”
Result: Process will be killed.
step 3
stop monitors after a period of time specified by “waiting_time”
Result: The monitor info will be aggregated.
step 4
verify the SLA
Result: The test case is passed or not.
postIt is the action when the test cases exist. It will check the status of the specified
action
and restart the process if it is not running for next test cases
test
Fails only if SLA is not passed, or if there is a test case execution problem.
verdict
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
Yardstick Test Case Description TC048
2.2. Testing User Guides
Control Node Openstack Service High Availability - Cinder Api
test
OPNFV_YARDSTICK_TC048: Control node Openstack service down - cinder
case id
test
This test case will verify the high availability of the volume service provided by
purpose (cinder-api) on control node.
test
This test case kills the processes of cinder-api service on a selected control nod
method
the request of the related Openstack command is OK and the killed processes a
attackIn this test case, an attacker called “kill-process” is needed. This attacker includ
ers
fault_type: which is used for finding the attacker’s scripts. It should be always s
this test case. 2) process_name: which is the process name of the specified Ope
are multiple processes use the same name on the host, all of them are killed by
case. This parameter should always set to “cinder- api”. 3) host: which is the na
being attacked.
e.g. -fault_type: “kill-process” -process_name: “cinder-api” -host: node1
moniIn this test case, two kinds of monitor are needed: 1. the “openstack-cmd” mon
tors
specific Openstack command, which needs two parameters: 1) monitor_type: w
the monitor class and related scritps. It should be always set to “openstack-cmd
command_name: which is the command name used for request. In this case, th
should be cinder related commands.
2. the “process” monitor check whether a process is running on a specific node
parameters: 1) monitor_type: which used for finding the monitor class and rela
always set to “process” for this monitor. 2) process_name: which is the process
host: which is the name of the node runing the process
e.g. monitor1: -monitor_type: “openstack-cmd” -command_name: “cinder list”
-monitor_type: “process” -process_name: “cinder-api” -host: node1
metrics
In this test case, there are two metrics: 1)service_outage_time: which indicates
time (seconds) of the specified Openstack command request. 2)process_recove
the maximun time (seconds) from the process being killed to recovered
test tool Developed by the project. Please see folder: “yardstick/benchmark/scenarios/av
referETSI NFV REL001
ences
configThis test case needs two configuration files: 1) test case file: opnfv_yardstick_t
uration
see above “attackers” discription -waiting_time: which is the time (seconds) fro
killed to stoping monitors the monitors -Monitors: see above “monitors” discrip
“metrics” discription
2)POD file: pod.yaml The POD configuration should record on pod.yaml first.
test case will use the node name in the pod.yaml.
test sedescription and expected result
quence
step 1
start monitors: each monitor will run with independently process
Result: The monitor info will be collected.
step 2
do attacker: connect the host through SSH, and then execute the kill process scr
specified by “process_name”
Result: Process will be killed.
step 3
stop monitors after a period of time specified by “waiting_time”
Result: The monitor info will be aggregated.
step 4
verify the SLA
Result: The test case is passed or not.
postIt is the action when the test cases exist. It will check the status of the specified
action
and restart the process if it is not running for next test cases
test
Fails only if SLA is not passed, or if there is a test case execution problem.
verdict
195
OPNFV Documentation, Release Danube
Yardstick Test Case Description TC049
196
Control Node Openstack Service High Availability - Swift Proxy
test
OPNFV_YARDSTICK_TC049: Control node Openstack service down - swift
case id
test
This test case will verify the high availability of the storage service provided by
purpose (swift-proxy) on control node.
test
This test case kills the processes of swift-proxy service on a selected control no
method
the request of the related Openstack command is OK and the killed processes a
attackIn this test case, an attacker called “kill-process” is needed. This attacker includ
ers
fault_type: which is used for finding the attacker’s scripts. It should be always s
this test case. 2) process_name: which is the process name of the specified Ope
are multiple processes use the same name on the host, all of them are killed by
case. This parameter should always set to “swift- proxy”. 3) host: which is the
being attacked.
e.g. -fault_type: “kill-process” -process_name: “swift-proxy” -host: node1
moniIn this test case, two kinds of monitor are needed: 1. the “openstack-cmd” mon
tors
specific Openstack command, which needs two parameters: 1) monitor_type: w
the monitor class and related scritps. It should be always set to “openstack-cmd
command_name: which is the command name used for request. In this case, th
should be swift related commands.
2. the “process” monitor check whether a process is running on a specific node
parameters: 1) monitor_type: which used for finding the monitor class and rela
always set to “process” for this monitor. 2) process_name: which is the process
host: which is the name of the node runing the process
e.g. monitor1: -monitor_type: “openstack-cmd” -command_name: “swift stat”
-monitor_type: “process” -process_name: “swift-proxy” -host: node1
metrics
In this test case, there are two metrics: 1)service_outage_time: which indicates
time (seconds) of the specified Openstack command request. 2)process_recove
the maximun time (seconds) from the process being killed to recovered
test tool Developed by the project. Please see folder: “yardstick/benchmark/scenarios/av
referETSI NFV REL001
ences
configThis test case needs two configuration files: 1) test case file: opnfv_yardstick_t
uration
see above “attackers” discription -waiting_time: which is the time (seconds) fro
killed to stoping monitors the monitors -Monitors: see above “monitors” discrip
“metrics” discription
2)POD file: pod.yaml The POD configuration should record on pod.yaml first.
test case will use the node name in the pod.yaml.
test sedescription and expected result
quence
step 1
start monitors: each monitor will run with independently process
Result: The monitor info will be collected.
step 2
do attacker: connect the host through SSH, and then execute the kill process scr
specified by “process_name”
Result: Process will be killed.
step 3
stop monitors after a period of time specified by “waiting_time”
Result: The monitor info will be aggregated.
step 4
verify the SLA
Result: The test case is passed or not.
postIt is the action when the test cases exist. It will check the status of the specified
action
and restart the process if it is not running for next test cases
test
Fails only if SLA is not passed, or if there is a test case execution problem.
verdict
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
Yardstick Test Case Description TC050
2.2. Testing User Guides
OpenStack Controller Node Network High Availability
test
OPNFV_YARDSTICK_TC050: OpenStack Controller Node Network High Av
case id
test
This test case will verify the high availability of control node. When one of the
purpose connect the network, which breaks down the Openstack services on this node. T
service should able to be accessed by other controller nodes, and the services o
should be isolated.
test
This test case turns off the network interfaces of a specified control node, then c
method
services provided by the control node are OK with some monitor tools.
attackIn this test case, an attacker called “close-interface” is needed. This attacker inc
ers
1) fault_type: which is used for finding the attacker’s scripts. It should be alway
“close-interface” in this test case. 2) host: which is the name of a control node
interface: the network interface to be turned off.
There are four instance of the “close-interface” monitor: attacker1(for public ne
“close-interface” -host: node1 -interface: “br-ex” attacker2(for management ne
“close-interface” -host: node1 -interface: “br-mgmt” attacker3(for storage neto
“close-interface” -host: node1 -interface: “br-storage” attacker4(for private neto
“close-interface” -host: node1 -interface: “br-mesh”
moniIn this test case, the monitor named “openstack-cmd” is needed. The monitor n
tors
parameters: 1) monitor_type: which is used for finding the monitor class and re
be always set to “openstack-cmd” for this monitor. 2) command_name: which
used for request
There are four instance of the “openstack-cmd” monitor: monitor1: -monitor_t
-command_name: “nova image-list” monitor2: -monitor_type: “openstack-cmd
“neutron router-list” monitor3: -monitor_type: “openstack-cmd” -command_na
monitor4: -monitor_type: “openstack-cmd” -command_name: “cinder list”
metrics
In this test case, there is one metric: 1)service_outage_time: which indicates the
(seconds) of the specified Openstack command request.
test tool Developed by the project. Please see folder: “yardstick/benchmark/scenarios/av
referETSI NFV REL001
ences
configThis test case needs two configuration files: 1) test case file: opnfv_yardstick_t
uration
see above “attackers” discription -waiting_time: which is the time (seconds) fro
killed to stoping monitors the monitors -Monitors: see above “monitors” discrip
“metrics” discription
2)POD file: pod.yaml The POD configuration should record on pod.yaml first.
test case will use the node name in the pod.yaml.
test sedescription and expected result
quence
step 1
start monitors: each monitor will run with independently process
Result: The monitor info will be collected.
step 2
do attacker: connect the host through SSH, and then execute the turnoff networ
param value specified by “interface”.
Result: Network interfaces will be turned down.
step 3
stop monitors after a period of time specified by “waiting_time”
Result: The monitor info will be aggregated.
step 4
verify the SLA
Result: The test case is passed or not.
postIt is the action when the test cases exist. It turns up the network interface of the
action
turned up.
test
Fails only if SLA is not passed, or if there is a test case execution problem.
verdict
197
OPNFV Documentation, Release Danube
Yardstick Test Case Description TC051
198
OpenStack Controller Node CPU Overload High Availability
test
OPNFV_YARDSTICK_TC051: OpenStack Controller Node CPU Overload H
case id
test
This test case will verify the high availability of control node. When the CPU u
purpose controller node is stressed to 100%, which breaks down the Openstack services
Openstack service should able to be accessed by other controller nodes, and the
controller node should be isolated.
test
This test case stresses the CPU uasge of a specified control node to 100%, then
method
services provided by the environment are OK with some monitor tools.
attackIn this test case, an attacker called “stress-cpu” is needed. This attacker include
ers
fault_type: which is used for finding the attacker’s scripts. It should be always s
this test case. 2) host: which is the name of a control node being attacked. e.g.
“stress-cpu” -host: node1
moniIn this test case, the monitor named “openstack-cmd” is needed. The monitor n
tors
parameters: 1) monitor_type: which is used for finding the monitor class and re
be always set to “openstack-cmd” for this monitor. 2) command_name: which
used for request
There are four instance of the “openstack-cmd” monitor: monitor1: -monitor_t
-command_name: “nova image-list” monitor2: -monitor_type: “openstack-cmd
“neutron router-list” monitor3: -monitor_type: “openstack-cmd” -command_na
monitor4: -monitor_type: “openstack-cmd” -command_name: “cinder list”
metrics
In this test case, there is one metric: 1)service_outage_time: which indicates the
(seconds) of the specified Openstack command request.
test tool Developed by the project. Please see folder: “yardstick/benchmark/scenarios/av
referETSI NFV REL001
ences
configThis test case needs two configuration files: 1) test case file: opnfv_yardstick_t
uration
see above “attackers” discription -waiting_time: which is the time (seconds) fro
killed to stoping monitors the monitors -Monitors: see above “monitors” discrip
“metrics” discription
2)POD file: pod.yaml The POD configuration should record on pod.yaml first.
test case will use the node name in the pod.yaml.
test sedescription and expected result
quence
step 1
start monitors: each monitor will run with independently process
Result: The monitor info will be collected.
step 2
do attacker: connect the host through SSH, and then execute the stress cpu scrip
Result: The CPU usage of the host will be stressed to 100%.
step 3
stop monitors after a period of time specified by “waiting_time”
Result: The monitor info will be aggregated.
step 4
verify the SLA
Result: The test case is passed or not.
postIt is the action when the test cases exist. It kills the process that stresses the CP
action
test
Fails only if SLA is not passed, or if there is a test case execution problem.
verdict
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
Yardstick Test Case Description TC052
2.2. Testing User Guides
OpenStack Controller Node Disk I/O Block High Availability
test case id
OPNFV_YARDSTICK_TC052
Node Disk I/O Block High Ava
test purpose
This test case will verify the hig
node. When the disk I/O of a s
which breaks down the Opensta
Read and write services should s
controller nodes, and the servi
node should be isolated.
test method
This test case blocks the disk I/
node, then checks whether the s
or wirte the disk of the control
monitor tools.
attackers
In this test case, an attacker
needed. This attacker includ
fault_type: which is used fo
scripts. It should be always se
test case. 2) host: which is the
being attacked. e.g. -fault_ty
node1
monitors
In this test case, two kinds of
the “openstack-cmd” monitor c
cific Openstack command, whic
1) monitor_type: which is use
tor class and related scripts. It
“openstack-cmd” for this moni
which is the command name us
e.g.
-monitor_type:
command_name: “nova flavor-l
2. the second monitor verifi
function by a “operation” and
“operation” have two paramet
which is used for finding the
lated scripts. 2) action_parame
operation. the “result checker”
1) checker_type: which is use
checker class and realted script
expected value for the output o
condition: whether the expected
checker script or is totally same
In this case, the “operation” add
checker” checks whether ths fla
rameters show as follows: ope
“nova-create-flavor” -action_pa
flavorconfig: “test-001 te
result checker: -checker_ty
expectedValue: “test-001” -con
metrics
In this test case, there is
vice_outage_time: which indic
age time (seconds) of the specifi
request.
test tool
Developed by the project. Pl
stick/benchmark/scenarios/avai
references
ETSI NFV REL001
configuration
This test case needs two configu
file: opnfv_yardstick_tc052.yam
199
“attackers”
discription -waiting
(seconds) from the process bein
itors the monitors -Monitors: se
cription -SLA: see above “metr
OPNFV Documentation, Release Danube
200
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
Yardstick Test Case Description TC053
2.2. Testing User Guides
OpenStack Controller Load Balance Service High Availability
test
OPNFV_YARDSTICK_TC053: OpenStack Controller Load Balance Service H
case id
test
This test case will verify the high availability of the load balance service(curren
purpose supports OpenStack on controller node. When the load balance service of a spe
killed, whether other load balancers on other controller nodes will work, and w
node will restart the load balancer are checked.
test
This test case kills the processes of load balance service on a selected control n
method
whether the request of the related Openstack command is OK and the killed pro
attackIn this test case, an attacker called “kill-process” is needed. This attacker includ
ers
fault_type: which is used for finding the attacker’s scripts. It should be always s
this test case. 2) process_name: which is the process name of the specified Ope
are multiple processes use the same name on the host, all of them are killed by
case. This parameter should always set to “swift- proxy”. 3) host: which is the
being attacked.
e.g. -fault_type: “kill-process” -process_name: “haproxy” -host: node1
moniIn this test case, two kinds of monitor are needed: 1. the “openstack-cmd” mon
tors
specific Openstack command, which needs two parameters: 1) monitor_type: w
the monitor class and related scritps. It should be always set to “openstack-cmd
command_name: which is the command name used for request.
2. the “process” monitor check whether a process is running on a specific node
parameters: 1) monitor_type: which used for finding the monitor class and rela
always set to “process” for this monitor. 2) process_name: which is the process
host: which is the name of the node runing the process In this case, the comman
should be services that is supported by load balancer and the process- name of
“haproxy”, for example:
e.g. monitor1: -monitor_type: “openstack-cmd” -command_name: “nova imag
-monitor_type: “process” -process_name: “haproxy” -host: node1
metrics
In this test case, there are two metrics: 1)service_outage_time: which indicates
time (seconds) of the specified Openstack command request. 2)process_recove
the maximun time (seconds) from the process being killed to recovered
test tool Developed by the project. Please see folder: “yardstick/benchmark/scenarios/av
referETSI NFV REL001
ences
configThis test case needs two configuration files: 1) test case file: opnfv_yardstick_t
uration
see above “attackers” discription -waiting_time: which is the time (seconds) fro
killed to stoping monitors the monitors -Monitors: see above “monitors” discrip
“metrics” discription
2)POD file: pod.yaml The POD configuration should record on pod.yaml first.
test case will use the node name in the pod.yaml.
test sedescription and expected result
quence
step 1
start monitors: each monitor will run with independently process
Result: The monitor info will be collected.
step 2
do attacker: connect the host through SSH, and then execute the kill process scr
specified by “process_name”
Result: Process will be killed.
step 3
stop monitors after a period of time specified by “waiting_time”
Result: The monitor info will be aggregated.
step 4
verify the SLA
Result: The test case is passed or not.
postIt is the action when the test cases exist. It will check the status of the specified
action
and restart the process if it is not running for next test cases.
test
Fails only if SLA is not passed, or if there is a test case execution problem.
verdict
201
OPNFV Documentation, Release Danube
Yardstick Test Case Description TC054
OpenStack Virtual IP High Availability
test
OPNFV_YARDSTICK_TC054: OpenStack Virtual IP High Availability
case id
test
This test case will verify the high availability for virtual ip in the environment.
purpose virtual ip is abnormally shutdown, connection to virtual ip and the services bind
should be OK.
test
This test case shutdowns the virtual IP master node with some fault injection to
method
whether virtual ips can be pinged and services binded to virtual ip are OK with
attackIn this test case, an attacker called “control-shutdown” is needed. This attacker
ers
parameters: 1) fault_type: which is used for finding the attacker’s scripts. It sho
“control-shutdown” in this test case. 2) host: which is the name of a control nod
In this case the host should be the virtual ip master node, that means the host ip
exapmle: -fault_type: “control-shutdown” -host: node1(the VIP Master node)
moniIn this test case, two kinds of monitor are needed: 1. the “ip_status” monitor th
tors
check the connectivity of this ip, which needs two parameters: 1) monitor_type
finding the monitor class and related scripts. It should be always set to “ip_statu
ip_address: The ip to be pinged. In this case, ip_address should be the virtual IP
2. the “openstack-cmd” monitor constantly request a specific Openstack comm
parameters: 1) monitor_type: which is used for finding the monitor class and re
be always set to “openstack-cmd” for this monitor. 2) command_name: which
used for request.
e.g. monitor1: -monitor_type: “ip_status” -host: 192.168.0.2 monitor2: -monito
“openstack-cmd” -command_name: “nova image-list”
metrics
In this test case, there are two metrics: 1) ping_outage_time: which-indicates th
time to ping the specified host. 2)service_outage_time: which indicates the max
(seconds) of the specified Openstack command request.
test tool Developed by the project. Please see folder: “yardstick/benchmark/scenarios/av
referETSI NFV REL001
ences
configThis test case needs two configuration files: 1) test case file: opnfv_yardstick_t
uration
see above “attackers” discription -waiting_time: which is the time (seconds) fro
killed to stoping monitors the monitors -Monitors: see above “monitors” discrip
“metrics” discription
2)POD file: pod.yaml The POD configuration should record on pod.yaml first.
test case will use the node name in the pod.yaml.
test sedescription and expected result
quence
step 1
start monitors: each monitor will run with independently process
Result: The monitor info will be collected.
step 2
do attacker: connect the host through SSH, and then execute the shutdown scrip
node.
Result: VIP master node will be shutdown
step 3
stop monitors after a period of time specified by “waiting_time”
Result: The monitor info will be aggregated.
step 4
verify the SLA
Result: The test case is passed or not.
postIt is the action when the test cases exist. It restarts the original VIP master node
action
test
Fails only if SLA is not passed, or if there is a test case execution problem.
verdict
IPv6
202
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
Yardstick Test Case Description TC027
IPv6 connectivity between nodes on the tenant network
test case id OPNFV_YARDSTICK_TC027_IPv6 connectivity
metric
RTT, Round Trip Time
test
To do a basic verification that IPv6 connectivity is within acceptable bounda
purpose
travel between hosts located on same or different compute blades. The purpo
spot trends. Test results, graphs and similar shall be stored for comparison re
evolution understanding between different OPNFV versions and/or configura
configurafile: opnfv_yardstick_tc027.yaml
tion
Packet size 56 bytes. SLA RTT is set to maximum 30 ms. ipv6 test case can
independent modules (setup, run, teardown). if you only want to setup ipv6 t
some tests as you want, “run_step” of task yaml file should be configured as
setup and run ping6 testing automatically, “run_step” should be configured a
you have had a environment which has been setup, you only wan to verify th
network, “run_step” should be “run”. Of course, default is that three module
test tool
ping6
Ping6 is normally part of Linux distribution, hence it doesn’t need to be insta
references
ipv6
ETSI-NFV-TST001
applicabil- Test case can be configured with different run step you can run setup, run be
ity
independently SLA is optional. The SLA in this test case serves as an examp
RTT is expected.
pre-test
The test case image needs to be installed into Glance with ping6 included in
conditions For Brahmaputra, a compass_os_nosdn_ha deploy scenario is need. more in
deploy scenario will be supported soon
test
description and expected result
sequence
step 1
To setup IPV6 testing environment: 1. disable security group 2. create (ipv6
and subnet 3. create VRouter, VM1, VM2
step 2
To run ping6 to verify IPV6 connectivity : 1. ssh to VM1 2. Ping6 to ipv6 ro
the result(RTT) and logs are stored
step 3
To teardown IPV6 testing environment 1. delete VRouter, VM1, VM2 2. del
network and subnet 3. enable security group
test verdict Test should not PASS if any RTT is above the optional SLA value, or if there
problem.
KVM
2.2. Testing User Guides
203
OPNFV Documentation, Release Danube
Yardstick Test Case Description TC028
KVM Latency measurements
test case id OPNFV_YARDSTICK_TC028_KVM Latency measurements
metric
min, avg and max latency
test
To evaluate the IaaS KVM virtualization capability with regards to min, avg
purpose
purpose is also to be able to spot trends. Test results, graphs and similar shal
comparison reasons and product evolution understanding between different O
configurations.
configurafile: samples/cyclictest-node-context.yaml
tion
test tool
Cyclictest
(Cyclictest is not always part of a Linux distribution, hence it needs to be ins
see the /yardstick/tools/ directory for how to generate a Linux image with cy
references
Cyclictest
applicabil- This test case is mainly for kvm4nfv project CI verify. Upgrade host linux k
ity
update it’s linux kernel, and then run the cyclictest to test the new kernel is w
pre-test
The test kernel rpm, test sequence scripts and test guest image need put the r
conditions in the test case yaml file. The test guest image needs with cyclictest included
No POD specific requirements have been identified.
test
description and expected result
sequence
step 1
The host and guest os kernel is upgraded. Cyclictest is invoked and logs are
Result: Logs are stored.
test verdict Fails only if SLA is not passed, or if there is a test case execution problem.
Parser
204
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
Verify Parser Yang-to-Tosca
test case id
metric
test purpose
configuration
test tool
Yardstick Test Case Description TC040
references
applicability
pre-test conditions
test sequence
step 1
test verdict
2.2. Testing User Guides
OPNFV_YARDSTICK_TC040
Tosca
1. tosca file which is conv
Parser
2. result whether the outpu
outcome
To verify the function of Yang-t
file: opnfv_yardstick_tc040.yam
yangfile: the path of the yangfile
vert toscafile: the path of the to
pected outcome.
Parser
(Parser is not part of a Linu
needs to be installed. As an
stick/benchmark/scenarios/pars
how to install it manual. Of co
and uninstalled automatically
case by yardstick)
Parser
Test can be configured with di
and toscafile to fit your real envi
No POD specific requirements
can be run without VM
description and expected result
parser is installed without VM,
module to convert yang file to t
put against expected outcome.
Result: Logs are stored.
Fails only if output is different
or if there is a test case executio
205
OPNFV Documentation, Release Danube
Storperf
test case id
metric
test purpose
configuration
test tool
references
applicability
Yardstick Test Case Description TC074
206
OPNFV_YARDSTICK_TC074
Storage performance
Storperf integration with yardsti
Perf is to provide a tool to measu
age performance in an NFVI. W
a characterization of typical VF
quirements, it can provide pass
staging, and production NFVI e
The benchmarks developed for
will be sufficiently varied to pr
expected storage performance b
VNF workload.
file: opnfv_yardstick_tc074.yam
• agent_count: 1 - the numb
• agent_image: “Ubuntu-1
creating VMs
• public_network: “ext-net
work
• volume_size: 2 - cinder v
• block_sizes: “4096” - dat
• queue_depths: “4”
• StorPerf_ip: “192.168.20
• query_interval: 10 - state
• timeout: 600 - maximum
Storperf
StorPerf is a tool to measure b
performance in an NFVI.
StorPerf is delivered as a D
https://hub.docker.com/r/opnfv/
Storperf
ETSI-NFV-TST001
Test can be configured with diff
• agent_count
• volume_size
• block_sizes
• queue_depths
• query_interval
• timeout
• target=[device or path] T
tached storage device (/d
tory path (/opt/storperf) t
cute the performance test
the entire device will be u
current directory will be u
• workload=[workload mod
default is to run all wor
types are:
– rs: 100% Read, sequ
– ws: 100% Write, se
– rr: 100% Read, rand
– wr: 100% Write, ran
– rw: 70% Read / 30%
• nossd: Do not perform SS
• nowarm: Do not perform
Chapter 2. Test Frameworks
surements.
• report= [job_id] Query th
job_id and report on metr
plied, will report on only
OPNFV Documentation, Release Danube
virtual Traffic Classifier
2.2. Testing User Guides
207
OPNFV Documentation, Release Danube
Network Performance
test case id
metric
test purpose
configuration
test tool
Yardstick Test Case Description TC006
references
applicability
pre-test
208
OPNFV_YARDSTICK_TC006
fier Data Plane Throughput Ben
Throughput
To measure the throughput supp
fic Classifier according to the
for a user-defined set of vTC dep
file: file: opnfv_yardstick_tc006
packet_size: size of the packet
throughput calculation. A
256, 512, 1024, 1280, 15
vnic_type: type of VNIC to be
Allowed values are:
• normal: for def
tion
• direct: for SR-I
Default value: None
vtc_flavor: OpenStack flavor
Default available valu
m1.medium, and m1.la
create his/her own flavo
Default value: None
vlan_sender: vlan tag of the n
receive traffic (VLAN Ne
ues: range (1, 4096)
vlan_receiver: vlan tag of the
will send traffic back t
(VLAN Network 2). Al
4096)
default_net_name: neutron n
is used for access to the
(vNIC 1).
default_subnet_name: subnet
(information available thr
vlan_net_1_name: Neutron N
(information available thr
vlan_subnet_1_name: Subnet
(information available thr
vlan_net_2_name: Neutron N
(information available thr
vlan_subnet_2_name: Subnet
(information available thr
DPDK pktgen
DPDK Pktgen is not part of a L
it needs to be installed by the us
DPDK Pktgen: DPDKpktgen
ETSI-NFV-TST001
RFC 2544: rfc2544
Test can be configured with diff
and packet sizes. Default values
The vNIC type and flavor MU
user.
The vTC has been successfully
ured. The user has correctly as
deployment
Chapter 2. Test Frameworks
configuration parameters.
• Multicast traffic MUST
The Data network s
figured in order
OPNFV Documentation, Release Danube
2.2. Testing User Guides
209
OPNFV Documentation, Release Danube
Network Performance
test case id
metric
test purpose
configuration
OPNFV_YARDSTICK_TC00
Throughput Benchmarki
Noisy neighbours
Throughput
To measure the throughput supp
fic Classifier according to the
for a user-defined set of vTC de
in the presence of noisy neighbo
file: opnfv_yardstick_tc007.yam
packet_size: size of the packet
throughput calculation. A
256, 512, 1024, 1280, 15
vnic_type: type of VNIC to be
Allowed values are:
• normal: for def
tion
• direct: for SR-I
vtc_flavor: OpenStack flavor
Default available valu
m1.medium, and m1.la
create his/her own flavor
num_of_neighbours: Number
instantiated during the ex
ues: range (1, 10)
amount_of_ram: RAM to be
Allowed values: [‘250M
‘6G’, ‘7G’, ‘8G’, ‘9
Deault value: 256M
number_of_cores: Number of
instantiated during the ex
ues: range (1, 10) Defaul
vlan_sender: vlan tag of the n
receive traffic (VLAN Ne
ues: range (1, 4096)
vlan_receiver: vlan tag of the
will send traffic back t
(VLAN Network 2). Al
4096)
default_net_name: neutron n
is used for access to the
(vNIC 1).
default_subnet_name: subnet
(information available thr
vlan_net_1_name: Neutron N
(information available thr
vlan_subnet_1_name: Subnet
(information available thr
vlan_net_2_name: Neutron N
(information available thr
vlan_subnet_2_name: Subnet
(information available thr
Yardstick Test Case Description TC007
210
test tool
references
DPDK pktgen
Chapter 2. Test Frameworks
DPDK Pktgen is not part of a L
it needs to be installed by the us
DPDKpktgen
OPNFV Documentation, Release Danube
2.2. Testing User Guides
211
OPNFV Documentation, Release Danube
Network Performance
test case id
metric
test purpose
configuration
test tool
Yardstick Test Case Description TC020
references
applicability
pre-test
212
OPNFV_YARDSTICK_TC002
fier Instantiation Test
Failure
To verify that a newly instanti
functional and its instantiation i
the infrastructure.
file: opnfv_yardstick_tc020.yam
vnic_type: type of VNIC to be
Allowed values are:
• normal: for def
tion
• direct: for SR-I
Default value: None
vtc_flavor: OpenStack flavor
Default available valu
m1.medium, and m1.la
create his/her own flavo
Default value: None
vlan_sender: vlan tag of the n
receive traffic (VLAN Ne
ues: range (1, 4096)
vlan_receiver: vlan tag of the
will send traffic back t
(VLAN Network 2). Al
4096)
default_net_name: neutron n
is used for access to the
(vNIC 1).
default_subnet_name: subnet
(information available thr
vlan_net_1_name: Neutron N
(information available thr
vlan_subnet_1_name: Subnet
(information available thr
vlan_net_2_name: Neutron N
(information available thr
vlan_subnet_2_name: Subnet
(information available thr
DPDK pktgen
DPDK Pktgen is not part of a L
it needs to be installed by the us
DPDKpktgen
ETSI-NFV-TST001
rfc2544
Test can be configured with diff
and packet sizes. Default values
The vNIC type and flavor MU
user.
The vTC has been successfully
ured. The user has correctly as
deployment
configuration parameters.
• Multicast traffic MUST
The Data network s
Chapter 2. Test Frameworks
figured in order to m
Installation and co
is required before
(For further instruc
OPNFV Documentation, Release Danube
2.2. Testing User Guides
213
OPNFV Documentation, Release Danube
Network Performance
test case id
metric
test purpose
configuration
OPNFV_YARDSTICK_TC002
fier Instantiation Test in Presenc
Failure
To verify that a newly instanti
functional and its instantiation i
the infrastructure in the presenc
file: opnfv_yardstick_tc021.yam
vnic_type: type of VNIC to be
Allowed values are:
• normal: for def
tion
• direct: for SR-I
Default value: None
vtc_flavor: OpenStack flavor
Default available valu
m1.medium, and m1.la
create his/her own flavo
Default value: None
num_of_neighbours: Number
instantiated during the ex
ues: range (1, 10)
amount_of_ram: RAM to be
Allowed values: [‘250M
‘6G’, ‘7G’, ‘8G’, ‘9
Deault value: 256M
number_of_cores: Number of
instantiated during the ex
ues: range (1, 10) Defaul
vlan_sender: vlan tag of the n
receive traffic (VLAN Ne
ues: range (1, 4096)
vlan_receiver: vlan tag of the
will send traffic back t
(VLAN Network 2). Al
4096)
default_net_name: neutron n
is used for access to the
(vNIC 1).
default_subnet_name: subnet
(information available thr
vlan_net_1_name: Neutron N
(information available thr
vlan_subnet_1_name: Subnet
(information available thr
vlan_net_2_name: Neutron N
(information available thr
vlan_subnet_2_name: Subnet
(information available thr
Yardstick Test Case Description TC021
test tool
references
214
applicability
DPDK pktgen
DPDK Pktgen is not part of a L
it needs to be installed by the us
DPDK Pktgen: DPDK Pktgen:
ETSI-NFV-TST001
Chapter 2. Test Frameworks
RFC 2544: rfc2544
Test can be configured with diff
and packet sizes. Default values
The vNIC type and flavor MU
OPNFV Documentation, Release Danube
Templates
Yardstick Test Case Description TCXXX
test case slogan e.g. Network Latency
test case id e.g. OPNFV_YARDSTICK_TC001_NW Latency
metric
what will be measured, e.g. latency
test
describe what is the purpose of the test case
purpose
configurawhat .yaml file to use, state SLA if applicable, state test duration, list and d
tion
options used in this TC and also list the options using default values.
test tool
e.g. ping
references
e.g. RFCxxx, ETSI-NFVyyy
applicabildescribe variations of the test case which can be performend, e.g. run the t
ity
sizes
pre-test
describe configuration in the tool(s) used to perform the measurements (e.g
conditions
POD-specific configuration required to enable running the test
test
description and expected result
sequence
step 1
use this to describe tests that require sveveral steps e.g collect logs.
Result: what happens in this step e.g. logs collected
step 2
remove interface
Result: interface down.
step N
what is done in step N
Result: what happens
test verdict expected behavior, or SLA, pass/fail criteria
Task Template Syntax
Basic template syntax A nice feature of the input task format used in Yardstick is that it supports the template
syntax based on Jinja2. This turns out to be extremely useful when, say, you have a fixed structure of your task but
you want to parameterize this task in some way. For example, imagine your input task file (task.yaml) runs a set of
Ping scenarios:
# Sample benchmark task config file
# measure network latency using ping
schema: "yardstick:task:0.1"
scenarios:
type: Ping
options:
packetsize: 200
host: athena.demo
target: ares.demo
runner:
type: Duration
duration: 60
interval: 1
sla:
max_rtt: 10
action: monitor
2.2. Testing User Guides
215
OPNFV Documentation, Release Danube
context:
...
Let’s say you want to run the same set of scenarios with the same runner/ context/sla, but you want to try another
packetsize to compare the performance. The most elegant solution is then to turn the packetsize name into a template
variable:
# Sample benchmark task config file
# measure network latency using ping
schema: "yardstick:task:0.1"
scenarios:
type: Ping
options:
packetsize: {{packetsize}}
host: athena.demo
target: ares.demo
runner:
type: Duration
duration: 60
interval: 1
sla:
max_rtt: 10
action: monitor
context:
...
and then pass the argument value for {{packetsize}} when starting a task with this configuration file. Yardstick
provides you with different ways to do that:
1.Pass the argument values directly in the command-line interface (with either a JSON or YAML dictionary):
yardstick task start samples/ping-template.yaml
--task-args'{"packetsize":"200"}'
2.Refer to a file that specifies the argument values (JSON/YAML):
yardstick task start samples/ping-template.yaml --task-args-file args.yaml
Using the default values Note that the Jinja2 template syntax allows you to set the default values for your parameters. With default values set, your task file will work even if you don’t parameterize it explicitly while starting a task.
The default values should be set using the {% set ... %} clause (task.yaml). For example:
# Sample benchmark task config file
# measure network latency using ping
schema: "yardstick:task:0.1"
{% set packetsize = packetsize or "100" %}
scenarios:
type: Ping
options:
packetsize: {{packetsize}}
host: athena.demo
target: ares.demo
216
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
runner:
type: Duration
duration: 60
interval: 1
...
If you don’t pass the value for {{packetsize}} while starting a task, the default one will be used.
Advanced templates Yardstick makes it possible to use all the power of Jinja2 template syntax, including the mechanism of built-in functions. As an example, let us make up a task file that will do a block storage performance test.
The input task file (fio-template.yaml) below uses the Jinja2 for-endfor construct to accomplish that:
#Test block sizes of 4KB, 8KB, 64KB, 1MB
#Test 5 workloads: read, write, randwrite, randread, rw
schema: "yardstick:task:0.1"
scenarios:
{% for bs in ['4k', '8k', '64k', '1024k' ] %}
{% for rw in ['read', 'write', 'randwrite', 'randread', 'rw' ] %}
type: Fio
options:
filename: /home/ubuntu/data.raw
bs: {{bs}}
rw: {{rw}}
ramp_time: 10
host: fio.demo
runner:
type: Duration
duration: 60
interval: 60
{% endfor %}
{% endfor %}
context
...
Glossary
API Application Programming Interface
DPDK Data Plane Development Kit
DPI Deep Packet Inspection
DSCP Differentiated Services Code Point
IGMP Internet Group Management Protocol
IOPS Input/Output Operations Per Second
NFVI Network Function Virtualization Infrastructure
NIC Network Interface Controller
PBFS Packet Based per Flow State
QoS Quality of Service
2.2. Testing User Guides
217
OPNFV Documentation, Release Danube
SR-IOV Single Root IO Virtualization
SUT System Under Test
ToS Type of Service
VLAN Virtual LAN
VM Virtual Machine
VNF Virtual Network Function
VNFC Virtual Network Function Component
VTC Virtual Traffic Classifier
References
OPNFV
• Parser wiki: https://wiki.opnfv.org/parser
• Pharos wiki: https://wiki.opnfv.org/pharos
• VTC: https://wiki.opnfv.org/vtc
• Yardstick CI: https://build.opnfv.org/ci/view/yardstick/
• Yardstick and ETSI TST001 presentation: https://wiki.opnfv.org/display/yardstick/Yardstick?preview=%2F2925202%2F2925205%
_bridging_opnfv_and_etsi.pdf
• Yardstick Project presentation: https://wiki.opnfv.org/display/yardstick/Yardstick?preview=%2F2925202%2F2925208%2Fopnfv_
_yardstick_project.pdf
• Yardstick wiki: https://wiki.opnfv.org/yardstick
References used in Test Cases
• cachestat: https://github.com/brendangregg/perf-tools/tree/master/fs
• cirros-image: https://download.cirros-cloud.net
• cyclictest: https://rt.wiki.kernel.org/index.php/Cyclictest
• DPDKpktgen: https://github.com/Pktgen/Pktgen-DPDK/
• DPDK supported NICs: http://dpdk.org/doc/nics
• fdisk: http://www.tldp.org/HOWTO/Partition/fdisk_partitioning.html
• fio: http://www.bluestop.org/fio/HOWTO.txt
• free: http://manpages.ubuntu.com/manpages/trusty/en/man1/free.1.html
• iperf3: https://iperf.fr/
• iostat: http://linux.die.net/man/1/iostat
• Lmbench man-pages: http://manpages.ubuntu.com/manpages/trusty/lat_mem_rd.8.html
• Memory bandwidth man-pages: http://manpages.ubuntu.com/manpages/trusty/bw_mem.8.html
• mpstat man-pages: http://manpages.ubuntu.com/manpages/trusty/man1/mpstat.1.html
• netperf: http://www.netperf.org/netperf/training/Netperf.html
• pktgen: https://www.kernel.org/doc/Documentation/networking/pktgen.txt
218
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
• RAMspeed: http://alasir.com/software/ramspeed/
• sar: http://linux.die.net/man/1/sar
• SR-IOV: https://wiki.openstack.org/wiki/SR-IOV-Passthrough-For-Networking
• Storperf: https://wiki.opnfv.org/display/storperf/Storperf
• unixbench: https://github.com/kdlucas/byte-unixbench/blob/master/UnixBench
Research
• NCSRD: http://www.demokritos.gr/?lang=en
• T-NOVA: http://www.t-nova.eu/
• T-NOVA Results: http://www.t-nova.eu/results/
Standards
• ETSI NFV: http://www.etsi.org/technologies-clusters/technologies/nfv
• ETSI GS-NFV TST 001: http://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/001/01.01.01_60/gs_NFVTST001v010101p.pdf
• RFC2544: https://www.ietf.org/rfc/rfc2544.txt
Testing Developer Guides
Bottlenecks
Bottlenecks - Testing Guide
Project Testing Guide
For each test suite, you can either setup test story or test case to run certain test. test story could include several test
cases as a set in one configuration file. You could then call the test story or test case by using Bottlencks CLI or Python
build process. Details will be shown in the following section.
Brief Introdcution of the Test suites in Project Releases Brahmaputra: rubbos is introduced, which is an end2end
NFVI perforamnce tool. Virtual switch test framework(VSTF) is also introduced, which is an test framework used for
vswitch performance test.
Colorado: rubbos is refactored by using puppet, which makes it quite flexible to configure with different number of
load generator (Client), worker (tomcat). vstf is refactored by extracting the test case’s configuration information.
Danube: posca testsuite is introduced to implementing stress (factor), scenario and tuning test in parametric manner.
Two testcases are developed and integrated into community CI pipeline. Rubbos and VSTF are not supported any
more.
Integration Description
Release
Brahmaputra
Colorado
Danube
2.3. Testing Developer Guides
Integrated Installer
Fuel
Compass
Compass
Supported Testsuite
Rubbos, VSTF
Rubbos, VSTF
POSCA
219
OPNFV Documentation, Release Danube
POSCA
Rubbos
Test suite & Test case Description
vstf
posca_factor_ping
posca_factor_system_bandwidth
rubbos_basic
rubbos_TC1101
rubbos_TC1201
rubbos_TC1301
rubbos_TC1401
rubbos_heavy_TC1101
vstf_Ti1
vstf_Ti2
vstf_Ti3
vstf_Tn1
vstf_Tn2
vstf_Tu1
vstf_Tu2
vstf_Tu3
POSCA Testsuite Guide
POSCA Introduction The POSCA (Parametric Bottlenecks Testing Catalogue) testsuite classifies the bottlenecks
test cases and results into 5 categories. Then the results will be analyzed and bottlenecks will be searched among these
categories.
The POSCA testsuite aims to locate the bottlenecks in parmetric manner and to decouple the bottlenecks regarding
the deployment requirements. The POSCA testsuite provides an user friendly way to profile and understand the E2E
system behavior and deployment requirements.
Goals of the POSCA testsuite:
1. Automatically locate the bottlenecks in a iterative manner.
2. Automatically generate the testing report for bottlenecks in different categories.
3. Implementing Automated Staging.
Scopes of the POSCA testsuite:
1. Modeling, Testing and Test Result analysis.
2. Parameters choosing and Algorithms.
Test stories of POSCA testsuite:
1. Factor test (Stress test): base test cases that Feature test and Optimization will be dependant on.
2. Feature test: test cases for features/scenarios.
3. Optimization test: test to tune the system parameter.
Detailed workflow is illutrated below.
• https://wiki.opnfv.org/display/bottlenecks
Preinstall Packages
• Docker: https://docs.docker.com/engine/installation/
– For Ubuntu, please refer to https://docs.docker.com/engine/installation/linux/ubuntu/
• Docker-Compose: https://docs.docker.com/compose/
220
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
if [ -d usr/local/bin/docker-compose ]; then
rm -rf usr/local/bin/docker-compose
fi
curl -L https://github.com/docker/compose/releases/download/1.11.0/docker-compose-`uname -s`-`uname chmod +x /usr/local/bin/docker-compose
Run POSCA Locally POSCA testsuite is highly automated regarding test environment preparation, installing testing tools, excuting tests and showing the report/analysis. A few steps are needed to run it locally.
It is presumed that a user is using Compass4nfv to deploy OPNFV Danube and the user logins jumper server as root.
Downloading Bottlenecks Software
mkdir /home/opnfv
cd /home/opnfv
git clone https://gerrit.opnfv.org/gerrit/bottlenecks
cd bottlenecks
Preparing Python Virtual Evnironment
. pre_virt_env.sh
Excuting Specified Testcase Bottlencks provide a CLI interface to run the tests, which is one of the most convenient
way since it is more close to our natural languge. An GUI interface with rest API will also be provided in later update.
bottlenecks [testcase run <testcase>] [teststory run <teststory>]
For the testcase command, testcase name should be as the same name of the test case configuration file located in
testsuites/posca/testcase_cfg. For stress tests in Danube, testcase should be replaced by either posca_factor_ping or
posca_factor_system_bandwidth. For the teststory command, a user could specified the test cases to be excuted by
defined it in a teststory configuration file located in testsuites/posca/testsuite_story. There is also an example there
named posca_factor_test.
There are also other 2 ways to run test cases and test stories. The first one is using shell script.
bash run_tests.sh [-h|--help] [-s <testsuite>] [-c <testcase>]
The second is using python interpreter.
docker-compose -f docker/bottleneck-compose/docker-compose.yml up -d
docker pull tutum/influxdb:0.13
sleep 5
POSCA_SCRIPT="/home/opnfv/bottlenecks/testsuites/posca"
docker exec bottleneckcompose_bottlenecks_1 python ${POSCA_SCRIPT}/run_posca.py [testcase <testcase>]
Showing Report Bottlenecks uses ELK to illustrate the testing results. Asumming IP of the SUT (System Under
Test) is denoted as ipaddr, then the address of Kibana is http://[ipaddr]:5601. One can visit this address to see the
illustrations. Address for elasticsearch is http://[ipaddr]:9200. One can use any Rest Tool to visit the testing data
stored in elasticsearch.
Cleaning Up Environment
. rm_virt_env.sh
If you want to clean the dockers that established during the test, you can excute the additional commands below.
2.3. Testing Developer Guides
221
OPNFV Documentation, Release Danube
docker-compose -f docker/bottleneck-compose/docker-compose.yml down -d
docker ps -a | grep 'influxdb' | awk '{print $1}' | xargs docker rm -f >/dev/stdout
Or you can just run the following command
bash run_tests.sh --cleanup
Note that you can also add cleanup parameter when you run a test case. Then environment will be automatically
cleaned up when completing the test.
Run POSCA through Community CI POSCA test cases are runned by OPNFV CI now. See https://build.opnfv.org
for details of the building jobs. Each building job is set up to execute a single test case. The test results/logs will be
printed on the web page and reported automatically to community MongoDB. There are two ways to report the results.
1. Report testing result by shell script
bash run_tests.sh [-h|--help] [-s <testsuite>] [-c <testcase>] --report
2. Report testing result by python interpreter
docker-compose -f docker/bottleneck-compose/docker-compose.yml up -d
docker pull tutum/influxdb:0.13
sleep 5
REPORT="True"
POSCA_SCRIPT="/home/opnfv/bottlenecks/testsuites/posca"
docker exec bottleneckcompose_bottlenecks_1 python ${POSCA_SCRIPT}/run_posca.py [testcase <testcase>]
Test Result Description
• Please refer to release notes and also https://wiki.opnfv.org/display/testing/Result+alignment+for+ELK+postprocessing
Dashbard guide
Scope This document provides an overview of the results of test cases developed by the OPNFV Bottlenecks Project,
executed on OPNFV community labs.
OPNFV CI(Continous Integration) system provides automated build, deploy and testing for the software developed in
OPNFV. Unless stated, the reported tests are automated via Jenkins Jobs.
Test results are visible in the following dashboard:
• Testing dashboard: uses Mongo DB to store test results and Bitergia for visualization, which includes the rubbos
test result, vstf test result.
Bottlenecks - Deprecated Testing Guide
Rubbos Testsuite Guide
Rubbos Introduction Rubbos is a bulletin board benchmark modeled after an online news forum like Slashdot. It
is an open source Middleware and an n-tier system model which is used to be deployed on multiple physical node and
to measure the whole performacne of OPNFV platform. Rubbos can deploy the Apache, tomcat, and DB. Based on
the deployment, rubbos gives the pressure to the whole system. When the system reaches to the peak, the throughput
222
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
will not grow more. This testcase can help to understand the bottlenecks of OPNFV plantform and improve the
performance of OPNFV platform.
Detailed workflow is illutrated below.
submodules/bottlenecks/docs/testing/developer/devguide/../images/Framework_Setup.png
Preinstall Packages There is a need to install some packages before running the rubbos, gcc, gettext, g++,
libaio1, libaio-dev, make and git are necessary. When the rubbos runs on the OPNFV community continuous
integration(CI) system, the required packages are installed automately as shown in the code repository, which is
/utils/infra_setup/vm_dev_setup/packages.conf, besides, the packages can be encapsulated in the images initially. If
someone wants to use rubbos locally, he/she has to install them by hand, such as in ubuntu 14.04,
apt-get update
apt-get install gettext
How does Rubbos Integrate into Installers 1.Community CI System
Rubbos has been successfully integrated into fuel and compass with NOSDN scenario in OPNFV community CI
system.
Heat is used to create 9 instances, which is shown in /utils/infra_setup/heat_template/HOT_create_instance.sh, the 9
instances are used for installing Apache, Tomcat, Mysql, Control, Benchmark and 4 Clients. The tools, such as rubbos,
sysstat, oprofile, etc, are installed in these instances to perform the test, the test results are stored in the Benchmark
instance initially, then they are copied to the Rubbos_result instance, finally, the test results are transferred to the
community dashboard.
There’s a need to store our pakages as large as M bytes or G bytes size, such as the images, jdk, apache-ant, apachetomcat, etc, the OPNFV community storage system, Google Cloud Storage, is used, the pakages can be downloaded
from https://artifacts.opnfv.org/bottlenecks/rubbos.
2.Local Deployment
If someone wants to run the rubbos in his own environment, he/she can keep to the following steps,
2.1 Start up instances by using heat, nova or libvert. In Openstack Environemnt, the heat script can refer
/utils/infra_setup/heat_template/HOT_create_instance.sh, if the openstack doesn’t support heat module, the script
/utils/infra_setup/create_instance.sh can be used. Without Openstack, there’s a way to set up instances by using libvert,
the scripts are shown under the directory /utils/rubbos_dev_env_setup.
The image can be downloaded from the community cloud storage
curl --connect-timeout 10 -o bottlenecks-trusty-server.img
http://artifacts.opnfv.org/bottlenecks/rubbos/bottlenecks-trusty-server.img
2.2 Ssh into the control node and clone the bottlenecks codes to the root directory.
git clone https://git.opnfv.org/bottlenecks /bottlenecks
2.3 Download the packages and decompress them into the proper directory.
curl --connect-timeout 10 -o app_tools.tar.gz
http://artifacts.opnfv.org/bottlenecks/rubbos/app_tools.tar.gz
2.3. Testing Developer Guides
223
OPNFV Documentation, Release Danube
curl --connect-timeout 10 -o rubbosMulini6.tar.gz
http://artifacts.opnfv.org/bottlenecks/rubbos/rubbosMulini6.tar.gz
tar zxf app_tools.tar.gz -C /bottlenecks/rubbos
tar zxf rubbosMulini6.tar.gz -C /bottlenecks/rubbos/rubbos_scripts
2.4 Ssh into the Control node and run the script
source /bottlenecks/rubbos/rubbos_scripts/1-1-1/scripts/run.sh
2.5 Check the test results under the directory /bottlenecks/rubbos/rubbos_results in Control node. The results are
stored in the format of xml, move them to the brower chrome, then you can see the results.
Test Result Description In OPNFV community, the result is shown in the following format
[{'client':
{'client':
{'client':
{'client':
{'client':
{'client':
{'client':
200, 'throughput': 27},
700, 'throughput': 102},
1200, 'throughput': 177},
1700, 'throughput': 252},
2200, 'throughput': 323},
2700, 'throughput': 399},
3200, 'throughput': 473}]
The results are transferred to the community database and a map is drawed on the dashboard. Along with the growth
of the number of the client, the throughput grows at first, then meets up with a point of inflexion, which is caused by
the bottlenecks of the measured system.
VSTF Testsuite Guide
VSTF Introduction VSTF(Virtual Switch Test Framework) is a system-level testing framework in the area of network virtualization, and it could help you estimate the system switch ability and find out the network bottlenecks by
main KPIs(bandwidth, latency, resource usage and so on), VSTF owns a methodology to define the test scenario and
testcases, Now we could support Tu testcases in the Openstack environment, More scenarios and cases will be added.
VSTF TestScenario
1. Tu - VM to VM
2. Tn - Physical Nic loopback
3. TnV - VNF loopback
4. Ti - VM to Physical Nic
Pre-install Packages on the ubuntu 14.04 VM
VSTF VM Preparation Steps
1. Create a ubuntu 14.04 VM
2. Install dependency inside VM
3. Install vstf python package inside VM
224
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
VM preparation Install python2.7 version and git
sudo apt-get install python2.7
sudo apt-get install git
Download Bottlenecks package
sudo cd /home/
sudo git clone https://gerrit.opnfv.org/gerrit/bottlenecks
Install the dependency
sudo
sudo
sudo
sudo
sudo
sudo
sudo
sudo
sudo
sudo
sudo
apt-get install python-pip
pip install --upgrade pip
dpkg-reconfigure dash
apt-get install libjpeg-dev
apt-get install libpng-dev
apt-get install python-dev
apt-get install python-testrepository
apt-get install git
apt-get install python-pika
apt-get install python-oslo.config
pip install -r /home/bottlenecks/vstf/requirements.txt
Install vstf package
sudo
sudo
sudo
sudo
sudo
mkdir -p /var/log/vstf/
cp -r /home/bottlenecks/vstf/etc/vstf/ /etc/
mkdir -p /opt/vstf/
cd /home/bottlenecks;sudo rm -rf build/
python setup.py install
Image on the Cloud
Name
URL
Format
Size
User
Passwd
vstf-image
http://artifacts.opnfv.org/bottlenecks/vstf-manager-new.img
QCOW2
5G
root
root
There is a complete vstf image on the cloud ,you could download it and use it to deploy and run cases ,but do not need
VM preparation steps.
How is VSTF Integrated into Installers
VM requirements
Name
FLAVOR
NETWORK
m1.large
IMAGE_NAME
vstf-image
vstfmanager
vstf-tester
m1.large
vstf-image
m1.large
vstf-image
control-plane(eth0)=XX.XX.XX.XX
test-plane(eth1)=XX.XX.XX.XX
control-plane(eth0)=XX.XX.XX.XX
test-plane(eth1)=XX.XX.XX.XX
vstf-target
control-plane=XX.XX.XX.XX
m1.large means 4U4G for the target image Size 5GB For the network used by VMs,network need two plane ,one plane
is control plane and the other plane is test plane.
2.3. Testing Developer Guides
225
OPNFV Documentation, Release Danube
OPNFV community Usage in the CI system
Project Name
bottlenecks-daily-fuel-vstf-lf-master
Project Categoty
bottlenecks
OPNFV community jenkins Project info
Main Entrance for the ci test:
cd /home/bottlenecks/ci;
bash -x vstf_run.sh
Test on local(Openstack Environment) download the image file
curl --connect-timeout 10 -o /tmp/vstf-manager.img \
http://artifacts.opnfv.org/bottlenecks/vstf-manager-new.img -v
create the image file by the glance
glance image-create --name $MANAGER_IMAGE_NAME \
--disk-format qcow2 \
--container-format bare \
--file /tmp/vstf-manager.img
create the keypair for the image(anyone will be ok)
cd /home/bottlenecks/utils/infra_setup/bottlenecks_key
nova keypair-add --pub_key $KEY_PATH/bottlenecks_key.pub $KEY_NAME
create the vstf three VMs in the openstack by heat
cd /home/bottlenecks/utils/infra_setup/heat_template/vstf_heat_template
heat stack-create vstf -f bottleneck_vstf.yaml
launch the vstf process inside the vstf-manager vstf-tester vstf-target VMs
cd /home/bottlenecks/utils/infra_setup/heat_template/vstf_heat_template
bash -x launch_vstf.sh
edit the test scenario and test packet list in the vstf_test.sh, now support the Tu-1/2/3
function fn_testing_scenario(){
...
local test_length_list="64 128 256 512 1024"
local test_scenario_list="Tu-1 Tu-3"
...
}
launch the vstf script
cd /home/bottlenecks/utils/infra_setup/heat_template/vstf_heat_template
bash -x vstf_test.sh
Test Result Description
Result Format For example after the test, The result will display as the following format
{ u'64': { u'AverageLatency': 0.063,
u'Bandwidth': 0.239,
u'CPU': 0.0,
u'Duration': 20,
226
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
u'MaximumLatency': 0.063,
u'MinimumLatency': 0.063,
u'MppspGhz': 0,
u'OfferedLoad': 100.0,
u'PercentLoss': 22.42,
u'RxFrameCount': 4309750.0,
u'RxMbps': 198.28,
u'TxFrameCount': 5555436.0,
u'TxMbps': 230.03}}
Option Description
Option Name
AverageLatency
Bandwidth
CPU
Duration
MaximumLatency
MinimumLatency
MppspGhz
OfferedLoad
PercentLoss
RxFrameCount
RxMbps
TxFrameCount
TxMbps
Description
The average latency data during the packet transmission (Unit:microsecond)
Network bandwidth(Unit:Million packets per second)
Total Resource Cpu usage(Unit: Ghz)
Test time(Unit: second)
The maximum packet latency during the packet transmission (Unit:microsecond)
The maximum packet latency during the packet transmission (Unit:microsecond)
Million Packets per second with per CPU resource Ghz(Unit: Mpps/Ghz)
The load of network offered
The percent of frame loss rate
The total frame on Nic rx
The received bandwidth per second
The total frame on Nic rx
The send bandwidth per second
Functest
OPNFV FUNCTEST developer guide
Introduction
Functest is a project dealing with functional testing. Functest produces its own internal test cases but can also be
considered as a framework to support feature and VNF onboarding project testing. Functest developed a TestAPI and
defined a test collection framework that can be used by any OPNFV project.
Therefore there are many ways to contribute to Functest. You can:
• Develop new internal test cases
• Integrate the tests from your feature project
• Develop the framework to ease the integration of external test cases
• Develop the API / Test collection framework
• Develop dashboards or automatic reporting portals
This document describes how, as a developer, you may interact with the Functest project. The first section details
the main working areas of the project. The Second part is a list of “How to” to help you to join the Functest family
whatever your field of interest is.
Functest developer areas
2.3. Testing Developer Guides
227
OPNFV Documentation, Release Danube
Functest High level architecture Functest is project delivering a test container dedicated to OPNFV. It includes the
tools, the scripts and the test scenarios.
Functest can be described as follow:
+----------------------+
|
|
|
+--------------+
|
+-------------------+
|
|
|
|
Public
|
|
|
| Tools
|
+------------------+
OPNFV
|
|
| Scripts
|
|
| System Under Test |
|
| Scenarios
|
+------------------+
|
|
|
|
|
Management
|
|
|
+--------------+
|
+-------------------+
|
|
|
Functest Docker
|
|
|
+----------------------+
Functest internal test cases The internal test cases in Danube are:
• api_check
• cloudify_ims
• connection_check
• vping_ssh
• vping_userdata
• odl
• rally_full
• rally_sanity
• snaps_health_check
• tempest_full_parallel
• tempest_smoke_serial
By internal, we mean that this particular test cases have been developped and/or integrated by functest contributors
and the associated code is hosted in the Functest repository. An internal case can be fully developped or a simple
integration of upstream suites (e.g. Tempest/Rally developped in OpenStack are just integrated in Functest). The
structure of this repository is detailed in [1]. The main internal test cases are in the opnfv_tests subfolder of the
repository, the internal test cases are:
• sdn: odl, onos
• openstack: api_check, connection_check, snaps_health_check, vping_ssh, vping_userdata, tempest_*, rally_*,
snaps_smoke
• vnf: cloudify_ims
If you want to create a new test case you will have to create a new folder under the testcases directory.
Functest external test cases The external test cases are inherited from other OPNFV projects, especially the feature
projects.
The external test cases are:
228
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
• barometer
• bgpvpn
• doctor
• domino
• odl-netvirt
• onos
• fds
• multisite
• netready
• orchestra_ims
• parser
• promise
• refstack_defcore
• security_scan
• snaps_smoke
• sfc-odl
• vyos_vrouter
The code to run these test cases may be directly in the repository of the project. We have also a features sub directory
under opnfv_tests directory that may be used (it can be usefull if you want to reuse Functest library).
Functest framework Functest can be considered as a framework. Functest is release as a docker file, including
tools, scripts and a CLI to prepare the environement and run tests. It simplifies the integration of external test suites in
CI pipeline and provide commodity tools to collect and display results.
Since Colorado, test categories also known as tiers have been created to group similar tests, provide consistant sub-lists
and at the end optimize test duration for CI (see How To section).
The definition of the tiers has been agreed by the testing working group.
The tiers are:
• healthcheck
• smoke
• features
• components
• performance
• vnf
• stress
2.3. Testing Developer Guides
229
OPNFV Documentation, Release Danube
Functest abstraction classes In order to harmonize test integration, 3 abstraction classes have been introduced in
Danube:
• testcase: base for any test case
• feature_base: abstraction for feature project
• vnf_base: abstraction for vnf onboarding
The goal is to unify the way to run test from Functest.
feature_base and vnf_base inherit from testcase:
+-----------------------------------------+
|
|
|
Testcase_base
|
|
|
|
- init()
|
|
- run()
|
|
- publish_report()
|
|
- check_criteria()
|
|
|
+-----------------------------------------+
|
|
V
V
+--------------------+
+--------------------------+
|
|
|
|
|
feature_base
|
|
vnf_base
|
|
|
|
|
| - prepare()
|
| - prepare()
|
| - execute()
|
| - deploy_orchestrator() |
| - post()
|
| - deploy_vnf()
|
| - parse_results() |
| - test_vnf()
|
|
|
| - clean()
|
|
|
| - execute()
|
|
|
|
|
+--------------------+
+--------------------------+
Functest util classes In order to simplify the creation of test cases, Functest develops some functions that can be
used by any feature or internal test cases. Several features are supported such as logger, configuration management
and Openstack capabilities (snapshot, clean, tacker,..). These functions can be found under <repo>/functest/utils and
can be described as follows:
functest/utils/ |– config.py |– constants.py |– env.py |– functest_logger.py |– functest_utils.py |– openstack_clean.py |–
openstack_snapshot.py |– openstack_tacker.py ‘– openstack_utils.py
Note that for Openstack, keystone v3 is now deployed by default by compass, fuel and joid in Danube. All installers
still support keysone v2 (deprecated in next version).
Test collection framework The OPNFV testing group created a test collection database to collect the test results
from CI:
http://testresults.opnfv.org/test/swagger/spec.html
Authentication: opnfv/[email protected]
Any test project running on any lab integrated in CI can push the results to this database. This database can be used to
see the evolution of the tests and compare the results versus the installers, the scenarios or the labs.
230
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
Overall Architecture The Test result management can be summarized as follows:
+-------------+
+-------------+
+-------------+
|
|
|
|
|
|
|
Test
|
|
Test
|
|
Test
|
| Project #1 |
| Project #2 |
| Project #N |
|
|
|
|
|
|
+-------------+
+-------------+
+-------------+
|
|
|
V
V
V
+-----------------------------------------+
|
|
|
Test Rest API front end
|
| http://testresults.opnfv.org/test
|
|
|
+-----------------------------------------+
A
|
|
V
|
+-------------------------+
|
|
|
|
|
Test Results DB
|
|
|
Mongo DB
|
|
|
|
|
+-------------------------+
|
|
+----------------------+
|
|
|
test Dashboard
|
|
|
+----------------------+
TestAPI description The TestAPI is used to declare pods, projects, test cases and test results. Pods are the pods
used to run the tests. The results pushed in the database are related to pods, projects and cases. If you try to push
results of test done on non referenced pod, the API will return an error message.
An additional method dashboard has been added to post-process the raw results in release Brahmaputra (deprecated in
Colorado).
The data model is very basic, 5 objects are created:
• Pods
• Projects
• Testcases
• Results
• Scenarios
The code of the API is hosted in the releng repository [6]. The static documentation of the API can be found at [17].
The TestAPI has been dockerized and may be installed locally in your lab. See [15] for details.
The deployment of the TestAPI has been automated. A jenkins job manages:
• the unit tests of the TestAPI
• the creation of a new docker file
• the deployment of the new TestAPI
2.3. Testing Developer Guides
231
OPNFV Documentation, Release Danube
• the archive of the old TestAPI
• the backup of the Mongo DB
TestAPI Authorization PUT/DELETE/POST operations of the TestAPI now require token based authorization. The
token needs to be added in the request using a header ‘X-Auth-Token’ for access to the database.
e.g:: headers[’X-Auth-Token’]
The value of the header i.e the token can be accessed in the jenkins environment variable TestApiToken. The token
value is added as a masked password.
headers['X-Auth-Token'] = os.environ.get('TestApiToken')
The above example is in Python. Token based authentication has been added so that only ci pods jenkins job can have
access to the database.
Please note that currently token authorization is implemented but is not yet enabled.
An automatic reporting page has been created in order to provide a consistant view of the scenarios. In
this page, each scenario is evaluated according to test criteria. The code for the automatic reporting is
available at [8].
The results are collected from the centralized database every day and, per scenario. A score is calculated
based on the results from the last 10 days. This score is the addition of single test scores. Each test case
has a success criteria reflected in the criteria field from the results.
Considering an instance of a scenario os-odl_l2-nofeature-ha, the scoring is the addition of the scores of
all the runnable tests from the categories (tiers healthcheck, smoke and features) corresponding to this
scenario.
Test
vPing_ssh
vPing_userdata
tempest_smoke_serial
rally_sanity
odl
promise
doctor
security_scan
parser
copper
Apex
X
X
X
X
X
X
X
Compass
X
X
X
X
X
Fuel
X
X
X
X
X
X
X
Joid
X
X
X
X
X
X
X
X
X
src: colorado (see release note for the last matrix version)
All the testcases listed in the table are runnable on os-odl_l2-nofeature scenarios. If no result is available
or if all the results are failed, the test case get 0 point. If it was succesfull at least once but not anymore
during the 4 runs, the case get 1 point (it worked once). If at least 3 of the last 4 runs were successful, the
case get 2 points. If the last 4 runs of the test are successful, the test get 3 points.
In the example above, the target score for fuel/os-odl_l2-nofeature-ha is 3x6 = 18 points.
The scenario is validated per installer when we got 3 points for all individual test cases (e.g 18/18). Please
note that complex or long duration tests are not considered for the scoring. The success criteria are not
always easy to define and may require specific hardware configuration. These results however provide a
good level of trust on the scenario.
A web page is automatically generated every day to display the status. This page can be found at [9]. For
the status, click on Status menu, you may also get feedback for vims and tempest_smoke_serial test cases.
232
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
Any validated scenario is stored in a local file on the web server. In fact as we are using a sliding windows
to get results, it may happen that a successful scenarios is no more run (because considered as stable) and
then the number of iterations (4 needed) would not be sufficient to get the green status.
Please note that other test cases (e.g. sfc_odl, bgpvpn) need also ODL configuration addons and as a
consequence specific scenario. There are not considered as runnable on the generic odl_l2 scenario.
Dashboard Dashboard is used to provide a consistant view of the results collected in CI. The results showed on the
dashboard are post processed from the Database, which only contains raw results.
In Brahmaputra, we created a basic dashboard. Since Colorado, it was decided to adopt ELK framework. Mongo DB
results are extracted to feed Elasticsearch database ([7]).
A script was developed to build elasticsearch data set. This script can be found in [16].
For next versions, it was decided to integrated bitergia dashboard. Bitergia already provides a dashboard for code and
infrastructure. A new Test tab will be added. The dataset will be built by consuming the TestAPI.
How TOs
How Functest works? The installation and configuration of the Functest docker image is described in [1].
The procedure to start tests is described in [2]
How can I contribute to Functest? If you are already a contributor of any OPNFV project, you can contribute to
functest. If you are totally new to OPNFV, you must first create your Linux Foundation account, then contact us in
order to declare you in the repository database.
We distinguish 2 levels of contributors:
• the standard contributor can push patch and vote +1/0/-1 on any Functest patch
• The commitor can vote -2/-1/0/+1/+2 and merge
Functest commitors are promoted by the Functest contributors.
Where can I find some help to start? This guide is made for you. You can also have a look at the project wiki page
[10]. There are references on documentation, video tutorials, tips...
You can also directly contact us by mail with [Functest] prefix in the title at [email protected] or on
the IRC chan #opnfv-functest.
What kind of testing do you do in Functest? Functest is focusing on Functional testing. The results must be PASS
or FAIL. We do not deal with performance and/or qualification tests. We consider OPNFV as a black box and execute
our tests from the jumphost according to Pharos reference technical architecture.
Upstream test suites are integrated (Rally/Tempest/ODL/ONOS,...). If needed Functest may bootstrap temporarily
testing activities if they are identified but not covered yet by an existing testing project (e.g security_scan before the
creation of the security repository)
How test constraints are defined? Test constraints are defined according to 2 paramaters:
• The scenario (DEPLOY_SCENARIO env variable)
• The installer (INSTALLER_TYPE env variable)
2.3. Testing Developer Guides
233
OPNFV Documentation, Release Danube
A scenario is a formal description of the system under test. The rules to define a scenario are described in [4]
These 2 constraints are considered to determinate if the test is runnable or not (e.g. no need to run onos suite on odl
scenario).
In the test declaration for CI, the test owner shall indicate these 2 constraints. The file testcases.yaml [5] must be
patched in git to include new test cases. A more elaborated system based on template is planned for next releases
For each dependency, it is possible to define a regex:
name: promise
criteria: 'success_rate == 100%'
description: >Test suite from Promise project.
dependencies:
installer: '(fuel)|(joid)'
scenario: ''
In the example above, it means that promise test will be runnable only with joid or fuel installers on any scenario.
The vims criteria means any installer and exclude onos and odl with bgpvpn scenarios:
name: vims
criteria: 'status == "PASS"'
description: >This test case deploys an OpenSource vIMS solution from Clearwater
using the Cloudify orchestrator. It also runs some signaling traffic.
dependencies:
installer: ''
scenario: '(ocl)|(nosdn)|^(os-odl)((?!bgpvpn).)*$'
How to write and check constaint regex? Regex are standard regex. You can have a look at [11]
You can also easily test your regex via an online regex checker such as [12]. Put your scenario in the TEST STRING
window (e.g. os-odl_l3-ovs-ha), put your regex in the REGULAR EXPRESSION window, then you can test your rule.
How to know which test I can run? You can use the API [13]. The static declaration is in git [5]
If you are in a Functest docker container (assuming that the environement has been prepared): just use the CLI.
You can get the list per Test cases or by Tier:
# functest testcase list
healthcheck
vping_ssh
vping_userdata
tempest_smoke_serial
rally_sanity
odl
doctor
security_scan
tempest_full_parallel
rally_full
vims
# functest tier list
- 0. healthcheck:
['healthcheck']
- 1. smoke:
['vping_ssh', 'vping_userdata', 'tempest_smoke_serial', 'rally_sanity']
- 2. sdn_suites:
234
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
['odl']
- 3. features:
['doctor', 'security_scan']
- 4. openstack:
['tempest_full_parallel', 'rally_full']
- 5. vnf:
['vims']
How to manually start Functest tests? Assuming that you are connected on the jumphost and that the system is
“Pharos compliant”, i.e the technical architecture is compatible with the one defined in the Pharos project:
# docker pull opnfv/functest:latest
# envs="-e INSTALLER_TYPE=fuel -e INSTALLER_IP=10.20.0.2 -e DEPLOY_SCENARIO=os-odl_l2-nofeature-ha -e
# sudo docker run --privileged=true -id ${envs} opnfv/functest:latest /bin/bash
Then you must connect to the docker container and source the credentials:
# docker ps (copy the id)
# docker exec -ti <container_id> bash
# source $creds
You must first check if the environment is ready:
# functest env status
Functest environment ready to run tests.
If not ready, prepare the env by launching:
# functest env prepare
Functest environment ready to run tests.
Once the Functest env is ready, you can use the CLI to start tests.
You can run test cases per test case or per tier: # functest testcase run <case name> or # functest tier run <tier
name>
e.g:
# functest testcase run tempest_smoke_serial
# functest tier run features
If you want to run all the tests you can type:
# functest testcase run all
If you want to run all the tiers (same at the end that running all the test cases) you can type:
# functest tier run all
How to declare my tests in Functest? If you want to add new internal test cases, you can submit patch under the
testcases directory of Functest repository.
For feature test integration, the code can be kept into your own repository. The Functest files to be modified are:
• functest/docker/Dockerfile: get your code in Functest container
• functest/ci/testcases.yaml: reference your test and its associated constraints
2.3. Testing Developer Guides
235
OPNFV Documentation, Release Danube
Dockerfile This file lists the repositories (internal or external) to be cloned in the Functest container. You can also
add external packages:
RUN git clone https://gerrit.opnfv.org/gerrit/<your project> ${REPOS_DIR}/<your project>
testcases.yaml All the test cases that must be run from CI / CLI must be declared in ci/testcases.yaml.
This file is used to get the constraints related to the test:
name: <my_super_test_case>
criteria: <not used yet in Colorado, could be > 'PASS', 'rate > 90%'
description: ><the description of your super test suite>
dependencies:
installer: regex related to installer e.g. 'fuel', '(apex)||(joid)'
scenario: regex related to the scenario e.g. 'ovs*no-ha'
You must declare your test case in one of the category (tier).
If you are integrating test suites from a feature project, the default category is features.
How to select my list of tests for CI? Functest can be run automatically from CI, a jenkins job is usually called
after an OPNFV fresh installation. By default we try to run all the possible tests (see [14] called from Functest jenkins
job):
cmd="python ${FUNCTEST_REPO_DIR}/ci/run_tests.py -t all ${flags}"
Each case can be configured as daily and/or weekly task. Weekly tasks are used for long duration or experimental
tests. Daily tasks correspond to the minimum set of test suites to validate a scenario.
When executing run_tests.py, a check based on the jenkins build tag will be considered to detect whether it is a daily
and/or a weekly test.
in your CI you can customize the list of test you want to run by case or by tier, just change the line:
cmd="python ${FUNCTEST_REPO_DIR}/ci/run_tests.py -t <whatever you want> ${flags}"
e.g.:
cmd="python ${FUNCTEST_REPO_DIR}/ci/run_tests.py -t healthcheck,smoke ${flags}"
This command will run all the test cases of the first 2 tiers, i.e. healthcheck, connection_check, api_check, vping_ssh,
vping_userdata, snaps_somke, tempest_smoke_serial and rally_sanity.
How to push your results into the Test Database The test database is used to collect test results. By default it is
enabled only for CI tests from Production CI pods.
The architecture and associated API is described in previous chapter. If you want to push your results from CI, you
just have to call the API at the end of your script.
You can also reuse a python function defined in functest_utils.py:
def push_results_to_db(db_url, case_name, logger, pod_name,version, payload):
"""
POST results to the Result target DB
"""
url = db_url + "/results"
installer = get_installer_type(logger)
params = {"project_name": "functest", "case_name": case_name,
236
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
"pod_name": pod_name, "installer": installer,
"version": version, "details": payload}
headers = {'Content-Type': 'application/json'}
try:
r = requests.post(url, data=json.dumps(params), headers=headers)
if logger:
logger.debug(r)
return True
except Exception, e:
print "Error [push_results_to_db('%s', '%s', '%s', '%s', '%s')]:" \
% (db_url, case_name, pod_name, version, payload), e
return False
Where can I find the documentation on the test API? http://artifacts.opnfv.org/releng/docs/testapi.html
How to exclude Tempest case from default Tempest smoke suite? Tempest default smoke suite deals with 165
test cases. Since Colorado the success criteria is 100%, i.e. if 1 test is failed the success criteria is not matched for the
scenario.
It is necessary to exclude some test cases that are expected to fail due to known upstream bugs (see release notes).
A file has been created for such operation: https://git.opnfv.org/cgit/functest/tree/functest/opnfv_tests/openstack/tempest/custom_tests/bla
It can be described as follows:
-
scenarios:
- os-odl_l2-bgpvpn-ha
- os-odl_l2-bgpvpn-noha
installers:
- fuel
- apex
tests:
- tempest.api.compute.servers.test_create_server.ServersTestJSON.test_list_servers
- tempest.api.compute.servers.test_create_server.ServersTestJSON.test_verify_server_details
- tempest.api.compute.servers.test_create_server.ServersTestManualDisk.test_list_servers
- tempest.api.compute.servers.test_create_server.ServersTestManualDisk.test_verify_server_det
- tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_reboot_server_ha
- tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops
- tempest.scenario.test_server_basic_ops.TestServerBasicOps.test_server_basic_ops
- tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_volume_boot_pattern
- tempest.scenario.test_volume_boot_pattern.TestVolumeBootPatternV2.test_volume_boot_pattern
Please note that each exclusion must be justified. the goal is not to exclude test cases because they do not pass. Several
scenarios reached the 100% criteria. So it is expected in the patch submited to exclude the cases to indicate the reasons
of the exclusion.
How do I know the Functest status of a scenario? A Functest automatic reporting page is generated daily. This
page is dynamically created through a cron job and is based on the results stored in the Test DB. You can access this
reporting page: http://testresults.opnfv.org/reporting
See https://wiki.opnfv.org/pages/viewpage.action?pageId=6828617 for details.
I have tests, to which category should I declare them? CATEGORIES/TIERS description:
2.3. Testing Developer Guides
237
OPNFV Documentation, Release Danube
healthcheck Simple OpenStack healtcheck tests case that validates the basic operations in OpenStack
Smoke
Set of smoke test cases/suites to validate the most common OpenStack and SDN Controller
operations
Features
Test cases that validate a specific feature on top of OPNFV. Those come from Feature projects and
need a bit of support for integration
CompoAdvanced Openstack tests: Full Tempest, Full Rally
nents
PerforOut of Functest Scope
mance
VNF
Test cases related to deploy an open source VNF including an orchestrator
The main ambiguity could be between features and VNF. In fact sometimes you have to spawn VMs to demonstrate
the capabilities of the feature you introduced. We recommend to declare your test in the feature category.
VNF category is really dedicated to test including:
• creation of resources
• deployement of an orchestrator/VNFM
• deployment of the VNF
• test of the VNFM
• free resources
The goal is not to study a particular feature on the infrastructure but to have a whole end to end test of a VNF
automatically deployed in CI. Moreover VNF are run in weekly jobs (one a week), feature tests are in daily jobs and
use to get a scenario score.
Where are the logs? Functest deals with internal and external testcases. Each testcase can generate logs.
Since Colorado we introduce the possibility to push the logs to the artifact.
(https://git.opnfv.org/releng/tree/utils/push-test-logs.sh) has been created for CI.
A new script
When called, and assuming that the POD is authorized to push the logs to artifacts, the script will push all the results
or logs locally stored under /home/opnfv/functest/results/.
If the POD is not connected to CI, logs are not pushed.
But in both cases, logs are stored in
/home/opnfv/functest/results in the container. Projects are encouraged to push their logs here.
Since Colorado it is also easy for feature project to integrate this feature by adding the log file as output_file parameter
when calling execute_command from functest_utils library
ret_val = functest_utils.execute_command(cmd, output_file=log_file)
How does Functest deal with VNF onboarding? VNF onboarding has been introduced in Brahmaputra through
the automation of a clearwater vIMS deployed thanks to cloudify orchestrator.
This automation has been described at OpenStack summit Barcelona: https://youtu.be/Jr4nG74glmY
The goal of Functest consists in testing OPNFV from a functional perspective: the NFVI and/or the features developed
in OPNFV. Feature test suites are provided by the feature project. Functest just simplifies the integration of the suite
into the CI and gives a consolidated view of the tests per scenario.
Functest does not develop VNFs.
Functest does not test any MANO stack.
238
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
OPNFV projects dealing with VNF onboarding Testing VNF is not the main goal however it gives interesting and
realistic feedback on OPNFV as a Telco cloud.
Onboarding VNF also allows to test a full stack: orchestrator + VNF.
Functest is VNF and MANO stack agnostic.
An internship has been initiated to reference the Open Source VNF: Intern Project Open Source VNF catalog
New projects dealing with orchestrators or VNFs are candidate for Danube.
The 2 projects dealing with orchestration are:
• orchestra (Openbaton)
• opera (Open-O)
The Models project address various goals for promoting availability and convergence of information and/or data models related to NFV service/VNF management, as being defined in standards (SDOs) and as developed in open source
projects.
Functest VNF onboarding In order to simplify VNF onboarding a new abstraction class has been developed in
Functest.
This class is based on vnf_base and can be described as follow:
+————+ +————–+ | test_base |————>| vnf_base | +————+ +————–+
|_ prepare
|_ deploy_orchestrator (optional)
|_ deploy_vnf
|_ test_vnf |_ clean
Several methods are declared in vnf_base:
• prepare
• deploy_orchestrator
• deploy_vnf
• test_vnf
• clean
deploy_vnf and test_vnf are mandatory.
prepare will create a user and a project.
How to declare your orchestrator/VNF?
1. test declaration
You must declare your testcase in the file <Functest repo>/functest/ci/testcases.yaml
2. configuration
You can precise some configuration parameters in config_functest.yaml
3. implement your test
2.3. Testing Developer Guides
239
OPNFV Documentation, Release Danube
Create your own VnfOnboarding file
you must create your entry point through a python clase as referenced in the configuration file
e.g. aaa => creation of the file <Functest repo>/functest/opnfv_tests/vnf/aaa/aaa.py
the class shall inherit vnf_base. You must implement the methods deploy_vnf() and test_vnf() and may implement
deploy_orchestrator()
you can call the code from your repo (but need to add the repo in Functest if it is not the case)
4. success criteria
So far we considered the test as PASS if vnf_deploy and test_vnf is PASS (see example in aaa).
References
[1]: http://artifacts.opnfv.org/functest/docs/configguide/index.html Functest configuration guide
[2]: http://artifacts.opnfv.org/functest/docs/userguide/index.html functest user guide
[3]: https://wiki.opnfv.org/opnfv_test_dashboard Brahmaputra dashboard
[4]: https://wiki.opnfv.org/display/INF/CI+Scenario+Naming
[5]: https://git.opnfv.org/cgit/functest/tree/ci/testcases.yaml
[6]: https://git.opnfv.org/cgit/releng/tree/utils/test/result_collection_api
[7]: https://git.opnfv.org/cgit/releng/tree/utils/test/scripts
[8]: https://git.opnfv.org/cgit/releng/tree/utils/test/reporting/functest
[9]: http://testresults.opnfv.org/reporting/
[10]: https://wiki.opnfv.org/opnfv_functional_testing
[11]: https://docs.python.org/2/howto/regex.html
[12]: https://regex101.com/
[13]: http://testresults.opnfv.org/test/api/v1/projects/functest/cases
[14]: https://git.opnfv.org/cgit/releng/tree/jjb/functest/functest-daily.sh
[15]: https://git.opnfv.org/cgit/releng/tree/utils/test/result_collection_api/README.rst
[16]: https://git.opnfv.org/cgit/releng/tree/utils/test/scripts/mongo_to_elasticsearch.py
[17]: http://artifacts.opnfv.org/releng/docs/testapi.html
OPNFV main site: http://www.opnfv.org
OPNFV functional test page: https://wiki.opnfv.org/opnfv_functional_testing
IRC support chan: #opnfv-functest
OpenRC: http://docs.openstack.org/user-guide/common/cli_set_environment_variables_using_openstack_rc.html
Rally installation procedure: https://rally.readthedocs.org/en/latest/tutorial/step_0_installation.html
config_functest.yaml : https://git.opnfv.org/cgit/functest/tree/functest/ci/config_functest.yaml
240
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
QTIP
QTIP Developer Guide
Overview
QTIP uses Python as primary programming language and build the framework from the following packages
Module
api
cli
template
docs
testing
Package
Connexion - API first applications with OpenAPI/Swagger and Flask
Click - the “Command Line Interface Creation Kit”
Jinja2 - a full featured template engine for Python
sphinx - a tool that makes it easy to create intelligent and beautiful documentation
pytest - a mature full-featured Python testing tool that helps you write better programs
Source Code The structure of repository is based on the recommended sample in The Hitchhiker’s Guide to Python
Path
./benchmarks/
./contrib/
./docker/
./docs/
./legacy/
./opt/
./qtip/
./tests/
./third-party/
Content
builtin benchmark assets including plan, QPI and metrics
independent project/plugin/code contributed to QTIP
configuration for building Docker image for QTIP deployment
release notes, user and developer documentation, design proposals
legacy obsoleted code that is unmaintained but kept for reference
optional component, e.g. scripts to setup infrastructure services for QTIP
the actual package
package functional and unit tests
third part included in QTIP project
Coding Style QTIP follows OpenStack Style Guidelines for source code and commit message.
Specially, it is recommended to link each patch set with a JIRA issue. Put:
JIRA: QTIP-n
in commit message to create an automatic link.
Testing All testing related code are stored in ./tests/
Path
./tests/data/
./tests/unit/
./conftest.py
Content
data fixtures for testing
unit test for each module, follow the same layout as ./qtip/
pytest configuration in project scope
tox is used to automate the testing tasks
cd <project_root>
pip install tox
tox
The test cases are written in pytest. You may run it selectively with
pytest tests/unit/reporter
2.3. Testing Developer Guides
241
OPNFV Documentation, Release Danube
Architecture
In Danube, QTIP releases its standalone mode, which is also know as solo:
The runner could be launched from CLI (command line interpreter) or API (application programming interface) and
drives the testing jobs. The generated data including raw performance data and testing environment are fed to collector.
Performance metrics will be parsed from the raw data and used for QPI calculation. Then the benchmark report is
rendered with the benchmarking results.
The execution can be detailed in the diagram below:
CLI - Command Line Interface
QTIP consists of different tools(metrics) to benchmark the NFVI. These metrics fall under different NFVI subsystems(QPI’s) such as compute, storage and network. A plan consists of one or more QPI’s, depending upon how
the end user would want to measure performance. CLI is designed to help the user, execute benchmarks and view
respective scores.
Framework QTIP CLI has been created using the Python package Click, Command Line Interface Creation Kit. It
has been chosen for number of reasons. It presents the user with a very simple yet powerful API to build complex
applications. One of the most striking features is command nesting.
As explained, QTIP consists of metrics, QPI’s and plans. CLI is designed to provide interface to all these components.
It is responsible for execution, as well as provide listing and details of each individual element making up these
components.
242
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
Design CLI’s entry point extends Click’s built in MultiCommand class object. It provides two methods, which are
overridden to provide custom configurations.
class QtipCli(click.MultiCommand):
def list_commands(self, ctx):
rv = []
for filename in os.listdir(cmd_folder):
if filename.endswith('.py') and \
filename.startswith('cmd_'):
rv.append(filename[4:-3])
rv.sort()
return rv
def get_command(self, ctx, name):
try:
if sys.version_info[0] == 2:
name = name.encode('ascii', 'replace')
mod = __import__('qtip.cli.commands.cmd_' + name,
None, None, ['cli'])
except ImportError:
return
return mod.cli
Commands and subcommands will then be loaded by the get_command method above.
Extending the Framework Framework can be easily extended, as per the users requirements. One such example
can be to override the builtin configurations with user defined ones. These can be written in a file, loaded via a Click
2.3. Testing Developer Guides
243
OPNFV Documentation, Release Danube
Context and passed through to all the commands.
class Context:
def __init__():
self.config = ConfigParser.ConfigParser()
self.config.read('path/to/configuration_file')
def get_paths():
paths = self.config.get('section', 'path')
return paths
The above example loads configuration from user defined paths, which then need to be provided to the actual command
definitions.
from qtip.cli.entry import Context
pass_context = click.make_pass_decorator(Context, ensure=False)
@cli.command('list', help='List the Plans')
@pass_context
def list(ctx):
plans = Plan.list_all(ctx.paths())
table = utils.table('Plans', plans)
click.echo(table)
API - Application Programming Interface
QTIP consists of different tools(metrics) to benchmark the NFVI. These metrics fall under different NFVI subsystems(QPI’s) such as compute, storage and network. A plan consists of one or more QPI’s, depending upon how the
end-user would want to measure performance. API is designed to expose a RESTful interface to the user for executing
benchmarks and viewing respective scores.
Framework QTIP API has been created using the Python package Connexion. It has been chosen for a number of
reasons. It follows API First approach to create micro-services. Hence, firstly the API specifications are defined from
the client side perspective, followed by the implementation of the micro-service. It decouples the business logic from
routing and resource mapping making design and implementation cleaner.
It has two major components:
API Specifications
The API specification is defined in a yaml or json file. Connexion follows Open API specification to
determine the design and maps the endpoints to methods in python.
Micro-service Implementation Connexion maps the operationId corresponding to every operation in API Specification to methods in python which handles request and responses.
As explained, QTIP consists of metrics, QPI’s and plans. The API is designed to provide a RESTful interface to
all these components. It is responsible to provide listing and details of each individual element making up these
components.
Design
244
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
Specification API’s entry point (main) runs connexion App class object after adding API Specification using
App.add_api method. It loads specification from swagger.yaml file by specifying specification_dir.
Connexion reads API’s endpoints(paths), operations, their request and response parameter details and response definitions from the API specification i.e. swagger.yaml in this case.
Following example demonstrates specification for the resource plans.
paths:
/plans/{name}:
get:
summary: Get a plan by plan name
operationId: qtip.api.controllers.plan.get_plan
tags:
- Plan
- Standalone
parameters:
- name: name
in: path
description: Plan name
required: true
type: string
responses:
200:
description: Plan information
schema:
$ref: '#/definitions/Plan'
404:
description: Plan not found
schema:
$ref: '#/definitions/Error'
501:
description: Resource not implemented
schema:
$ref: '#/definitions/Error'
default:
description: Unexpected error
schema:
$ref: '#/definitions/Error'
definitions:
Plan:
type: object
required:
- name
properties:
name:
type: string
description:
type: string
info:
type: object
config:
type: object
Every operationId in above operations corresponds to a method in controllers. QTIP has three controller modules
each for plan, QPI and metric. Connexion will read these mappings and automatically route endpoints to business
logic.
Swagger Editor can be explored to play with more such examples and to validate the specification.
2.3. Testing Developer Guides
245
OPNFV Documentation, Release Danube
Controllers The request is handled through these methods and response is sent back to the client. Connexion takes
care of data validation.
@common.check_endpoint_for_error(resource='Plan')
def get_plan(name):
plan_spec = plan.Plan(name)
return plan_spec.content
In above code get_plan takes a plan name and return its content.
The decorator check_endpoint_for_error defined in common is used to handle error and return a suitable
error response.
During Development the server can be run by passing specification file(swagger.yaml in this case) to connexion
cli connexion run <path_to_specification_file> -v
Extending the Framework
Modifying Existing API:
API can be modified by adding entries in swagger.yaml and adding the corresponding controller
mapped from operationID.
Adding endpoints:
New endpoints can be defined in paths section in swagger.yaml. To add a new resource
dummy paths:
/dummies:
get:
summary: Get all dummies
operationId: qtip.api.controllers.dummy.get_dummies
tags:
- dummy
responses:
200:
description: Foo information
schema:
$ref: '#/definitions/Dummy
default:
description: Unexpected error
schema:
$ref: '#/definitions/Error'
And then model of the resource can be defined in the definitions section.
definitions:
Dummy:
type: object
required:
- name
properties:
name:
type: string
description:
246
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
type: string
id:
type: string
Adding controller methods: Methods for handling requests and responses for every operation for the
endpoint added can be implemented in controller.
In controllers.dummy
def get_dummies():
all_dummies = [<code to get all dummies>]
return all_dummies, httplib.OK
Adding error responses Decorators for handling errors are defined in common.py in api.
from qtip.api import common
@common.check_endpoint_for_error(resource='dummy',operation='get')
def get_dummies()
all_dummies = [<code to get all dummies>]
return all_dummies
Adding new API:
API can easily be extended by adding more APIs to Connexion.App class object using add_api
class method.
In __main__
def get_app():
app = connexion.App(__name__, specification_dir=swagger_dir)
app.add_api('swagger.yaml', base_path='/v1.0', strict_validation=True)
return app
Extending it to add new APIs. The new API should have all endpoints mapped using operationId.
from qtip.api import __main__
my_app = __main__.get_app()
my_app.add_api('new_api.yaml',base_path'api2',strict_validation=True)
my_app.run(host="0.0.0.0", port=5000)
Compute QPI
The compute QPI gives user an overall score for system compute performace.
Summary The compute QPI are calibrated a ZTE E9000 server as a baseline with score of 2500 points. Higher
scores are better, with double the score indicating double the performance. The compute QPI provides three different
kinds of scores:
• Workload Scores
• Section Scores
• Compute QPI Scores
Baseline ZTE E9000 server with an 2 Deca core Intel Xeon CPU processor,128560.0MB Memory.
2.3. Testing Developer Guides
247
OPNFV Documentation, Release Danube
Workload Scores Each time a workload is executed QTIP calculates a score based on the computer’s performance
compared to the baseline performance.
Section Scores QTIP uses a number of different tests, or workloads, to measure performance. The workloads are
divided into five different sections:
Section
Integer
Floating
point
Memory
DPI
SSL
Detail
Indication
Integer workloads measure the integer instruction
performace of host or vm by performing Dhrystone
test.
Floating point workloads measure the floating
pointperfo rmance by performing Whetstone test.
All app relies on integer performance
Memory workloads measure memory bandwidth
by performing RamSpeed test.
DPI workloads measure deep-packet inspection
speed by performing nDPI test.
SSL Performance workloads measure cipher
speeds by using the OpenSSL tool.
Floating point performance is especially
important in video games,digital content creation
applications.
Software working with cipher large amounts data
relies on SSL Performace.
Software working with network packet anlysis
relies on DPI performance.
Software working with cipher large amounts data
relies on SSL Performace
A section score is the geometric mean of all the workload scores for workloads that are part of the section. These
scores are useful for determining the performance of the computer in a particular area.
Compute QPI Scores The compute QPI score is the weighted arithmetic mean of the five section scores. The
compute QPI score provides a way to quickly compare performance across different computers and different platforms
without getting bogged down in details.
VSPERF
VSPERF
VSPERF is an OPNFV testing project.
VSPERF provides an automated test-framework and comprehensive test suite based on industry standards for measuring data-plane performance of Telco NFV switching technologies as well as physical and virtual network interfaces
(NFVI). The VSPERF architecture is switch and traffic generator agnostic and provides full control of software component versions and configurations as well as test-case customization.
The Danube release of VSPERF includes improvements in documentation and capabilities. This includes additional
test-cases such as RFC 5481 Latency test and RFC-2889 address-learning-rate test. Hardware traffic generator support
is now provided for Spirent and Xena in addition to Ixia. The Moongen software traffic generator is also now fully
supported. VSPERF can be used in a variety of modes for configuration and setup of the network and/or for control of
the test-generator and test execution.
• Wiki: https://wiki.opnfv.org/characterize_vswitch_performance_for_telco_nfv_use_cases
• Repository: https://git.opnfv.org/vswitchperf
• Artifacts: https://artifacts.opnfv.org/vswitchperf.html
• Continuous Integration status: https://build.opnfv.org/ci/view/vswitchperf/
248
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
VSPERF Developer Guide
Traffic Generator Integration Guide
Intended Audience This document is intended to aid those who want to integrate new traffic generator into the
vsperf code. It is expected, that reader has already read generic part of VSPERF Design Document.
Let us create a sample traffic generator called sample_tg, step by step.
Step 1 - create a directory Implementation of trafficgens is located at tools/pkt_gen/ directory, where every implementation has its dedicated sub-directory. It is required to create a new directory for new traffic generator implementations.
E.g.
$ mkdir tools/pkt_gen/sample_tg
Step 2 - create a trafficgen module Every trafficgen class must inherit from generic ITrafficGenerator interface
class. VSPERF during its initialization scans content of pkt_gen directory for all python modules, that inherit from
ITrafficGenerator. These modules are automatically added into the list of supported traffic generators.
Example:
Let us create a draft of tools/pkt_gen/sample_tg/sample_tg.py module.
from tools.pkt_gen import trafficgen
class SampleTG(trafficgen.ITrafficGenerator):
"""
A sample traffic generator implementation
"""
pass
VSPERF is immediately aware of the new class:
$ ./vsperf --list-trafficgen
Output should look like:
Classes derived from: ITrafficGenerator
======
* Ixia:
A wrapper around the IXIA traffic generator.
* IxNet:
A wrapper around IXIA IxNetwork applications.
* Dummy:
A dummy traffic generator whose data is generated by the user.
* SampleTG:
A sample traffic generator implementation
* TestCenter:
Spirent TestCenter
Step 3 - configuration All configuration values, required for correct traffic generator function, are passed from
VSPERF to the traffic generator in a dictionary. Default values shared among all traffic generators are defined in
conf/03_traffic.conf within TRAFFIC dictionary. Default values are loaded by ITrafficGenerator interface class
2.3. Testing Developer Guides
249
OPNFV Documentation, Release Danube
automatically, so it is not needed to load them explicitly. In case that there are any traffic generator specific default
values, then they should be set within class specific __init__ function.
VSPERF passes test specific configuration within traffic dictionary to every start and send function. So implementation of these functions must ensure, that default values are updated with the testcase specific values. Proper merge of
values is assured by call of merge_spec function from conf module.
Example of merge_spec usage in tools/pkt_gen/sample_tg/sample_tg.py module:
from conf import merge_spec
def start_rfc2544_throughput(self, traffic=None, duration=30):
self._params = {}
self._params['traffic'] = self.traffic_defaults.copy()
if traffic:
self._params['traffic'] = merge_spec(
self._params['traffic'], traffic)
Step 4 - generic functions There are some generic functions, which every traffic generator should provide. Although these functions are mainly optional, at least empty implementation must be provided. This is required, so that
developer is explicitly aware of these functions.
The connect function is called from the traffic generator controller from its __enter__ method. This function should
assure proper connection initialization between DUT and traffic generator. In case, that such implementation is not
needed, empty implementation is required.
The disconnect function should perform clean up of any connection specific actions called from the connect function.
Example in tools/pkt_gen/sample_tg/sample_tg.py module:
def connect(self):
pass
def disconnect(self):
pass
Step 5 - supported traffic types Currently VSPERF supports three different types of tests for traffic generators,
these are identified in vsperf through the traffic type, which include:
• RFC2544 throughput - Send fixed size packets at different rates, using traffic configuration, until minimum rate at which no packet loss is detected is found. Methods with its implementation have suffix
_rfc2544_throughput.
• RFC2544 back2back - Send fixed size packets at a fixed rate, using traffic configuration, for specified time
interval. Methods with its implementation have suffix _rfc2544_back2back.
• continuous flow - Send fixed size packets at given framerate, using traffic configuration, for specified time
interval. Methods with its implementation have suffix _cont_traffic.
In general, both synchronous and asynchronous interfaces must be implemented for each traffic type. Synchronous
functions start with prefix send_. Asynchronous with prefixes start_ and wait_ in case of throughput and back2back
and start_ and stop_ in case of continuous traffic type.
Example of synchronous interfaces:
def send_rfc2544_throughput(self, traffic=None, tests=1, duration=20,
lossrate=0.0):
def send_rfc2544_back2back(self, traffic=None, tests=1, duration=20,
250
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
lossrate=0.0):
def send_cont_traffic(self, traffic=None, duration=20):
Example of asynchronous interfaces:
def start_rfc2544_throughput(self, traffic=None, tests=1, duration=20,
lossrate=0.0):
def wait_rfc2544_throughput(self):
def start_rfc2544_back2back(self, traffic=None, tests=1, duration=20,
lossrate=0.0):
def wait_rfc2544_back2back(self):
def start_cont_traffic(self, traffic=None, duration=20):
def stop_cont_traffic(self):
Description of parameters used by send, start, wait and stop functions:
• param traffic: A dictionary with detailed definition of traffic pattern. It contains following parameters to be
implemented by traffic generator.
Note: Traffic dictionary has also virtual switch related parameters, which are not listed below.
Note: There are parameters specific to testing of tunnelling protocols, which are discussed in detail at Integration
tests userguide.
– param traffic_type: One of the supported traffic types, e.g. rfc2544_throughput, rfc2544_continuous or
rfc2544_back2back.
– param frame_rate: Defines desired percentage of frame rate used during continuous stream tests.
– param bidir: Specifies if generated traffic will be full-duplex (true) or half-duplex (false).
– param multistream: Defines number of flows simulated by traffic generator. Value 0 disables MultiStream
feature.
– param stream_type: Stream Type defines ISO OSI network layer used for simulation of multiple streams.
Supported values:
* L2 - iteration of destination MAC address
* L3 - iteration of destination IP address
* L4 - iteration of destination port of selected transport protocol
– param l2: A dictionary with data link layer details, e.g. srcmac, dstmac and framesize.
– param l3: A dictionary with network layer details, e.g. srcip, dstip and proto.
– param l3: A dictionary with transport layer details, e.g. srcport, dstport.
– param vlan: A dictionary with vlan specific parameters, e.g. priority, cfi, id and vlan on/off switch
enabled.
• param tests: Number of times the test is executed.
• param duration: Duration of continuous test or per iteration duration in case of RFC2544 throughput or
back2back traffic types.
• param lossrate: Acceptable lossrate percentage.
2.3. Testing Developer Guides
251
OPNFV Documentation, Release Danube
Step 6 - passing back results It is expected that methods send, wait and stop will return values measured by traffic generator within a dictionary. Dictionary keys are defined in ResultsConstants implemented
in core/results/results_constants.py. Please check sections for RFC2544 Throughput & Continuous and for
Back2Back. The same key names should be used by all traffic generator implementations.
VSPERF Design Document
Intended Audience This document is intended to aid those who want to modify the vsperf code. Or to extend it for example to add support for new traffic generators, deployment scenarios and so on.
Usage
Example Connectivity to DUT Establish connectivity to the VSPERF DUT Linux host. If this is in an OPNFV lab
following the steps provided by Pharos
to access the POD
The followign steps establish the VSPERF environment.
Example Command Lines List all the cli options:
$ ./vsperf -h
Run all tests that have tput in their name - phy2phy_tput, pvp_tput etc.:
$ ./vsperf --tests 'tput'
As above but override default configuration with settings in ‘10_custom.conf’. This is useful as modifying configuration directly in the configuration files in conf/NN_*.py shows up as changes under git source control:
$ ./vsperf --conf-file=<path_to_custom_conf>/10_custom.conf --tests 'tput'
Override specific test parameters. Useful for shortening the duration of tests for development purposes:
$ ./vsperf --test-params 'TRAFFICGEN_DURATION=10;TRAFFICGEN_RFC2544_TESTS=1;' \
'TRAFFICGEN_PKT_SIZES=(64,)' pvp_tput
Typical Test Sequence This is a typical flow of control for a test.
252
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
Configuration The conf package contains the configuration files (*.conf) for all system components, it also provides a settings object that exposes all of these settings.
Settings are not passed from component to component. Rather they are available globally to all components once they
import the conf package.
2.3. Testing Developer Guides
253
OPNFV Documentation, Release Danube
from conf import settings
...
log_file = settings.getValue('LOG_FILE_DEFAULT')
Settings files (*.conf) are valid python code so can be set to complex types such as lists and dictionaries as well as
scalar types:
first_packet_size = settings.getValue('PACKET_SIZE_LIST')[0]
Configuration Procedure and Precedence Configuration files follow a strict naming convention that allows them
to be processed in a specific order. All the .conf files are named NN_name.conf, where NN is a decimal number.
The files are processed in order from 00_name.conf to 99_name.conf so that if the name setting is given in both a
lower and higher numbered conf file then the higher numbered file is the effective setting as it is processed after the
setting in the lower numbered file.
The values in the file specified by --conf-file takes precedence over all the other configuration files and does not
have to follow the naming convention.
Configuration of PATHS dictionary VSPERF uses external tools like Open vSwitch and Qemu for execution of
testcases. These tools may be downloaded and built automatically (see Installation) or installed manually by user from
binary packages. It is also possible to use a combination of both approaches, but it is essential to correctly set paths
to all required tools. These paths are stored within a PATHS dictionary, which is evaluated before execution of each
testcase, in order to setup testcase specific environment. Values selected for testcase execution are internally stored
inside TOOLS dictionary, which is used by VSPERF to execute external tools, load kernel modules, etc.
The default configuration of PATHS dictionary is spread among three different configuration files to follow logical grouping of configuration options.
Basic description of PATHS dictionary is placed inside conf/00_common.conf.
The configuration specific to DPDK and vswitches is located at
conf/02_vswitch.conf. The last part related to the Qemu is defined inside conf/04_vnf.conf. Default
configuration values can be used in case, that all required tools were downloaded and built automatically by vsperf
itself. In case, that some of tools were installed manually from binary packages, then it will be necessary to modify
the content of PATHS dictionary accordingly.
Dictionary has a specific section of configuration options for every tool type, it means:
• PATHS[’vswitch’] - contains a separete dictionary for each of vswitches supported by VSPEF
Example:
PATHS['vswitch'] = {
'OvsDpdkVhost': { ... },
'OvsVanilla' : { ... },
...
}
• PATHS[’dpdk’] - contains paths to the dpdk sources, kernel modules and tools (e.g. testpmd)
Example:
PATHS['dpdk'] = {
'type' : 'src',
'src': {
'path': os.path.join(ROOT_DIR, 'src/dpdk/dpdk/'),
'modules' : ['uio', os.path.join(RTE_TARGET, 'kmod/igb_uio.ko')],
'bind-tool': 'tools/dpdk*bind.py',
'testpmd': os.path.join(RTE_TARGET, 'app', 'testpmd'),
},
254
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
...
}
• PATHS[’qemu’] - contains paths to the qemu sources and executable file
Example:
PATHS['qemu'] = {
'type' : 'bin',
'bin': {
'qemu-system': 'qemu-system-x86_64'
},
...
}
Every section specific to the particular vswitch, dpdk or qemu may contain following types of configuration options:
• option type - is a string, which defines the type of configured paths (‘src’ or ‘bin’) to be selected for a given
section:
– value src means, that VSPERF will use vswitch, DPDK or QEMU built from sources e.g. by execution
of systems/build_base_machine.sh script during VSPERF installation
– value bin means, that VSPERF will use vswitch, DPDK or QEMU binaries installed directly in the
operating system, e.g. via OS specific packaging system
• option path - is a string with a valid system path; Its content is checked for existence, prefixed with section
name and stored into TOOLS for later use e.g. TOOLS[’dpdk_src’] or TOOLS[’vswitch_src’]
• option modules - is list of strings with names of kernel modules; Every module name from given list is checked
for a ‘.ko’ suffix. In case that it matches and if it is not an absolute path to the module, then module name is
prefixed with value of path option defined for the same section
Example:
"""
snippet of PATHS definition from the configuration file:
"""
PATHS['vswitch'] = {
'OvsVanilla' = {
'type' : 'src',
'src': {
'path': '/tmp/vsperf/src_vanilla/ovs/ovs/',
'modules' : ['datapath/linux/openvswitch.ko'],
...
},
...
}
...
}
"""
Final content of TOOLS dictionary used during runtime:
"""
TOOLS['vswitch_modules'] = ['/tmp/vsperf/src_vanilla/ovs/ovs/datapath/linux/openvswitch.ko']
• all other options are strings with names and paths to specific tools; If a given string contains a relative path
and option path is defined for a given section, then string content will be prefixed with content of the path.
Otherwise the name of the tool will be searched within standard system directories. In case that filename
contains OS specific wildcards, then they will be expanded to the real path. At the end of the processing, every
absolute path will be checked for its existence. In case that temporary path (i.e. path with a _tmp suffix) does
2.3. Testing Developer Guides
255
OPNFV Documentation, Release Danube
not exist, then log will be written and vsperf will continue. If any other path will not exist, then vsperf execution
will be terminated with a runtime error.
Example:
"""
snippet of PATHS definition from the configuration file:
"""
PATHS['vswitch'] = {
'OvsDpdkVhost': {
'type' : 'src',
'src': {
'path': '/tmp/vsperf/src_vanilla/ovs/ovs/',
'ovs-vswitchd': 'vswitchd/ovs-vswitchd',
'ovsdb-server': 'ovsdb/ovsdb-server',
...
}
...
}
...
}
"""
Final content of TOOLS dictionary used during runtime:
"""
TOOLS['ovs-vswitchd'] = '/tmp/vsperf/src_vanilla/ovs/ovs/vswitchd/ovs-vswitchd'
TOOLS['ovsdb-server'] = '/tmp/vsperf/src_vanilla/ovs/ovs/ovsdb/ovsdb-server'
Note: In case that bin type is set for DPDK, then TOOLS[’dpdk_src’] will be set to the value of
PATHS[’dpdk’][’src’][’path’]. The reason is, that VSPERF uses downloaded DPDK sources to copy
DPDK and testpmd into the GUEST, where testpmd is built. In case, that DPDK sources are not available, then vsperf
will continue with test execution, but testpmd can’t be used as a guest loopback. This is useful in case, that other guest
loopback applications (e.g. buildin or l2fwd) are used.
Note: In case of RHEL 7.3 OS usage, binary package configuration is required for Vanilla OVS tests. With the
installation of a supported rpm for OVS there is a section in the conf\10_custom.conf file that can be used.
Configuration of TRAFFIC dictionary TRAFFIC dictionary is used for configuration of traffic generator. Default
values can be found in configuration file conf/03_traffic.conf. These default values can be modified by (first
option has the highest priorty):
1. Parameters section of testcase defintion
2. command line options specified by --test-params argument
3. custom configuration file
It is to note, that in case of option 1 and 2, it is possible to specify only values, which should be changed. In case of
custom configuration file, it is required to specify whole TRAFFIC dictionary with its all values or explicitly call and
update() method of TRAFFIC dictionary.
Detailed description of TRAFFIC dictionary items follows:
'traffic_type'
'bidir'
256
- One of the supported traffic types.
E.g. rfc2544_throughput, rfc2544_back2back
or rfc2544_continuous
Data type: str
Default value: "rfc2544_throughput".
- Specifies if generated traffic will be full-duplex (True)
or half-duplex (False)
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
Data type: str
Supported values: "True", "False"
Default value: "False".
'frame_rate'
- Defines desired percentage of frame rate used during
continuous stream tests.
Data type: int
Default value: 100.
'multistream'
- Defines number of flows simulated by traffic generator.
Value 0 disables multistream feature
Data type: int
Supported values: 0-65535
Default value: 0.
'stream_type'
- Stream type is an extension of the "multistream" feature.
If multistream is disabled, then stream type will be
ignored. Stream type defines ISO OSI network layer used
for simulation of multiple streams.
Data type: str
Supported values:
"L2" - iteration of destination MAC address
"L3" - iteration of destination IP address
"L4" - iteration of destination port
of selected transport protocol
Default value: "L4".
'pre_installed_flows'
- Pre-installed flows is an extension of the "multistream"
feature. If enabled, it will implicitly insert a flow
for each stream. If multistream is disabled, then
pre-installed flows will be ignored.
Note: It is supported only for p2p deployment scenario.
Data type: str
Supported values:
"Yes" - flows will be inserted into OVS
"No" - flows won't be inserted into OVS
Default value: "No".
'flow_type'
- Defines flows complexity.
Data type: str
Supported values:
"port" - flow is defined by ingress ports
"IP"
- flow is defined by ingress ports
and src and dst IP addresses
Default value: "port"
'l2'
- A dictionary with l2 network layer details. Supported
values are:
'srcmac'
- Specifies source MAC address filled by traffic generator.
NOTE: It can be modified by vsperf in some scenarios.
Data type: str
Default value: "00:00:00:00:00:00".
'dstmac'
- Specifies destination MAC address filled by traffic generator.
NOTE: It can be modified by vsperf in some scenarios.
Data type: str
Default value: "00:00:00:00:00:00".
'framesize' - Specifies default frame size. This value should not be
changed directly. It will be overridden during testcase
execution by values specified by list TRAFFICGEN_PKT_SIZES.
Data type: int
Default value: 64
'l3'
- A dictionary with l3 network layer details. Supported
values are:
2.3. Testing Developer Guides
257
OPNFV Documentation, Release Danube
'srcip'
'dstip'
'proto'
'l4'
'srcport'
'dstport'
'vlan'
'enabled'
'id'
'priority'
'cfi'
- Specifies source MAC address filled by traffic generator.
NOTE: It can be modified by vsperf in some scenarios.
Data type: str
Default value: "1.1.1.1".
- Specifies destination MAC address filled by traffic generator.
NOTE: It can be modified by vsperf in some scenarios.
Data type: str
Default value: "90.90.90.90".
- Specifies deflaut protocol type.
Please check particular traffic generator implementation
for supported protocol types.
Data type: str
Default value: "udp".
- A dictionary with l4 network layer details. Supported
values are:
- Specifies source port of selected transport protocol.
NOTE: It can be modified by vsperf in some scenarios.
Data type: int
Default value: 3000
- Specifies destination port of selected transport protocol.
NOTE: It can be modified by vsperf in some scenarios.
Data type: int
Default value: 3001
- A dictionary with vlan encapsulation details. Supported
values are:
- Specifies if vlan encapsulation should be enabled or
disabled.
Data type: bool
Default value: False
- Specifies vlan id.
Data type: int (NOTE: must fit to 12 bits)
Default value: 0
- Specifies a vlan priority (PCP header field).
Data type: int (NOTE: must fit to 3 bits)
Default value: 0
- Specifies if frames can or cannot be dropped during
congestion (DEI header field).
Data type: int (NOTE: must fit to 1 bit)
Default value: 0
Configuration of GUEST options VSPERF is able to setup scenarios involving a number of VMs in series or in
parallel. All configuration options related to a particular VM instance are defined as lists and prefixed with GUEST_
label. It is essential, that there is enough items in all GUEST_ options to cover all VM instances involved in the test.
In case there is not enough items, then VSPERF will use the first item of particular GUEST_ option to expand the list
to required length.
Example of option expansion for 4 VMs:
"""
Original values:
"""
GUEST_SMP = ['2']
GUEST_MEMORY = ['2048', '4096']
"""
Values after automatic expansion:
"""
258
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
GUEST_SMP = ['2', '2', '2', '2']
GUEST_MEMORY = ['2048', '4096', '2048', '2048']
First option can contain macros starting with # to generate VM specific values. These macros can be used only for
options of list or str types with GUEST_ prefix.
Example of macros and their expnasion for 2 VMs:
"""
Original values:
"""
GUEST_SHARE_DIR = ['/tmp/qemu#VMINDEX_share']
GUEST_BRIDGE_IP = ['#IP(1.1.1.5)/16']
"""
Values after automatic expansion:
"""
GUEST_SHARE_DIR = ['/tmp/qemu0_share', '/tmp/qemu1_share']
GUEST_BRIDGE_IP = ['1.1.1.5/16', '1.1.1.6/16']
Additional examples are available at 04_vnf.conf.
Note: In case, that macro is detected in the first item of the list, then all other items are ignored and list content is
created automatically.
Multiple macros can be used inside one configuration option definition, but macros cannot be used inside other macros.
The only exception is macro #VMINDEX, which is expanded first and thus it can be used inside other macros.
Following macros are supported:
• #VMINDEX - it is replaced by index of VM being executed; This macro is expanded first, so it can be used
inside other macros.
Example:
GUEST_SHARE_DIR = ['/tmp/qemu#VMINDEX_share']
• #MAC(mac_address[, step]) - it will iterate given mac_address with optional step. In case that
step is not defined, then it is set to 1. It means, that first VM will use the value of mac_address, second VM
value of mac_address increased by step, etc.
Example:
GUEST_NICS = [[{'mac' : '#MAC(00:00:00:00:00:01,2)'}]]
• #IP(ip_address[, step]) - it will iterate given ip_address with optional step. In case that step is
not defined, then it is set to 1. It means, that first VM will use the value of ip_address, second VM value of
ip_address increased by step, etc.
Example:
GUEST_BRIDGE_IP = ['#IP(1.1.1.5)/16']
• #EVAL(expression) - it will evaluate given expression as python code; Only simple expressions should
be used. Call of the functions is not supported.
Example:
GUEST_CORE_BINDING = [('#EVAL(6+2*#VMINDEX)', '#EVAL(7+2*#VMINDEX)')]
Other Configuration conf.settings also loads configuration from the command line and from the environment.
2.3. Testing Developer Guides
259
OPNFV Documentation, Release Danube
PXP Deployment Every testcase uses one of the supported deployment scenarios to setup test environment. The
controller responsible for a given scenario configures flows in the vswitch to route traffic among physical interfaces
connected to the traffic generator and virtual machines. VSPERF supports several deployments including PXP deployment, which can setup various scenarios with multiple VMs.
These scenarios are realized by VswitchControllerPXP class, which can configure and execute given number of VMs
in serial or parallel configurations. Every VM can be configured with just one or an even number of interfaces. In case
that VM has more than 2 interfaces, then traffic is properly routed among pairs of interfaces.
Example of traffic routing for VM with 4 NICs in serial configuration:
+------------------------------------------+
| VM with 4 NICs
|
| +---------------+
+---------------+ |
| | Application |
| Application | |
| +---------------+
+---------------+ |
|
^
|
^
|
|
|
|
v
|
v
|
| +---------------+
+---------------+ |
| | logical ports |
| logical ports | |
| |
0
1
|
|
2
3
| |
+--+---------------+----+---------------+--+
^
:
^
:
|
|
|
|
:
v
:
v
+-----------+---------------+----+---------------+----------+
| vSwitch
|
0
1
|
|
2
3
|
|
|
| logical ports |
| logical ports |
|
| previous +---------------+
+---------------+
next
|
| VM or PHY
^
|
^
|
VM or PHY|
|
port
-----+
+------------+
+--->
port
|
+-----------------------------------------------------------+
It is also possible to define different number of interfaces for each VM to better simulate real scenarios.
Example of traffic routing for 2 VMs in serial configuration, where 1st VM has 4 NICs and 2nd VM 2 NICs:
+------------------------------------------+ +---------------------+
| 1st VM with 4 NICs
| | 2nd VM with 2 NICs |
| +---------------+
+---------------+ | | +---------------+ |
| | Application |
| Application | | | | Application | |
| +---------------+
+---------------+ | | +---------------+ |
|
^
|
^
|
| |
^
|
|
|
|
v
|
v
| |
|
v
|
| +---------------+
+---------------+ | | +---------------+ |
| | logical ports |
| logical ports | | | | logical ports | |
| |
0
1
|
|
2
3
| | | |
0
1
| |
+--+---------------+----+---------------+--+ +--+---------------+--+
^
:
^
:
^
:
|
|
|
|
|
|
:
v
:
v
:
v
+-----------+---------------+----+---------------+-------+---------------+----------+
| vSwitch
|
0
1
|
|
2
3
|
|
4
5
|
|
|
| logical ports |
| logical ports |
| logical ports |
|
| previous +---------------+
+---------------+
+---------------+
next
|
| VM or PHY
^
|
^
|
^
|
VM or PHY|
|
port
-----+
+------------+
+---------------+
+----> port
|
+-----------------------------------------------------------------------------------+
The number of VMs involved in the test and the type of their connection is defined by deployment name as follows:
260
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
• pvvp[number] - configures scenario with VMs connected in series with optional number of VMs. In case
that number is not specified, then 2 VMs will be used.
Example of 2 VMs in a serial configuration:
+----------------------+ +----------------------+
|
1st VM
| |
2nd VM
|
|
+---------------+ | |
+---------------+ |
|
| Application | | |
| Application | |
|
+---------------+ | |
+---------------+ |
|
^
|
| |
^
|
|
|
|
v
| |
|
v
|
|
+---------------+ | |
+---------------+ |
|
| logical ports | | |
| logical ports | |
|
|
0
1
| | |
|
0
1
| |
+---+---------------+--+ +---+---------------+--+
^
:
^
:
|
|
|
|
:
v
:
v
+---+---------------+---------+---------------+--+
|
|
0
1
|
|
3
4
| |
|
| logical ports | vSwitch | logical ports | |
|
+---------------+
+---------------+ |
|
^
|
^
|
|
|
|
+-----------------+
v
|
|
+----------------------------------------+
|
|
|
physical ports
|
|
|
|
0
1
|
|
+---+----------------------------------------+---+
^
:
|
|
:
v
+------------------------------------------------+
|
|
|
traffic generator
|
|
|
+------------------------------------------------+
• pvpv[number] - configures scenario with VMs connected in parallel with optional number of VMs. In case
that number is not specified, then 2 VMs will be used. Multistream feature is used to route traffic to particular
VMs (or NIC pairs of every VM). It means, that VSPERF will enable multistream feaure and sets the number
of streams to the number of VMs and their NIC pairs. Traffic will be dispatched based on Stream Type, i.e. by
UDP port, IP address or MAC address.
Example of 2 VMs in a parallel configuration, where traffic is dispatched based on the UDP port.
+----------------------+
|
1st VM
|
|
+---------------+ |
|
| Application | |
|
+---------------+ |
|
^
|
|
|
|
v
|
|
+---------------+ |
|
| logical ports | |
|
|
0
1
| |
+---+---------------+--+
^
:
|
|
:
v
2.3. Testing Developer Guides
+----------------------+
|
2nd VM
|
|
+---------------+ |
|
| Application | |
|
+---------------+ |
|
^
|
|
|
|
v
|
|
+---------------+ |
|
| logical ports | |
|
|
0
1
| |
+---+---------------+--+
^
:
|
|
:
v
261
OPNFV Documentation, Release Danube
+---+---------------+---------+---------------+--+
|
|
0
1
|
|
3
4
| |
|
| logical ports | vSwitch | logical ports | |
|
+---------------+
+---------------+ |
|
^
|
^
:
|
|
|
......................:
:
|
| UDP | UDP :
|
:
|
| port| port:
+--------------------+
:
|
|
0 | 1 :
|
:
|
|
|
:
v
v
|
|
+----------------------------------------+
|
|
|
physical ports
|
|
|
|
0
1
|
|
+---+----------------------------------------+---+
^
:
|
|
:
v
+------------------------------------------------+
|
|
|
traffic generator
|
|
|
+------------------------------------------------+
PXP deployment is backward compatible with PVP deployment, where pvp is an alias for pvvp1 and it executes just
one VM.
The number of interfaces used by VMs is defined by configuration option GUEST_NICS_NR. In case that more than
one pair of interfaces is defined for VM, then:
• for pvvp (serial) scenario every NIC pair is connected in serial before connection to next VM is created
• for pvpv (parallel) scenario every NIC pair is directly connected to the physical ports and unique traffic stream
is assigned to it
Examples:
• Deployment pvvp10 will start 10 VMs and connects them in series
• Deployment pvpv4 will start 4 VMs and connects them in parallel
• Deployment pvpv1 and GUEST_NICS_NR = [4] will start 1 VM with 4 interfaces and every NIC pair is
directly connected to the physical ports
• Deployment pvvp and GUEST_NICS_NR = [2, 4] will start 2 VMs; 1st VM will have 2 interfaces and 2nd
VM 4 interfaces. These interfaces will be connected in serial, i.e. traffic will flow as follows: PHY1 -> VM1_1
-> VM1_2 -> VM2_1 -> VM2_2 -> VM2_3 -> VM2_4 -> PHY2
Note: In case that only 1 or more than 2 NICs are configured for VM, then testpmd should be used as forwarding
application inside the VM. As it is able to forward traffic between multiple VM NIC pairs.
Note: In case of linux_bridge, all NICs are connected to the same bridge inside the VM.
VM, vSwitch, Traffic Generator Independence VSPERF supports different vSwithes, Traffic Generators, VNFs
and Forwarding Applications by using standard object-oriented polymorphism:
• Support for vSwitches is implemented by a class inheriting from IVSwitch.
• Support for Traffic Generators is implemented by a class inheriting from ITrafficGenerator.
• Support for VNF is implemented by a class inheriting from IVNF.
• Support for Forwarding Applications is implemented by a class inheriting from IPktFwd.
262
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
By dealing only with the abstract interfaces the core framework can support many implementations of different
vSwitches, Traffic Generators, VNFs and Forwarding Applications.
IVSwitch
class IVSwitch:
start(self)
stop(self)
add_switch(switch_name)
del_switch(switch_name)
add_phy_port(switch_name)
add_vport(switch_name)
get_ports(switch_name)
del_port(switch_name, port_name)
add_flow(switch_name, flow)
del_flow(switch_name, flow=None)
ITrafficGenerator
class ITrafficGenerator:
connect()
disconnect()
send_burst_traffic(traffic, numpkts, time, framerate)
send_cont_traffic(traffic, time, framerate)
start_cont_traffic(traffic, time, framerate)
stop_cont_traffic(self):
send_rfc2544_throughput(traffic, tests, duration, lossrate)
start_rfc2544_throughput(traffic, tests, duration, lossrate)
wait_rfc2544_throughput(self)
send_rfc2544_back2back(traffic, tests, duration, lossrate)
start_rfc2544_back2back(traffic, , tests, duration, lossrate)
wait_rfc2544_back2back()
Note send_xxx() blocks whereas start_xxx() does not and must be followed by a subsequent call to
wait_xxx().
IVnf
class IVnf:
start(memory, cpus,
monitor_path, shared_path_host,
shared_path_guest, guest_prompt)
stop()
execute(command)
wait(guest_prompt)
execute_and_wait (command)
IPktFwd
class IPktFwd:
start()
stop()
2.3. Testing Developer Guides
263
OPNFV Documentation, Release Danube
Controllers Controllers are used in conjunction with abstract interfaces as way of decoupling the control of
vSwtiches, VNFs, TrafficGenerators and Forwarding Applications from other components.
The controlled classes provide basic primitive operations. The Controllers sequence and co-ordinate these primitive
operation in to useful actions. For instance the vswitch_controller_p2p can be used to bring any vSwitch (that implements the primitives defined in IVSwitch) into the configuration required by the Phy-to-Phy Deployment Scenario.
In order to support a new vSwitch only a new implementation of IVSwitch needs be created for the new vSwitch to be
capable of fulfilling all the Deployment Scenarios provided for by existing or future vSwitch Controllers.
Similarly if a new Deployment Scenario is required it only needs to be written once as a new vSwitch Controller and
it will immediately be capable of controlling all existing and future vSwitches in to that Deployment Scenario.
Similarly the Traffic Controllers can be used to co-ordinate basic operations provided by implementers of ITrafficGenerator to provide useful tests. Though traffic generators generally already implement full test cases i.e. they both
generate suitable traffic and analyse returned traffic in order to implement a test which has typically been predefined
in an RFC document. However the Traffic Controller class allows for the possibility of further enhancement - such as
iterating over tests for various packet sizes or creating new tests.
Traffic Controller’s Role
264
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
Loader & Component Factory The working of the Loader package (which is responsible for finding arbitrary
classes based on configuration data) and the Component Factory which is responsible for choosing the correct class
for a particular situation - e.g. Deployment Scenario can be seen in this diagram.
Routing Tables Vsperf uses a standard set of routing tables in order to allow tests to easily mix and match Deployment Scenarios (PVP, P2P topology), Tuple Matching and Frame Modification requirements.
+--------------+
|
|
| Table 0
|
|
|
|
|
+--------------+
|
|
v
+--------------+
|
|
| Table 1
|
|
|
|
|
+--------------+
|
|
v
+--------------+
|
|
| Table 2
|
|
|
|
|
|
|
+--------------+
|
|
v
+--------------+
|
|
| Table 3
|
|
|
+--------------+
table#0 - Match table. Flows designed to force 5 & 10
tuple matches go here.
table#1 - Routing table. Flow entries to forward
packets between ports goes here.
The chosen port is communicated to subsequent tables by
setting the metadata value to the egress port number.
Generally this table is set-up by by the
vSwitchController.
table#2 - Frame modification table. Frame modification
flow rules are isolated in this table so that they can
be turned on or off without affecting the routing or
tuple-matching flow rules. This allows the frame
modification and tuple matching required by the tests
in the VSWITCH PERFORMANCE FOR TELCO NFV test
specification to be independent of the Deployment
Scenario set up by the vSwitchController.
table#3 - Egress table. Egress packets on the ports
setup in Table 1.
VSPERF LEVEL TEST DESIGN (LTD)
2.3. Testing Developer Guides
265
OPNFV Documentation, Release Danube
Introduction The intention of this Level Test Design (LTD) document is to specify the set of tests to carry out
in order to objectively measure the current characteristics of a virtual switch in the Network Function Virtualization
Infrastructure (NFVI) as well as the test pass criteria. The detailed test cases will be defined in details-of-LTD,
preceded by the doc-id-of-LTD and the scope-of-LTD.
This document is currently in draft form.
Document identifier The document id will be used to uniquely identify versions of the LTD. The format for the document id will be: OPNFV_vswitchperf_LTD_REL_STATUS, where by the status is one of: draft, reviewed, corrected
or final. The document id for this version of the LTD is: OPNFV_vswitchperf_LTD_Brahmaputra_REVIEWED.
Scope The main purpose of this project is to specify a suite of performance tests in order to objectively measure
the current packet transfer characteristics of a virtual switch in the NFVI. The intent of the project is to facilitate
testing of any virtual switch. Thus, a generic suite of tests shall be developed, with no hard dependencies to a single
implementation. In addition, the test case suite shall be architecture independent.
The test cases developed in this project shall not form part of a separate test framework, all of these tests may be
inserted into the Continuous Integration Test Framework and/or the Platform Functionality Test Framework - if a
vSwitch becomes a standard component of an OPNFV release.
References
• RFC 1242 Benchmarking Terminology for Network Interconnection Devices
• RFC 2544 Benchmarking Methodology for Network Interconnect Devices
• RFC 2285 Benchmarking Terminology for LAN Switching Devices
• RFC 2889 Benchmarking Methodology for LAN Switching Devices
• RFC 3918 Methodology for IP Multicast Benchmarking
• RFC 4737 Packet Reordering Metrics
• RFC 5481 Packet Delay Variation Applicability Statement
• RFC 6201 Device Reset Characterization
Details of the Level Test Design This section describes the features to be tested (FeaturesToBeTested-of-LTD), and
identifies the sets of test cases or scenarios (TestIdentification-of-LTD).
Features to be tested Characterizing virtual switches (i.e. Device Under Test (DUT) in this document) includes
measuring the following performance metrics:
• Throughput
• Packet delay
• Packet delay variation
• Packet loss
• Burst behaviour
• Packet re-ordering
• Packet correctness
• Availability and capacity of the DUT
266
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
Test identification
Throughput tests The following tests aim to determine the maximum forwarding rate that can be achieved with a
virtual switch. The list is not exhaustive but should indicate the type of tests that should be required. It is expected
that more will be added.
Test ID: LTD.Throughput.RFC2544.PacketLossRatio
Title: RFC 2544 X% packet loss ratio Throughput and Latency Test
Prerequisite Test: N/A
Priority:
Description:
This test determines the DUT’s maximum forwarding rate with X% traffic loss for a constant load (fixed
length frames at a fixed interval time). The default loss percentages to be tested are: - X = 0% - X =
10^-7%
Note: Other values can be tested if required by the user.
The selected frame sizes are those previously defined under Default Test Parameters. The test can also be
used to determine the average latency of the traffic.
Under the RFC2544 test methodology, the test duration will include a number of trials; each trial should
run for a minimum period of 60 seconds. A binary search methodology must be applied for each trial to
obtain the final result.
Expected Result: At the end of each trial, the presence or absence of loss determines the modification
of offered load for the next trial, converging on a maximum rate, or RFC2544 Throughput with X% loss.
The Throughput load is re-used in related RFC2544 tests and other tests.
Metrics Collected:
The following are the metrics collected for this test:
• The maximum forwarding rate in Frames Per Second (FPS) and Mbps of the DUT for each frame
size with X% packet loss.
• The average latency of the traffic flow when passing through the DUT (if testing for latency, note
that this average is different from the test specified in Section 26.3 of RFC2544).
• CPU and memory utilization may also be collected as part of this test, to determine the vSwitch’s
performance footprint on the system.
Test ID: LTD.Throughput.RFC2544.PacketLossRatioFrameModification
Title: RFC 2544 X% packet loss Throughput and Latency Test with packet modification
Prerequisite Test: N/A
Priority:
Description:
This test determines the DUT’s maximum forwarding rate with X% traffic loss for a constant load (fixed
length frames at a fixed interval time). The default loss percentages to be tested are: - X = 0% - X =
10^-7%
Note: Other values can be tested if required by the user.
2.3. Testing Developer Guides
267
OPNFV Documentation, Release Danube
The selected frame sizes are those previously defined under Default Test Parameters. The test can also be
used to determine the average latency of the traffic.
Under the RFC2544 test methodology, the test duration will include a number of trials; each trial should
run for a minimum period of 60 seconds. A binary search methodology must be applied for each trial to
obtain the final result.
During this test, the DUT must perform the following operations on the traffic flow:
• Perform packet parsing on the DUT’s ingress port.
• Perform any relevant address look-ups on the DUT’s ingress ports.
• Modify the packet header before forwarding the packet to the DUT’s egress port. Packet modifications include:
– Modifying the Ethernet source or destination MAC address.
– Modifying/adding a VLAN tag. (Recommended).
– Modifying/adding a MPLS tag.
– Modifying the source or destination ip address.
– Modifying the TOS/DSCP field.
– Modifying the source or destination ports for UDP/TCP/SCTP.
– Modifying the TTL.
Expected Result: The Packet parsing/modifications require some additional degree of processing resource, therefore the RFC2544 Throughput is expected to be somewhat lower than the Throughput level
measured without additional steps. The reduction is expected to be greatest on tests with the smallest
packet sizes (greatest header processing rates).
Metrics Collected:
The following are the metrics collected for this test:
• The maximum forwarding rate in Frames Per Second (FPS) and Mbps of the DUT for each frame
size with X% packet loss and packet modification operations being performed by the DUT.
• The average latency of the traffic flow when passing through the DUT (if testing for latency, note
that this average is different from the test specified in Section 26.3 of RFC2544).
• The RFC5481 PDV form of delay variation on the traffic flow, using the 99th percentile.
• CPU and memory utilization may also be collected as part of this test, to determine the vSwitch’s
performance footprint on the system.
Test ID: LTD.Throughput.RFC2544.Profile
Title: RFC 2544 Throughput and Latency Profile
Prerequisite Test: N/A
Priority:
Description:
This test reveals how throughput and latency degrades as the offered rate varies in the region of the
DUT’s maximum forwarding rate as determined by LTD.Throughput.RFC2544.PacketLossRatio (0%
Packet Loss). For example it can be used to determine if the degradation of throughput and latency
as the offered rate increases is slow and graceful or sudden and severe.
The selected frame sizes are those previously defined under Default Test Parameters.
268
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
The offered traffic rate is described as a percentage delta with respect to the DUT’s RFC 2544 Throughput
as determined by LTD.Throughput.RFC2544.PacketLoss Ratio (0% Packet Loss case). A delta of 0% is
equivalent to an offered traffic rate equal to the RFC 2544 Maximum Throughput; A delta of +50%
indicates an offered rate half-way between the Maximum RFC2544 Throughput and line-rate, whereas a
delta of -50% indicates an offered rate of half the RFC 2544 Maximum Throughput. Therefore the range
of the delta figure is natuarlly bounded at -100% (zero offered traffic) and +100% (traffic offered at line
rate).
The following deltas to the maximum forwarding rate should be applied:
• -50%, -10%, 0%, +10% & +50%
Expected Result: For each packet size a profile should be produced of how throughput and latency vary
with offered rate.
Metrics Collected:
The following are the metrics collected for this test:
• The forwarding rate in Frames Per Second (FPS) and Mbps of the DUT for each delta to the maximum forwarding rate and for each frame size.
• The average latency for each delta to the maximum forwarding rate and for each frame size.
• CPU and memory utilization may also be collected as part of this test, to determine the vSwitch’s
performance footprint on the system.
• Any failures experienced (for example if the vSwitch crashes, stops processing packets, restarts or
becomes unresponsive to commands) when the offered load is above Maximum Throughput MUST
be recorded and reported with the results.
Test ID: LTD.Throughput.RFC2544.SystemRecoveryTime
Title: RFC 2544 System Recovery Time Test
Prerequisite Test LTD.Throughput.RFC2544.PacketLossRatio
Priority:
Description:
The aim of this test is to determine the length of time it takes the DUT to recover from an overload
condition for a constant load (fixed length frames at a fixed interval time). The selected frame sizes are
those previously defined under Default Test Parameters, traffic should be sent to the DUT under normal
conditions. During the duration of the test and while the traffic flows are passing though the DUT, at
least one situation leading to an overload condition for the DUT should occur. The time from the end of
the overload condition to when the DUT returns to normal operations should be measured to determine
recovery time. Prior to overloading the DUT, one should record the average latency for 10,000 packets
forwarded through the DUT.
The overload condition SHOULD be to transmit traffic at a very high frame rate to the DUT (150% of the
maximum 0% packet loss rate as determined by LTD.Throughput.RFC2544.PacketLossRatio or line-rate
whichever is lower), for at least 60 seconds, then reduce the frame rate to 75% of the maximum 0% packet
loss rate. A number of time-stamps should be recorded: - Record the time-stamp at which the frame rate
was reduced and record a second time-stamp at the time of the last frame lost. The recovery time is the
difference between the two timestamps. - Record the average latency for 10,000 frames after the last
frame loss and continue to record average latency measurements for every 10,000 frames, when latency
returns to within 10% of pre-overload levels record the time-stamp.
Expected Result:
Metrics collected
2.3. Testing Developer Guides
269
OPNFV Documentation, Release Danube
The following are the metrics collected for this test:
• The length of time it takes the DUT to recover from an overload condition.
• The length of time it takes the DUT to recover the average latency to pre-overload conditions.
Deployment scenario:
• Physical → virtual switch → physical.
Test ID: LTD.Throughput.RFC2544.BackToBackFrames
Title: RFC2544 Back To Back Frames Test
Prerequisite Test: N
Priority:
Description:
The aim of this test is to characterize the ability of the DUT to process back-to-back frames. For each
frame size previously defined under Default Test Parameters, a burst of traffic is sent to the DUT with the
minimum inter-frame gap between each frame. If the number of received frames equals the number of
frames that were transmitted, the burst size should be increased and traffic is sent to the DUT again. The
value measured is the back-to-back value, that is the maximum burst size the DUT can handle without
any frame loss. Please note a trial must run for a minimum of 2 seconds and should be repeated 50 times
(at a minimum).
Expected Result:
Tests of back-to-back frames with physical devices have produced unstable results in some cases. All
tests should be repeated in multiple test sessions and results stability should be examined.
Metrics collected
The following are the metrics collected for this test:
• The average back-to-back value across the trials, which is the number of frames in the longest burst
that the DUT will handle without the loss of any frames.
• CPU and memory utilization may also be collected as part of this test, to determine the vSwitch’s
performance footprint on the system.
Deployment scenario:
• Physical → virtual switch → physical.
Test ID: LTD.Throughput.RFC2889.MaxForwardingRateSoak
Title: RFC 2889 X% packet loss Max Forwarding Rate Soak Test
Prerequisite Test LTD.Throughput.RFC2544.PacketLossRatio
Priority:
Description:
The aim of this test is to understand the Max Forwarding Rate stability over an extended test duration in
order to uncover any outliers. To allow for an extended test duration, the test should ideally run for 24
hours or, if this is not possible, for at least 6 hours. For this test, each frame size must be sent at the highest
Throughput rate with X% packet loss, as determined in the prerequisite test. The default loss percentages
to be tested are: - X = 0% - X = 10^-7%
Note: Other values can be tested if required by the user.
270
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
Expected Result:
Metrics Collected:
The following are the metrics collected for this test:
• Max Forwarding Rate stability of the DUT.
– This means reporting the number of packets lost per time interval and reporting any time intervals with packet loss. The RFC2889 Forwarding Rate shall be measured in each interval. An
interval of 60s is suggested.
• CPU and memory utilization may also be collected as part of this test, to determine the vSwitch’s
performance footprint on the system.
• The RFC5481 PDV form of delay variation on the traffic flow, using the 99th percentile.
Test ID: LTD.Throughput.RFC2889.MaxForwardingRateSoakFrameModification
Title: RFC 2889 Max Forwarding Rate Soak Test with Frame Modification
Prerequisite Test: LTD.Throughput.RFC2544.PacketLossRatioFrameModification (0% Packet Loss)
Priority:
Description:
The aim of this test is to understand the Max Forwarding Rate stability over an extended test duration in
order to uncover any outliers. To allow for an extended test duration, the test should ideally run for 24
hours or, if this is not possible, for at least 6 hour. For this test, each frame size must be sent at the highest
Throughput rate with 0% packet loss, as determined in the prerequisite test.
During this test, the DUT must perform the following operations on the traffic flow:
• Perform packet parsing on the DUT’s ingress port.
• Perform any relevant address look-ups on the DUT’s ingress ports.
• Modify the packet header before forwarding the packet to the DUT’s egress port. Packet modifications include:
– Modifying the Ethernet source or destination MAC address.
– Modifying/adding a VLAN tag (Recommended).
– Modifying/adding a MPLS tag.
– Modifying the source or destination ip address.
– Modifying the TOS/DSCP field.
– Modifying the source or destination ports for UDP/TCP/SCTP.
– Modifying the TTL.
Expected Result:
Metrics Collected:
The following are the metrics collected for this test:
• Max Forwarding Rate stability of the DUT.
– This means reporting the number of packets lost per time interval and reporting any time intervals with packet loss. The RFC2889 Forwarding Rate shall be measured in each interval. An
interval of 60s is suggested.
2.3. Testing Developer Guides
271
OPNFV Documentation, Release Danube
• CPU and memory utilization may also be collected as part of this test, to determine the vSwitch’s
performance footprint on the system.
• The RFC5481 PDV form of delay variation on the traffic flow, using the 99th percentile.
Test ID: LTD.Throughput.RFC6201.ResetTime
Title: RFC 6201 Reset Time Test
Prerequisite Test: N/A
Priority:
Description:
The aim of this test is to determine the length of time it takes the DUT to recover from a reset.
Two reset methods are defined - planned and unplanned. A planned reset requires stopping and restarting
the virtual switch by the usual ‘graceful’ method defined by it’s documentation. An unplanned reset
requires simulating a fatal internal fault in the virtual switch - for example by using kill -SIGKILL on a
Linux environment.
Both reset methods SHOULD be exercised.
For each frame size previously defined under Default Test Parameters, traffic should be sent to the DUT
under normal conditions. During the duration of the test and while the traffic flows are passing through
the DUT, the DUT should be reset and the Reset time measured. The Reset time is the total time that
a device is determined to be out of operation and includes the time to perform the reset and the time to
recover from it (cf. RFC6201).
RFC6201 defines two methods to measure the Reset time:
• Frame-Loss Method: which requires the monitoring of the number of lost frames and calculates
the Reset time based on the number of frames lost and the offered rate according to the following
formula:
Frames_lost (packets)
Reset_time = ------------------------------------Offered_rate (packets per second)
• Timestamp Method: which measures the time from which the last frame is forwarded from the DUT
to the time the first frame is forwarded after the reset. This involves time-stamping all transmitted
frames and recording the timestamp of the last frame that was received prior to the reset and also
measuring the timestamp of the first frame that is received after the reset. The Reset time is the
difference between these two timestamps.
According to RFC6201 the choice of method depends on the test tool’s capability; the Frame-Loss method
SHOULD be used if the test tool supports:
• Counting the number of lost frames per stream.
• Transmitting test frame despite the physical link status.
whereas the Timestamp method SHOULD be used if the test tool supports:
• Timestamping each frame.
• Monitoring received frame’s timestamp.
• Transmitting frames only if the physical link status is up.
Expected Result:
Metrics collected
272
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
The following are the metrics collected for this test:
• Average Reset Time over the number of trials performed.
Results of this test should include the following information:
• The reset method used.
• Throughput in Fps and Mbps.
• Average Frame Loss over the number of trials performed.
• Average Reset Time in milliseconds over the number of trials performed.
• Number of trials performed.
• Protocol: IPv4, IPv6, MPLS, etc.
• Frame Size in Octets
• Port Media: Ethernet, Gigabit Ethernet (GbE), etc.
• Port Speed: 10 Gbps, 40 Gbps etc.
• Interface Encapsulation: Ethernet, Ethernet VLAN, etc.
Deployment scenario:
• Physical → virtual switch → physical.
Test ID: LTD.Throughput.RFC2889.MaxForwardingRate
Title: RFC2889 Forwarding Rate Test
Prerequisite Test: LTD.Throughput.RFC2544.PacketLossRatio
Priority:
Description:
This test measures the DUT’s Max Forwarding Rate when the Offered Load is varied between the throughput and the Maximum Offered Load for fixed length frames at a fixed time interval. The selected frame
sizes are those previously defined under Default Test Parameters. The throughput is the maximum offered load with 0% frame loss (measured by the prerequisite test), and the Maximum Offered Load (as
defined by RFC2285) is “the highest number of frames per second that an external source can transmit
to a DUT/SUT for forwarding to a specified output interface or interfaces”.
Traffic should be sent to the DUT at a particular rate (TX rate) starting with TX rate equal to the throughput
rate. The rate of successfully received frames at the destination counted (in FPS). If the RX rate is equal
to the TX rate, the TX rate should be increased by a fixed step size and the RX rate measured again until
the Max Forwarding Rate is found.
The trial duration for each iteration should last for the period of time needed for the system to reach steady
state for the frame size being tested. Under RFC2889 (Sec. 5.6.3.1) test methodology, the test duration
should run for a minimum period of 30 seconds, regardless whether the system reaches steady state before
the minimum duration ends.
Expected Result: According to RFC2889 The Max Forwarding Rate is the highest forwarding rate of
a DUT taken from an iterative set of forwarding rate measurements. The iterative set of forwarding rate
measurements are made by setting the intended load transmitted from an external source and measuring
the offered load (i.e what the DUT is capable of forwarding). If the Throughput == the Maximum Offered
Load, it follows that Max Forwarding Rate is equal to the Maximum Offered Load.
Metrics Collected:
2.3. Testing Developer Guides
273
OPNFV Documentation, Release Danube
The following are the metrics collected for this test:
• The Max Forwarding Rate for the DUT for each packet size.
• CPU and memory utilization may also be collected as part of this test, to determine the vSwitch’s
performance footprint on the system.
Deployment scenario:
• Physical → virtual switch → physical. Note: Full mesh tests with multiple ingress and egress ports
are a key aspect of RFC 2889 benchmarks, and scenarios with both 2 and 4 ports should be tested.
In any case, the number of ports used must be reported.
Test ID: LTD.Throughput.RFC2889.ForwardPressure
Title: RFC2889 Forward Pressure Test
Prerequisite Test: LTD.Throughput.RFC2889.MaxForwardingRate
Priority:
Description:
The aim of this test is to determine if the DUT transmits frames with an inter-frame gap that is less than
12 bytes. This test overloads the DUT and measures the output for forward pressure. Traffic should be
transmitted to the DUT with an inter-frame gap of 11 bytes, this will overload the DUT by 1 byte per
frame. The forwarding rate of the DUT should be measured.
Expected Result: The forwarding rate should not exceed the maximum forwarding rate of the DUT
collected by LTD.Throughput.RFC2889.MaxForwardingRate.
Metrics collected
The following are the metrics collected for this test:
• Forwarding rate of the DUT in FPS or Mbps.
• CPU and memory utilization may also be collected as part of this test, to determine the vSwitch’s
performance footprint on the system.
Deployment scenario:
• Physical → virtual switch → physical.
Test ID: LTD.Throughput.RFC2889.ErrorFramesFiltering
Title: RFC2889 Error Frames Filtering Test
Prerequisite Test: N/A
Priority:
Description:
The aim of this test is to determine whether the DUT will propagate any erroneous frames it receives or
whether it is capable of filtering out the erroneous frames. Traffic should be sent with erroneous frames
included within the flow at random intervals. Illegal frames that must be tested include: - Oversize Frames.
- Undersize Frames. - CRC Errored Frames. - Dribble Bit Errored Frames - Alignment Errored Frames
The traffic flow exiting the DUT should be recorded and checked to determine if the erroneous frames
where passed through the DUT.
Expected Result: Broken frames are not passed!
Metrics collected
274
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
No Metrics are collected in this test, instead it determines:
• Whether the DUT will propagate erroneous frames.
• Or whether the DUT will correctly filter out any erroneous frames from traffic flow with out removing correct frames.
Deployment scenario:
• Physical → virtual switch → physical.
Test ID: LTD.Throughput.RFC2889.BroadcastFrameForwarding
Title: RFC2889 Broadcast Frame Forwarding Test
Prerequisite Test: N
Priority:
Description:
The aim of this test is to determine the maximum forwarding rate of the DUT when forwarding broadcast
traffic. For each frame previously defined under Default Test Parameters, the traffic should be set up as
broadcast traffic. The traffic throughput of the DUT should be measured.
The test should be conducted with at least 4 physical ports on the DUT. The number of ports used MUST
be recorded.
As broadcast involves forwarding a single incoming packet to several destinations, the latency of a single
packet is defined as the average of the latencies for each of the broadcast destinations.
The incoming packet is transmitted on each of the other physical ports, it is not transmitted on the port
on which it was received. The test MAY be conducted using different broadcasting ports to uncover any
performance differences.
Expected Result:
Metrics collected:
The following are the metrics collected for this test:
• The forwarding rate of the DUT when forwarding broadcast traffic.
• The minimum, average & maximum packets latencies observed.
Deployment scenario:
• Physical → virtual switch 3x physical. In the Broadcast rate testing, four test ports are required. One
of the ports is connected to the test device, so it can send broadcast frames and listen for miss-routed
frames.
Test ID: LTD.Throughput.RFC2544.WorstN-BestN
Title: Modified RFC 2544 X% packet loss ratio Throughput and Latency Test
Prerequisite Test: N/A
Priority:
Description:
This test determines the DUT’s maximum forwarding rate with X% traffic loss for a constant load (fixed
length frames at a fixed interval time). The default loss percentages to be tested are: X = 0%, X = 10^-7%
2.3. Testing Developer Guides
275
OPNFV Documentation, Release Danube
Modified RFC 2544 throughput benchmarking methodology aims to quantify the throughput measurement variations observed during standard RFC 2544 benchmarking measurements of virtual switches and
VNFs. The RFC2544 binary search algorithm is modified to use more samples per test trial to drive
the binary search and yield statistically more meaningful results. This keeps the heart of the RFC2544
methodology, still relying on the binary search of throughput at specified loss tolerance, while providing
more useful information about the range of results seen in testing. Instead of using a single traffic trial per
iteration step, each traffic trial is repeated N times and the success/failure of the iteration step is based on
these N traffic trials. Two types of revised tests are defined - Worst-of-N and Best-of-N.
Worst-of-N
Worst-of-N indicates the lowest expected maximum throughput for ( packet size, loss tolerance) when
repeating the test.
1. Repeat the same test run N times at a set packet rate, record each result.
2. Take the WORST result (highest packet loss) out of N result samples, called the Worst-of-N sample.
3. If Worst-of-N sample has loss less than the set loss tolerance, then the step is successful - increase
the test traffic rate.
4. If Worst-of-N sample has loss greater than the set loss tolerance then the step failed - decrease the
test traffic rate.
5. Go to step 1.
Best-of-N
Best-of-N indicates the highest expected maximum throughput for ( packet size, loss tolerance) when
repeating the test.
1. Repeat the same traffic run N times at a set packet rate, record each result.
2. Take the BEST result (least packet loss) out of N result samples, called the Best-of-N sample.
3. If Best-of-N sample has loss less than the set loss tolerance, then the step is successful - increase the
test traffic rate.
4. If Best-of-N sample has loss greater than the set loss tolerance, then the step failed - decrease the
test traffic rate.
5. Go to step 1.
Performing both Worst-of-N and Best-of-N benchmark tests yields lower and upper bounds of expected
maximum throughput under the operating conditions, giving a very good indication to the user of the
deterministic performance range for the tested setup.
Expected Result: At the end of each trial series, the presence or absence of loss determines the modification of offered load for the next trial series, converging on a maximum rate, or RFC2544 Throughput
with X% loss. The Throughput load is re-used in related RFC2544 tests and other tests.
Metrics Collected:
The following are the metrics collected for this test:
• The maximum forwarding rate in Frames Per Second (FPS) and Mbps of the DUT for each frame
size with X% packet loss.
• The average latency of the traffic flow when passing through the DUT (if testing for latency, note
that this average is different from the test specified in Section 26.3 of RFC2544).
• Following may also be collected as part of this test, to determine the vSwitch’s performance footprint
on the system:
• CPU core utilization.
276
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
• CPU cache utilization.
• Memory footprint.
• System bus (QPI, PCI, ...) utilization.
• CPU cycles consumed per packet.
Test ID: LTD.Throughput.Overlay.Network.<tech>.RFC2544.PacketLossRatio
Title: <tech> Overlay Network RFC 2544 X% packet loss ratio Throughput and
Latency Test
NOTE: Throughout this test, four interchangeable overlay technologies are covered
by the same test description. They are: VXLAN, GRE, NVGRE and GENEVE.
Prerequisite Test: N/A
Priority:
Description: This test evaluates standard switch performance benchmarks for the scenario
where an Overlay Network is deployed for all paths through the vSwitch. Overlay Technologies covered (replacing <tech> in the test name) include:
• VXLAN
• GRE
• NVGRE
• GENEVE
Performance will be assessed for each of the following overlay network functions:
• Encapsulation only
• De-encapsulation only
• Both Encapsulation and De-encapsulation
For each native packet, the DUT must perform the following operations:
• Examine the packet and classify its correct overlay net (tunnel) assignment
• Encapsulate the packet
• Switch the packet to the correct port
For each encapsulated packet, the DUT must perform the following operations:
• Examine the packet and classify its correct native network assignment
• De-encapsulate the packet, if required
• Switch the packet to the correct port
The selected frame sizes are those previously defined under Default Test Parameters.
Thus, each test comprises an overlay technology, a network function, and a packet size with overlay
network overhead included (but see also the discussion at https://etherpad.opnfv.org/p/vSwitchTestsDrafts
).
The test can also be used to determine the average latency of the traffic.
Under the RFC2544 test methodology, the test duration will include a number of trials; each trial should
run for a minimum period of 60 seconds. A binary search methodology must be applied for each trial to
obtain the final result for Throughput.
2.3. Testing Developer Guides
277
OPNFV Documentation, Release Danube
Expected Result: At the end of each trial, the presence or absence of loss determines the modification
of offered load for the next trial, converging on a maximum rate, or RFC2544 Throughput with X% loss
(where the value of X is typically equal to zero). The Throughput load is re-used in related RFC2544 tests
and other tests.
Metrics Collected: The following are the metrics collected for this test:
• The maximum Throughput in Frames Per Second (FPS) and Mbps of the DUT for each frame size
with X% packet loss.
• The average latency of the traffic flow when passing through the DUT and VNFs (if testing for
latency, note that this average is different from the test specified in Section 26.3 of RFC2544).
• CPU and memory utilization may also be collected as part of this test, to determine the vSwitch’s
performance footprint on the system.
Test ID: LTD.Throughput.RFC2544.MatchAction.PacketLossRatio
Title: RFC 2544 X% packet loss ratio match action Throughput and Latency Test
Prerequisite Test: LTD.Throughput.RFC2544.PacketLossRatio
Priority:
Description:
The aim of this test is to determine the cost of carrying out match action(s) on the DUT’s
RFC2544 Throughput with X% traffic loss for a constant load (fixed length frames at a fixed
interval time).
Each test case requires:
• selection of a specific match action(s),
• specifying a percentage of total traffic that is elligible for the match action,
• determination of the specific test configuration (number of flows, number of test ports,
presence of an external controller, etc.), and
• measurement of the RFC 2544 Throughput level with X% packet loss: Traffic shall be
bi-directional and symmetric.
Note: It would be ideal to verify that all match action-elligible traffic was forwarded to the
correct port, and if forwarded to an unintended port it should be considered lost.
A match action is an action that is typically carried on a frame or packet that matches a set of
flow classification parameters (typically frame/packet header fields). A match action may or
may not modify a packet/frame. Match actions include [1]:
• output : outputs a packet to a particular port.
• normal: Subjects the packet to traditional L2/L3 processing (MAC learning).
• flood: Outputs the packet on all switch physical ports other than the port on which it was
received and any ports on which flooding is disabled.
• all: Outputs the packet on all switch physical ports other than the port on which it was
received.
• local: Outputs the packet on the local port, which corresponds to the network device
that has the same name as the bridge.
• in_port: Outputs the packet on the port from which it was received.
278
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
• Controller: Sends the packet and its metadata to the OpenFlow controller as a packet
in message.
• enqueue: Enqueues the packet on the specified queue within port.
• drop: discard the packet.
Modifications include [1]:
• mod vlan: covered by LTD.Throughput.RFC2544.PacketLossRatioFrameModification
• mod_dl_src: Sets the source Ethernet address.
• mod_dl_dst: Sets the destination Ethernet address.
• mod_nw_src: Sets the IPv4 source address.
• mod_nw_dst: Sets the IPv4 destination address.
• mod_tp_src: Sets the TCP or UDP or SCTP source port.
• mod_tp_dst: Sets the TCP or UDP or SCTP destination port.
• mod_nw_tos: Sets the DSCP bits in the IPv4 ToS/DSCP or IPv6 traffic class field.
• mod_nw_ecn: Sets the ECN bits in the appropriate IPv4 or IPv6 field.
• mod_nw_ttl: Sets the IPv4 TTL or IPv6 hop limit field.
Note: This comprehensive list requires extensive traffic generator capabilities.
The match action(s) that were applied as part of the test should be reported in the final test
report.
During this test, the DUT must perform the following operations on the traffic flow:
• Perform packet parsing on the DUT’s ingress port.
• Perform any relevant address look-ups on the DUT’s ingress ports.
• Carry out one or more of the match actions specified above.
The default loss percentages to be tested are: - X = 0% - X = 10^-7% Other values can be tested
if required by the user. The selected frame sizes are those previously defined under Default
Test Parameters.
The test can also be used to determine the average latency of the traffic when a match action
is applied to packets in a flow. Under the RFC2544 test methodology, the test duration will
include a number of trials; each trial should run for a minimum period of 60 seconds. A binary
search methodology must be applied for each trial to obtain the final result.
Expected Result:
At the end of each trial, the presence or absence of loss determines the modification of offered
load for the next trial, converging on a maximum rate, or RFC2544Throughput with X% loss.
The Throughput load is re-used in related RFC2544 tests and other tests.
Metrics Collected:
The following are the metrics collected for this test:
• The RFC 2544 Throughput in Frames Per Second (FPS) and Mbps of the DUT for each
frame size with X% packet loss.
• The average latency of the traffic flow when passing through the DUT (if testing
for latency, note that this average is different from the test specified in Section 26.3
ofRFC2544).
2.3. Testing Developer Guides
279
OPNFV Documentation, Release Danube
• CPU and memory utilization may also be collected as part of this test, to determine the
vSwitch’s performance footprint on the system.
The metrics collected can be compared to that of the prerequisite test to determine the cost of
the match action(s) in the pipeline.
Deployment scenario:
• Physical → virtual switch → physical (and others are possible)
[1] ovs-ofctl - administer OpenFlow switches [http://openvswitch.org/support/distdocs/ovs-ofctl.8.txt ]
Packet Latency tests These tests will measure the store and forward latency as well as the packet delay variation
for various packet types through the virtual switch. The following list is not exhaustive but should indicate the type of
tests that should be required. It is expected that more will be added.
Test ID: LTD.PacketLatency.InitialPacketProcessingLatency
Title: Initial Packet Processing Latency
Prerequisite Test: N/A
Priority:
Description:
In some virtual switch architectures, the first packets of a flow will take the system longer to process than
subsequent packets in the flow. This test determines the latency for these packets. The test will measure
the latency of the packets as they are processed by the flow-setup-path of the DUT. There are two methods
for this test, a recommended method and a nalternative method that can be used if it is possible to disable
the fastpath of the virtual switch.
Recommended method: This test will send 64,000 packets to the DUT, each belonging to a different flow.
Average packet latency will be determined over the 64,000 packets.
Alternative method: This test will send a single packet to the DUT after a fixed interval of time. The time
interval will be equivalent to the amount of time it takes for a flow to time out in the virtual switch plus
10%. Average packet latency will be determined over 1,000,000 packets.
This test is intended only for non-learning virtual switches; For learning virtual switches use RFC2889.
For this test, only unidirectional traffic is required.
Expected Result: The average latency for the initial packet of all flows should be greater than the latency
of subsequent traffic.
Metrics Collected:
The following are the metrics collected for this test:
• Average latency of the initial packets of all flows that are processed by the DUT.
Deployment scenario:
• Physical → Virtual Switch → Physical.
Test ID: LTD.PacketDelayVariation.RFC3393.Soak
Title: Packet Delay Variation Soak Test
Prerequisite Tests: LTD.Throughput.RFC2544.PacketLossRatio (0% Packet Loss)
280
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
Priority:
Description:
The aim of this test is to understand the distribution of packet delay variation for different frame sizes
over an extended test duration and to determine if there are any outliers. To allow for an extended test
duration, the test should ideally run for 24 hours or, if this is not possible, for at least 6 hour. For this test,
each frame size must be sent at the highest possible throughput with 0% packet loss, as determined in the
prerequisite test.
Expected Result:
Metrics Collected:
The following are the metrics collected for this test:
• The packet delay variation value for traffic passing through the DUT.
• The RFC5481 PDV form of delay variation on the traffic flow, using the 99th percentile, for each
60s interval during the test.
• CPU and memory utilization may also be collected as part of this test, to determine the vSwitch’s
performance footprint on the system.
Scalability tests The general aim of these tests is to understand the impact of large flow table size and flow lookups
on throughput. The following list is not exhaustive but should indicate the type of tests that should be required. It is
expected that more will be added.
Test ID: LTD.Scalability.Flows.RFC2544.0PacketLoss
Title: RFC 2544 0% loss Flow Scalability throughput test
Prerequisite Test: LTD.Throughput.RFC2544.PacketLossRatio, IF the delta Throughput between the
single-flow RFC2544 test and this test with a variable number of flows is desired.
Priority:
Description:
The aim of this test is to measure how throughput changes as the number of flows in the DUT increases.
The test will measure the throughput through the fastpath, as such the flows need to be installed on the
DUT before passing traffic.
For each frame size previously defined under Default Test Parameters and for each of the following
number of flows:
• 1,000
• 2,000
• 4,000
• 8,000
• 16,000
• 32,000
• 64,000
• Max supported number of flows.
This test will be conducted under two conditions following the establishment of all flows as required by
RFC 2544, regarding the flow expiration time-out:
2.3. Testing Developer Guides
281
OPNFV Documentation, Release Danube
1. The time-out never expires during each trial.
2) The time-out expires for all flows periodically. This would require a short time-out compared with flow
re-appearance for a small number of flows, and may not be possible for all flow conditions.
The maximum 0% packet loss Throughput should be determined in a manner identical to
LTD.Throughput.RFC2544.PacketLossRatio.
Expected Result:
Metrics Collected:
The following are the metrics collected for this test:
• The maximum number of frames per second that can be forwarded at the specified number of flows
and the specified frame size, with zero packet loss.
Test ID: LTD.MemoryBandwidth.RFC2544.0PacketLoss.Scalability
Title: RFC 2544 0% loss Memory Bandwidth Scalability test
Prerequisite Tests: LTD.Throughput.RFC2544.PacketLossRatio, IF the delta Throughput between an
undisturbed RFC2544 test and this test with the Throughput affected by cache and memory bandwidth
contention is desired.
Priority:
Description:
The aim of this test is to understand how the DUT’s performance is affected by cache sharing and memory
bandwidth between processes.
During the test all cores not used by the vSwitch should be running a memory intensive application. This
application should read and write random data to random addresses in unused physical memory. The
random nature of the data and addresses is intended to consume cache, exercise main memory access (as
opposed to cache) and exercise all memory buses equally. Furthermore:
• the ratio of reads to writes should be recorded. A ratio of 1:1 SHOULD be used.
• the reads and writes MUST be of cache-line size and be cache-line aligned.
• in NUMA architectures memory access SHOULD be local to the core’s node. Whether only local
memory or a mix of local and remote memory is used MUST be recorded.
• the memory bandwidth (reads plus writes) used per-core MUST be recorded; the test MUST be run
with a per-core memory bandwidth equal to half the maximum system memory bandwidth divided
by the number of cores. The test MAY be run with other values for the per-core memory bandwidth.
• the test MAY also be run with the memory intensive application running on all cores.
Under these conditions the DUT’s 0%
LTD.Throughput.RFC2544.PacketLossRatio.
packet
loss
throughput
is
determined
as
per
Expected Result:
Metrics Collected:
The following are the metrics collected for this test:
• The DUT’s 0% packet loss throughput in the presence of cache sharing and memory bandwidth
between processes.
282
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
Test ID: LTD.Scalability.VNF.RFC2544.PacketLossRatio
Title: VNF Scalability RFC 2544 X% packet loss ratio Throughput and Latency Test
Prerequisite Test: N/A
Priority:
Description:
This test determines the DUT’s throughput rate with X% traffic loss for a constant load (fixed length
frames at a fixed interval time) when the number of VNFs on the DUT increases. The default loss percentages to be tested are: - X = 0% - X = 10^-7% . The minimum number of VNFs to be tested are
3.
Flow classification should be conducted with L2, L3 and L4 matching to understand the matching and
scaling capability of the vSwitch. The matching fields which were used as part of the test should be
reported as part of the benchmark report.
The vSwitch is responsible for forwarding frames between the VNFs
The SUT (vSwitch and VNF daisy chain) operation should be validated before running the test. This
may be completed by running a burst or continuous stream of traffic through the SUT to ensure proper
operation before a test.
Note: The traffic rate used to validate SUT operation should be low enough not to stress the SUT.
Note: Other values can be tested if required by the user.
Note: The same VNF should be used in the “daisy chain” formation. Each addition of a VNF should
be conducted in a new test setup (The DUT is brought down, then the DUT is brought up again). An
atlernative approach would be to continue to add VNFs without bringing down the DUT. The approach
used needs to be documented as part of the test report.
The selected frame sizes are those previously defined under Default Test Parameters. The test can also be
used to determine the average latency of the traffic.
Under the RFC2544 test methodology, the test duration will include a number of trials; each trial should
run for a minimum period of 60 seconds. A binary search methodology must be applied for each trial to
obtain the final result for Throughput.
Expected Result: At the end of each trial, the presence or absence of loss determines the modification
of offered load for the next trial, converging on a maximum rate, or RFC2544 Throughput with X% loss.
The Throughput load is re-used in related RFC2544 tests and other tests.
If the test VNFs are rather light-weight in terms of processing, the test provides a view of multiple passes
through the vswitch on logical interfaces. In other words, the test produces an optimistic count of daisychained VNFs, but the cumulative effect of traffic on the vSwitch is “real” (assuming that the vSwitch has
some dedicated resources, and the effects on shared resources is understood).
Metrics Collected: The following are the metrics collected for this test:
• The maximum Throughput in Frames Per Second (FPS) and Mbps of the DUT for each frame size
with X% packet loss.
• The average latency of the traffic flow when passing through the DUT and VNFs (if testing for
latency, note that this average is different from the test specified in Section 26.3 of RFC2544).
• CPU and memory utilization may also be collected as part of this test, to determine the vSwitch’s
performance footprint on the system.
2.3. Testing Developer Guides
283
OPNFV Documentation, Release Danube
Test ID: LTD.Scalability.VNF.RFC2544.PacketLossProfile
Title: VNF Scalability RFC 2544 Throughput and Latency Profile
Prerequisite Test: N/A
Priority:
Description:
This test reveals how throughput and latency degrades as the number of VNFs increases and
offered rate varies in the region of the DUT’s maximum forwarding rate as determined by
LTD.Throughput.RFC2544.PacketLossRatio (0% Packet Loss). For example it can be used to
determine if the degradation of throughput and latency as the number of VNFs and offered rate
increases is slow and graceful, or sudden and severe. The minimum number of VNFs to be
tested is 3.
The selected frame sizes are those previously defined under Default Test Parameters.
The offered traffic rate is described as a percentage delta with respect to the DUT’s RFC 2544
Throughput as determined by LTD.Throughput.RFC2544.PacketLoss Ratio (0% Packet Loss
case). A delta of 0% is equivalent to an offered traffic rate equal to the RFC 2544 Throughput;
A delta of +50% indicates an offered rate half-way between the Throughput and line-rate,
whereas a delta of -50% indicates an offered rate of half the maximum rate. Therefore the
range of the delta figure is natuarlly bounded at -100% (zero offered traffic) and +100% (traffic
offered at line rate).
The following deltas to the maximum forwarding rate should be applied:
• -50%, -10%, 0%, +10% & +50%
Note: Other values can be tested if required by the user.
Note: The same VNF should be used in the “daisy chain” formation. Each addition of a VNF should
be conducted in a new test setup (The DUT is brought down, then the DUT is brought up again). An
atlernative approach would be to continue to add VNFs without bringing down the DUT. The approach
used needs to be documented as part of the test report.
Flow classification should be conducted with L2, L3 and L4 matching to understand the matching and
scaling capability of the vSwitch. The matching fields which were used as part of the test should be
reported as part of the benchmark report.
The SUT (vSwitch and VNF daisy chain) operation should be validated before running the test. This
may be completed by running a burst or continuous stream of traffic through the SUT to ensure proper
operation before a test.
Note: the traffic rate used to validate SUT operation should be low enough not to stress the SUT
Expected Result: For each packet size a profile should be produced of how throughput and latency vary
with offered rate.
Metrics Collected:
The following are the metrics collected for this test:
• The forwarding rate in Frames Per Second (FPS) and Mbps of the DUT for each delta to the maximum forwarding rate and for each frame size.
• The average latency for each delta to the maximum forwarding rate and for each frame size.
• CPU and memory utilization may also be collected as part of this test, to determine the vSwitch’s
performance footprint on the system.
284
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
• Any failures experienced (for example if the vSwitch crashes, stops processing packets, restarts or
becomes unresponsive to commands) when the offered load is above Maximum Throughput MUST
be recorded and reported with the results.
Activation tests The general aim of these tests is to understand the capacity of the and speed with which the vswitch
can accommodate new flows.
Test ID: LTD.Activation.RFC2889.AddressCachingCapacity
Title: RFC2889 Address Caching Capacity Test
Prerequisite Test: N/A
Priority:
Description:
Please note this test is only applicable to virtual switches that are capable of MAC learning. The aim
of this test is to determine the address caching capacity of the DUT for a constant load (fixed length
frames at a fixed interval time). The selected frame sizes are those previously defined under Default Test
Parameters.
In order to run this test the aging time, that is the maximum time the DUT will keep a learned address in
its flow table, and a set of initial addresses, whose value should be >= 1 and <= the max number supported
by the implementation must be known. Please note that if the aging time is configurable it must be longer
than the time necessary to produce frames from the external source at the specified rate. If the aging time
is fixed the frame rate must be brought down to a value that the external source can produce in a time that
is less than the aging time.
Learning Frames should be sent from an external source to the DUT to install a number of flows. The
Learning Frames must have a fixed destination address and must vary the source address of the frames.
The DUT should install flows in its flow table based on the varying source addresses. Frames should then
be transmitted from an external source at a suitable frame rate to see if the DUT has properly learned
all of the addresses. If there is no frame loss and no flooding, the number of addresses sent to the DUT
should be increased and the test is repeated until the max number of cached addresses supported by the
DUT determined.
Expected Result:
Metrics collected:
The following are the metrics collected for this test:
• Number of cached addresses supported by the DUT.
• CPU and memory utilization may also be collected as part of this test, to determine the vSwitch’s
performance footprint on the system.
Deployment scenario:
• Physical → virtual switch → 2 x physical (one receiving, one listening).
Test ID: LTD.Activation.RFC2889.AddressLearningRate
Title: RFC2889 Address Learning Rate Test
Prerequisite Test: LTD.Memory.RFC2889.AddressCachingCapacity
Priority:
Description:
2.3. Testing Developer Guides
285
OPNFV Documentation, Release Danube
Please note this test is only applicable to virtual switches that are capable of MAC learning. The aim
of this test is to determine the rate of address learning of the DUT for a constant load (fixed length
frames at a fixed interval time). The selected frame sizes are those previously defined under Default
Test Parameters, traffic should be sent with each IPv4/IPv6 address incremented by one. The rate
at which the DUT learns a new address should be measured. The maximum caching capacity from
LTD.Memory.RFC2889.AddressCachingCapacity should be taken into consideration as the maximum
number of addresses for which the learning rate can be obtained.
Expected Result: It may be worthwhile to report the behaviour when operating beyond address capacity
- some DUTs may be more friendly to new addresses than others.
Metrics collected:
The following are the metrics collected for this test:
• The address learning rate of the DUT.
Deployment scenario:
• Physical → virtual switch → 2 x physical (one receiving, one listening).
Coupling between control path and datapath Tests The following tests aim to determine how tightly coupled the
datapath and the control path are within a virtual switch. The following list is not exhaustive but should indicate the
type of tests that should be required. It is expected that more will be added.
Test ID: LTD.CPDPCouplingFlowAddition
Title: Control Path and Datapath Coupling
Prerequisite Test:
Priority:
Description:
The aim of this test is to understand how exercising the DUT’s control path affects datapath performance.
Initially a certain number of flow table entries are installed in the vSwitch. Then over the duration of an
RFC2544 throughput test flow-entries are added and removed at the rates specified below. No traffic is
‘hitting’ these flow-entries, they are simply added and removed.
The test MUST be repeated with the following initial number of flow-entries installed: - < 10 - 1000 100,000 - 10,000,000 (or the maximum supported number of flow-entries)
The test MUST be repeated with the following rates of flow-entry addition and deletion per second: - 0 1 (i.e. 1 addition plus 1 deletion) - 100 - 10,000
Expected Result:
Metrics Collected:
The following are the metrics collected for this test:
• The maximum forwarding rate in Frames Per Second (FPS) and Mbps of the DUT.
• The average latency of the traffic flow when passing through the DUT (if testing for latency, note
that this average is different from the test specified in Section 26.3 of RFC2544).
• CPU and memory utilization may also be collected as part of this test, to determine the vSwitch’s
performance footprint on the system.
Deployment scenario:
• Physical → virtual switch → physical.
286
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
CPU and memory consumption The following tests will profile a virtual switch’s CPU and memory utilization
under various loads and circumstances. The following list is not exhaustive but should indicate the type of tests that
should be required. It is expected that more will be added.
Test ID: LTD.Stress.RFC2544.0PacketLoss
Title: RFC 2544 0% Loss CPU OR Memory Stress Test
Prerequisite Test:
Priority:
Description:
The aim of this test is to understand the overall performance of the system when a CPU or Memory intensive application is run on the same DUT as the Virtual Switch. For each frame size, an
LTD.Throughput.RFC2544.PacketLossRatio (0% Packet Loss) test should be performed. Throughout
the entire test a CPU or Memory intensive application should be run on all cores on the system not in use
by the Virtual Switch. For NUMA system only cores on the same NUMA node are loaded.
It is recommended that stress-ng be used for loading the non-Virtual Switch cores but any stress tool MAY
be used.
Expected Result:
Metrics Collected:
The following are the metrics collected for this test:
• Memory and CPU utilization of the cores running the Virtual Switch.
• The number of identity of the cores allocated to the Virtual Switch.
• The configuration of the stress tool (for example the command line parameters used to start it.)
Note: Stress in the test ID can be replaced with the name of the component being stressed, when reporting the results: LTD.CPU.RFC2544.0PacketLoss or LTD.Memory.RFC2544.0PacketLoss
Summary List of Tests
1. Throughput tests
• Test ID: LTD.Throughput.RFC2544.PacketLossRatio
• Test ID: LTD.Throughput.RFC2544.PacketLossRatioFrameModification
• Test ID: LTD.Throughput.RFC2544.Profile
• Test ID: LTD.Throughput.RFC2544.SystemRecoveryTime
• Test ID: LTD.Throughput.RFC2544.BackToBackFrames
• Test ID: LTD.Throughput.RFC2889.Soak
• Test ID: LTD.Throughput.RFC2889.SoakFrameModification
• Test ID: LTD.Throughput.RFC6201.ResetTime
• Test ID: LTD.Throughput.RFC2889.MaxForwardingRate
• Test ID: LTD.Throughput.RFC2889.ForwardPressure
• Test ID: LTD.Throughput.RFC2889.ErrorFramesFiltering
• Test ID: LTD.Throughput.RFC2889.BroadcastFrameForwarding
2.3. Testing Developer Guides
287
OPNFV Documentation, Release Danube
• Test ID: LTD.Throughput.RFC2544.WorstN-BestN
• Test ID: LTD.Throughput.Overlay.Network.<tech>.RFC2544.PacketLossRatio
2. Packet Latency tests
• Test ID: LTD.PacketLatency.InitialPacketProcessingLatency
• Test ID: LTD.PacketDelayVariation.RFC3393.Soak
3. Scalability tests
• Test ID: LTD.Scalability.Flows.RFC2544.0PacketLoss
• Test ID: LTD.MemoryBandwidth.RFC2544.0PacketLoss.Scalability
• LTD.Scalability.VNF.RFC2544.PacketLossProfile
• LTD.Scalability.VNF.RFC2544.PacketLossRatio
4. Activation tests
• Test ID: LTD.Activation.RFC2889.AddressCachingCapacity
• Test ID: LTD.Activation.RFC2889.AddressLearningRate
5. Coupling between control path and datapath Tests
• Test ID: LTD.CPDPCouplingFlowAddition
6. CPU and memory consumption
• Test ID: LTD.Stress.RFC2544.0PacketLoss
VSPERF LEVEL TEST PLAN (LTP)
Introduction The objective of the OPNFV project titled Characterize vSwitch Performance for Telco NFV Use
Cases, is to evaluate the performance of virtual switches to identify its suitability for a Telco Network Function
Virtualization (NFV) environment. The intention of this Level Test Plan (LTP) document is to specify the scope,
approach, resources, and schedule of the virtual switch performance benchmarking activities in OPNFV. The test
cases will be identified in a separate document called the Level Test Design (LTD) document.
This document is currently in draft form.
Document identifier The document id will be used to uniquely identify versions of the LTP. The format for the document id will be: OPNFV_vswitchperf_LTP_REL_STATUS, where by the status is one of: draft, reviewed, corrected
or final. The document id for this version of the LTP is: OPNFV_vswitchperf_LTP_Colorado_REVIEWED.
Scope The main purpose of this project is to specify a suite of performance tests in order to objectively measure
the current packet transfer characteristics of a virtual switch in the NFVI. The intent of the project is to facilitate the
performance testing of any virtual switch. Thus, a generic suite of tests shall be developed, with no hard dependencies
to a single implementation. In addition, the test case suite shall be architecture independent.
The test cases developed in this project shall not form part of a separate test framework, all of these tests may be
inserted into the Continuous Integration Test Framework and/or the Platform Functionality Test Framework - if a
vSwitch becomes a standard component of an OPNFV release.
288
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
References
• RFC 1242 Benchmarking Terminology for Network Interconnection Devices
• RFC 2544 Benchmarking Methodology for Network Interconnect Devices
• RFC 2285 Benchmarking Terminology for LAN Switching Devices
• RFC 2889 Benchmarking Methodology for LAN Switching Devices
• RFC 3918 Methodology for IP Multicast Benchmarking
• RFC 4737 Packet Reordering Metrics
• RFC 5481 Packet Delay Variation Applicability Statement
• RFC 6201 Device Reset Characterization
Level in the overall sequence The level of testing conducted by vswitchperf in the overall testing sequence (among
all the testing projects in OPNFV) is the performance benchmarking of a specific component (the vswitch) in the
OPNFV platfrom. It’s expected that this testing will follow on from the functional and integration testing conducted
by other testing projects in OPNFV, namely Functest and Yardstick.
Test classes and overall test conditions A benchmark is defined by the IETF as: A standardized test that serves as
a basis for performance evaluation and comparison. It’s important to note that benchmarks are not Functional tests.
They do not provide PASS/FAIL criteria, and most importantly ARE NOT performed on live networks, or performed
with live network traffic.
In order to determine the packet transfer characteristics of a virtual switch, the benchmarking tests will be broken
down into the following categories:
• Throughput Tests to measure the maximum forwarding rate (in frames per second or fps) and bit rate (in Mbps)
for a constant load (as defined by RFC1242) without traffic loss.
• Packet and Frame Delay Tests to measure average, min and max packet and frame delay for constant loads.
• Stream Performance Tests (TCP, UDP) to measure bulk data transfer performance, i.e. how fast systems can
send and receive data through the virtual switch.
• Request/Response Performance Tests (TCP, UDP) the measure the transaction rate through the virtual switch.
• Packet Delay Tests to understand latency distribution for different packet sizes and over an extended test run to
uncover outliers.
• Scalability Tests to understand how the virtual switch performs as the number of flows, active ports, complexity
of the forwarding logic’s configuration... it has to deal with increases.
• Control Path and Datapath Coupling Tests, to understand how closely coupled the datapath and the control
path are as well as the effect of this coupling on the performance of the DUT.
• CPU and Memory Consumption Tests to understand the virtual switch’s footprint on the system, this includes:
– CPU core utilization.
– CPU cache utilization.
– Memory footprint.
– System bus (QPI, PCI, ..) utilization.
– Memory lanes utilization.
– CPU cycles consumed per packet.
– Time To Establish Flows Tests.
2.3. Testing Developer Guides
289
OPNFV Documentation, Release Danube
• Noisy Neighbour Tests, to understand the effects of resource sharing on the performance of a virtual switch.
Note: some of the tests above can be conducted simultaneously where the combined results would be insightful, for
example Packet/Frame Delay and Scalability.
Details of the Level Test Plan This section describes the following items: * Test items and their identifiers
(TestItems) * Test Traceability Matrix (TestMatrix) * Features to be tested (FeaturesToBeTested) * Features not to
be tested (FeaturesNotToBeTested) * Approach (Approach) * Item pass/fail criteria (PassFailCriteria) * Suspension
criteria and resumption requirements (SuspensionResumptionReqs)
Test items and their identifiers The test item/application vsperf is trying to test are virtual switches and in particular
their performance in an nfv environment. vsperf will first try to measure the maximum achievable performance by a
virtual switch and then it will focus in on usecases that are as close to real life deployment scenarios as possible.
Test Traceability Matrix vswitchperf leverages the “3x3” matrix (introduced in https://tools.ietf.org/html/draft-ietfbmwg-virtual-net-02) to achieve test traceability. The matrix was expanded to 3x4 to accommodate scale metrics when
displaying the coverage of many metrics/benchmarks). Test case covreage in the LTD is tracked using the following
catagories:
Activation
Operation
De-activation
SPEED
X
X
ACCURACY
X
X
RELIABILITY
X
X
SCALE
X
X
X = denotes a test catagory that has 1 or more test cases defined.
Features to be tested Characterizing virtual switches (i.e. Device Under Test (DUT) in this document) includes
measuring the following performance metrics:
• Throughput as defined by RFC1242: The maximum rate at which none of the offered frames are dropped by
the DUT. The maximum frame rate and bit rate that can be transmitted by the DUT without any error should be
recorded. Note there is an equivalent bit rate and a specific layer at which the payloads contribute to the bits.
Errors and improperly formed frames or packets are dropped.
• Packet delay introduced by the DUT and its cumulative effect on E2E networks. Frame delay can be measured
equivalently.
• Packet delay variation: measured from the perspective of the VNF/application. Packet delay variation is
sometimes called “jitter”. However, we will avoid the term “jitter” as the term holds different meaning to
different groups of people. In this document we will simply use the term packet delay variation. The preferred
form for this metric is the PDV form of delay variation defined in RFC5481. The most relevant measurement of
PDV considers the delay variation of a single user flow, as this will be relevant to the size of end-system buffers to
compensate for delay variation. The measurement system’s ability to store the delays of individual packets in the
flow of interest is a key factor that determines the specific measurement method. At the outset, it is ideal to view
the complete PDV distribution. Systems that can capture and store packets and their delays have the freedom
to calculate the reference minimum delay and to determine various quantiles of the PDV distribution accurately
(in post-measurement processing routines). Systems without storage must apply algorithms to calculate delay
and statistical measurements on the fly. For example, a system may store temporary estimates of the mimimum
delay and the set of (100) packets with the longest delays during measurement (to calculate a high quantile, and
update these sets with new values periodically. In some cases, a limited number of delay histogram bins will
be available, and the bin limits will need to be set using results from repeated experiments. See section 8 of
RFC5481.
• Packet loss (within a configured waiting time at the receiver): All packets sent to the DUT should be accounted
for.
290
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
• Burst behaviour: measures the ability of the DUT to buffer packets.
• Packet re-ordering: measures the ability of the device under test to maintain sending order throughout transfer
to the destination.
• Packet correctness: packets or Frames must be well-formed, in that they include all required fields, conform to
length requirements, pass integrity checks, etc.
• Availability and capacity of the DUT i.e. when the DUT is fully “up” and connected, following measurements
should be captured for DUT without any network packet load:
– Includes average power consumption of the CPUs (in various power states) and system over specified
period of time. Time period should not be less than 60 seconds.
– Includes average per core CPU utilization over specified period of time. Time period should not be less
than 60 seconds.
– Includes the number of NIC interfaces supported.
– Includes headroom of VM workload processing cores (i.e. available for applications).
Features not to be tested
performance.
vsperf doesn’t intend to define or perform any functional tests. The aim is to focus on
Approach The testing approach adoped by the vswitchperf project is black box testing, meaning the test inputs can
be generated and the outputs captured and completely evaluated from the outside of the System Under Test. Some
metrics can be collected on the SUT, such as cpu or memory utilization if the collection has no/minimal impact on
benchmark. This section will look at the deployment scenarios and the general methodology used by vswitchperf. In
addition, this section will also specify the details of the Test Report that must be collected for each of the test cases.
Deployment Scenarios The following represents possible deployment test scenarios which can help to determine
the performance of both the virtual switch and the datapaths to physical ports (to NICs) and to logical ports (to VNFs):
Physical port → vSwitch → physical port
_
+--------------------------------------------------+ |
|
+--------------------+
| |
|
|
|
| |
|
|
v
| |
|
+--------------+
+--------------+
| |
|
|
phy port
| vSwitch
|
phy port
|
| |
+---+--------------+------------+--------------+---+ _|
^
:
|
|
:
v
+--------------------------------------------------+
|
|
|
traffic generator
|
|
|
+--------------------------------------------------+
Host
Physical port → vSwitch → VNF → vSwitch → physical port
_
+---------------------------------------------------+
|
|
|
+-------------------------------------------+
|
2.3. Testing Developer Guides
|
|
|
291
OPNFV Documentation, Release Danube
|
|
Application
|
| |
|
+-------------------------------------------+
| |
|
^
:
| |
|
|
|
| | Guest
|
:
v
| |
|
+---------------+
+---------------+
| |
|
| logical port 0|
| logical port 1|
| |
+---+---------------+-----------+---------------+---+ _|
^
:
|
|
:
v
_
+---+---------------+----------+---------------+---+ |
|
| logical port 0|
| logical port 1|
| |
|
+---------------+
+---------------+
| |
|
^
:
| |
|
|
|
| | Host
|
:
v
| |
|
+--------------+
+--------------+
| |
|
|
phy port
| vSwitch
|
phy port
|
| |
+---+--------------+------------+--------------+---+ _|
^
:
|
|
:
v
+--------------------------------------------------+
|
|
|
traffic generator
|
|
|
+--------------------------------------------------+
Physical port → vSwitch → VNF → vSwitch → VNF → vSwitch → physical port
_
+----------------------+ +----------------------+ |
|
Guest 1
| |
Guest 2
| |
|
+---------------+ | |
+---------------+ | |
|
| Application | | |
| Application | | |
|
+---------------+ | |
+---------------+ | |
|
^
|
| |
^
|
| |
|
|
v
| |
|
v
| |
|
+---------------+ | |
+---------------+ | |
|
| logical ports | | |
| logical ports | | |
|
|
0
1
| | |
|
0
1
| | |
+---+---------------+--+ +---+---------------+--+ _|
^
:
^
:
|
|
|
|
:
v
:
v
_
+---+---------------+---------+---------------+--+ |
|
|
0
1
|
|
3
4
| | |
|
| logical ports |
| logical ports | | |
|
+---------------+
+---------------+ | |
|
^
|
^
|
| |
|
|
L-----------------+
v
| |
|
+--------------+
+--------------+
| |
|
|
phy ports | vSwitch |
phy ports |
| |
+---+--------------+----------+--------------+---+ _|
^
^
:
:
|
|
|
|
:
:
v
v
292
Guests
Host
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
+--------------------------------------------------+
|
|
|
traffic generator
|
|
|
+--------------------------------------------------+
Physical port → VNF → vSwitch → VNF → physical port
_
+----------------------+ +----------------------+
|
|
Guest 1
| |
Guest 2
|
|
|+-------------------+ | | +-------------------+|
|
||
Application
| | | |
Application
||
|
|+-------------------+ | | +-------------------+|
|
|
^
|
| |
^
|
|
|
|
|
v
| |
|
v
|
|
|+-------------------+ | | +-------------------+|
|
||
logical ports
| | | |
logical ports
||
|
|| 0
1 | | | | 0
1 ||
|
++--------------------++ ++--------------------++ _|
^
:
^
:
(PCI passthrough) |
|
(PCI passthrough)
|
v
:
|
_
+--------++------------+-+------------++---------+
|
|
|
||
0
| |
1
||
|
|
|
|
|
||logical port| |logical port||
|
|
|
|
|
|+------------+ +------------+|
|
|
|
|
|
|
|
^
|
|
|
|
|
|
|
L-----------------+
|
|
|
|
|
|
|
|
|
|
|
|
|
|
vSwitch
|
|
|
|
|
|
+-----------------------------+
|
|
|
|
|
|
|
|
|
|
v
|
|
| +--------------+
+--------------+ |
|
| | phy port/VF |
| phy port/VF | |
|
+-+--------------+--------------+--------------+-+ _|
^
:
|
|
:
v
+--------------------------------------------------+
|
|
|
traffic generator
|
|
|
+--------------------------------------------------+
Guests
Host
Physical port → vSwitch → VNF
_
+---------------------------------------------------+
|
|
|
+-------------------------------------------+
|
|
|
Application
|
|
|
+-------------------------------------------+
|
|
^
|
|
|
|
|
:
|
|
+---------------+
|
|
| logical port 0|
|
2.3. Testing Developer Guides
|
|
|
|
|
|
|
|
|
|
Guest
293
OPNFV Documentation, Release Danube
+---+---------------+-------------------------------+ _|
^
|
:
_
+---+---------------+------------------------------+ |
|
| logical port 0|
| |
|
+---------------+
| |
|
^
| |
|
|
| | Host
|
:
| |
|
+--------------+
| |
|
|
phy port
| vSwitch
| |
+---+--------------+------------ -------------- ---+ _|
^
|
:
+--------------------------------------------------+
|
|
|
traffic generator
|
|
|
+--------------------------------------------------+
VNF → vSwitch → physical port
_
+---------------------------------------------------+ |
|
| |
|
+-------------------------------------------+
| |
|
|
Application
|
| |
|
+-------------------------------------------+
| |
|
:
| |
|
|
| | Guest
|
v
| |
|
+---------------+
| |
|
| logical port |
| |
+-------------------------------+---------------+---+ _|
:
|
v
_
+------------------------------+---------------+---+ |
|
| logical port |
| |
|
+---------------+
| |
|
:
| |
|
|
| | Host
|
v
| |
|
+--------------+
| |
|
vSwitch
|
phy port
|
| |
+-------------------------------+--------------+---+ _|
:
|
v
+--------------------------------------------------+
|
|
|
traffic generator
|
|
|
+--------------------------------------------------+
VNF → vSwitch → VNF → vSwitch
294
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
_
+-------------------------+ +-------------------------+ |
|
Guest 1
| |
Guest 2
| |
|
+-----------------+
| |
+-----------------+
| |
|
|
Application
|
| |
|
Application
|
| |
|
+-----------------+
| |
+-----------------+
| |
|
:
| |
^
| |
|
|
| |
|
| | Guest
|
v
| |
:
| |
|
+---------------+
| |
+---------------+
| |
|
| logical port 0|
| |
| logical port 0|
| |
+-----+---------------+---+ +---+---------------+-----+ _|
:
^
|
|
v
:
_
+----+---------------+------------+---------------+-----+ |
|
|
port 0
|
|
port 1
|
| |
|
+---------------+
+---------------+
| |
|
:
^
| |
|
|
|
| | Host
|
+--------------------+
| |
|
| |
|
vswitch
| |
+-------------------------------------------------------+ _|
HOST 1(Physical port → virtual switch → VNF → virtual switch → Physical port) → HOST 2(Physical port →
virtual switch → VNF → virtual switch → Physical port)
HOST 1 (PVP) → HOST 2 (PVP)
_
+----------------------+ +----------------------+ |
|
Guest 1
| |
Guest 2
| |
|
+---------------+ | |
+---------------+ | |
|
| Application | | |
| Application | | |
|
+---------------+ | |
+---------------+ | |
|
^
|
| |
^
|
| |
|
|
v
| |
|
v
| |
|
+---------------+ | |
+---------------+ | |
|
| logical ports | | |
| logical ports | | |
|
|
0
1
| | |
|
0
1
| | |
+---+---------------+--+ +---+---------------+--+ _|
^
:
^
:
|
|
|
|
:
v
:
v
_
+---+---------------+--+ +---+---------------+--+ |
|
|
0
1
| | |
|
3
4
| | |
|
| logical ports | | |
| logical ports | | |
|
+---------------+ | |
+---------------+ | |
|
^
|
| |
^
|
| |
|
|
v
| |
|
v
| |
|
+--------------+
| |
+--------------+
| |
|
|
phy ports |
| |
|
phy ports |
| |
+---+--------------+---+ +---+--------------+---+ _|
^
:
:
:
|
+-----------------+
|
:
v
+--------------------------------------------------+
|
|
2.3. Testing Developer Guides
Guests
Hosts
295
OPNFV Documentation, Release Danube
|
traffic generator
|
|
|
+--------------------------------------------------+
Note: For tests where the traffic generator and/or measurement receiver are implemented on VM and connected to the
virtual switch through vNIC, the issues of shared resources and interactions between the measurement devices and the
device under test must be considered.
Note: Some RFC 2889 tests require a full-mesh sending and receiving pattern involving more than two ports. This
possibility is illustrated in the Physical port → vSwitch → VNF → vSwitch → VNF → vSwitch → physical port
diagram above (with 2 sending and 2 receiving ports, though all ports could be used bi-directionally).
Note: When Deployment Scenarios are used in RFC 2889 address learning or cache capacity testing, an additional
port from the vSwitch must be connected to the test device. This port is used to listen for flooded frames.
General Methodology: To establish the baseline performance of the virtual switch, tests would initially be run
with a simple workload in the VNF (the recommended simple workload VNF would be DPDK‘s testpmd application
forwarding packets in a VM or vloop_vnf a simple kernel module that forwards traffic between two network interfaces
inside the virtualized environment while bypassing the networking stack). Subsequently, the tests would also be
executed with a real Telco workload running in the VNF, which would exercise the virtual switch in the context of
higher level Telco NFV use cases, and prove that its underlying characteristics and behaviour can be measured and
validated. Suitable real Telco workload VNFs are yet to be identified.
Default Test Parameters The following list identifies the default parameters for suite of tests:
• Reference application: Simple forwarding or Open Source VNF.
• Frame size (bytes): 64, 128, 256, 512, 1024, 1280, 1518, 2K, 4k OR Packet size based on
use-case (e.g.
RTP 64B, 256B) OR Mix of packet sizes as maintained by the Functest project
<https://wiki.opnfv.org/traffic_profile_management>.
• Reordering check: Tests should confirm that packets within a flow are not reordered.
• Duplex: Unidirectional / Bidirectional. Default: Full duplex with traffic transmitting in both directions, as
network traffic generally does not flow in a single direction. By default the data rate of transmitted traffic should
be the same in both directions, please note that asymmetric traffic (e.g. downlink-heavy) tests will be mentioned
explicitly for the relevant test cases.
• Number of Flows: Default for non scalability tests is a single flow. For scalability tests the goal is to test with
maximum supported flows but where possible will test up to 10 Million flows. Start with a single flow and scale
up. By default flows should be added sequentially, tests that add flows simultaneously will explicitly call out
their flow addition behaviour. Packets are generated across the flows uniformly with no burstiness. For multicore tests should consider the number of packet flows based on vSwitch/VNF multi-thread implementation and
behavior.
• Traffic Types: UDP, SCTP, RTP, GTP and UDP traffic.
• Deployment scenarios are:
• Physical → virtual switch → physical.
• Physical → virtual switch → VNF → virtual switch → physical.
• Physical → virtual switch → VNF → virtual switch → VNF → virtual switch → physical.
• Physical → VNF → virtual switch → VNF → physical.
• Physical → virtual switch → VNF.
• VNF → virtual switch → Physical.
296
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
• VNF → virtual switch → VNF.
Tests MUST have these parameters unless otherwise stated. Test cases with non default parameters will be stated
explicitly.
Note: For throughput tests unless stated otherwise, test configurations should ensure that traffic traverses the installed
flows through the virtual switch, i.e. flows are installed and have an appropriate time out that doesn’t expire before
packet transmission starts.
Flow Classification Virtual switches classify packets into flows by processing and matching particular header fields
in the packet/frame and/or the input port where the packets/frames arrived. The vSwitch then carries out an action on
the group of packets that match the classification parameters. Thus a flow is considered to be a sequence of packets
that have a shared set of header field values or have arrived on the same port and have the same action applied to them.
Performance results can vary based on the parameters the vSwitch uses to match for a flow. The recommended flow
classification parameters for L3 vSwitch performance tests are: the input port, the source IP address, the destination
IP address and the Ethernet protocol type field. It is essential to increase the flow time-out time on a vSwitch before
conducting any performance tests that do not measure the flow set-up time. Normally the first packet of a particular
flow will install the flow in the vSwitch which adds an additional latency, subsequent packets of the same flow are not
subject to this latency if the flow is already installed on the vSwitch.
Test Priority Tests will be assigned a priority in order to determine which tests should be implemented immediately
and which tests implementations can be deferred.
Priority can be of following types: - Urgent: Must be implemented immediately. - High: Must be implemented in the
next release. - Medium: May be implemented after the release. - Low: May or may not be implemented at all.
SUT Setup The SUT should be configured to its “default” state. The SUT’s configuration or set-up must not change
between tests in any way other than what is required to do the test. All supported protocols must be configured and
enabled for each test set up.
Port Configuration The DUT should be configured with n ports where n is a multiple of 2. Half of the ports on the
DUT should be used as ingress ports and the other half of the ports on the DUT should be used as egress ports. Where
a DUT has more than 2 ports, the ingress data streams should be set-up so that they transmit packets to the egress ports
in sequence so that there is an even distribution of traffic across ports. For example, if a DUT has 4 ports 0(ingress),
1(ingress), 2(egress) and 3(egress), the traffic stream directed at port 0 should output a packet to port 2 followed by a
packet to port 3. The traffic stream directed at port 1 should also output a packet to port 2 followed by a packet to port
3.
Frame Formats
Frame formats Layer 2 (data link layer) protocols
• Ethernet II
+---------------------------+-----------+
| Ethernet Header | Payload | Check Sum |
+-----------------+---------+-----------+
|_________________|_________|___________|
14 Bytes
46 - 1500
4 Bytes
Bytes
Layer 3 (network layer) protocols
• IPv4
2.3. Testing Developer Guides
297
OPNFV Documentation, Release Danube
+-----------------+-----------+---------+-----------+
| Ethernet Header | IP Header | Payload | Checksum |
+-----------------+-----------+---------+-----------+
|_________________|___________|_________|___________|
14 Bytes
20 bytes 26 - 1480
4 Bytes
Bytes
• IPv6
+-----------------+-----------+---------+-----------+
| Ethernet Header | IP Header | Payload | Checksum |
+-----------------+-----------+---------+-----------+
|_________________|___________|_________|___________|
14 Bytes
40 bytes 26 - 1460
4 Bytes
Bytes
Layer 4 (transport layer) protocols
• TCP
• UDP
• SCTP
+-----------------+-----------+-----------------+---------+-----------+
| Ethernet Header | IP Header | Layer 4 Header | Payload | Checksum |
+-----------------+-----------+-----------------+---------+-----------+
|_________________|___________|_________________|_________|___________|
14 Bytes
40 bytes
20 Bytes
6 - 1460
4 Bytes
Bytes
Layer 5 (application layer) protocols
• RTP
• GTP
+-----------------+-----------+-----------------+---------+-----------+
| Ethernet Header | IP Header | Layer 4 Header | Payload | Checksum |
+-----------------+-----------+-----------------+---------+-----------+
|_________________|___________|_________________|_________|___________|
14 Bytes
20 bytes
20 Bytes
>= 6 Bytes
4 Bytes
Packet Throughput There is a difference between an Ethernet frame, an IP packet, and a UDP datagram. In the
seven-layer OSI model of computer networking, packet refers to a data unit at layer 3 (network layer). The correct
term for a data unit at layer 2 (data link layer) is a frame, and at layer 4 (transport layer) is a segment or datagram.
Important concepts related to 10GbE performance are frame rate and throughput. The MAC bit rate of 10GbE, defined
in the IEEE standard 802 .3ae, is 10 billion bits per second. Frame rate is based on the bit rate and frame format
definitions. Throughput, defined in IETF RFC 1242, is the highest rate at which the system under test can forward the
offered load, without loss.
The frame rate for 10GbE is determined by a formula that divides the 10 billion bits per second by the preamble +
frame length + inter-frame gap.
The maximum frame rate is calculated using the minimum values of the following parameters, as described in the
IEEE 802 .3ae standard:
• Preamble: 8 bytes * 8 = 64 bits
• Frame Length: 64 bytes (minimum) * 8 = 512 bits
298
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
• Inter-frame Gap: 12 bytes (minimum) * 8 = 96 bits
Therefore, Maximum Frame Rate (64B Frames) = MAC Transmit Bit Rate / (Preamble + Frame Length + Inter-frame
Gap) = 10,000,000,000 / (64 + 512 + 96) = 10,000,000,000 / 672 = 14,880,952.38 frame per second (fps)
RFCs for testing virtual switch performance The starting point for defining the suite of tests for benchmarking
the performance of a virtual switch is to take existing RFCs and standards that were designed to test their physical
counterparts and adapting them for testing virtual switches. The rationale behind this is to establish a fair comparison between the performance of virtual and physical switches. This section outlines the RFCs that are used by this
specification.
RFC 1242 Benchmarking Terminology for Network Interconnection Devices RFC 1242 defines the terminology
that is used in describing performance benchmarking tests and their results. Definitions and discussions covered
include: Back-to-back, bridge, bridge/router, constant load, data link frame size, frame loss rate, inter frame gap,
latency, and many more.
RFC 2544 Benchmarking Methodology for Network Interconnect Devices RFC 2544 outlines a benchmarking
methodology for network Interconnect Devices. The methodology results in performance metrics such as latency,
frame loss percentage, and maximum data throughput.
In this document network “throughput” (measured in millions of frames per second) is based on RFC 2544, unless
otherwise noted. Frame size refers to Ethernet frames ranging from smallest frames of 64 bytes to largest frames of
9K bytes.
Types of tests are:
1. Throughput test defines the maximum number of frames per second that can be transmitted without any error, or
0% loss ratio. In some Throughput tests (and those tests with long duration), evaluation of an additional frame
loss ratio is suggested. The current ratio (10^-7 %) is based on understanding the typical user-to-user packet
loss ratio needed for good application performance and recognizing that a single transfer through a vswitch must
contribute a tiny fraction of user-to-user loss. Further, the ratio 10^-7 % also recognizes practical limitations
when measuring loss ratio.
2. Latency test measures the time required for a frame to travel from the originating device through the network to
the destination device. Please note that RFC2544 Latency measurement will be superseded with a measurement
of average latency over all successfully transferred packets or frames.
3. Frame loss test measures the network’s response in overload conditions - a critical indicator of the network’s
ability to support real-time applications in which a large amount of frame loss will rapidly degrade service
quality.
4. Burst test assesses the buffering capability of a virtual switch. It measures the maximum number of frames
received at full line rate before a frame is lost. In carrier Ethernet networks, this measurement validates the
excess information rate (EIR) as defined in many SLAs.
5. System recovery to characterize speed of recovery from an overload condition.
6. Reset to characterize speed of recovery from device or software reset. This type of test has been updated by
RFC6201 as such, the methodology defined by this specification will be that of RFC 6201.
Although not included in the defined RFC 2544 standard, another crucial measurement in Ethernet networking is
packet delay variation. The definition set out by this specification comes from RFC5481.
RFC 2285 Benchmarking Terminology for LAN Switching Devices RFC 2285 defines the terminology that is
used to describe the terminology for benchmarking a LAN switching device. It extends RFC 1242 and defines: DUTs,
SUTs, Traffic orientation and distribution, bursts, loads, forwarding rates, etc.
2.3. Testing Developer Guides
299
OPNFV Documentation, Release Danube
RFC 2889 Benchmarking Methodology for LAN Switching RFC 2889 outlines a benchmarking methodology
for LAN switching, it extends RFC 2544. The outlined methodology gathers performance metrics for forwarding,
congestion control, latency, address handling and finally filtering.
RFC 3918 Methodology for IP Multicast Benchmarking RFC 3918 outlines a methodology for IP Multicast
benchmarking.
RFC 4737 Packet Reordering Metrics RFC 4737 describes metrics for identifying and counting re-ordered packets
within a stream, and metrics to measure the extent each packet has been re-ordered.
RFC 5481 Packet Delay Variation Applicability Statement RFC 5481 defined two common, but different forms
of delay variation metrics, and compares the metrics over a range of networking circumstances and tasks. The most
suitable form for vSwitch benchmarking is the “PDV” form.
RFC 6201 Device Reset Characterization RFC 6201 extends the methodology for characterizing the speed of
recovery of the DUT from device or software reset described in RFC 2544.
Item pass/fail criteria vswitchperf does not specify Pass/Fail criteria for the tests in terms of a threshold, as benchmarks do not (and should not do this). The results/metrics for a test are simply reported. If it had to be defined, a test
is considered to have passed if it succesfully completed and a relavent metric was recorded/reported for the SUT.
Suspension criteria and resumption requirements In the case of a throughput test, a test should be suspended if a
virtual switch is failing to forward any traffic. A test should be restarted from a clean state if the intention is to carry
out the test again.
Test deliverables Each test should produce a test report that details SUT information as well as the test results.
There are a number of parameters related to the system, DUT and tests that can affect the repeatability of a test results
and should be recorded. In order to minimise the variation in the results of a test, it is recommended that the test report
includes the following information:
• Hardware details including:
– Platform details.
– Processor details.
– Memory information (see below)
– Number of enabled cores.
– Number of cores used for the test.
– Number of physical NICs, as well as their details (manufacturer, versions, type and the PCI slot they are
plugged into).
– NIC interrupt configuration.
– BIOS version, release date and any configurations that were modified.
• Software details including:
– OS version (for host and VNF)
– Kernel version (for host and VNF)
– GRUB boot parameters (for host and VNF).
300
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
– Hypervisor details (Type and version).
– Selected vSwitch, version number or commit id used.
– vSwitch launch command line if it has been parameterised.
– Memory allocation to the vSwitch – which NUMA node it is using, and how many memory channels.
– Where the vswitch is built from source: compiler details including versions and the flags that were used to
compile the vSwitch.
– DPDK or any other SW dependency version number or commit id used.
– Memory allocation to a VM - if it’s from Hugpages/elsewhere.
– VM storage type: snapshot/independent persistent/independent non-persistent.
– Number of VMs.
– Number of Virtual NICs (vNICs), versions, type and driver.
– Number of virtual CPUs and their core affinity on the host.
– Number vNIC interrupt configuration.
– Thread affinitization for the applications (including the vSwitch itself) on the host.
– Details of Resource isolation, such as CPUs designated for Host/Kernel (isolcpu) and CPUs designated for
specific processes (taskset).
• Memory Details
– Total memory
– Type of memory
– Used memory
– Active memory
– Inactive memory
– Free memory
– Buffer memory
– Swap cache
– Total swap
– Used swap
– Free swap
• Test duration.
• Number of flows.
• Traffic Information:
– Traffic type - UDP, TCP, IMIX / Other.
– Packet Sizes.
• Deployment Scenario.
Note: Tests that require additional parameters to be recorded will explicitly specify this.
2.3. Testing Developer Guides
301
OPNFV Documentation, Release Danube
Test management This section will detail the test activities that will be conducted by vsperf as well as the infrastructure that will be used to complete the tests in OPNFV.
Planned activities and tasks; test progression A key consideration when conducting any sort of benchmark is
trying to ensure the consistency and repeatability of test results between runs. When benchmarking the performance
of a virtual switch there are many factors that can affect the consistency of results. This section describes these factors
and the measures that can be taken to limit their effects. In addition, this section will outline some system tests to
validate the platform and the VNF before conducting any vSwitch benchmarking tests.
System Isolation:
When conducting a benchmarking test on any SUT, it is essential to limit (and if reasonable, eliminate) any noise that
may interfere with the accuracy of the metrics collected by the test. This noise may be introduced by other hardware or
software (OS, other applications), and can result in significantly varying performance metrics being collected between
consecutive runs of the same test. In the case of characterizing the performance of a virtual switch, there are a number
of configuration parameters that can help increase the repeatability and stability of test results, including:
• OS/GRUB configuration:
– maxcpus = n where n >= 0; limits the kernel to using ‘n’ processors. Only use exactly what you need.
– isolcpus: Isolate CPUs from the general scheduler. Isolate all CPUs bar one which will be used by the OS.
– use taskset to affinitize the forwarding application and the VNFs onto isolated cores. VNFs and the vSwitch
should be allocated their own cores, i.e. must not share the same cores. vCPUs for the VNF should be
affinitized to individual cores also.
– Limit the amount of background applications that are running and set OS to boot to runlevel 3. Make sure
to kill any unnecessary system processes/daemons.
– Only enable hardware that you need to use for your test – to ensure there are no other interrupts on the
system.
– Configure NIC interrupts to only use the cores that are not allocated to any other process (VNF/vSwitch).
• NUMA configuration: Any unused sockets in a multi-socket system should be disabled.
• CPU pinning: The vSwitch and the VNF should each be affinitized to separate logical cores using a combination
of maxcpus, isolcpus and taskset.
• BIOS configuration: BIOS should be configured for performance where an explicit option exists, sleep states
should be disabled, any virtualization optimization technologies should be enabled, and hyperthreading should
also be enabled, turbo boost and overclocking should be disabled.
System Validation:
System validation is broken down into two sub-categories: Platform validation and VNF validation. The validation
test itself involves verifying the forwarding capability and stability for the sub-system under test. The rationale behind
system validation is two fold. Firstly to give a tester confidence in the stability of the platform or VNF that is being
tested; and secondly to provide base performance comparison points to understand the overhead introduced by the
virtual switch.
• Benchmark platform forwarding capability: This is an OPTIONAL test used to verify the platform and measure the base performance (maximum forwarding rate in fps and latency) that can be achieved by the platform
without a vSwitch or a VNF. The following diagram outlines the set-up for benchmarking Platform forwarding
capability:
__
+--------------------------------------------------+
|
+------------------------------------------+
|
|
|
|
|
|
|
l2fw or DPDK L2FWD app
|
|
302
|
|
|
Host
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
|
|
|
|
|
|
+------------------------------------------+
|
|
|
|
NIC
|
|
|
+---+------------------------------------------+---+ __|
^
:
|
|
:
v
+--------------------------------------------------+
|
|
|
traffic generator
|
|
|
+--------------------------------------------------+
• Benchmark VNF forwarding capability: This test is used to verify the VNF and measure the base performance
(maximum forwarding rate in fps and latency) that can be achieved by the VNF without a vSwitch. The performance metrics collected by this test will serve as a key comparison point for NIC passthrough technologies and
vSwitches. VNF in this context refers to the hypervisor and the VM. The following diagram outlines the set-up
for benchmarking VNF forwarding capability:
__
+--------------------------------------------------+
|
|
+------------------------------------------+
|
|
|
|
|
|
|
|
|
VNF
|
|
|
|
|
|
|
|
|
+------------------------------------------+
|
|
|
|
Passthrough/SR-IOV
|
| Host
|
+------------------------------------------+
|
|
|
|
NIC
|
|
|
+---+------------------------------------------+---+ __|
^
:
|
|
:
v
+--------------------------------------------------+
|
|
|
traffic generator
|
|
|
+--------------------------------------------------+
Methodology to benchmark Platform/VNF forwarding capability
The recommended methodology for the platform/VNF validation and benchmark is: - Run RFC2889 Maximum Forwarding Rate test, this test will produce maximum forwarding rate and latency results that will serve as the expected
values. These expected values can be used in subsequent steps or compared with in subsequent validation tests. Transmit bidirectional traffic at line rate/max forwarding rate (whichever is higher) for at least 72 hours, measure
throughput (fps) and latency. - Note: Traffic should be bidirectional. - Establish a baseline forwarding rate for what
the platform can achieve. - Additional validation: After the test has completed for 72 hours run bidirectional traffic
at the maximum forwarding rate once more to see if the system is still functional and measure throughput (fps) and
latency. Compare the measure the new obtained values with the expected values.
NOTE 1: How the Platform is configured for its forwarding capability test (BIOS settings, GRUB configuration,
runlevel...) is how the platform should be configured for every test after this
NOTE 2: How the VNF is configured for its forwarding capability test (# of vCPUs, vNICs, Memory, affinitization. . . )
is how it should be configured for every test that uses a VNF after this.
Methodology to benchmark the VNF to vSwitch to VNF deployment scenario
vsperf has identified the following concerns when benchmarking the VNF to vSwitch to VNF deployment scenario:
2.3. Testing Developer Guides
303
OPNFV Documentation, Release Danube
• The accuracy of the timing synchronization between VNFs/VMs.
• The clock accuracy of a VNF/VM if they were to be used as traffic generators.
• VNF traffic generator/receiver may be using resources of the system under test, causing at least three forms of
workload to increase as the traffic load increases (generation, switching, receiving).
The recommendation from vsperf is that tests for this sceanario must include an external HW traffic generator to act
as the tester/traffic transmitter and receiver. The perscribed methodology to benchmark this deployment scanrio with
an external tester involves the following three steps:
#. Determine the forwarding capability and latency through the virtual interface connected to the VNF/VM.
Fig. 2.1: Virtual interfaces performance benchmark
1. Determine the forwarding capability and latency through the VNF/hypervisor.
1. Determine the forwarding capability and latency for the VNF to vSwitch to VNF taking the information from
the previous two steps into account.
vsperf also identified an alternative configuration for the final step:
Environment/infrastructure VSPERF CI jobs are run using the OPNFV lab infrastructure as described by
the ‘Pharos Project <https://www.opnfv.org/community/projects/pharos>‘_ . A VSPERF POD is described here
https://wiki.opnfv.org/display/pharos/VSPERF+in+Intel+Pharos+Lab+-+Pod+12
vsperf CI vsperf CI jobs are broken down into:
• Daily job:
– Runs everyday takes about 10 hours to complete.
– TESTCASES_DAILY=’phy2phy_tput back2back phy2phy_tput_mod_vlan phy2phy_scalability pvp_tput
pvp_back2back pvvp_tput pvvp_back2back’.
304
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
Fig. 2.2: Hypervisor performance benchmark
Fig. 2.3: VNF to vSwitch to VNF performance benchmark
2.3. Testing Developer Guides
305
OPNFV Documentation, Release Danube
Fig. 2.4: VNF to vSwitch to VNF alternative performance benchmark
– TESTPARAM_DAILY=’–test-params TRAFFICGEN_PKT_SIZES=(64,128,512,1024,1518)’.
• Merge job:
– Runs whenever patches are merged to master.
– Runs a basic Sanity test.
• Verify job:
– Runs every time a patch is pushed to gerrit.
– Builds documentation.
Scripts: There are 2 scripts that are part of VSPERFs CI:
• build-vsperf.sh: Lives in the VSPERF repository in the ci/ directory and is used to run vsperf with the appropriate
cli parameters.
• vswitchperf.yml: YAML description of our jenkins job. lives in the RELENG repository.
More info on vsperf CI can be found here: https://wiki.opnfv.org/display/vsperf/VSPERF+CI
Responsibilities and authority The group responsible for managing, designing, preparing and executing the tests
listed in the LTD are the vsperf committers and contributors. The vsperf committers and contributors should work
with the relavent OPNFV projects to ensure that the infrastructure is in place for testing vswitches, and that the results
are published to common end point (a results database).
306
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
VSPERF IETF Internet Draft
This IETF INternet Draft on Benchmarking Virtual Switches in OPNFV was developed by VSPERF contributors and
is maintained in the IETF repo. at https://tools.ietf.org/htm
VSPERF Scenarios and CI Results
VSPERF Test Scenarios Predefined Tests run with CI:
Test
phy2phy_tput
back2back
phy2phy_tput_mod_vlan
phy2phy_cont
pvp_cont
pvvp_cont
phy2phy_scalability
pvp_tput
pvp_back2back
pvvp_tput
pvvp_back2back
phy2phy_cpu_load
phy2phy_mem_load
Definition
PacketLossRatio for Phy2Phy
BackToBackFrames for Phy2Phy
PacketLossRatioFrameModification for Phy2Phy
Phy2Phy blast vswitch at x% TX rate and measure throughput
PVP blast vswitch at x% TX rate and measure throughput
PVVP blast vswitch at x% TX rate and measure throughput
Scalability0PacketLoss for Phy2Phy
PacketLossRatio for PVP
BackToBackFrames for PVP
PacketLossRatio for PVVP
BackToBackFrames for PVVP
CPU0PacketLoss for Phy2Phy
Same as CPU0PacketLoss but using a memory intensive app
Deployment topologies:
• Phy2Phy: Physical port -> vSwitch -> Physical port.
• PVP: Physical port -> vSwitch -> VNF -> vSwitch -> Physical port.
• PVVP: Physical port -> vSwitch -> VNF -> vSwitch -> VNF -> vSwitch -> Physical port.
Loopback applications in the Guest:
• DPDK testpmd.
• Linux Bridge.
• l2fwd Kernel Module
Supported traffic generators:
• Spirent Testcenter
• Ixia: IxOS and IxNet.
• Xena
• MoonGen
• Dummy
OPNFV Test Results VSPERF CI jobs are
https://wiki.opnfv.org/display/vsperf/Vsperf+Results
run
daily
and
sample
results
can
be
found
at
The following example maps the results in the test dashboard to the appropriate test case in the VSPERF Framework
and specifies the metric the vertical/Y axis is plotting. Please note, the presence of dpdk within a test name signifies
that the vswitch under test was OVS with DPDK, while its absence indicates that the vswitch under test was stock
OVS.
2.3. Testing Developer Guides
307
OPNFV Documentation, Release Danube
Dashboard Test
tput_ovsdpdk
tput_ovs
b2b_ovsdpdk
b2b_ovs
tput_mod_vlan_ovs
tput_mod_vlan_ovsdpdk
scalability_ovs
scalability_ovsdpdk
pvp_tput_ovsdpdkuser
pvp_tput_ovsvirtio
pvp_b2b_ovsdpdkuser
pvp_b2b_ovsvirtio
pvvp_tput_ovsdpdkuser
pvvp_tput_ovsvirtio
pvvp_b2b_ovsdpdkuser
pvvp_b2b_ovsvirtio
Framework Test
phy2phy_tput
phy2phy_tput
back2back
back2back
phy2phy_tput_mod_vlan
phy2phy_tput_mod_vlan
phy2phy_scalability
phy2phy_scalability
pvp_tput
pvp_tput
pvp_back2back
pvp_back2back
pvvp_tput
pvvp_tput
pvvp_back2back
pvvp_back2back
Metric
Throughput (FPS)
Throughput (FPS)
Back-to-back value
Back-to-back value
Throughput (FPS)
Throughput (FPS)
Throughput (FPS)
Throughput (FPS)
Throughput (FPS)
Throughput (FPS)
Back-to-back value
Back-to-back value
Throughput (FPS)
Throughput (FPS)
Throughput (FPS)
Throughput (FPS)
Guest Interface
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
vhost-user
virtio-net
vhost-user
virtio-net
vhost-user
virtio-net
vhost-user
virtio-net
The loopback application in the VNF was used for PVP and PVVP scenarios was DPDK testpmd.
Indices
• search
Yardstick
OPNFV Yardstick developer guide
Introduction
Yardstick is a project dealing with performance testing. Yardstick produces its own test cases but can also be considered
as a framework to support feature project testing.
Yardstick developed a test API that can be used by any OPNFV project. Therefore there are many ways to contribute
to Yardstick.
You can:
• Develop new test cases
• Review codes
• Develop Yardstick API / framework
• Develop Yardstick grafana dashboards and Yardstick reporting page
• Write Yardstick documentation
This developer guide describes how to interact with the Yardstick project. The first section details the main working
areas of the project. The Second part is a list of “How to” to help you to join the Yardstick family whatever your field
of interest is.
Where can I find some help to start? This guide is made for you. You can have a look at the user guide. There
are also references on documentation, video tutorials, tips in the project wiki page. You can also directly contact us by
mail with [Yardstick] prefix in the title at [email protected] or on the IRC chan #opnfv-yardstick.
308
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
Yardstick developer areas
Yardstick framework Yardstick can be considered as a framework. Yardstick is release as a docker file, including
tools, scripts and a CLI to prepare the environement and run tests. It simplifies the integration of external test suites in
CI pipeline and provide commodity tools to collect and display results.
Since Danube, test categories also known as tiers have been created to group similar tests, provide consistant sub-lists
and at the end optimize test duration for CI (see How To section).
The definition of the tiers has been agreed by the testing working group.
The tiers are:
• smoke
• features
• components
• performance
• vnf
How Todos?
How Yardstick works? The installation and configuration of the Yardstick is described in the user guide.
How can I contribute to Yardstick? If you are already a contributor of any OPNFV project, you can contribute to
Yardstick. If you are totally new to OPNFV, you must first create your Linux Foundation account, then contact us in
order to declare you in the repository database.
We distinguish 2 levels of contributors:
• the standard contributor can push patch and vote +1/0/-1 on any Yardstick patch
• The commitor can vote -2/-1/0/+1/+2 and merge
Yardstick commitors are promoted by the Yardstick contributors.
Gerrit & JIRA introduction OPNFV uses Gerrit for web based code review and repository management for the
Git Version Control System. You can access OPNFV Gerrit. Please note that you need to have Linux Foundation ID
in order to use OPNFV Gerrit. You can get one from this link.
OPNFV uses JIRA for issue management. An important principle of change management is to have two-way traceability between issue management (i.e. JIRA) and the code repository (via Gerrit). In this way, individual commits
can be traced to JIRA issues and we also know which commits were used to resolve a JIRA issue.
If you want to contribute to Yardstick, you can pick a issue from Yardstick’s JIRA dashboard or you can create you
own issue and submit it to JIRA.
Install Git and Git-reviews Installing and configuring Git and Git-Review is necessary in order to submit code to
Gerrit. The Getting to the code page will provide you with some help for that.
2.3. Testing Developer Guides
309
OPNFV Documentation, Release Danube
Verify your patch locally before submitting Once you finish a patch, you can submit it to Gerrit for code review.
A developer sends a new patch to Gerrit will trigger patch verify job on Jenkins CI. The yardstick patch verify job
includes python flake8 check, unit test and code coverage test. Before you submit your patch, it is recommended to
run the patch verification in your local environment first.
Open a terminal window and set the project’s directory to the working directory using the cd command. Assume that
YARDSTICK_REPO_DIR is the path to the Yardstick project folder on your computer:
cd $YARDSTICK_REPO_DIR
Verify your patch:
./run_tests.sh
It is used in CI but also by the CLI.
Submit the code with Git Tell Git which files you would like to take into account for the next commit. This is
called ‘staging’ the files, by placing them into the staging area, using the git add command (or the synonym git
stage command):
git add $YARDSTICK_REPO_DIR/samples/sample.yaml
Alternatively, you can choose to stage all files that have been modified (that is the files you have worked on) since the
last time you generated a commit, by using the -a argument:
git add -a
Git won’t let you push (upload) any code to Gerrit if you haven’t pulled the latest changes first. So the next step is to
pull (download) the latest changes made to the project by other collaborators using the pull command:
git pull
Now that you have the latest version of the project and you have staged the files you wish to push, it is time to actually
commit your work to your local Git repository:
git commit --signoff -m "Title of change"
Test of change that describes in high level what was done. There is a lot of
documentation in code so you do not need to repeat it here.
JIRA: YARDSTICK-XXX
The message that is required for the commit should follow a specific set of rules. This practice allows to standardize
the description messages attached to the commits, and eventually navigate among the latter more easily.
This document happened to be very clear and useful to get started with that.
Push the code to Gerrit for review Now that the code has been comitted into your local Git repository the following
step is to push it online to Gerrit for it to be reviewed. The command we will use is git review:
git review
This will automatically push your local commit into Gerrit. You can add Yardstick committers and contributors to
review your codes.
310
Chapter 2. Test Frameworks
OPNFV Documentation, Release Danube
You can find Yardstick people info here.
Modify the code under review in Gerrit At the same time the code is being reviewed in Gerrit, you may need to
edit it to make some changes and then send it back for review. The following steps go through the procedure.
Once you have modified/edited your code files under your IDE, you will have to stage them. The ‘status’ command is
very helpful at this point as it provides an overview of Git’s current state:
git status
The output of the command provides us with the files that have been modified after the latest commit.
You can now stage the files that have been modified as part of the Gerrit code review edition/modification/improvement
using git add command. It is now time to commit the newly modified files, but the objective here is not to create
a new commit, we simply want to inject the new changes into the previous commit. You can achieve that with the
‘–amend’ option on the git commit command:
git commit --amend
If the commit was successful, the git status command should not return the updated files as about to be commited.
The final step consists in pushing the newly modified commit to Gerrit:
git review
Plugins
For information about Yardstick plugins, refer to the chapter Installing a plug-in into Yardstick in the user guide.
2.3. Testing Developer Guides
311
OPNFV Documentation, Release Danube
312
Chapter 2. Test Frameworks
CHAPTER 3
Developer
Documentation Guide
Documentation Guide
This page intends to cover the documentation handling for OPNFV. OPNFV projects are expected to create a variety
of document types, according to the nature of the project. Some of these are common to projects that develop/integrate
features into the OPNFV platform, e.g. Installation Instructions and User/Configurations Guides. Other document
types may be project-specific.
• Getting Started with Documentation for Your Project
– Licencing your documentation
– How and where to store the document content files in your repository
• Document structure and contribution
– Release documentation
– Testing documentation
– Development Documentation
Getting Started with Documentation for Your Project
OPNFV documentation is automated and integrated into our git & gerrit toolchains.
We use RST document templates in our repositories and automatically render to HTML and PDF versions of the
documents in our artifact store, our WiKi is also able to integrate these rendered documents directly allowing projects
to use the revision controlled documentation process for project information, content and deliverables. Read this page
which elaborates on how documentation is to be included within opnfvdocs.
Licencing your documentation
All contributions to the OPNFV project are done in accordance with the OPNFV licensing requirements. Documentation in OPNFV is contributed in accordance with the Creative Commons 4.0 and the ‘SPDX https://spdx.org/>‘_
licence. All documentation files need to be licensed using the text below. The license may be applied in the first lines
of all contributed RST files:
.. This work is licensed under a Creative Commons Attribution 4.0 International License.
.. SPDX-License-Identifier: CC-BY-4.0
.. (c) <optionally add copywriters name>
313
OPNFV Documentation, Release Danube
These lines will not be rendered in the html and pdf files.
How and where to store the document content files in your repository
All documentation for your project should be structured and stored in the <repo>/docs/ directory. The documentation toolchain will look in these directories and be triggered on events in these directories when generating
documents.
Document structure and contribution
A general structure is proposed for storing and handling documents that are common across many projects but also
for documents that may be project specific. The documentation is divided into three areas Release, Development and
Testing. Templates for these areas can be found under opnfvdocs/docs/templates/.
Project teams are encouraged to use templates provided by the opnfvdocs project to ensure that there is consistency
across the community. Following representation shows the expected structure:
docs/
-- development
|
-- design
|
-- overview
|
-- requirements
-- release
|
-- configguide
|
-- installation
|
-- release-notes
|
-- scenarios
|
|
-- scenario.name
|
-- userguide
-- testing
-- developer
-- user
Release documentation
Release documentation is the set of documents that are published for each OPNFV release. These documents are
created and developed following the OPNFV release process and milestones and should reflect the content of the
OPNFV release. These documents have a master index.rst file in the <opnfvdocs> repository and extract content
from other repositories. To provide content into these documents place your <content>.rst files in a directory in your
repository that matches the master document and add a reference to that file in the correct place in the corresponding
index.rst file in opnfvdocs/docs/release/.
Platform Overview: opnfvdocs/docs/release/overview
• Note this document is not a contribution driven document
• Content for this is prepared by the Marketing team together with the opnfvdocs team
Installation Instruction: <repo>/docs/release/installation
• Folder for documents describing how to deploy each installer and scenario descriptions
• Release notes will be included here <To Confirm>
• Security related documents will be included here
314
Chapter 3. Developer
OPNFV Documentation, Release Danube
• Note that this document will be compiled into ‘OPNFV Installation Instruction’
User Guide: <repo>/docs/release/userguide
• Folder for manuals to use specific features
• Folder for documents describing how to install/configure project specific components and features
• Can be the directory where API reference for project specific features are stored
• Note this document will be compiled into ‘OPNFV userguide’
Configuration Guide: <repo>/docs/release/configguide
• Brief introduction to configure OPNFV with its dependencies.
Release Notes: <repo>/docs/release/release-notes
• Changes brought about in the release cycle.
• Include version details.
Testing documentation
Documentation created by test projects can be stored under two different sub directories /user or /developemnt. Release
notes will be stored under <repo>/docs/release/release-notes
User documentation: <repo>/testing/user/ Will collect the documentation of the test projects allowing the
end user to perform testing towards a OPNFV SUT e.g. Functest/Yardstick/Vsperf/Storperf/Bottlenecks/Qtip installation/config & user guides.
Development documentation: <repo>/testing/developent/ Will collect documentation to explain how to
create your own test case and leverage existing testing frameworks e.g. developer guides.
Development Documentation
Project specific documents such as design documentation, project overview or requirement documentation can be
stored under /docs/development. Links to generated documents will be dislayed under Development Documentaiton
section on docs.opnfv.org. You are encouraged to establish the following basic structure for your project as needed:
Requirement Documentation: <repo>/docs/development/requirements/
• Folder for your requirement documentation
• For details on requirements projects’ structures see the Requirements Projects page.
Design Documentation: <repo>/docs/development/design
• Folder for your upstream design documents (blueprints, development proposals, etc..)
Project overview: <repo>/docs/development/overview
• Folder for any project specific documentation.
Including your Documentation
3.1. Documentation Guide
315
OPNFV Documentation, Release Danube
• In your project repository
• In OPNFVDocs Composite Documentation
– In toctree
– As Hyperlink
• ‘doc8’ Validation
• Testing: Build Documentation Locally
– Composite OPNFVDOCS documentation
– Individual project documentation
In your project repository
Add your documentation to your repository in the folder structure and according to the templates listed above. The
documentation templates you will require are available in opnfvdocs/docs/templates/ repository, you should copy the
relevant templates to your <repo>/docs/ directory in your repository. For instance if you want to document userguide,
then your steps shall be as follows:
git clone ssh://<your_id>@gerrit.opnfv.org:29418/opnfvdocs.git
cp -p opnfvdocs/docs/userguide/* <my_repo>/docs/userguide/
You should then add the relevant information to the template that will explain the documentation. When you are done
writing, you can commit the documentation to the project repository.
git add .
git commit --signoff --all
git review
In OPNFVDocs Composite Documentation
In toctree
To import project documents from project repositories, we use submodules. Each
opnfvdocs/docs/submodule/ as follows:
project
is
stored
in
To include your project specific documentation in the composite documentation, first identify where your project
documentation should be included. Say your project userguide should figure in the ‘OPNFV Userguide’, then:
vim opnfvdocs/docs/release/userguide.introduction.rst
This opens the text editor. Identify where you want to add the userguide. If the userguide is to be added to the toctree,
simply include the path to it, example:
316
Chapter 3. Developer
OPNFV Documentation, Release Danube
.. toctree::
:maxdepth: 1
submodules/functest/docs/userguide/index
submodules/bottlenecks/docs/userguide/index
submodules/yardstick/docs/userguide/index
<submodules/path-to-your-file>
As Hyperlink
It’s pretty common to want to reference another location in the OPNFV documentation and it’s pretty easy to do with
reStructuredText. This is a quick primer, more information is in the Sphinx section on Cross-referencing arbitrary
locations.
Within a single document, you can reference another section simply by:
This is a reference to `The title of a section`_
Assuming that somewhere else in the same file there a is a section title something like:
The title of a section
^^^^^^^^^^^^^^^^^^^^^^
It’s typically better to use :ref: syntax and labels to provide links as they work across files and are resilient to
sections being renamed. First, you need to create a label something like:
.. _a-label:
The title of a section
^^^^^^^^^^^^^^^^^^^^^^
Note: The underscore (_) before the label is required.
Then you can reference the section anywhere by simply doing:
This is a reference to :ref:`a-label`
or:
This is a reference to :ref:`a section I really liked <a-label>`
Note: When using :ref:-style links, you don’t need a trailing underscore (_).
Because the labels have to be unique, it usually makes sense to prefix the labels with the project name to help share
the label space, e.g., sfc-user-guide instead of just user-guide.
Once you have made these changes you need to push the patch back to the opnfvdocs team for review and integration.
git add .
git commit --signoff --all
git review
Be sure to add the project leader of the opnfvdocs project as a reviewer of the change you just pushed in gerrit.
‘doc8’ Validation
It is recommended that all rst content is validated by doc8 standards. To validate your rst files using doc8, install doc8.
3.1. Documentation Guide
317
OPNFV Documentation, Release Danube
sudo pip install doc8
doc8 can now be used to check the rst files. Execute as,
doc8 --ignore D000,D001 <file>
Testing: Build Documentation Locally
Composite OPNFVDOCS documentation
To build whole documentation under opnfvdocs/, follow these steps:
Install virtual environment.
sudo pip install virtualenv
cd /local/repo/path/to/project
Download the OPNFVDOCS repository.
git clone https://gerrit.opnfv.org/gerrit/opnfvdocs
Change directory to opnfvdocs & install requirements.
cd opnfvdocs
sudo pip install -r etc/requirements.txt
Update submodules, build documentation using tox & then open using any browser.
cd opnfvdocs
git submodule update --init
tox -edocs
firefox docs/_build/html/index.html
Note: Make sure to run tox -edocs and not just tox.
Individual project documentation
To test how the documentation renders in HTML, follow these steps:
Install virtual environment.
sudo pip install virtualenv
cd /local/repo/path/to/project
Download the opnfvdocs repository.
git clone https://gerrit.opnfv.org/gerrit/opnfvdocs
Change directory to opnfvdocs & install requirements.
cd opnfvdocs
sudo pip install -r etc/requirements.txt
Move the conf.py file to your project folder where RST files have been kept:
mv opnfvdocs/docs/conf.py <path-to-your-folder>/
318
Chapter 3. Developer
OPNFV Documentation, Release Danube
Move the static files to your project folder:
mv opnfvdocs/_static/ <path-to-your-folder>/
Build the documentation from within your project folder:
sphinx-build -b html <path-to-your-folder> <path-to-output-folder>
Your documentation shall be built as HTML inside the specified output folder directory.
Note: Be sure to remove the conf.py, the static/ files and the output folder from the <project>/docs/. This is for
testing only. Only commit the rst files and related content.
Addendum
Index File
The index file must relatively refence your other rst files in that directory.
Here is an example index.rst :
*******************
Documentation Title
*******************
.. toctree::
:numbered:
:maxdepth: 2
documentation-example
Source Files
Document source files have to be written in reStructuredText format (rst). Each file would be build as an html page.
Here is an example source rst file :
=============
Chapter Title
=============
Section Title
=============
Subsection Title
---------------Hello!
Writing RST Markdown
See http://sphinx-doc.org/rest.html .
Hint: You can add dedicated contents by using ‘only’ directive with build type (‘html’ and ‘singlehtml’) for OPNFV
document. But, this is not encouraged to use since this may make different views.
3.1. Documentation Guide
319
OPNFV Documentation, Release Danube
.. only:: html
This line will be shown only in html version.
Verify Job
The verify job name is docs-verify-rtd-{branch}.
When you send document changes to gerrit, jenkins will create your documents in HTML formats (normal and singlepage) to verify that new document can be built successfully. Please check the jenkins log and artifact carefully. You
can improve your document even though if the build job succeeded.
Merge Job
The merge job name is docs-merge-rtd-{branch}.
Once the patch is merged, jenkins will automatically trigger building of the new documentation. This might take about
15 minutes while readthedocs builds the documentatation. The newly built documentation shall show up as appropriate
placed in docs.opnfv.org/{branch}/path-to-file.
OPNFV Projects
Apex
OPNFV Installation instructions (Apex)
Contents:
Abstract
This document describes how to install the Danube release of OPNFV when using Apex as a deployment tool covering
it’s limitations, dependencies and required system resources.
License
Danube release of OPNFV when using Apex as a deployment tool Docs (c) by Tim Rozet (Red Hat) and Dan Radez
(Red Hat)
Danube release of OPNFV when using Apex as a deployment tool Docs are licensed under a Creative Commons
Attribution 4.0 International License. You should have received a copy of the license along with this. If not, see
<http://creativecommons.org/licenses/by/4.0/>.
Introduction
This document describes the steps to install an OPNFV Danube reference platform, as defined by the Genesis Project
using the Apex installer.
The audience is assumed to have a good background in networking and Linux administration.
320
Chapter 3. Developer
OPNFV Documentation, Release Danube
Preface
Apex uses Triple-O from the RDO Project OpenStack distribution as a provisioning tool. The Triple-O image based
life cycle installation tool provisions an OPNFV Target System (3 controllers, 2 or more compute nodes) with OPNFV
specific configuration provided by the Apex deployment tool chain.
The Apex deployment artifacts contain the necessary tools to deploy and configure an OPNFV target system using the Apex deployment toolchain. These artifacts offer the choice of using the Apex bootable ISO
(opnfv-apex-danube.iso) to both install CentOS 7 and the necessary materials to deploy or the Apex RPMs
(opnfv-apex*.rpm), and their associated dependencies, which expects installation to a CentOS 7 libvirt enabled
host. The RPM contains a collection of configuration files, prebuilt disk images, and the automatic deployment script
(opnfv-deploy).
An OPNFV install requires a “Jumphost” in order to operate. The bootable ISO will allow you to install a customized CentOS 7 release to the Jumphost, which includes the required packages needed to run opnfv-deploy.
If you already have a Jumphost with CentOS 7 installed, you may choose to skip the ISO step and simply install the
(opnfv-apex*.rpm) RPMs. The RPMs are the same RPMs included in the ISO and include all the necessary disk
images and configuration files to execute an OPNFV deployment. Either method will prepare a host to the same ready
state for OPNFV deployment.
opnfv-deploy instantiates a Triple-O Undercloud VM server using libvirt as its provider. This VM is then configured and used to provision the OPNFV target deployment (3 controllers, n compute nodes). These nodes can be either
virtual or bare metal. This guide contains instructions for installing either method.
Triple-O Deployment Architecture
Apex is based on the OpenStack Triple-O project as distributed by the RDO Project. It is important to understand the
basics of a Triple-O deployment to help make decisions that will assist in successfully deploying OPNFV.
Triple-O stands for OpenStack On OpenStack. This means that OpenStack will be used to install OpenStack. The
target OPNFV deployment is an OpenStack cloud with NFV features built-in that will be deployed by a smaller allin-one deployment of OpenStack. In this deployment methodology there are two OpenStack installations. They are
referred to as the undercloud and the overcloud. The undercloud is used to deploy the overcloud.
The undercloud is the all-in-one installation of OpenStack that includes baremetal provisioning capability. The undercloud will be deployed as a virtual machine on a jumphost. This VM is pre-built and distributed as part of the Apex
RPM.
The overcloud is OPNFV. Configuration will be passed into undercloud and the undercloud will use OpenStack’s
orchestration component, named Heat, to execute a deployment that will provision the target OPNFV nodes.
Apex High Availability Architecture
Undercloud The undercloud is not Highly Available. End users do not depend on the undercloud. It is only for
management purposes.
Overcloud Apex will deploy three control nodes in an HA deployment. Each of these nodes will run the following
services:
• Stateless OpenStack services
• MariaDB / Galera
• RabbitMQ
• OpenDaylight
3.2. OPNFV Projects
321
OPNFV Documentation, Release Danube
• HA Proxy
• Pacemaker & VIPs
• Ceph Monitors and OSDs
Stateless OpenStack services All running stateless OpenStack services are load balanced by HA Proxy. Pacemaker
monitors the services and ensures that they are running.
Stateful OpenStack services All running stateful OpenStack services are load balanced by HA Proxy. They are
monitored by pacemaker in an active/passive failover configuration.
MariaDB / Galera The MariaDB database is replicated across the control nodes using Galera. Pacemaker is responsible for a proper start up of the Galera cluster. HA Proxy provides and active/passive failover methodology to
connections to the database.
RabbitMQ The message bus is managed by Pacemaker to ensure proper start up and establishment of clustering
across cluster members.
OpenDaylight OpenDaylight is currently installed on all three control nodes and started as an HA cluster unless otherwise noted for that scenario. OpenDaylight’s database, known as MD-SAL, breaks up pieces of the database
into “shards”. Each shard will have its own election take place, which will determine which OpenDaylight node
is the leader for that shard. The other OpenDaylight nodes in the cluster will be in standby. Every Open vSwitch
node connects to every OpenDaylight to enable HA.
HA Proxy HA Proxy is monitored by Pacemaker to ensure it is running across all nodes and available to balance
connections.
Pacemaker & VIPs Pacemaker has relationships and restraints setup to ensure proper service start up order and
Virtual IPs associated with specific services are running on the proper host.
Ceph Monitors & OSDs The Ceph monitors run on each of the control nodes. Each control node also has a Ceph
OSD running on it. By default the OSDs use an autogenerated virtual disk as their target device. A nonautogenerated device can be specified in the deploy file.
VM Migration is configured and VMs can be evacuated as needed or as invoked by tools such as heat as part of a
monitored stack deployment in the overcloud.
OPNFV Scenario Architecture
OPNFV distinguishes different types of SDN controllers, deployment options, and features into “scenarios”. These
scenarios are universal across all OPNFV installers, although some may or may not be supported by each installer.
The standard naming convention for a scenario is: <VIM platform>-<SDN type>-<feature>-<ha/noha>
The only supported VIM type is “OS” (OpenStack), while SDN types can be any supported SDN controller. “feature”
includes things like ovs_dpdk, sfc, etc. “ha” or “noha” determines if the deployment will be highly available. If “ha”
is used at least 3 control nodes are required.
OPNFV Scenarios in Apex
Apex provides pre-built scenario files in /etc/opnfv-apex which a user can select from to deploy the desired scenario.
Simply pass the desired file to the installer as a (-d) deploy setting. Read further in the Apex documentation to
learn more about invoking the deploy command. Below is quick reference matrix for OPNFV scenarios supported
in Apex. Please refer to the respective OPNFV Docs documentation for each scenario in order to see a full scenario
description. Also, please refer to release-notes for information about known issues per scenario. The following
scenarios correspond to a supported <Scenario>.yaml deploy settings file:
322
Chapter 3. Developer
OPNFV Documentation, Release Danube
Scenario
os-nosdn-nofeature-ha
os-nosdn-nofeature-noha
os-nosdn-ovs-ha
os-nosdn-ovs-noha
os-nosdn-fdio-ha
os-nosdn-fdio-noha
os-nosdn-kvm-ha
os-nosdn-kvm-noha
os-nosdn-performance-ha
os-odl_l3-nofeature-ha
os-odl_l3-nofeature-noha
os-odl_l3-ovs-ha
os-odl_l3-ovs-noha
os-odl-bgpvpn-ha
os-odl-bgpvpn-noha
os-odl-gluon-noha
os-odl_l3-csit-noha
os-odl_l3-fdio-ha
os-odl_l3-fdio-noha
os-odl_l2-fdio-ha
os-odl_l2-fdio-noha
os-odl_l2-sfc-noha
os-onos-nofeature-ha
os-onos-sfc-ha
os-ovn-nofeature-noha
Owner
Apex
Apex
OVS for NFV
OVS for NFV
FDS
FDS
KVM for NFV
KVM for NFV
Apex
Apex
Apex
OVS for NFV
OVS for NFV
SDNVPN
SDNVPN
GluOn
Apex
FDS
FDS
FDS
FDS
SFC
ONOSFW
ONOSFW
Apex
Supported
Yes
Yes
Yes
Yes
No
No
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
No
No
No
Yes
Setup Requirements
Jumphost Requirements The Jumphost requirements are outlined below:
1. CentOS 7 (from ISO or self-installed).
2. Root access.
3. libvirt virtualization support.
4. minimum 1 networks and maximum 5 networks, multiple NIC and/or VLAN combinations are supported. This
is virtualized for a VM deployment.
5. The Danube Apex RPMs and their dependencies.
6. 16 GB of RAM for a bare metal deployment, 64 GB of RAM for a VM deployment.
Network Requirements Network requirements include:
1. No DHCP or TFTP server running on networks used by OPNFV.
2. 1-5 separate networks with connectivity between Jumphost and nodes.
• Control Plane (Provisioning)
• Private Tenant-Networking Network*
• External Network*
• Storage Network*
3.2. OPNFV Projects
323
OPNFV Documentation, Release Danube
• Internal API Network* (required for IPv6 **)
3. Lights out OOB network access from Jumphost with IPMI node enabled (bare metal deployment only).
4. External network is a routable network from outside the cloud, deployment. The External network is where
public internet access would reside if available.
*These networks can be combined with each other or all combined on the Control Plane network.
**Internal API network, by default, is collapsed with provisioning in IPv4 deployments, this is not possible with
the current lack of PXE boot support and therefore the API network is required to be its own network in an IPv6
deployment.
Bare Metal Node Requirements Bare metal nodes require:
1. IPMI enabled on OOB interface for power control.
2. BIOS boot priority should be PXE first then local hard disk.
3. BIOS PXE interface should include Control Plane network mentioned above.
Execution Requirements (Bare Metal Only)
information:
In order to execute a deployment, one must gather the following
1. IPMI IP addresses for the nodes.
2. IPMI login information for the nodes (user/pass).
3. MAC address of Control Plane / Provisioning interfaces of the overcloud nodes.
Installation High-Level Overview - Bare Metal Deployment
The setup presumes that you have 6 or more bare metal servers already setup with network connectivity on at least 1
or more network interfaces for all servers via a TOR switch or other network implementation.
The physical TOR switches are not automatically configured from the OPNFV reference platform. All the networks
involved in the OPNFV infrastructure as well as the provider networks and the private tenant VLANs needs to be
manually configured.
The Jumphost can be installed using the bootable ISO or by using the (opnfv-apex*.rpm) RPMs and their dependencies. The Jumphost should then be configured with an IP gateway on its admin or public interface and configured
with a working DNS server. The Jumphost should also have routable access to the lights out network for the overcloud
nodes.
opnfv-deploy is then executed in order to deploy the undercloud VM and to provision the overcloud nodes.
opnfv-deploy uses three configuration files in order to know how to install and provision the OPNFV target
system. The information gathered under section Execution Requirements (Bare Metal Only) is put into the YAML
file /etc/opnfv-apex/inventory.yaml configuration file. Deployment options are put into the YAML file
/etc/opnfv-apex/deploy_settings.yaml. Alternatively there are pre-baked deploy_settings files available in /etc/opnfv-apex/. These files are named with the naming convention os-sdn_controller-enabled_feature[no]ha.yaml. These files can be used in place of the /etc/opnfv-apex/deploy_settings.yaml file if one
suites your deployment needs. Networking definitions gathered under section Network Requirements are put into the
YAML file /etc/opnfv-apex/network_settings.yaml. opnfv-deploy will boot the undercloud VM
and load the target deployment configuration into the provisioning toolchain. This information includes MAC address,
IPMI, Networking Environment and OPNFV deployment options.
Once configuration is loaded and the undercloud is configured it will then reboot the overcloud nodes via IPMI. The
nodes should already be set to PXE boot first off the admin interface. The nodes will first PXE off of the undercloud
PXE server and go through a discovery/introspection process.
324
Chapter 3. Developer
OPNFV Documentation, Release Danube
Introspection boots off of custom introspection PXE images. These images are designed to look at the properties of
the hardware that is being booted and report the properties of it back to the undercloud node.
After introspection the undercloud will execute a Heat Stack Deployment to continue node provisioning and configuration. The nodes will reboot and PXE from the undercloud PXE server again to provision each node using Glance
disk images provided by the undercloud. These disk images include all the necessary packages and configuration for
an OPNFV deployment to execute. Once the disk images have been written to node’s disks the nodes will boot locally
and execute cloud-init which will execute the final node configuration. This configuration is largely completed by
executing a puppet apply on each node.
Installation High-Level Overview - VM Deployment
The VM nodes deployment operates almost the same way as the bare metal deployment with a few differences mainly
related to power management. opnfv-deploy still deploys an undercloud VM. In addition to the undercloud VM
a collection of VMs (3 control nodes + 2 compute for an HA deployment or 1 control node and 1 or more compute
nodes for a Non-HA Deployment) will be defined for the target OPNFV deployment. The part of the toolchain that
executes IPMI power instructions calls into libvirt instead of the IPMI interfaces on baremetal servers to operate the
power management. These VMs are then provisioned with the same disk images and configuration that baremetal
would be.
To Triple-O these nodes look like they have just built and registered the same way as bare metal nodes, the main
difference is the use of a libvirt driver for the power management.
Installation Guide - Bare Metal Deployment
This section goes step-by-step on how to correctly install and provision the OPNFV target system to bare metal nodes.
Install Bare Metal Jumphost
1a. If your Jumphost does not have CentOS 7 already on it, or you would like to do a fresh install, then download the Apex bootable ISO from the OPNFV artifacts site <http://artifacts.opnfv.org/apex.html>. There have
been isolated reports of problems with the ISO having trouble completing installation successfully. In the unexpected event the ISO does not work please workaround this by downloading the CentOS 7 DVD and performing
a “Virtualization Host” install. If you perform a “Minimal Install” or install type other than “Virtualization
Host” simply run sudo yum groupinstall "Virtualization Host" chkconfig libvirtd
on && reboot to install virtualzation support and enable libvirt on boot. If you use the CentOS 7 DVD
proceed to step 1b once the CentOS 7 with “Virtualzation Host” support is completed.
1b. If your Jump host already has CentOS 7 with libvirt running on it then install the install the RDO Newton
Release RPM and epel-release:
sudo yum install https://repos.fedorapeople.org/repos/openstack/openstack-newton/rdo-re
sudo yum install epel-release
The RDO Project release repository is needed to install OpenVSwitch, which is a dependency of opnfv-apex. If
you do not have external connectivity to use this repository you need to download the OpenVSwitch RPM from
the RDO Project repositories and install it with the opnfv-apex RPM.
2a. Boot the ISO off of a USB or other installation media and walk through installing OPNFV CentOS 7. The
ISO comes prepared to be written directly to a USB drive with dd as such:
dd if=opnfv-apex.iso of=/dev/sdX bs=4M
Replace /dev/sdX with the device assigned to your usb drive. Then select the USB device as the boot media on
your Jumphost
3.2. OPNFV Projects
325
OPNFV Documentation, Release Danube
2b. If your Jump host already has CentOS 7 with libvirt running on it then install the opnfv-apex RPMs using
the OPNFV artifacts yum repo. This yum repo is created at release. It will not exist before release day.
sudo yum install http://artifacts.opnfv.org/apex/danube/opnfv-apex-release-danube.noarc
Once you have installed the repo definitions for Apex, RDO and EPEL then yum install Apex:
sudo yum install opnfv-apex
If ONOS will be used, install the ONOS rpm instead of the opnfv-apex rpm.
sudo yum install opnfv-apex-onos
2c. If you choose not to use the Apex yum repo or you choose to use pre-released RPMs you can download and install the required RPMs from the artifacts site <http://artifacts.opnfv.org/apex.html>. The following RPMs are
available for installation:
• opnfv-apex - OpenDaylight L2 / L3 and ODL SFC support *
• opnfv-apex-onos - ONOS support *
• opnfv-apex-undercloud - (reqed) Undercloud Image
• opnfv-apex-common - (reqed) Supporting config files and scripts
• python34-markupsafe - (reqed) Dependency of opnfv-apex-common **
• python3-jinja2 - (reqed) Dependency of opnfv-apex-common **
• python3-ipmi - (reqed) Dependency of opnfv-apex-common **
* One or more of these RPMs is required Only one of opnfv-apex or opnfv-apex-onos is required. It is safe to
leave the unneeded SDN controller’s RPMs uninstalled if you do not intend to use them.
** These RPMs are not yet distributed by CentOS or EPEL. Apex has built these for distribution with Apex
while CentOS and EPEL do not distribute them. Once they are carried in an upstream channel Apex will no
longer carry them and they will not need special handling for installation.
The EPEL and RDO yum repos are still required: sudo yum install epel-release sudo yum
install https://repos.fedorapeople.org/repos/openstack/openstack-newton/rdo-release-new
Once the apex RPMs are downloaded install them by passing the file names directly to yum: sudo yum
install python34-markupsafe-<version>.rpm python3-jinja2-<version>.rpm
python3-ipmi-<version>.rpm
sudo yum install opnfv-apex-<version>.rpm
opnfv-apex-undercloud-<version>.rpm opnfv-apex-common-<version>.rpm
3. After the operating system and the opnfv-apex RPMs are installed, login to your Jumphost as root.
4. Configure IP addresses on the interfaces that you have selected as your networks.
5. Configure the IP gateway to the Internet either, preferably on the public interface.
6. Configure your /etc/resolv.conf to point to a DNS server (8.8.8.8 is provided by Google).
Creating a Node Inventory File IPMI configuration information gathered in section Execution Requirements (Bare
Metal Only) needs to be added to the inventory.yaml file.
1. Copy /usr/share/doc/opnfv/inventory.yaml.example as your inventory file template to
/etc/opnfv-apex/inventory.yaml.
2. The nodes dictionary contains a definition block for each baremetal host that will be deployed. 1 or more
compute nodes and 3 controller nodes are required. (The example file contains blocks for each of these already).
It is optional at this point to add more compute nodes into the node list.
3. Edit the following values for each node:
326
Chapter 3. Developer
OPNFV Documentation, Release Danube
• mac_address: MAC of the interface that will PXE boot from undercloud
• ipmi_ip: IPMI IP Address
• ipmi_user: IPMI username
• ipmi_password: IPMI password
• pm_type: Power Management driver to use for the node values: pxe_ipmitool (tested) or pxe_wol
(untested) or pxe_amt (untested)
• cpus: (Introspected*) CPU cores available
• memory: (Introspected*) Memory available in Mib
• disk: (Introspected*) Disk space available in Gb
• disk_device: (Opt***) Root disk device to use for installation
• arch: (Introspected*) System architecture
• capabilities: (Opt**) Node’s role in deployment values: profile:control or profile:compute
* Introspection looks up the overcloud node’s resources and overrides these value. You can leave default values
and Apex will get the correct values when it runs introspection on the nodes.
** If capabilities profile is not specified then Apex will select node’s roles in the OPNFV cluster in a nondeterministic fashion.
*** disk_device declares which hard disk to use as the root device for installation. The format is a comma
delimited list of devices, such as “sda,sdb,sdc”. The disk chosen will be the first device in the list which is found
by introspection to exist on the system. Currently, only a single definition is allowed for all nodes. Therefore if
multiple disk_device definitions occur within the inventory, only the last definition on a node will be used for
all nodes.
Creating the Settings Files
customize them.
Edit the 2 settings files in /etc/opnfv-apex/. These files have comments to help you
1. deploy_settings.yaml This file includes basic configuration options deployment, and also documents all available options. Alternatively, there are pre-built deploy_settings files available in (/etc/opnfv-apex/).
These files are named with the naming convention os-sdn_controller-enabled_feature-[no]ha.yaml. These
files can be used in place of the (/etc/opnfv-apex/deploy_settings.yaml) file if one suites
your deployment needs. If a pre-built deploy_settings file is chosen there is no need to customize
(/etc/opnfv-apex/deploy_settings.yaml). The pre-built file can be used in place of the
(/etc/opnfv-apex/deploy_settings.yaml) file.
2. network_settings.yaml This file provides Apex with the networking information that satisfies the prerequisite
Network Requirements. These are specific to your environment.
Running opnfv-deploy You are now ready to deploy OPNFV using Apex! opnfv-deploy will use the inventory and settings files to deploy OPNFV.
Follow the steps below to execute:
1. Execute
opnfv-deploy
sudo opnfv-deploy -n network_settings.yaml -i
inventory.yaml -d deploy_settings.yaml If you need more information about the options
that can be passed to opnfv-deploy use opnfv-deploy --help. -n network_settings.yaml allows you to
customize your networking topology.
2. Wait while deployment is executed. If something goes wrong during this part of the process, start by reviewing
your network or the information in your configuration files. It’s not uncommon for something small to be
overlooked or mis-typed. You will also notice outputs in your shell as the deployment progresses.
3.2. OPNFV Projects
327
OPNFV Documentation, Release Danube
3. When the deployment is complete the undercloud IP and ovecloud dashboard url will be printed. OPNFV has
now been deployed using Apex.
Installation High-Level Overview - Virtual Deployment
The VM nodes deployment operates almost the same way as the bare metal deployment with a few differences.
opnfv-deploy still deploys an undercloud VM. In addition to the undercloud VM a collection of VMs (3 control
nodes + 2 compute for an HA deployment or 1 control node and 1 or more compute nodes for a non-HA Deployment)
will be defined for the target OPNFV deployment. The part of the toolchain that executes IPMI power instructions
calls into libvirt instead of the IPMI interfaces on baremetal servers to operate the power management. These VMs
are then provisioned with the same disk images and configuration that baremetal would be. To Triple-O these nodes
look like they have just built and registered the same way as bare metal nodes, the main difference is the use of a
libvirt driver for the power management. Finally, the default network_settings file will deploy without modification.
Customizations are welcome but not needed if a generic set of network_settings are acceptable.
Installation Guide - Virtual Deployment
This section goes step-by-step on how to correctly install and provision the OPNFV target system to VM nodes.
Special Requirements for Virtual Deployments In scenarios where advanced performance options or features are
used, such as using huge pages with nova instances, DPDK, or iommu; it is required to enabled nested KVM support.
This allows hardware extensions to be passed to the overcloud VMs, which will allow the overcloud compute nodes
to bring up KVM guest nova instances, rather than QEMU. This also provides a great performance increase even in
non-required scenarios and is recommended to be enabled.
During deployment the Apex installer will detect if nested KVM is enabled, and if not, it will attempt to enable it;
while printing a warning message if it cannot. Check to make sure before deployment that Nested Virtualization is
enabled in BIOS, and that the output of cat /sys/module/kvm_intel/parameters/nested returns “Y”.
Also verify using lsmod that the kvm_intel module is loaded for x86_64 machines, and kvm_amd is loaded for
AMD64 machines.
Install Jumphost Follow the instructions in the Install Bare Metal Jumphost section.
Running opnfv-deploy You are now ready to deploy OPNFV! opnfv-deploy has virtual deployment capability that includes all of the configuration necessary to deploy OPNFV with no modifications.
If no modifications are made to the included configurations the target environment will deploy with the following
architecture:
• 1 undercloud VM
• The option of 3 control and 2 or more compute VMs (HA Deploy / default) or 1 control and 1 or more compute
VM (Non-HA deploy / pass -n)
• 1-5 networks: provisioning, private tenant networking, external, storage and internal API. The API, storage and
tenant networking networks can be collapsed onto the provisioning network.
Follow the steps below to execute:
1. sudo opnfv-deploy -v [ --virtual-computes n ] [ --virtual-cpus n ] [
--virtual-ram n ] -n network_settings.yaml -d deploy_settings.yaml
2. It will take approximately 45 minutes to an hour to stand up undercloud, define the target virtual machines,
configure the deployment and execute the deployment. You will notice different outputs in your shell.
328
Chapter 3. Developer
OPNFV Documentation, Release Danube
3. When the deployment is complete the IP for the undercloud and a url for the OpenStack dashboard will be
displayed
Verifying the Setup - VMs To verify the set you can follow the instructions in the Verifying the Setup section.
Verifying the Setup
Once the deployment has finished, the OPNFV deployment can be accessed via the undercloud node. From the jump
host ssh to the undercloud host and become the stack user. Alternativly ssh keys have been setup such that the root user
on the jump host can ssh to undercloud directly as the stack user. For convenience a utility script has been provided
to look up the undercloud’s ip address and ssh to the undercloud all in one command. An optional user name can be
passed to indicate whether to connect as the stack or root user. The stack user is default if a username is not specified.
opnfv-util undercloud root
su - stack
Once connected to undercloud as the stack user look for two keystone files that can be used to interact with the
undercloud and the overcloud. Source the appropriate RC file to interact with the respective OpenStack deployment.
source stackrc (undercloud)
source overcloudrc (overcloud / OPNFV)
The contents of these files include the credentials for the administrative user for undercloud and OPNFV respectivly.
At this point both undercloud and OPNFV can be interacted with just as any OpenStack installation can be. Start by
listing the nodes in the undercloud that were used to deploy the overcloud.
source stackrc
openstack server list
The control and compute nodes will be listed in the output of this server list command. The IP addresses that are listed
are the control plane addresses that were used to provision the nodes. Use these IP addresses to connect to these nodes.
Initial authentication requires using the user heat-admin.
ssh [email protected]
To begin creating users, images, networks, servers, etc in OPNFV source the overcloudrc file or retrieve the admin
user’s credentials from the overcloudrc file and connect to the web Dashboard.
You are now able to follow the OpenStack Verification section.
OpenStack Verification
Once connected to the OPNFV Dashboard make sure the OPNFV target system is working correctly:
3.2. OPNFV Projects
329
OPNFV Documentation, Release Danube
1. In the left pane, click Compute -> Images, click Create Image.
2. Insert a name “cirros”, Insert an Image Location http://download.cirros-cloud.net/0.3.5/cirros-0.3.5-x86
3. Select format “QCOW2”, select Public, then click Create Image.
4. Now click Project -> Network -> Networks, click Create Network.
5. Enter a name “internal”, click Next.
6. Enter a subnet name “internal_subnet”, and enter Network Address 172.16.1.0/24, click Next.
7. Now go to Project -> Compute -> Instances, click Launch Instance.
8. Enter Instance Name “first_instance”, select Instance Boot Source “Boot from image”, and then select Image
Name “cirros”.
9. Click Launch, status will cycle though a couple states before becoming “Active”.
10. Steps 7 though 9 can be repeated to launch more instances.
11. Once an instance becomes “Active” their IP addresses will display on the Instances page.
12. Click the name of an instance, then the “Console” tab and login as “cirros”/”cubswin:)”
13. To verify storage is working, click Project -> Compute -> Volumes, Create Volume
14. Give the volume a name and a size of 1 GB
15. Once the volume becomes “Available” click the dropdown arrow and attach it to an instance.
Congratulations you have successfully installed OPNFV!
Developer Guide and Troubleshooting
This section aims to explain in more detail the steps that Apex follows to make a deployment. It also tries to explain
possible issues you might find in the process of building or deploying an environment.
After installing the Apex RPMs in the jumphost, some files will be located around the system.
1. /etc/opnfv-apex: this directory contains a bunch of scenarios to be deployed with different characteristics such
HA (High Availability), SDN controller integration (OpenDaylight/ONOS), BGPVPN, FDIO, etc. Having a
look at any of these files will give you an idea of how to make a customized scenario setting up different flags.
2. /usr/bin/: it contains the binaries for the commands opnfv-deploy, opnfv-clean and opnfv-util.
3. /var/opt/opnfv/: it contains several files and directories.
3.1. images/: this folder contains the images that will be deployed according to the chosen scenario.
3.2. lib/: bunch of scripts that will be executed in the different phases of deployment.
Utilization of Images As mentioned earlier in this guide, the Undercloud VM will be in charge of deploying OPNFV
(Overcloud VMs). Since the Undercloud is an all-in-one OpenStack deployment, it will use Glance to manage the
images that will be deployed as the Overcloud.
So whatever customization that is done to the images located in the jumpserver (/var/opt/opnfv/images) will be uploaded to the undercloud and consequently, to the overcloud.
Make sure, the customization is performed on the right image. For example, if I virt-customize the following image
overcloud-full-opendaylight.qcow2, but then I deploy OPNFV with the following command:
sudo opnfv-deploy -n network_settings.yaml -d /etc/opnfv-apex/os-onos-nofeature-ha.yaml
330
Chapter 3. Developer
OPNFV Documentation, Release Danube
It will not have any effect over the deployment, since the customized image is the opendaylight one, and the scenario
indicates that the image to be deployed is the overcloud-full-onos.qcow2.
Post-deployment Configuration Post-deployment scripts will perform some configuration tasks such ssh-key injection, network configuration, NATing, OpenVswitch creation. It will take care of some OpenStack tasks such creation
of endpoints, external networks, users, projects, etc.
If any of these steps fail, the execution will be interrupted. In some cases, the interruption occurs at very early stages,
so a new deployment must be executed. However, some other cases it could be worth it to try to debug it.
1. There is not external connectivity from the overcloud nodes:
Post-deployment scripts will configure the routing, nameservers and a bunch of other things between
the overcloud and the undercloud. If local connectivity, like pinging between the different nodes,
is working fine, script must have failed when configuring the NAT via iptables. The main rules to
enable external connectivity would look like these:
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
iptables
-t nat -A POSTROUTING -s ${external_cidr} -o eth0 -j MASQUERADE
iptables -A FORWARD -i eth2 -j ACCEPT
iptables -A FORWARD -s
${external_cidr} -m state --state ESTABLISHED,RELATED -j ACCEPT
service iptables save
These rules must be executed as root (or sudo) in the undercloud machine.
OpenDaylight Integration When a user deploys a scenario that starts with os-odl*:
OpenDaylight (ODL) SDN controller will be deployed and integrated with OpenStack. ODL will run as a systemd
service, and can be managed as as a regular service:
systemctl start/restart/stop opendaylight.service
This command must be executed as root in the controller node of the overcloud, where OpenDaylight is running. ODL
files are located in /opt/opendaylight. ODL uses karaf as a Java container management system that allows the users to
install new features, check logs and configure a lot of things. In order to connect to Karaf’s console, use the following
command:
opnfv-util opendaylight
This command is very easy to use, but in case it is not connecting to Karaf, this is the command that is executing
underneath:
ssh -p 8101 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no
[email protected]
Of course, localhost when the command is executed in the overcloud controller, but you use its public IP to connect
from elsewhere.
Debugging Failures This section will try to gather different type of failures, the root cause and some possible
solutions or workarounds to get the process continued.
1. I can see in the output log a post-deployment error messages:
Heat resources will apply puppet manifests during this phase. If one of these processes fail, you
could try to see the error and after that, re-run puppet to apply that manifest. Log into the controller
(see verification section for that) and check as root /var/log/messages. Search for the error you have
encountered and see if you can fix it. In order to re-run the puppet manifest, search for “puppet
apply” in that same log. You will have to run the last “puppet apply” before the error. And It should
look like this:
3.2. OPNFV Projects
331
OPNFV Documentation, Release Danube
FACTER_heat_outputs_path="/var/run/heat-config/heat-config-puppet/5b4c7a01-0d63FACTER_fqdn="overcloud-controller-0.localdomain.com" \
FACTER_deploy_config_name="ControllerOvercloudServicesDeployment_Step4"
puppet apply --detailed-exitcodes -l syslog -l console \
/var/lib/heat-config/heat-config-puppet/5b4c7a01-0d63-4a71-81e9-d5ee6f0a1f2f.pp
As a comment, Heat will trigger the puppet run via os-apply-config and it will pass a different value
for step each time. There is a total of five steps. Some of these steps will not be executed depending
on the type of scenario that is being deployed.
Frequently Asked Questions
License
All Apex and “common” entities are protected by the Apache 2.0 License.
References
OPNFV OPNFV Home Page
OPNFV Apex project page
OPNFV Apex Release Notes
OpenStack OpenStack Newton Release artifacts
OpenStack documentation
OpenDaylight Upstream OpenDaylight provides a number of packaging and deployment options meant for consumption by downstream projects like OPNFV.
Currently, OPNFV Apex uses OpenDaylight’s Puppet module, which in turn depends on OpenDaylight’s RPM.
RDO Project RDO Project website
Authors Tim Rozet ([email protected])
Authors Dan Radez ([email protected])
Version 4.0
Indices and tables
• search
Armband
Installation instruction for [email protected] on AArch64
332
Chapter 3. Developer
OPNFV Documentation, Release Danube
Abstract
This document describes how to install the Danube release of OPNFV when using Fuel as a deployment tool, with an
AArch64 (only) target node pool. It covers its usage, limitations, dependencies and required system resources.
Introduction
This document provides guidelines on how to install and configure the Danube release of OPNFV when using Fuel as
a deployment tool, with an AArch64 (only) target node pool, including required software and hardware configurations.
Although the available installation options give a high degree of freedom in how the system is set-up, including architecture, services and features, etc., said permutations may not provide an OPNFV compliant reference architecture.
This instruction provides a step-by-step guide that results in an OPNFV Danube compliant deployment.
The audience of this document is assumed to have good knowledge in networking and Unix/Linux administration.
Preface
Before starting the installation of the AArch64 Danube release of OPNFV, using Fuel as a deployment tool, some
planning must be done.
Retrieving the ISO image First of all, the Fuel deployment ISO image needs to be retrieved, the ArmbandFuel .iso
image of the AArch64 Danube release can be found at OPNFV Downloads.
Building the ISO image Alternatively, you may build the Armband Fuel .iso from source by cloning the opnfv/armband git repository. To retrieve the repository for the AArch64 Danube release use the following command:
$ git clone https://gerrit.opnfv.org/gerrit/armband
Check-out the Danube release tag to set the HEAD to the baseline required to replicate the Danube release:
$ git checkout danube.3.0
Go to the armband directory and build the .iso:
$ cd armband; make all
For more information on how to build, please see Build instruction for [email protected]
Other preparations Next, familiarize yourself with Fuel by reading the following documents:
• Fuel Installation Guide
• Fuel User Guide
• Fuel Developer Guide
• Fuel Plugin Developers Guide
Prior to installation, a number of deployment specific parameters must be collected, those are:
1. Provider sub-net and gateway information
2. Provider VLAN information
3. Provider DNS addresses
3.2. OPNFV Projects
333
OPNFV Documentation, Release Danube
4. Provider NTP addresses
5. Network overlay you plan to deploy (VLAN, VXLAN, FLAT)
6. How many nodes and what roles you want to deploy (Controllers, Storage, Computes)
7. Monitoring options you want to deploy (Ceilometer, Syslog, etc.).
8. Other options not covered in the document are available in the links above
This information will be needed for the configuration procedures provided in this document.
Hardware requirements
The following minimum hardware requirements must be met for the installation of AArch64 Danube using Fuel:
HW Aspect
# of AArch64 nodes
Requirement
Minimum 5 (3 for non redundant deployment):
• 1 Fuel deployment master (may be virtualized)
• 3(1) Controllers (1 colocated mongo/ceilometer
role, 2 Ceph-OSD roles)
• 1 Compute (1 co-located Ceph-OSD role)
CPU
Minimum 1 socket AArch64 (ARMv8) with Virtualization support
Minimum 16GB/server (Depending on VNF work load)
UEFI compatible (e.g. EDK2) with PXE support
Minimum 256GB 10kRPM spinning disks
4 Tagged VLANs (PUBLIC, MGMT, STORAGE, PRIVATE)
1 Un-Tagged VLAN for PXE Boot - ADMIN Network
Note: These can be allocated to a single NIC - or spread
out over multiple NICs as your hardware supports.
RAM
Firmware
Disk
Networks
1 x86_64 node
• 1 Fuel deployment master, x86 (may be virtualized)
Help with Hardware Requirements
Calculate hardware requirements:
For information on compatible hardware types available for use, please see Fuel OpenStack Hardware Compatibility
List.
When choosing the hardware on which you will deploy your OpenStack environment, you should think about:
• CPU – Consider the number of virtual machines that you plan to deploy in your cloud environment and the CPU
per virtual machine.
• Memory – Depends on the amount of RAM assigned per virtual machine and the controller node.
• Storage – Depends on the local drive space per virtual machine, remote volumes that can be attached to a virtual
machine, and object storage.
• Networking – Depends on the Choose Network Topology, the network bandwidth per virtual machine, and
network storage.
334
Chapter 3. Developer
OPNFV Documentation, Release Danube
Top of the rack (TOR) Configuration requirements
The switching infrastructure provides connectivity for the OPNFV infrastructure operations, tenant networks
(East/West) and provider connectivity (North/South); it also provides needed connectivity for the Storage Area Network (SAN). To avoid traffic congestion, it is strongly suggested that three physically separated networks are used,
that is: 1 physical network for administration and control, one physical network for tenant private and public networks,
and one physical network for SAN. The switching connectivity can (but does not need to) be fully redundant, in such
case it comprises a redundant 10GE switch pair for each of the three physically separated networks.
The physical TOR switches are not automatically configured from the Fuel OPNFV reference platform. All the
networks involved in the OPNFV infrastructure as well as the provider networks and the private tenant VLANs needs
to be manually configured.
Manual configuration of the Danube hardware platform should be carried out according to the OPNFV Pharos Specification.
OPNFV Software installation and deployment
This section describes the installation of the OPNFV installation server (Fuel master) as well as the deployment of the
full OPNFV reference platform stack across a server cluster.
Install Fuel master
1. Mount the Danube Armband Fuel ISO file/media as a boot device to the jump host server.
2. Reboot the jump host to establish the Fuel server.
• The system now boots from the ISO image.
• Select “Fuel Install (Static IP)” (See figure below)
• Press [Enter].
3. Wait until the Fuel setup screen is shown (Note: This can take up to 30 minutes).
4. In the “Fuel User” section - Confirm/change the default password (See figure below)
• Enter “admin” in the Fuel password input
• Enter “admin” in the Confirm password input
• Select “Check” and press [Enter]
5. In the “Network Setup” section - Configure DHCP/Static IP information for your FUEL node - For example,
ETH0 is 10.20.0.2/24 for FUEL booting and ETH1 is DHCP in your corporate/lab network (see figure below).
• Configure eth1 or other network interfaces here as well (if you have them present on your FUEL server).
6. In the “PXE Setup” section (see figure below) - Change the following fields to appropriate values (example
below):
• DHCP Pool Start 10.20.0.4
• DHCP Pool End 10.20.0.254
• DHCP Pool Gateway 10.20.0.2 (IP address of Fuel node)
7. In the “DNS & Hostname” section (see figure below) - Change the following fields to appropriate values:
• Hostname
• Domain
3.2. OPNFV Projects
335
OPNFV Documentation, Release Danube
336
Chapter 3. Developer
OPNFV Documentation, Release Danube
3.2. OPNFV Projects
337
OPNFV Documentation, Release Danube
• Search Domain
• External DNS
• Hostname to test DNS
• Select <Check> and press [Enter]
8. OPTION TO ENABLE PROXY SUPPORT - In the “Bootstrap Image” section (see figure below), edit the
following fields to define a proxy. (NOTE: cannot be used in tandem with local repository support)
• Navigate to “HTTP proxy” and enter your http proxy address
• Select <Check> and press [Enter]
9. In the “Time Sync” section (see figure below) - Change the following fields to appropriate values:
• NTP Server 1 <Customer NTP server 1>
• NTP Server 2 <Customer NTP server 2>
• NTP Server 3 <Customer NTP server 3>
10. In the “Feature groups” section - Enable “Experimental features” if you plan on using Ceilometer and/or MongoDB.
NOTE: Ceilometer and MongoDB are experimental features starting with Danube.1.0.
11. Start the installation.
NOTE: Saving each section and hitting <F8> does not apply all settings!
• Select Quit Setup and press Save and Quit.
• The installation will now start, wait until the login screen is shown.
338
Chapter 3. Developer
OPNFV Documentation, Release Danube
3.2. OPNFV Projects
339
OPNFV Documentation, Release Danube
Boot the Node Servers After the Fuel Master node has rebooted from the above steps and is at the login prompt,
you should boot the Node Servers (Your Compute/Control/Storage blades, nested or real) with a PXE booting scheme
so that the FUEL Master can pick them up for control.
NOTE: AArch64 target nodes are expected to support PXE booting an EFI binary, i.e. an EFI-stubbed GRUB2
bootloader.
NOTE: UEFI (EDK2) firmware is highly recommended, becoming the de facto standard for ARMv8 nodes.
1. Enable PXE booting
• For every controller and compute server: enable PXE Booting as the first boot device in the UEFI (EDK2)
boot order menu, and hard disk as the second boot device in the same menu.
2. Reboot all the control and compute blades.
3. Wait for the availability of nodes showing up in the Fuel GUI.
• Connect to the FUEL UI via the URL provided in the Console (default: https://10.20.0.2:8443)
• Wait until all nodes are displayed in top right corner of the Fuel GUI: Total nodes and Unallocated nodes
(see figure below).
Install additional Plugins/Features on the FUEL node
1. SSH to your FUEL node (e.g. [email protected] pwd: r00tme)
2. Select wanted plugins/features from the /opt/opnfv/ directory.
3. Install the wanted plugin with the command
$ fuel plugins --install /opt/opnfv/<plugin-name>-<version>.<arch>.rpm
Expected output (see figure below):
340
Chapter 3. Developer
OPNFV Documentation, Release Danube
Plugin ....... was successfully installed.
NOTE: AArch64 Danube 3.0 ships only with ODL, OVS, BGPVPN, SFC and Tacker plugins, see Reference
15.
Create an OpenStack Environment
1. Connect to Fuel WEB UI with a browser (default: https://10.20.0.2:8443) (login: admin/admin)
2. Create and name a new OpenStack environment, to be installed.
3. Select “<Newton on Ubuntu 16.04 (aarch64)>” and press <Next>
4. Select “compute virtulization method”.
• Select “QEMU-KVM as hypervisor” and press <Next>
5. Select “network mode”.
• Select “Neutron with ML2 plugin”
• Select “Neutron with tunneling segmentation” (Required when using the ODL plugin)
• Press <Next>
6. Select “Storage Back-ends”.
• Select “Ceph for block storage” and press <Next>
7. Select “additional services” you wish to install.
• Check option “Install Ceilometer and Aodh” and press <Next>
8. Create the new environment.
• Click <Create> Button
Configure the network environment
1. Open the environment you previously created.
2. Open the networks tab and select the “default” Node Networks group to on the left pane (see figure below).
3. Update the Public network configuration and change the following fields to appropriate values:
• CIDR to <CIDR for Public IP Addresses>
3.2. OPNFV Projects
341
OPNFV Documentation, Release Danube
342
Chapter 3. Developer
OPNFV Documentation, Release Danube
3.2. OPNFV Projects
343
OPNFV Documentation, Release Danube
• IP Range Start to <Public IP Address start>
• IP Range End to <Public IP Address end>
• Gateway to <Gateway for Public IP Addresses>
• Check <VLAN tagging>.
• Set appropriate VLAN id.
4. Update the Storage Network Configuration
• Set CIDR to appropriate value (default 192.168.1.0/24)
• Set IP Range Start to appropriate value (default 192.168.1.1)
• Set IP Range End to appropriate value (default 192.168.1.254)
• Set vlan to appropriate value (default 102)
5. Update the Management network configuration.
• Set CIDR to appropriate value (default 192.168.0.0/24)
• Set IP Range Start to appropriate value (default 192.168.0.1)
• Set IP Range End to appropriate value (default 192.168.0.254)
• Check <VLAN tagging>.
• Set appropriate VLAN id. (default 101)
6. Update the Private Network Information
• Set CIDR to appropriate value (default 192.168.2.0/24
• Set IP Range Start to appropriate value (default 192.168.2.1)
• Set IP Range End to appropriate value (default 192.168.2.254)
• Check <VLAN tagging>.
• Set appropriate VLAN tag (default 103)
7. Select the “Neutron L3” Node Networks group on the left pane.
8. Update the Floating Network configuration.
• Set the Floating IP range start (default 172.16.0.130)
• Set the Floating IP range end (default 172.16.0.254)
• Set the Floating network name (default admin_floating_net)
9. Update the Internal Network configuration.
• Set Internal network CIDR to an appropriate value (default 192.168.111.0/24)
• Set Internal network gateway to an appropriate value
• Set the Internal network name (default admin_internal_net)
10. Update the Guest OS DNS servers.
• Set Guest OS DNS Server values appropriately
11. Save Settings.
12. Select the “Other” Node Networks group on the left pane (see figure below).
13. Update the Public network assignment.
344
Chapter 3. Developer
OPNFV Documentation, Release Danube
3.2. OPNFV Projects
345
OPNFV Documentation, Release Danube
346
Chapter 3. Developer
OPNFV Documentation, Release Danube
• Check the box for “Assign public network to all nodes” (Required by OpenDaylight)
14. Update Host OS DNS Servers.
• Provide the DNS server settings
15. Update Host OS NTP Servers.
• Provide the NTP server settings
Select Hypervisor type
1. In the FUEL UI of your Environment, click the “Settings” Tab
2. Select “Compute” on the left side pane (see figure below)
• Check the KVM box and press “Save settings”
Enable Plugins
1. In the FUEL UI of your Environment, click the “Settings” Tab
2. Select Other on the left side pane (see figure below)
• Enable and configure the plugins of your choice
Allocate nodes to environment and assign functional roles
1. Click on the “Nodes” Tab in the FUEL WEB UI (see figure below).
2. Assign roles (see figure below).
• Click on the <+Add Nodes> button
3.2. OPNFV Projects
347
OPNFV Documentation, Release Danube
348
Chapter 3. Developer
OPNFV Documentation, Release Danube
• Check <Controller>, <Telemetry - MongoDB> and optionally an SDN Controller role (OpenDaylight controller) in the “Assign Roles” Section.
• Check one node which you want to act as a Controller from the bottom half of the screen
• Click <Apply Changes>.
• Click on the <+Add Nodes> button
• Check the <Controller> and <Storage - Ceph OSD> roles.
• Check the two next nodes you want to act as Controllers from the bottom half of the screen
• Click <Apply Changes>
• Click on <+Add Nodes> button
• Check the <Compute> and <Storage - Ceph OSD> roles.
• Check the Nodes you want to act as Computes from the bottom half of the screen
• Click <Apply Changes>.
3. Configure interfaces (see figure below).
• Check Select <All> to select all allocated nodes
• Click <Configure Interfaces>
• Assign interfaces (bonded) for mgmt-, admin-, private-, public- and storage networks
• Click <Apply>
OPTIONAL - Set Local Mirror Repos NOTE: Support for local mirrors is incomplete in Danube 3.0. You may
opt in for it to fetch less packages from internet during deployment, but an internet connection is still required.
The following steps must be executed if you are in an environment with no connection to the Internet. The Fuel server
delivers a local repo that can be used for installation / deployment of openstack.
1. In the Fuel UI of your Environment, click the Settings Tab and select General from the left pane.
• Replace the URI values for the “Name” values outlined below:
• “ubuntu” URI=”deb http://<ip-of-fuel-server>:8080/mirrors/ubuntu/ xenial main”
• “mos” URI=”deb http://<ip-of-fuel-server>::8080/newton-10.0/ubuntu/x86_64 mos10.0 main restricted”
• “Auxiliary” URI=”deb http://<ip-of-fuel-server>:8080/newton-10.0/ubuntu/auxiliary auxiliary main restricted”
• Click <Save Settings> at the bottom to Save your changes
3.2. OPNFV Projects
349
OPNFV Documentation, Release Danube
Target specific configuration
1. [AArch64 specific] Configure MySQL WSREP SST provider
NOTE: This option is only available for [email protected], since it currently only affects AArch64 targets
(see Reference 15).
When using some AArch64 platforms as controller nodes, WSREP SST synchronisation using default backend
provider (xtrabackup-v2) used to fail, so a mechanism that allows selecting a different WSREP SST provider
has been introduced.
In the FUEL UI of your Environment, click the <Settings> tab, click <OpenStack Services> on the left side
pane (see figure below), then select one of the following options:
• xtrabackup-v2 (default provider, AArch64 stability issues);
• rsync (AArch64 validated, better or comparable speed to xtrabackup, takes the donor node offline during
state transfer);
• mysqldump (untested);
2. [AArch64 specific] Using a different kernel
NOTE: By default, a 4.8 based kernel is used, for enabling experimental GICv3 features (e.g. live migration)
and SFC support (required by OVS-NSH).
To use Ubuntu Xenial LTS generic kernel (also available in offline mirror), in the FUEL UI of your Environment,
click the <Settings> tab, click <General> on the left side pane, then at the bottom of the page, in the <Provision>
subsection, amend the package list:
• add <linux-headers-generic-lts-xenial>;
350
Chapter 3. Developer
OPNFV Documentation, Release Danube
3.2. OPNFV Projects
351
OPNFV Documentation, Release Danube
• add <linux-image-generic-lts-xenial>;
• add <linux-image-extra-lts-xenial> (optional);
• remove <linux-image-4.8.0-9944-generic>;
• remove <linux-headers-4.8.0-9944-generic>;
• remove <linux-image-extra-4.8.0-9944-generic>;
3. Set up targets for provisioning with non-default “Offloading Modes”
Some target nodes may require additional configuration after they are PXE booted (bootstrapped); the most
frequent changes are in defaults for ethernet devices’ “Offloading Modes” settings (e.g. some targets’ ethernet
drivers may strip VLAN traffic by default).
If your target ethernet drivers have wrong “Offloading Modes” defaults, in “Configure interfaces” page (described above), expand affected interface’s “Offloading Modes” and [un]check the relevant settings (see figure
below):
4. Set up targets for “Verify Networks” with non-default “Offloading Modes”
NOTE: Check Reference 15 for an updated and comprehensive list of known issues and/or limitations, including
“Offloading Modes” not being applied during “Verify Networks” step.
Setting custom “Offloading Modes” in Fuel GUI will only apply those settings during provisiong and not during
“Verify Networks”, so if your targets need this change, you have to apply “Offloading Modes” settings by hand
to bootstrapped nodes.
E.g.: Our driver has “rx-vlan-filter” default “on” (expected “off”) on the Openstack interface(s) “eth1”, preventing VLAN traffic from passing during “Verify Networks”.
352
Chapter 3. Developer
OPNFV Documentation, Release Danube
• From Fuel master console identify target nodes admin IPs (see figure below):
$ fuel nodes
• SSH into each of the target nodes and disable “rx-vlan-filter” on the affected physical interface(s) allocated
for OpenStack traffic (eth1):
$ ssh [email protected] ethtool -K eth1 rx-vlan-filter off
• Repeat the step above for all affected nodes/interfaces in the POD.
Verify Networks It is important that the Verify Networks action is performed as it will verify that communicate
works for the networks you have setup, as well as check that packages needed for a successful deployment can be
fetched.
1. From the FUEL UI in your Environment, Select the Networks Tab and select “Connectivity check” on the left
pane (see figure below)
• Select <Verify Networks>
• Continue to fix your topology (physical switch, etc) until the “Verification Succeeded” and “Your network
is configured correctly” message is shown
3.2. OPNFV Projects
353
OPNFV Documentation, Release Danube
Deploy Your Environment
1. Deploy the environment.
• In the Fuel GUI, click on the “Dashboard” Tab.
• Click on <Deploy Changes> in the “Ready to Deploy?” section
• Examine any information notice that pops up and click <Deploy>
Wait for your deployment to complete, you can view the “Dashboard” Tab to see the progress and
status of your deployment.
Installation health-check
1. Perform system health-check (see figure below)
• Click the “Health Check” tab inside your Environment in the FUEL Web UI
• Check <Select All> and Click <Run Tests>
• Allow tests to run and investigate results where appropriate
• Check Reference 15 for known issues / limitations on AArch64
Release Notes
Please refer to the Release Notes article.
354
Chapter 3. Developer
OPNFV Documentation, Release Danube
References
OPNFV
1. OPNFV Home Page
2. OPNFV documentation- and software downloads
OpenStack
3. OpenStack Newton Release Artifacts
4. OpenStack Documentation
OpenDaylight
5. OpenDaylight Artifacts
Fuel
6. The Fuel OpenStack Project
7. Fuel Documentation Overview
8. Fuel Installation Guide
9. Fuel User Guide
10. Fuel Developer Guide
11. Fuel Plugin Developers Guide
12. (N/A on AArch64) Fuel OpenStack Hardware Compatibility List
Armband Fuel in OPNFV
13. OPNFV Installation instruction for the AArch64 Danube release of OPNFV when using Fuel as a deployment
tool
14. OPNFV Build instruction for the AArch64 Danube release of OPNFV when using Fuel as a deployment tool
15. OPNFV Release Note for the AArch64 Danube release of OPNFV when using Fuel as a deployment tool
Availability
PROPOSAL: Reach-thru Guest Monitoring and Services for High Availability
1 Overview
author Greg Waines
organization Wind River Systems
organization OPNFV - High Availability
status Draft - PROPOSAL
date March 2017
revision 1.5
abstract This document presents a PROPOSAL for a set of new optional capabilities where the OpenStack Cloud messages into the Guest VMs in order to provide improved Availability of the hosted
3.2. OPNFV Projects
355
OPNFV Documentation, Release Danube
VMs. The initial set of new capabilities include: enabling the detection of and recovery from internal VM faults and providing a simple out-of-band messaging service to prevent scenarios such as
split brain.
Table of Contents
•
•
•
•
•
•
1
2
3
4
5
6
Overview
Introduction
Messaging Layer
VM Heartbeating and Health Checking
VM Peer State Notification and Messaging
Conclusion
2 Introduction
This document provides an overview and rationale for a PROPOSAL of a set of new capabilities where the
OpenStack Cloud messages into the Guest VMs in order to provide improved Availability of the hosted
VMs.
The initial set of new capabilities specifically include:
• VM Heartbeating and Health Checking
• VM Peer State Notification and Messaging
All of these capabilities leverage Host-to-Guest Messaging Interfaces / APIs which are built on a messaging service between the OpenStack Host and the Guest VM that uses a simple low-bandwidth datagram
messaging capability in the hypervisor and therefore has no requirements on OpenStack Networking, and
is available very early after spawning the VM.
For each capability, the document outlines the interaction with the Guest VM, any key technologies
involved, the integration into the larger OpenStack and OPNFV Architectures (e.g. interactions with
VNFM), specific OPNFV HA Team deliverables, and the use cases for how availability of the hosted VM
is improved.
The intent is for the OPNFV HA Team to review the proposals of this document with the related other
teams in OPNFV (Doctor and Management & Orchestration (MANO)) and OpenStack (Nova).
3 Messaging Layer
The Host-to-Guest messaging APIs used by the services discussed in this document use a JSON-formatted
application messaging layer on top of a ‘virtio serial device??between QEMU on the OpenStack Host and
the Guest VM. JSON formatting provides a simple, humanly readable messaging format which can be
easily parsed and formatted using any high level programming language being used in the Guest VM (e.g.
C/C++, Python, Java, etc.). Use of the ‘virtio serial device??provides a simple, direct communication
channel between host and guest which is independent of the Guest’s L2/L3 networking.
The upper layer JSON messaging format is actually structured as a hierarchical JSON format containing
a Base JSON Message Layer and an Application JSON Message Layer:
• the Base Layer provides the ability to multiplex different groups of message types on top of a single
‘virtio serial device?? e.g.
– heartbeating and healthchecks,
– server group messaging,
356
Chapter 3. Developer
OPNFV Documentation, Release Danube
and
• the Application Layer provides the specific message types and fields of a particular group of message
types.
4 VM Heartbeating and Health Checking
Normally OpenStack monitoring of the health of a Guest VM is limited to a black-box approach of simply
monitoring the presence of the QEMU/KVM PID containing the VM.
VM Heartbeating and Health Checking provides a heartbeat service to monitor the health of guest application(s) within a VM running under the OpenStack Cloud. Loss of heartbeat or a failed health check status
will result in a fault event being reported to OPNFV’s DOCTOR infrastructure for alarm identification,
impact analysis and reporting. This would then enable VNF Managers (VNFMs) listening to OPNFV’s
DOCTOR External Alarm Reporting through Telemetry’s AODH, to initiate any required fault recovery
actions.
submodules/availability/docs/development/overview/OPNFV_HA_Guest_APIs-Overview_HLD-Gue
The VM Heartbeating and Health Checking functionality is enabled on a VM through a new flavor extraspec indicating that the VM supports and wants to enable Guest Heartbeating. An extension to Nova
Compute uses this extraspec to setup the required ‘virtio serial device’ for Host-to-Guest messaging, on
the QEMU/KVM instance created for the VM.
A daemon within the Guest VM will register with the OpenStack Guest Heartbeat Service on the compute
node to initiate the heartbeating on itself (i.e. the Guest VM). The OpenStack Compute Node will start
heartbeating the Guest VM, and if the heartbeat fails, the OpenStack Compute Node will report the VM
Fault thru DOCTOR and ultimately VNFM will see this thru NOVA VM State Chagne Notifications thru
AODH. I.e. VNFM wouild see the VM Heartbeat Failure events in teh same way it sees all other VM
Faults, thru DOCTOR initiated VM state changes.
Part of the Guest VM’s registration process is the specification of the heartbeat interval in msecs. I.e. the
registering Guest VM specifies the heartbeating interval.
Guest heartbeat works on a challenge response model. The OpenStack Guest Heartbeat Service on the
compute node will challenge the registered Guest VM daemon with a message each interval. The registered Guest VM daemon must respond prior to the next interval with a message indicating good health.
If the OpenStack Host does not receive a valid response, or if the response specifies that the VM is in ill
health, then a fault event for the Guest VM is reported to the OpenStack Guest Heartbeat Service on the
controller node which will report the event to OPNFV’s DOCTOR (i.e. thru the OpenStack Vitrage data
source APIs).
The registered Guest VM daemon’s response to the challenge can be as simple as just immediately responding with OK. This alone allows for detection of a failed or hung QEMU/KVM instance, or a failure
of the OS within the VM to schedule the registered Guest VM’s daemon or failure to route basic IO within
the Guest VM.
However the registered Guest VM daemon’s response to the challenge can be more complex, running
anything from a quick simple sanity check of the health of applications running in the Guest VM, to a
more thorough audit of the application state and data. In either case returning the status of the health check
enables the OpenStack host to detect and report the event in order to initiate recovery from application
level errors or failures within the Guest VM.
3.2. OPNFV Projects
357
OPNFV Documentation, Release Danube
In summary, the deliverables of this activity would be:
• Host Deliverables: (OpenStack and OPNFV blueprints and implementation)
• an OpenStack Nova or libvirt extension to interpret the new flavor extraspec and if present setup
the required ‘virtio serial device’ for Host-to-Guest heartbeat / health-check messaging, on the
QEMU/KVM instance created for the VM,
• an OPNFV Base Host-to-Guest Msging Layer Agent for multiplexing of Application Layer messaging over the ‘virtio serial device’ to the VM,
• an OPNFV Heartbeat / Health-Check Compute Agent for local heartbeating of VM and reporting of
failures to the OpenStack Controller,
• an OPNFV Heartbeat / Health-check Server on the OpenStack Controller for receiving VM failure
notifications and reporting these to Vitrage thru Vitrage’s Data Source API,
• Guest Deliverables:
• a Heartbeat / Health-Check Message Specification covering
– Heartbeat / Health-Check Application Layer JSON Protocol,
– Base Host-to-Guest JSON Protocol,
– Details on the use of the underlying ‘virtio serial device’,
• a Reference Implementation of the Guest-side support of Heartbeat / Health-check containing the
peer protocol layers within the Guest.
– will provide code and compile instructions,
– Guest will compile based on its specific OS.
This proposal requires review with OPNFV’s Doctor and Management & Orchestration teams, and OpenStack’s Nova Team.
5 VM Peer State Notification and Messaging
Server Group State Notification and Messaging is a service to provide simple low-bandwidth datagram
messaging and notifications for servers that are part of the same server group. This messaging channel is
available regardless of whether IP networking is functional within the server, and it requires no knowledge
within the server about the other members of the group.
NOTE: A Server Group here is the OpenStack Nova Server Group concept where VMs are grouped
together for purposes of scheduling. E.g. A specific Server Group instance can specify whether the VMs
within the group should be scheduled to run on the same compute host or different compute hosts. A
‘peer’ VM in the context of this section refers to a VM within the same Nova Server Group.
This Server Group Messaging service provides three types of messaging:
• Broadcast: this allows a server to send a datagram (size of up to 3050 bytes) to all other servers
within the server group.
• Notification: this provides servers with information about changes to the (Nova) state of other servers
within the server group.
• Status: this allows a server to query the current (Nova) state of all servers within the server group
(including itself).
A Server Group Messaging entity on both the controller node and the compute nodes manage the routing
of of VM-to-VM messages through the platform, leveraging Nova to determine Server Group membership
and compute node locations of VMs. The Server Group Messaging entity on the controller also listens
358
Chapter 3. Developer
OPNFV Documentation, Release Danube
to Nova VM state change notifications and querys VM state data from Nova, in order to provide the VM
query and notification functionality of this service.
submodules/availability/docs/development/overview/OPNFV_HA_Guest_APIs-Overview_HLD-Pee
This service is not intended for high bandwidth or low-latency operations. It is best-effort, not reliable.
Applications should do end-to-end acks and retries if they care about reliability.
This service provides building block type capabilities for the Guest VMs that contribute to higher availability of the VMs in the Guest VM Server Group. Notifications of VM Status changes potentially provide
a faster and more accurate notification of failed peer VMs than traditional peer VM monitoring over Tenant
Networks. While the Broadcast Messaging mechanism provides an out-of-band messaging mechanism to
monitor and control a peer VM under fault conditions; e.g. providing the ability to avoid potential split
brain scenarios between 1:1 VMs when faults in Tenant Networking occur.
In summary, the deliverables for Server Group Messaging would be:
• Host Deliverables:
• a Nova or libvirt extension to interpret the new flavor extraspec and if present setup the required ‘virtio serial device’ for Host-to-Guest Server Group Messaging, on the QEMU/KVM instance created
for the VM,
• [ leveraging the Base Host-to-Guest Msging Layer Agent from previous section ],
• a Server Group Messaging Compute Agent for implementing the Application Layer Server Group
Messaging JSON Protocol with the VM, and forwarding the messages to/from the Server Group
Messaging Server on the Controller,
• a Server Group Messaging Server on the Controller for routing broadcast messages to the proper
Computes and VMs, as well as listening for Nova VM State Change Notifications and forwarding
these to applicable Computes and VMs,
• Guest Deliverables:
• a Server Group Messaging Message Specification covering
– Server Group Messaging Application Layer JSON Protocol,
– [ leveraging Base Host-to-Guest JSON Protocol from previous section ],
– [ leveraging Details on the use of the underlying ‘virtio serial device’ from previous section ],
• a Reference Implementation of the Guest-side support of Server Group Messaging containing the
peer protocol layers and Guest Application hooks within the Guest.
This proposal requires review with OPNFV’s Management & Orchestration team and OpenStack’s Nova
Team.
6 Conclusion
The PROPOSAL of Reach-thru Guest Monitoring and Services described in this document leverage Hostto-Guest messaging to provide a number of extended capabilities that improve the Availability of the
hosted VMs. These new capabilities enable detection of and recovery from internal VM faults and provides a simple out-of-band messaging service to prevent scenarios such as split brain.
3.2. OPNFV Projects
359
OPNFV Documentation, Release Danube
The integration of these proposed new capabilities into the larger OpenStack and OPNFV Architectures
need to be reviewed with the other related teams in OPNFV (Doctor and Management & Orchestration
(MANO)) and OpenStack (Nova).
>>>>>>> e83e826... Modify format issues
Barometer
OPNFV Barometer Requirements
Problem Statement
Providing carrier grade Service Assurance is critical in the network transformation to a software defined and virtualized
network (NFV). Medium-/large-scale cloud environments account for between hundreds and hundreds of thousands
of infrastructure systems. It is vital to monitor systems for malfunctions that could lead to users application service
disruption and promptly react to these fault events to facilitate improving overall system performance. As the size of
infrastructure and virtual resources grow, so does the effort of monitoring back-ends. SFQM aims to expose as much
useful information as possible off the platform so that faults and errors in the NFVI can be detected promptly and
reported to the appropriate fault management entity.
The OPNFV platform (NFVI) requires functionality to:
• Create a low latency, high performance packet processing path (fast path) through the NFVI that VNFs can take
advantage of;
• Measure Telco Traffic and Performance KPIs through that fast path;
• Detect and report violations that can be consumed by VNFs and higher level EMS/OSS systems
Examples of local measurable QoS factors for Traffic Monitoring which impact both Quality of Experience and five
9’s availability would be (using Metro Ethernet Forum Guidelines as reference):
• Packet loss
• Packet Delay Variation
• Uni-directional frame delay
Other KPIs such as Call drops, Call Setup Success Rate, Call Setup time etc. are measured by the VNF.
In addition to Traffic Monitoring, the NFVI must also support Performance Monitoring of the physical interfaces
themselves (e.g. NICs), i.e. an ability to monitor and trace errors on the physical interfaces and report them.
All these traffic statistics for Traffic and Performance Monitoring must be measured in-service and must be capable of
being reported by standard Telco mechanisms (e.g. SNMP traps), for potential enforcement actions.
Barometer updated scope
The scope of the project is to provide interfaces to support monitoring of the NFVI. The project will develop plugins
for telemetry frameworks to enable the collection of platform stats and events and relay gathered information to fault
management applications or the VIM. The scope is limited to collecting/gathering the events and stats and relaying
them to a relevant endpoint. The project will not enforce or take any actions based on the gathered information.
Scope of SFQM NOTE: The SFQM project has been replaced by Barometer. The output of the project will provide
interfaces and functions to support monitoring of Packet Latency and Network Interfaces while the VNF is in service.
The DPDK interface/API will be updated to support:
360
Chapter 3. Developer
OPNFV Documentation, Release Danube
• Exposure of NIC MAC/PHY Level Counters
• Interface for Time stamp on RX
• Interface for Time stamp on TX
• Exposure of DPDK events
collectd will be updated to support the exposure of DPDK metrics and events.
Specific testing and integration will be carried out to cover:
• Unit/Integration Test plans: A sample application provided to demonstrate packet latency monitoring and interface monitoring
The following list of features and functionality will be developed:
• DPDK APIs and functions for latency and interface monitoring
• A sample application to demonstrate usage
• collectd plugins
The scope of the project involves developing the relavant DPDK APIs, OVS APIs, sample applications, as well as the
utilities in collectd to export all the relavent information to a telemetry and events consumer.
VNF specific processing, Traffic Monitoring, Performance Monitoring and Management Agent are out of scope.
The Proposed Interface counters include:
• Packet RX
• Packet TX
• Packet loss
• Interface errors + other stats
The Proposed Packet Latency Monitor include:
• Cycle accurate stamping on ingress
• Supports latency measurements on egress
Support for failover of DPDK enabled cores is also out of scope of the current proposal. However, this is an important
requirement and must-have functionality for any DPDK enabled framework in the NFVI. To that end, a second phase
of this project will be to implement DPDK Keep Alive functionality that would address this and would report to a
VNF-level Failover and High Availability mechanism that would then determine what actions, including failover, may
be triggered.
Consumption Models In reality many VNFs will have an existing performance or traffic monitoring utility used to
monitor VNF behavior and report statistics, counters, etc.
The consumption of performance and traffic related information/events provided by this project should be a logical
extension of any existing VNF/NFVI monitoring framework. It should not require a new framework to be developed.
We do not see the Barometer gathered metrics and evetns as major additional effort for monitoring frameworks to
consume; this project would be sympathetic to existing monitoring frameworks. The intention is that this project
represents an interface for NFVI monitoring to be used by higher level fault management entities (see below).
Allowing the Barometer metrics and events to be handled within existing telemetry frameoworks makes it simpler for
overall interfacing with higher level management components in the VIM, MANO and OSS/BSS. The Barometer proposal would be complementary to the Doctor project, which addresses NFVI Fault Management support in the VIM,
and the VES project, which addresses the integration of VNF telemetry-related data into automated VNF management
systems. To that end, the project committers and contributors for the Barometer project wish to collaborate with the
Doctor and VES projects to facilitate this.
3.2. OPNFV Projects
361
OPNFV Documentation, Release Danube
collectd
collectd is a daemon which collects system performance statistics periodically and provides a variety of mechanisms
to publish the collected metrics. It supports more than 90 different input and output plugins. Input plugins retrieve
metrics and publish them to the collectd deamon, while output plugins publish the data they receive to an end point.
collectd also has infrastructure to support thresholding and notification.
collectd statistics and Notifications
Within collectd notifications and performance data are dispatched in the same way. There are producer plugins (plugins
that create notifications/metrics), and consumer plugins (plugins that receive notifications/metrics and do something
with them).
Statistics in collectd consist of a value list. A value list includes:
• Values, can be one of:
– Derive: used for values where a change in the value since it’s last been read is of interest. Can be used to
calculate and store a rate.
– Counter: similar to derive values, but take the possibility of a counter wrap around into consideration.
– Gauge: used for values that are stored as is.
– Absolute: used for counters that are reset after reading.
• Value length: the number of values in the data set.
• Time: timestamp at which the value was collected.
• Interval: interval at which to expect a new value.
• Host: used to identify the host.
• Plugin: used to identify the plugin.
• Plugin instance (optional): used to group a set of values together. For e.g. values belonging to a DPDK interface.
• Type: unit used to measure a value. In other words used to refer to a data set.
• Type instance (optional): used to distinguish between values that have an identical type.
• meta data: an opaque data structure that enables the passing of additional information about a value list. “Meta
data in the global cache can be used to store arbitrary information about an identifier” [7].
Host, plugin, plugin instance, type and type instance uniquely identify a collectd value.
Values lists are often accompanied by data sets that describe the values in more detail. Data sets consist of:
• A type: a name which uniquely identifies a data set.
• One or more data sources (entries in a data set) which include:
– The name of the data source. If there is only a single data source this is set to “value”.
– The type of the data source, one of: counter, gauge, absolute or derive.
– A min and a max value.
Types in collectd are defined in types.db. Examples of types in types.db:
bitrate
counter
if_octets
362
value:GAUGE:0:4294967295
value:COUNTER:U:U
rx:COUNTER:0:4294967295, tx:COUNTER:0:4294967295
Chapter 3. Developer
OPNFV Documentation, Release Danube
In the example above if_octets has two data sources: tx and rx.
Notifications in collectd are generic messages containing:
• An associated severity, which can be one of OKAY, WARNING, and FAILURE.
• A time.
• A Message
• A host.
• A plugin.
• A plugin instance (optional).
• A type.
• A types instance (optional).
• Meta-data.
DPDK Enhancements
This section will discuss the Barometer features that were integrated with DPDK.
Measuring Telco Traffic and Performance KPIs
Measuring Telco Traffic and Performance KPIs.
This section will discuss the Barometer features that enable
Fig. 3.1: Measuring Telco Traffic and Performance KPIs
• The very first thing Barometer enabled was a call-back API in DPDK and an associated application that used the
API to demonstrate how to timestamp packets and measure packet latency in DPDK (the sample app is called
rxtx_callbacks). This was upstreamed to DPDK 2.0 and is represented by the interfaces 1 and 2 in Figure 1.2.
3.2. OPNFV Projects
363
OPNFV Documentation, Release Danube
• The second thing Barometer implemented in DPDK is the extended NIC statistics API, which exposes NIC stats
including error stats to the DPDK user by reading the registers on the NIC. This is represented by interface 3 in
Figure 1.2.
– For DPDK 2.1 this API was only implemented for the ixgbe (10Gb) NIC driver, in association with a
sample application that runs as a DPDK secondary process and retrieves the extended NIC stats.
– For DPDK 2.2 the API was implemented for igb, i40e and all the Virtual Functions (VFs) for all drivers.
– For DPDK 16.07 the API migrated from using string value pairs to using id value pairs, improving the
overall performance of the API.
Monitoring DPDK interfaces With the features Barometer enabled in DPDK to enable measuring Telco traffic and
performance KPIs, we can now retrieve NIC statistics including error stats and relay them to a DPDK user. The next
step is to enable monitoring of the DPDK interfaces based on the stats that we are retrieving from the NICs, by relaying
the information to a higher level Fault Management entity. To enable this Barometer has been enabling a number of
plugins for collectd.
DPDK Keep Alive description SFQM aims to enable fault detection within DPDK, the very first feature to meet
this goal is the DPDK Keep Alive Sample app that is part of DPDK 2.2.
DPDK Keep Alive or KA is a sample application that acts as a heartbeat/watchdog for DPDK packet processing cores,
to detect application thread failure. The application supports the detection of ‘failed’ DPDK cores and notification to
a HA/SA middleware. The purpose is to detect Packet Processing Core fails (e.g. infinite loop) and ensure the failure
of the core does not result in a fault that is not detectable by a management entity.
Fig. 3.2: DPDK Keep Alive Sample Application
Essentially the app demonstrates how to detect ‘silent outages’ on DPDK packet processing cores. The application
can be decomposed into two specific parts: detection and notification.
• The detection period is programmable/configurable but defaults to 5ms if no timeout is specified.
364
Chapter 3. Developer
OPNFV Documentation, Release Danube
• The Notification support is enabled by simply having a hook function that where this can be ‘call back support’
for a fault management application with a compliant heartbeat mechanism.
DPDK Keep Alive Sample App Internals This section provides some explanation of the The KeepAlive/’Liveliness’ conceptual scheme as well as the DPDK Keep Alive App. The initialization and run-time paths
are very similar to those of the L2 forwarding application (see L2 Forwarding Sample Application (in Real and Virtualized Environments) for more information).
There are two types of cores: a Keep Alive Monitor Agent Core (master DPDK core) and Worker cores
(Tx/Rx/Forwarding cores). The Keep Alive Monitor Agent Core will supervise worker cores and report any failure (2
successive missed pings). The Keep-Alive/’Liveliness’ conceptual scheme is:
• DPDK worker cores mark their liveliness as they forward traffic.
• A Keep Alive Monitor Agent Core runs a function every N Milliseconds to inspect worker core liveliness.
• If keep-alive agent detects time-outs, it notifies the fault management entity through a call-back function.
Note: Only the worker cores state is monitored. There is no mechanism or agent to monitor the Keep Alive Monitor
Agent Core.
DPDK Keep Alive Sample App Code Internals The following section provides some explanation of the code
aspects that are specific to the Keep Alive sample application.
The heartbeat functionality is initialized with a struct rte_heartbeat and the callback function to invoke in the case of a
timeout.
rte_global_keepalive_info = rte_keepalive_create(&dead_core, NULL);
if (rte_global_hbeat_info == NULL)
rte_exit(EXIT_FAILURE, "keepalive_create() failed");
The function that issues the pings hbeat_dispatch_pings() is configured to run every check_period milliseconds.
if (rte_timer_reset(&hb_timer,
(check_period * rte_get_timer_hz()) / 1000,
PERIODICAL,
rte_lcore_id(),
&hbeat_dispatch_pings, rte_global_keepalive_info
) != 0 )
rte_exit(EXIT_FAILURE, "Keepalive setup failure.\n");
The rest of the initialization and run-time path follows the same paths as the the L2 forwarding application. The only
addition to the main processing loop is the mark alive functionality and the example random failures.
rte_keepalive_mark_alive(&rte_global_hbeat_info);
cur_tsc = rte_rdtsc();
/* Die randomly within 7 secs for demo purposes.. */
if (cur_tsc - tsc_initial > tsc_lifetime)
break;
The rte_keepalive_mark_alive() function simply sets the core state to alive.
static inline void
rte_keepalive_mark_alive(struct rte_heartbeat *keepcfg)
{
keepcfg->state_flags[rte_lcore_id()] = 1;
}
3.2. OPNFV Projects
365
OPNFV Documentation, Release Danube
Keep Alive Monitor Agent Core Monitoring Options The application can run on either a host or a guest. As such there
are a number of options for monitoring the Keep Alive Monitor Agent Core through a Local Agent on the compute
node:
Application Location
HOST
GUEST
DPDK KA
X
X
LOCAL AGENT
HOST/GUEST
HOST/GUEST
For the first implementation of a Local Agent SFQM will enable:
Application Location
HOST
DPDK KA
X
LOCAL AGENT
HOST
Through extending the dpdkstat plugin for collectd with KA functionality, and integrating the extended plugin with
Monasca for high performing, resilient, and scalable fault detection.
OPNFV Barometer configuration Guide
Barometer Configuration
This document provides guidelines on how to install and configure the Barometer plugin when using Fuel as a deployment tool. The plugin name is: Collectd Ceilometer Plugin. This plugin installs collectd on a compute node and
enables a number of collectd plugins to collect metrics and events from the platform and send them to ceilometer.
•
•
•
•
Pre-configuration activities
Hardware configuration
Feature configuration
Upgrading the plugin
Pre-configuration activities The Barometer Fuel plugin can be found in /opt/opnfv on the fuel master. To enable
this plugin:
$ cd /opt/opnfv
$ fuel plugins --install fuel-plugin-collectd-ceilometer-1.0-1.0.0-1.noarch.rpm
On the Fuel UI, create a new environment. * In Settings > OpenStack Services * Enable “Install Ceilometer and
Aodh” * In Settings > Other * Enable “Deploy Collectd Ceilometer Plugin” * Enable the barometer plugins you’d like
to deploy using the checkboxes * Continue with environment configuration and deployment as normal.
Hardware configuration There’s no specific Hardware configuration required for this the barometer fuel plugin.
Feature configuration Describe the procedures to configure your feature on the platform in order that it is ready to
use according to the feature instructions in the platform user guide. Where applicable you should add content in the
postinstall.rst to validate the feature is configured for use. (checking components are installed correctly etc...)
Upgrading the plugin From time to time new versions of the plugin may become available.
The plugin cannot be upgraded if an active environment is using the plugin.
In order to upgrade the plugin:
• Copy the updated plugin file to the fuel-master.
366
Chapter 3. Developer
OPNFV Documentation, Release Danube
• On the Fuel UI, reset the environment.
• On the Fuel CLI “fuel plugins –update <fuel-plugin-file>”
• On the Fuel UI, re-deploy the environment.
Barometer post installation procedures
Add a brief introduction to the methods of validating the installation according to this specific installer or feature.
Automated post installation activities Describe specific post installation activities performed by the OPNFV deployment pipeline including testing activities and reports. Refer to the relevant testing guides, results, and release
notes.
note: this section should be singular and derived from the test projects once we have one test suite to run for all deploy
tools. This is not the case yet so each deploy tool will need to provide (hopefully very simillar) documentation of this.
Barometer post configuration procedures The fuel plugin installs collectd and its plugins on compute nodes.
separate config files for each of the collectd plugins. These configuration files can be found on the compute node
@ /etc/collectd/collectd.conf.d/ directory. Each collectd plugin will have its own configuration file with a default
configuration for each plugin. You can override any of the plugin configurations, by modifying the configuration file
and restarting the collectd service on the compute node.
Platform components validation
1. SSH to a compute node and ensure that the collectd service is running.
2. On the compute node, you need to inject a corrected memory error:
$
$
$
$
git clone https://git.kernel.org/pub/scm/utils/cpu/mce/mce-inject.git
cd mce-inject
make
modprobe mce-inject
Modify the test/corrected script to include the following:
CPU 0 BANK 0
STATUS 0xcc00008000010090
ADDR 0x0010FFFFFFF
Inject the error:
$ ./mce-inject < test/corrected
3. SSH to openstack controller node and query the ceilometer DB:
$
$
$
$
$
source openrc
ceilometer sample-list
ceilometer sample-list
ceilometer sample-list
ceilometer sample-list
-m
-m
-m
-m
interface.if_packets
hugepages.vmpage_number
ovs_events.gauge
mcelog.errors
As you run each command above, you should see output similar to the examples below:
3.2. OPNFV Projects
367
OPNFV Documentation, Release Danube
OPNFV Barometer User Guide
OPNFV Barometer User Guide
• Barometer collectd plugins description
• collectd capabilities and usage
– Building all Barometer upstreamed plugins from scratch
– DPDK statistics plugin
– DPDK events plugin
– Hugepages Plugin
– Intel RDT Plugin
– IPMI Plugin
– Mcelog Plugin
– Open vSwitch Plugins
– SNMP Agent Plugin
– Installing collectd as a service
– Additional useful plugins
– Monitoring Interfaces and Openstack Support
– References
Barometer collectd plugins description collectd is a daemon which collects system performance statistics periodically and provides a variety of mechanisms to publish the collected metrics. It supports more than 90 different
input and output plugins. Input plugins retrieve metrics and publish them to the collectd deamon, while output plugins
publish the data they receive to an end point. collectd also has infrastructure to support thresholding and notification.
Barometer has enabled the following collectd plugins:
• dpdkstat plugin: A read plugin that retrieve stats from the DPDK extended NIC stats API.
• dpdkevents plugin: A read plugin that retrieves DPDK link status and DPDK forwarding cores liveliness status
(DPDK Keep Alive).
• ceilometer plugin: A write plugin that pushes the retrieved stats to Ceilometer. It’s capable of pushing any stats
read through collectd to Ceilometer, not just the DPDK stats.
• hugepages plugin: A read plugin that retrieves the number of available and free hugepages on a platform as well
as what is available in terms of hugepages per socket.
• Open vSwitch events Plugin: A read plugin that retrieves events from OVS.
• Open vSwitch stats Plugin: A read plugin that retrieve flow and interface stats from OVS.
• mcelog plugin: A read plugin that uses mcelog client protocol to check for memory Machine Check Exceptions
and sends the stats for reported exceptions
• RDT plugin: A read plugin that provides the last level cache utilitzation and memory bandwidth utilization
All the plugins above are available on the collectd master, except for the ceilometer plugin as it’s a python based plugin
and only C plugins are accepted by the collectd community. The ceilometer plugin lives in the OpenStack repositories.
Other plugins existing as a pull request into collectd master:
• SNMP Agent: A write plugin that will act as a AgentX subagent that receives and handles queries from SNMP
master agent and returns the data collected by read plugins. The SNMP Agent plugin handles requests only for
OIDs specified in configuration file. To handle SNMP queries the plugin gets data from collectd and translates
requested values from collectd’s internal format to SNMP format. Supports SNMP: get, getnext and walk
requests.
368
Chapter 3. Developer
OPNFV Documentation, Release Danube
• Legacy/IPMI: A read plugin that reports platform thermals, voltages, fanspeed, current, flow, power etc. Also,
the plugin monitors Intelligent Platform Management Interface (IPMI) System Event Log (SEL) and sends the
Plugins included in the Danube release:
• Hugepages
• Open vSwitch Events
• Ceilometer
• Mcelog
collectd capabilities and usage
Note: Plugins included in the OPNFV D release will be built-in to the fuel plugin and available in the /opt/opnfv
directory on the fuel master. You don’t need to clone the barometer/collectd repos to use these, but you can configure
them as shown in the examples below.
The collectd plugins in OPNFV are configured with reasonable defaults, but can be overriden.
Building all Barometer upstreamed plugins from scratch The plugins that have been merged to the collectd
master branch can all be built and configured through the barometer repository.
Note:
• sudo permissions are required to install collectd.
• These are instructions for Ubuntu 16.04
To build and install these dependencies, clone the barometer repo:
$ git clone https://gerrit.opnfv.org/gerrit/barometer
Install the build dependencies
$ ./src/install_build_deps.sh
To install collectd as a service and install all it’s dependencies:
$ cd barometer/src && sudo make && sudo make install
This will install collectd as a service and the base install directory will be /opt/collectd.
Sample configuration files can be found in ‘/opt/collectd/etc/collectd.conf.d’
Note:
• If you plan on using the Exec plugin, the plugin requires non-root user to execute scripts. By default, collectd_exec user is used. Barometer scripts do not create this user. It needs to be manually added or exec plugin
configuration has to be changed to use other, existing user before starting collectd service.
• If you don’t want to use one of the Barometer plugins, simply remove the sample config file from
‘/opt/collectd/etc/collectd.conf.d’
• If you are using any Open vSwitch plugins you need to run:
$ sudo ovs-vsctl set-manager ptcp:6640
Below is the per plugin installation and configuration guide, if you only want to install some/particular plugins.
3.2. OPNFV Projects
369
OPNFV Documentation, Release Danube
DPDK statistics plugin Repo: https://github.com/collectd/collectd
Branch: master
Dependencies: DPDK (http://dpdk.org/) Min_Version: 16.04
To build and install DPDK to /usr please see: https://github.com/collectd/collectd/blob/master/docs/BUILD.dpdkstat.md
Building and installing collectd:
$
$
$
$
$
$
git clone https://github.com/collectd/collectd.git
cd collectd
./build.sh
./configure --enable-syslog --enable-logfile --enable-debug
make
sudo make install
This will install collectd to /opt/collectd The collectd configuration file can be found at /opt/collectd/etc To configure
the dpdkstats plugin you need to modify the configuration file to include:
LoadPlugin dpdkstat
<Plugin "dpdkstat">
<EAL>
Coremask "0x2"
MemoryChannels "4"
ProcessType "secondary"
FilePrefix "rte"
</EAL>
EnabledPortMask 0xffff
PortName "interface1"
PortName "interface2"
</Plugin>
For more information on the plugin parameters, please see: https://github.com/collectd/collectd/blob/master/src/collectd.conf.pod
Note: If you are not building and installing DPDK system-wide you will need to specify the specific paths to the
header files and libraries using LIBDPDK_CPPFLAGS and LIBDPDK_LDFLAGS. You will also need to add the
DPDK library symbols to the shared library path using ldconfig. Note that this update to the shared library path is not
persistant (i.e. it will not survive a reboot).
DPDK events plugin Repo: https://github.com/collectd/collectd
Branch: master
Dependencies: DPDK (http://dpdk.org/)
To build and install DPDK to /usr please see: https://github.com/collectd/collectd/blob/master/docs/BUILD.dpdkstat.md
Building and installing collectd:
$
$
$
$
$
$
git clone https://github.com/maryamtahhan/collectd.git
cd collectd
./build.sh
./configure --enable-syslog --enable-logfile --enable-debug
make
sudo make install
This will install collectd to /opt/collectd The collectd configuration file can be found at /opt/collectd/etc To configure
the dpdkevents plugin you need to modify the configuration file to include:
370
Chapter 3. Developer
OPNFV Documentation, Release Danube
LoadPlugin dpdkevents
<Plugin "dpdkevents">
Interval 1
<EAL>
Coremask "0x1"
MemoryChannels "4"
ProcessType "secondary"
FilePrefix "rte"
</EAL>
<Event "link_status">
SendEventsOnUpdate true
EnabledPortMask 0xffff
PortName "interface1"
PortName "interface2"
SendNotification false
</Event>
<Event "keep_alive">
SendEventsOnUpdate true
LCoreMask "0xf"
KeepAliveShmName "/dpdk_keepalive_shm_name"
SendNotification false
</Event>
</Plugin>
For more information on the plugin parameters, please see: https://github.com/collectd/collectd/blob/master/src/collectd.conf.pod
Note: If you are not building and installing DPDK system-wide you will need to specify the specific paths to the
header files and libraries using LIBDPDK_CPPFLAGS and LIBDPDK_LDFLAGS. You will also need to add the
DPDK library symbols to the shared library path using ldconfig. Note that this update to the shared library path is not
persistant (i.e. it will not survive a reboot).
$ ./configure LIBDPDK_CPPFLAGS="path to DPDK header files" LIBDPDK_LDFLAGS="path to DPDK libraries"
Hugepages Plugin Repo: https://github.com/collectd/collectd
Branch: master
Dependencies: None, but assumes hugepages are configured.
To configure some hugepages:
sudo mkdir -p /mnt/huge
sudo mount -t hugetlbfs nodev /mnt/huge
sudo echo 14336 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
Building and installing collectd:
$
$
$
$
$
$
git clone https://github.com/collectd/collectd.git
cd collectd
./build.sh
./configure --enable-syslog --enable-logfile --enable-hugepages --enable-debug
make
sudo make install
This will install collectd to /opt/collectd The collectd configuration file can be found at /opt/collectd/etc To configure
the hugepages plugin you need to modify the configuration file to include:
3.2. OPNFV Projects
371
OPNFV Documentation, Release Danube
LoadPlugin hugepages
<Plugin hugepages>
ReportPerNodeHP
ReportRootHP
ValuesPages
ValuesBytes
ValuesPercentage
</Plugin>
true
true
true
false
false
For more information on the plugin parameters, please see: https://github.com/collectd/collectd/blob/master/src/collectd.conf.pod
Intel RDT Plugin Repo: https://github.com/collectd/collectd
Branch: master
Dependencies:
• PQoS/Intel RDT library https://github.com/01org/intel-cmt-cat.git
• msr kernel module
Building and installing PQoS/Intel RDT library:
$
$
$
$
git clone https://github.com/01org/intel-cmt-cat.git
cd intel-cmt-cat
make
make install PREFIX=/usr
You will need to insert the msr kernel module:
$ modprobe msr
Building and installing collectd:
$
$
$
$
$
$
git clone https://github.com/collectd/collectd.git
cd collectd
./build.sh
./configure --enable-syslog --enable-logfile --with-libpqos=/usr/ --enable-debug
make
sudo make install
This will install collectd to /opt/collectd The collectd configuration file can be found at /opt/collectd/etc To configure
the RDT plugin you need to modify the configuration file to include:
<LoadPlugin intel_rdt>
Interval 1
</LoadPlugin>
<Plugin "intel_rdt">
Cores ""
</Plugin>
For more information on the plugin parameters, please see: https://github.com/collectd/collectd/blob/master/src/collectd.conf.pod
IPMI Plugin Repo: https://github.com/maryamtahhan/collectd
Branch: feat_ipmi_events, feat_ipmi_analog
Dependencies: OpenIPMI library
372
Chapter 3. Developer
OPNFV Documentation, Release Danube
The IPMI plugin is already implemented in the latest collectd and sensors like temperature, voltage, fanspeed, current
are already supported there. The list of supported IPMI sensors has been extended and sensors like flow, power are
supported now. Also, a System Event Log (SEL) notification feature has been introduced.
• The feat_ipmi_events branch includes new SEL feature support in collectd IPMI plugin. If this feature is enabled, the collectd IPMI plugin will dispatch notifications about new events in System Event Log.
• The feat_ipmi_analog branch includes the support of extended IPMI sensors in collectd IPMI plugin.
On Ubuntu, install the dependencies:
$ sudo apt-get install libopenipmi-dev
Enable IPMI support in the kernel:
$ sudo modprobe ipmi_devintf
$ sudo modprobe ipmi_si
Note: If HW supports IPMI, the /dev/ipmi0 character device will be created.
Clone and install the collectd IPMI plugin:
$
$
$
$
$
$
$
git clone https://github.com/maryamtahhan/collectd
cd collectd
git checkout $BRANCH
./build.sh
./configure --enable-syslog --enable-logfile --enable-debug
make
sudo make install
Where $BRANCH is feat_ipmi_events or feat_ipmi_analog.
This will install collectd to default folder /opt/collectd. The collectd configuration file (collectd.conf)
can be found at /opt/collectd/etc. To configure the IPMI plugin you need to modify the file to include:
LoadPlugin ipmi
<Plugin ipmi>
SELEnabled true # only feat_ipmi_events branch supports this
</Plugin>
Note: By default, IPMI plugin will read all available analog sensor values, dispatch the values to collectd and send
SEL notifications.
For more information on the IPMI plugin parameters and SEL feature configuration,
https://github.com/maryamtahhan/collectd/blob/feat_ipmi_events/src/collectd.conf.pod
please see:
Extended analog sensors support doesn’t require additional configuration. The usual collectd IPMI documentation can
be used:
• https://collectd.org/wiki/index.php/Plugin:IPMI
• https://collectd.org/documentation/manpages/collectd.conf.5.shtml#plugin_ipmi
IPMI documentation:
• https://www.kernel.org/doc/Documentation/IPMI.txt
• http://www.intel.com/content/www/us/en/servers/ipmi/ipmi-second-gen-interface-spec-v2-rev1-1.html
Mcelog Plugin Repo: https://github.com/collectd/collectd
Branch: master
3.2. OPNFV Projects
373
OPNFV Documentation, Release Danube
Dependencies: mcelog
Start by installing mcelog. Note: The kernel has to have CONFIG_X86_MCE enabled. For 32bit kernels you need at
least a 2.6,30 kernel.
On ubuntu:
$ apt-get update && apt-get install mcelog
Or build from source
$ git clone git://git.kernel.org/pub/scm/utils/cpu/mce/mcelog.git
$ cd mcelog
$ make
... become root ...
$ make install
$ cp mcelog.service /etc/systemd/system/
$ systemctl enable mcelog.service
$ systemctl start mcelog.service
Verify you got a /dev/mcelog. You can verify the daemon is running completely by running:
$ mcelog --client
This should query the information in the running daemon. If it prints nothing that is fine (no errors logged yet). More
info @ http://www.mcelog.org/installation.html
Modify the mcelog configuration file “/etc/mcelog/mcelog.conf” to include or enable:
socket-path = /var/run/mcelog-client
Clone and install the collectd mcelog plugin:
$
$
$
$
$
$
$
git clone https://github.com/maryamtahhan/collectd
cd collectd
git checkout feat_ras
./build.sh
./configure --enable-syslog --enable-logfile --enable-debug
make
sudo make install
This will install collectd to /opt/collectd The collectd configuration file can be found at /opt/collectd/etc To configure
the mcelog plugin you need to modify the configuration file to include:
<LoadPlugin mcelog>
Interval 1
</LoadPlugin>
<Plugin "mcelog">
McelogClientSocket "/var/run/mcelog-client"
</Plugin>
For more information on the plugin parameters, please see: https://github.com/maryamtahhan/collectd/blob/feat_ras/src/collectd.conf.pod
Simulating a Machine Check Exception can be done in one of 3 ways:
• Running $make test in the mcelog cloned directory - mcelog test suite
• using mce-inject
• using mce-test
mcelog test suite:
374
Chapter 3. Developer
OPNFV Documentation, Release Danube
It is always a good idea to test an error handling mechanism before it is really needed. mcelog includes a test suite.
The test suite relies on mce-inject which needs to be installed and in $PATH.
You also need the mce-inject kernel module configured (with CONFIG_X86_MCE_INJECT=y), compiled, installed
and loaded:
$ modprobe mce-inject
Then you can run the mcelog test suite with
$ make test
This will inject different classes of errors and check that the mcelog triggers runs. There will be some kernel messages
about page offlining attempts. The test will also lose a few pages of memory in your system (not significant) Note this
test will kill any running mcelog, which needs to be restarted manually afterwards. mce-inject:
A utility to inject corrected, uncorrected and fatal machine check exceptions
$
$
$
$
git clone https://git.kernel.org/pub/scm/utils/cpu/mce/mce-inject.git
cd mce-inject
make
modprobe mce-inject
Modify the test/corrected script to include the following:
CPU 0 BANK 0
STATUS 0xcc00008000010090
ADDR 0x0010FFFFFFF
Inject the error: .. code:: bash
$ ./mce-inject < test/corrected
Note: the uncorrected and fatal scripts under test will cause a platform reset. Only the fatal script generates the
memory errors. In order to quickly emulate uncorrected memory errors and avoid host reboot following test errors
from mce-test suite can be injected:
$ mce-inject
mce-test/cases/coverage/soft-inj/recoverable_ucr/data/srao_mem_scrub
mce-test:
In addition an more in-depth test of the Linux kernel machine check facilities can be done with the mce-test test suite.
mce-test supports testing uncorrected error handling, real error injection, handling of different soft offlining cases, and
other tests.
Corrected memory error injection:
To inject corrected memory errors:
• Remove sb_edac and edac_core kernel modules: rmmod sb_edac rmmod edac_core
• Insert einj module: modprobe einj param_extension=1
• Inject an error by specifying details (last command should be repeated at least two times):
$
$
$
$
$
$
APEI_IF=/sys/kernel/debug/apei/einj
echo 0x8 > $APEI_IF/error_type
echo 0x01f5591000 > $APEI_IF/param1
echo 0xfffffffffffff000 > $APEI_IF/param2
echo 1 > $APEI_IF/notrigger
echo 1 > $APEI_IF/error_inject
• Check the MCE statistic: mcelog –client. Check the mcelog log for injected error details: less /var/log/mcelog.
3.2. OPNFV Projects
375
OPNFV Documentation, Release Danube
Open vSwitch Plugins OvS Events Repo: https://github.com/collectd/collectd
OvS Stats Repo: https://github.com/maryamtahhan/collectd
OvS Events Branch: master
OvS Stats Branch:feat_ovs_stats
OvS Events MIBs: The SNMP OVS interface link status is provided by standard IF-MIB (http://www.netsnmp.org/docs/mibs/IF-MIB.txt)
Dependencies: Open vSwitch, Yet Another JSON Library (https://github.com/lloyd/yajl)
On Ubuntu, install the dependencies:
$ sudo apt-get install libyajl-dev openvswitch-switch
Start the Open vSwitch service:
$ sudo service openvswitch-switch start
configure the ovsdb-server manager:
$ sudo ovs-vsctl set-manager ptcp:6640
Clone and install the collectd ovs plugin:
$
$
$
$
$
$
$
git clone $REPO
cd collectd
git checkout $BRANCH
./build.sh
./configure --enable-syslog --enable-logfile --enable-debug
make
sudo make install
where $REPO is one of the repos listed at the top of this section.
Where $BRANCH is master or feat_ovs_stats.
This will install collectd to /opt/collectd. The collectd configuration file can be found at /opt/collectd/etc. To configure
the OVS events plugin you need to modify the configuration file to include:
<LoadPlugin ovs_events>
Interval 1
</LoadPlugin>
<Plugin "ovs_events">
Port 6640
Socket "/var/run/openvswitch/db.sock"
Interfaces "br0" "veth0"
SendNotification false
DispatchValues true
</Plugin>
To configure the OVS stats plugin you need to modify the configuration file to include:
<LoadPlugin ovs_stats>
Interval 1
</LoadPlugin>
<Plugin ovs_stats>
Port "6640"
Address "127.0.0.1"
Socket "/var/run/openvswitch/db.sock"
376
Chapter 3. Developer
OPNFV Documentation, Release Danube
Bridges "br0" "br_ext"
</Plugin>
For more information on the plugin parameters, please see: https://github.com/collectd/collectd/blob/master/src/collectd.conf.pod
and https://github.com/maryamtahhan/collectd/blob/feat_ovs_stats/src/collectd.conf.pod
SNMP Agent Plugin Repo: https://github.com/maryamtahhan/collectd/
Branch: feat_snmp
Dependencies: NET-SNMP library
Start by installing net-snmp and dependencies.
On ubuntu:
$ apt-get install snmp snmp-mibs-downloader snmpd libsnmp-dev
$ systemctl start snmpd.service
Or build from source
Become root to install net-snmp dependencies
$ apt-get install libperl-dev
Clone and build net-snmp
$
$
$
$
git clone https://github.com/haad/net-snmp.git
cd net-snmp
./configure --with-persistent-directory="/var/net-snmp" --with-systemd --enable-shared --prefix=/us
make
Become root
$ make install
Copy default configuration to persistent folder
$ cp EXAMPLE.conf /usr/share/snmp/snmpd.conf
Set library path and default MIB configuration
$
$
$
$
cd ~/
echo export LD_LIBRARY_PATH=/usr/lib >> .bashrc
net-snmp-config --default-mibdirs
net-snmp-config --snmpconfpath
Configure snmpd as a service
$
$
$
$
cd net-snmp
cp ./dist/snmpd.service /etc/systemd/system/
systemctl enable snmpd.service
systemctl start snmpd.service
Add the following line to snmpd.conf configuration file “/usr/share/snmp/snmpd.conf” to make all OID tree visible
for SNMP clients:
view
systemonly
included
.1
To verify that SNMP is working you can get IF-MIB table using SNMP client to view the list of Linux interfaces:
3.2. OPNFV Projects
377
OPNFV Documentation, Release Danube
$ snmpwalk -v 2c -c public localhost IF-MIB::interfaces
Clone and install the collectd snmp_agent plugin:
$
$
$
$
$
$
$
git clone https://github.com/maryamtahhan/collectd
cd collectd
git checkout feat_snmp
./build.sh
./configure --enable-syslog --enable-logfile --enable-debug --enable-snmp --with-libnetsnmp
make
sudo make install
This will install collectd to /opt/collectd The collectd configuration file can be found at /opt/collectd/etc SNMP Agent
plugin is a generic plugin and cannot work without configuration. To configure the snmp_agent plugin you
need to modify the configuration file to include OIDs mapped to collectd types. The following example maps scalar
memAvailReal OID to value represented as free memory type of memory plugin:
LoadPlugin snmp_agent
<Plugin "snmp_agent">
<Data "memAvailReal">
Plugin "memory"
Type "memory"
TypeInstance "free"
OIDs "1.3.6.1.4.1.2021.4.6.0"
</Data>
</Plugin>
For more information on the plugin parameters, please see: https://github.com/maryamtahhan/collectd/blob/feat_snmp/src/collectd.conf.p
For more details on AgentX subagent, please see: http://www.net-snmp.org/tutorial/tutorial-5/toolkit/demon/
Installing collectd as a service
NOTE: In an OPNFV installation, collectd is installed and configured as a service.
Collectd service scripts are available in the collectd/contrib directory. To install collectd as a service:
$
$
$
$
sudo cp contrib/systemd.collectd.service /etc/systemd/system/
cd /etc/systemd/system/
sudo mv systemd.collectd.service collectd.service
sudo chmod +x collectd.service
Modify collectd.service
[Service]
ExecStart=/opt/collectd/sbin/collectd
EnvironmentFile=-/opt/collectd/etc/
EnvironmentFile=-/opt/collectd/etc/
CapabilityBoundingSet=CAP_SETUID CAP_SETGID
Reload
$ sudo systemctl daemon-reload
$ sudo systemctl start collectd.service
$ sudo systemctl status collectd.service should show success
Additional useful plugins
• Exec Plugin : Can be used to show you when notifications are being
378
Chapter 3. Developer
OPNFV Documentation, Release Danube
generated by calling a bash script that dumps notifications to file.
/opt/collectd/etc/collectd.conf:
(handy for debug).
Modify
LoadPlugin exec
<Plugin exec>
#
Exec "user:group" "/path/to/exec"
NotificationExec "user" "<path to barometer>/barometer/src/collectd/collectd_sample_configs/write_
</Plugin>
write_notification.sh (just writes the notification passed from exec through STDIN to a file (/tmp/notifications)):
#!/bin/bash
rm -f /tmp/notifications
while read x y
do
echo $x$y >> /tmp/notifications
done
output to /tmp/notifications should look like:
Severity:WARNING
Time:1479991318.806
Host:localhost
Plugin:ovs_events
PluginInstance:br-ex
Type:gauge
TypeInstance:link_status
uuid:f2aafeec-fa98-4e76-aec5-18ae9fc74589
linkstate of "br-ex" interface has been changed to "DOWN"
• logfile plugin: Can be used to log collectd activity. Modify /opt/collectd/etc/collectd.conf to include:
LoadPlugin logfile
<Plugin logfile>
LogLevel info
File "/var/log/collectd.log"
Timestamp true
PrintSeverity false
</Plugin>
Monitoring Interfaces and Openstack Support The figure above shows the DPDK L2 forwarding application
running on a compute node, sending and receiving traffic. collectd is also running on this compute node retrieving the
stats periodically from DPDK through the dpdkstat plugin and publishing the retrieved stats to Ceilometer through the
ceilometer plugin.
To see this demo in action please checkout: Barometer OPNFV Summit demo
References
collectd VES plugin User Guide
The Barometer repository contains a python based write plugin for VES.
The plugin currently supports pushing platform relevant metrics through the additional measurements field for VES.
Please note: Hardcoded configuration values will be modified so that they are configurable through the configuration
file.
3.2. OPNFV Projects
379
OPNFV Documentation, Release Danube
Fig. 3.3: Monitoring Interfaces and Openstack Support
Installation Instructions:
1. Clone this repo
2. Install collectd
$ sudo apt-get install collectd
3. Modify the collectd configuration script: /etc/collectd/collectd.conf
<LoadPlugin python>
Globals true
</LoadPlugin>
<Plugin python>
ModulePath "/path/to/your/python/modules"
LogTraces true
Interactive false
Import "ves_plugin"
<Module ves_plugin>
# VES plugin configuration (see next section below)
</Module>
</Plugin>
where “/path/to/your/python/modules” is the path to where you cloned this repo
VES python plugin configuration description:
Note Details of the Vendor Event Listener REST service
REST resources are defined with respect to a ServerRoot:
380
Chapter 3. Developer
OPNFV Documentation, Release Danube
ServerRoot = https://{Domain}:{Port}/{optionalRoutingPath}
REST resources are of the form:
{ServerRoot}/eventListener/v{apiVersion}`
{ServerRoot}/eventListener/v{apiVersion}/{topicName}`
{ServerRoot}/eventListener/v{apiVersion}/eventBatch`
Domain “host” * VES domain name. It can be IP address or hostname of VES collector (default: 127.0.0.1)
Port port * VES port (default: 30000)
Path “path” * Used as the “optionalRoutingPath” element in the REST path (default: empty)
Topic “path” * Used as the “topicName” element in the REST path (default: empty)
UseHttps true|false * Allow plugin to use HTTPS instead of HTTP (default: false)
Username “username” * VES collector user name (default: empty)
Password “passwd” * VES collector password (default: empty)
FunctionalRole “role” * Used as the ‘functionalRole’ field of ‘commonEventHeader’ event (default: Collectd VES
Agent)
GuestRunning true|false * This option is used if the collectd is running on a guest machine, e.g this option should be
set to true in this case. Defaults to false.
Other collectd.conf configurations Please ensure that FQDNLookup is set to false
FQDNLookup
false
Please ensure that the virt plugin is enabled and configured as follows. This configuration is is required only on a host
side (‘GuestRunning’ = false).
LoadPlugin virt
<Plugin virt>
Connection "qemu:///system"
RefreshInterval 60
HostnameFormat uuid
</Plugin>
Please ensure that the cpu plugin is enabled and configured as follows
LoadPlugin cpu
<Plugin cpu>
ReportByCpu false
ValuesPercentage true
</Plugin>
Please ensure that the aggregation plugin is enabled and configured as follows
LoadPlugin aggregation
<Plugin aggregation>
<Aggregation>
Plugin "cpu"
Type "percent"
GroupBy "Host"
3.2. OPNFV Projects
381
OPNFV Documentation, Release Danube
GroupBy "TypeInstance"
SetPlugin "cpu-aggregation"
CalculateAverage true
</Aggregation>
</Plugin>
If plugin is running on a guest side, it is important to enable uuid plugin too. In this case the hostname in event message
will be represented as UUID instead of system host name.
LoadPlugin uuid
If custom UUID needs to be provided, the following configuration is required in collectd.conf file:
<Plugin uuid>
UUIDFile "/etc/uuid"
</Plugin>
Where “/etc/uuid” is a file containing custom UUID.
Please also ensure that the following plugins are enabled:
LoadPlugin disk
LoadPlugin interface
LoadPlugin memory
VES plugin notification example A good example of collectD notification is monitoring of CPU load on a host or
guest using ‘threshold’ plugin. The following configuration will setup VES plugin to send ‘Fault’ event every time a
CPU idle value is out of range (e.g.: WARNING: CPU-IDLE < 50%, CRITICAL: CPU-IDLE < 30%) and send ‘Fault’
NORMAL event if CPU idle value is back to normal.
LoadPlugin threshold
<Plugin "threshold">
<Plugin "cpu-aggregation">
<Type "percent">
WarningMin
50.0
WarningMax
100.0
FailureMin
30.0
FailureMax
100.0
Instance "idle"
Hits 1
</Type>
</Plugin>
</Plugin>
More detailed information on how to configure collectD thresholds(memory, cpu etc.)
https://collectd.org/documentation/manpages/collectd-threshold.5.shtml
can be found here at
Compass4Nfv
Compass4nfv Installation Instructions
Abstract
This document describes how to install the Danube release of OPNFV when using Compass4nfv as a deployment tool
covering it’s limitations, dependencies and required system resources.
382
Chapter 3. Developer
OPNFV Documentation, Release Danube
Version history
Date
2017-02-21
2016-09-13
2016-09-12
2016-01-17
2015-12-16
2015-09-12
Ver.
3.0.0
2.1.0
2.0.0
1.0.0
0.0.2
0.0.1
Author
Justin chi (HUAWEI)
Yuenan Li (HUAWEI)
Yuenan Li (HUAWEI)
Justin chi (HUAWEI)
Matthew Li (HUAWEI)
Chen Shuai (HUAWEI)
Comment
Changes for D release
Adjusted the docs structure
Rewritten for Compass4nfv C release
Rewritten for Compass4nfv B release
Minor changes & formatting
First draft
Features
Supported Openstack Version and OS
CentOS 7
Ubuntu trusty
Ubuntu xenial
Supported Openstack Flavor and Features
OS only
yes
yes
yes
OpenStack Liberty
yes
yes
no
Virtual Deployment
Baremetal Deployment
HA
Ceph
SDN ODL/ONOS
Compute Node Expansion
Multi-Nic Support
Boot Recovery
OpenStack Mitaka
yes
yes
yes
OpenStack Liberty
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
OpenStack Newton
yes
no
yes
OpenStack Mitaka
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
• ONOS support will Release in Danube 2.0 or 3.0
Compass4nfv configuration
This document describes providing guidelines on how to install and configure the Danube release of OPNFV when
using Compass as a deployment tool including required software and hardware configurations.
Installation and configuration of host OS, OpenStack, OpenDaylight, ONOS, Ceph etc. can be supported by Compass
on Virtual nodes or Bare Metal nodes.
The audience of this document is assumed to have good knowledge in networking and Unix/Linux administration.
Preconditions Before starting the installation of the Danube release of OPNFV, some planning must be done.
Retrieving the installation ISO image First of all, The installation ISO is needed for deploying your OPNFV
environment, it included packages of Compass, OpenStack, OpenDaylight, ONOS and so on.
The stable release ISO can be retrieved via OPNFV software download page
The daily build ISO can be retrieved via OPNFV artifacts repository:
http://artifacts.opnfv.org/compass4nfv.html
NOTE: Search the keyword “compass4nfv/Danube” to locate the ISO image.
E.g. compass4nfv/danube/opnfv-2017-03-29_08-55-09.iso
3.2. OPNFV Projects
383
OpenStack N
Yes
Yes
Yes
Yes
Yes*
Yes
Yes
Yes
OPNFV Documentation, Release Danube
The name of iso image includes the time of iso building, you can get the daily ISO according the building time. The git
url and sha1 of Compass4nfv are recorded in properties files, According these, the corresponding deployment scripts
can be retrieved.
Getting the deployment scripts To retrieve the repository of Compass4nfv on Jumphost use the following command:
• git clone https://gerrit.opnfv.org/gerrit/compass4nfv
NOTE: PLEASE DO NOT GIT CLONE COMPASS4NFV IN ROOT DIRECTORY(INCLUDE SUBFOLDERS).
To get stable /Danube release, you can use the following command:
• git checkout Danube.1.0
Setup Requirements If you have only 1 Bare Metal server, Virtual deployment is recommended. if more than
or equal 3 servers, the Bare Metal deployment is recommended. The minimum number of servers for Bare metal
deployment is 3, 1 for JumpServer(Jumphost), 1 for controller, 1 for compute.
Jumphost Requirements The Jumphost requirements are outlined below:
1. Ubuntu 14.04 (Pre-installed).
2. Root access.
3. libvirt virtualization support.
4. Minimum 2 NICs.
• PXE installation Network (Receiving PXE request from nodes and providing OS provisioning)
• IPMI Network (Nodes power control and set boot PXE first via IPMI interface)
• External Network (Optional: Internet access)
5. 16 GB of RAM for a Bare Metal deployment, 64 GB of RAM for a Virtual deployment.
6. CPU cores: 32, Memory: 64 GB, Hard Disk: 500 GB, (Virtual Deployment needs 1 TB Hard Disk)
Bare Metal Node Requirements Bare Metal nodes require:
1. IPMI enabled on OOB interface for power control.
2. BIOS boot priority should be PXE first then local hard disk.
3. Minimum 3 NICs.
• PXE installation Network (Broadcasting PXE request)
• IPMI Network (Receiving IPMI command from Jumphost)
• External Network (OpenStack mgmt/external/storage/tenant network)
Network Requirements Network requirements include:
1. No DHCP or TFTP server running on networks used by OPNFV.
2. 2-6 separate networks with connectivity between Jumphost and nodes.
• PXE installation Network
• IPMI Network
384
Chapter 3. Developer
OPNFV Documentation, Release Danube
• Openstack mgmt Network*
• Openstack external Network*
• Openstack tenant Network*
• Openstack storage Network*
3. Lights out OOB network access from Jumphost with IPMI node enabled (Bare Metal deployment only).
4. External network has Internet access, meaning a gateway and DNS availability.
The networks with(*) can be share one NIC(Default configuration) or use an exclusive NIC(Reconfigurated in
network.yml).
Execution Requirements (Bare Metal Only)
information:
In order to execute a deployment, one must gather the following
1. IPMI IP addresses of the nodes.
2. IPMI login information for the nodes (user/pass).
3. MAC address of Control Plane / Provisioning interfaces of the Bare Metal nodes.
Configurations There are three configuration files a user needs to modify for a cluster deployment.
network_cfg.yaml for openstack networks on hosts. dha file for host role, IPMI credential and host nic
idenfitication (MAC address). deploy.sh for os and openstack version.
Configure network
network_cfg.yaml file describes networks configuration for openstack on hosts. It specifies host network mapping
and ip assignment of networks to be installed on hosts. Compass4nfv includes a sample network_cfg.yaml under
compass4nfv/deploy/conf/network_cfg.yaml
There are three openstack networks to be installed: external, mgmt and storage. These three networks can be shared
on one physical nic or on separate nics (multi-nic). The sample included in compass4nfv uses one nic. For multi-nic
configuration, see multi-nic configuration.
Configure openstack network
mac address !**
**! All interface name in network_cfg.yaml must be identified in dha file by
Compass4nfv will install networks on host as described in this configuration. It will look for physical nic on host by
mac address from dha file and rename nic to the name with that mac address. Therefore, any network interface name
that is not identified by mac address in dha file will not be installed correctly as compass4nfv cannot find the nic.
Configure provider network
provider_net_mappings:
- name: br-prv
network: physnet
interface: eth1
type: ovs
role:
- controller
- compute
3.2. OPNFV Projects
385
OPNFV Documentation, Release Danube
The external nic in dha file must be named eth1 with mac address. If user uses a different interface name in dha
file, change eth1 to that name here. Note: User cannot use eth0 for external interface name as install/pxe network is
named as such.
Configure openstack mgmt&storage network:
sys_intf_mappings:
- name: mgmt
interface: eth1
vlan_tag: 101
type: vlan
role:
- controller
- compute
- name: storage
interface: eth1
vlan_tag: 102
type: vlan
role:
- controller
- compute
Change vlan_tag of mgmt and storage to corresponding vlan tag configured on switch.
Note: for virtual deployment, there is no need to modify mgmt&storage network.
If using multi-nic feature, i.e, separate nic for mgmt or storage network, user needs to change name to desired nic
name (need to match dha file). Please see multi-nic configuration.
Assign IP address to networks ip_settings section specifics ip assignment for openstack networks.
User can use default ip range for mgmt&storage network.
for external networks:
- name: external
ip_ranges:
- - "192.168.50.210"
- "192.168.50.220"
cidr: "192.168.50.0/24"
gw: "192.168.50.1"
role:
- controller
- compute
Provide at least number of hosts available ip for external IP range(these ips will be assigned to each host). Provide
actual cidr and gateway in cidr and gw fields.
configure public IP for horizon dashboard
public_vip:
ip: 192.168.50.240
netmask: "24"
interface: external
Provide an external ip in ip field. This ip cannot be within the ip range assigned to external network configured in
pervious section. It will be used for horizon address.
See section 6.2 (Vitual) and 7.2 (BareMetal) for graphs illustrating network topology.
386
Chapter 3. Developer
OPNFV Documentation, Release Danube
Installation on Bare Metal
Nodes Configuration (Bare Metal Deployment) The below file is the inventory template of deployment nodes:
“compass4nfv/deploy/conf/hardware_environment/huawei-pod1/dha.yml”
The “dha.yml” is a collectively name for “os-nosdn-nofeature-ha.yml os-ocl-nofeature-ha.yml os-odl_l2-moon-ha.yml
etc”.
You can write your own IPMI IP/User/Password/Mac address/roles reference to it.
• name – Host name for deployment node after installation.
• ipmiVer – IPMI interface version for deployment node support. IPMI 1.0 or IPMI 2.0 is available.
• ipmiIP – IPMI IP address for deployment node. Make sure it can access from Jumphost.
• ipmiUser – IPMI Username for deployment node.
• ipmiPass – IPMI Password for deployment node.
• mac – MAC Address of deployment node PXE NIC.
• interfaces – Host NIC renamed according to NIC MAC addresses when OS provisioning.
• roles – Components deployed.
Set TYPE/FLAVOR and POWER TOOL
E.g. .. code-block:: yaml
TYPE: baremetal FLAVOR: cluster POWER_TOOL: ipmitool
Set ipmiUser/ipmiPass and ipmiVer
E.g.
ipmiUser: USER
ipmiPass: PASSWORD
ipmiVer: '2.0'
Assignment of different roles to servers
E.g. Openstack only deployment roles setting
hosts:
- name: host1
mac: 'F8:4A:BF:55:A2:8D'
interfaces:
- eth1: 'F8:4A:BF:55:A2:8E'
ipmiIp: 172.16.130.26
roles:
- controller
- ha
- name: host2
mac: 'D8:49:0B:DA:5A:B7'
interfaces:
- eth1: 'D8:49:0B:DA:5A:B8'
ipmiIp: 172.16.130.27
roles:
- compute
NOTE: THE ‘ha’ role MUST BE SELECTED WITH CONTROLLERS, EVEN THERE IS ONLY ONE CONTROLLER NODE.
3.2. OPNFV Projects
387
OPNFV Documentation, Release Danube
E.g. Openstack and ceph deployment roles setting
hosts:
- name: host1
mac: 'F8:4A:BF:55:A2:8D'
interfaces:
- eth1: 'F8:4A:BF:55:A2:8E'
ipmiIp: 172.16.130.26
roles:
- controller
- ha
- ceph-adm
- ceph-mon
- name: host2
mac: 'D8:49:0B:DA:5A:B7'
interfaces:
- eth1: 'D8:49:0B:DA:5A:B8'
ipmiIp: 172.16.130.27
roles:
- compute
- ceph-osd
E.g. Openstack and ODL deployment roles setting
hosts:
- name: host1
mac: 'F8:4A:BF:55:A2:8D'
interfaces:
- eth1: 'F8:4A:BF:55:A2:8E'
ipmiIp: 172.16.130.26
roles:
- controller
- ha
- odl
- name: host2
mac: 'D8:49:0B:DA:5A:B7'
interfaces:
- eth1: 'D8:49:0B:DA:5A:B8'
ipmiIp: 172.16.130.27
roles:
- compute
E.g. Openstack and ONOS deployment roles setting
hosts:
- name: host1
mac: 'F8:4A:BF:55:A2:8D'
interfaces:
- eth1: 'F8:4A:BF:55:A2:8E'
ipmiIp: 172.16.130.26
roles:
- controller
- ha
- onos
- name: host2
mac: 'D8:49:0B:DA:5A:B7'
interfaces:
388
Chapter 3. Developer
OPNFV Documentation, Release Danube
- eth1: 'D8:49:0B:DA:5A:B8'
ipmiIp: 172.16.130.27
roles:
- compute
Network Configuration (Bare Metal Deployment) Before deployment, there are some network configuration to be checked based on your network topology.Compass4nfv network default configuration file is
“compass4nfv/deploy/conf/hardware_environment/huawei-pod1/network.yml”. This file is an example, you can customize by yourself according to specific network environment.
In this network.yml, there are several config sections listed following(corresponed to the ordre of the config file):
Provider Mapping
• name – provider network name.
• network – default as physnet, do not change it.
• interfaces – the NIC or Bridge attached by the Network.
• type – the type of the NIC or Bridge(vlan for NIC and ovs for Bridge, either).
• roles – all the possible roles of the host machines which connected by this network(mostly put both controller
and compute).
System Interface
• name – Network name.
• interfaces – the NIC or Bridge attached by the Network.
• vlan_tag – if type is vlan, add this tag before ‘type’ tag.
• type – the type of the NIC or Bridge(vlan for NIC and ovs for Bridge, either).
• roles – all the possible roles of the host machines which connected by this network(mostly put both controller
and compute).
IP Settings
• name – network name corresponding the the network name in System Interface section one by one.
• ip_ranges – ip addresses range provided for this network.
• cidr – the IPv4 address and its associated routing prefix and subnet mask?
• gw – need to add this line only if network is external.
• roles – all the possible roles of the host machines which connected by this network(mostly put both controller
and compute).
Internal VIP(virtual or proxy IP)
• ip – virtual or proxy ip address, must be in the same subnet with mgmt network but must not be in the range of
mgmt network.
• netmask – the length of netmask
• interface – mostly mgmt.
3.2. OPNFV Projects
389
OPNFV Documentation, Release Danube
Public VIP
• ip – virtual or proxy ip address, must be in the same subnet with external network but must not be in the range
of external network.
• netmask – the length of netmask
• interface – mostly external.
ONOS NIC
• the NIC for ONOS, if there is no ONOS configured, leave it unchanged.
Public Network
• enable – must be True(if False, you need to set up provider network manually).
• network – leave it ext-net.
• type – the type of the ext-net above, such as flat or vlan.
• segment_id – when the type is vlan, this should be id of vlan.
• subnet – leave it ext-subnet.
• provider_network – leave it physnet.
• router – leave it router-ext.
• enable_dhcp – must be False.
• no_gateway – must be False.
• external_gw – same as gw in ip_settings.
• floating_ip_cidr – cidr for floating ip, see explanation in ip_settings.
• floating_ip_start – define range of floating ip with floating_ip_end(this defined range must not be included in ip
range of external configured in ip_settings section).
• floating_ip_end – define range of floating ip with floating_ip_start.
The following figure shows the default network configuration.
+--+
+--+
+--+
| |
| |
| |
| |
+------------+
| |
| |
| +------+ Jumphost +------+ |
| |
| |
+------+-----+
| |
| |
| |
|
| |
| |
| |
+------------+ +-----+ |
| |
| |
| |
| |
+------------+
| |
| |
| +------+
host1
+------+ |
| |
| |
+------+-----+
| |
| |
| |
|
| |
| |
| |
+------------+ +-----+ |
| |
| |
| |
| |
+------------+
| |
| |
| +------+
host2
+------+ |
| |
| |
+------+-----+
| |
| |
| |
|
| |
| |
| |
+------------+ +-----+ |
390
Chapter 3. Developer
OPNFV Documentation, Release Danube
| |
| |
| |
| |
+------------+
| |
| |
| +------+
host3
+------+ |
| |
| |
+------+-----+
| |
| |
| |
|
| |
| |
| |
+------------+ +-----+ |
| |
| |
| |
| |
| |
| |
+-++
++-+
+-++
^
^
^
|
|
|
|
|
|
+-+-------------------------+ |
|
|
External Network
| |
|
+---------------------------+ |
|
+-----------------------+---+
|
|
IPMI Network
|
|
+---------------------------+
|
+-------------------------+-+
| PXE(Installation) Network |
+---------------------------+
The following figure shows the interfaces and nics of JumpHost and deployment nodes in huawei-pod1 network
configuration(default one nic for openstack networks).
+--------------JumpHost-------------+
|
|
|
+-+Compass+-+
|
|
|
+
+--------+
|
External-network
|
|
eth2+---+br-ext +-+eth0+----------------------+
|
|
+
+--------+
|
|
|
|
|
|
|
|
|
|
|
|
|
|
+
+--------+
|
Install-network
|
|
|
eth1+---+install +-+eth1+-----------------+
|
|
|
+
+--------+
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
+
|
IPMI-network
|
|
|
|
eth0
eth2+-----------+
|
|
|
|
+
|
|
|
|
|
+---+VM+----+
|
|
|
|
+-----------------------------------+
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
+---------------Host1---------------+
|
|
|
|
|
|
|
|
|
eth0+----------------+
|
|
|
|
|
|
|
mgmt +--------+ |
|
|
|
|
| |
|
|
|
|
+-----------+
| |
|
|
|
|
external+----+ br-prv
+----+eth1+---------------------+
|
+-----------+
| |
|
|
|
|
| |
|
|
|
|
storage +-----+ |
|
|
|
|
|
|
|
|
3.2. OPNFV Projects
391
OPNFV Documentation, Release Danube
+-----------------------------------+
|
|
|
|
IPMI+-----------+
|
|
+-----------------------------------+
|
|
|
|
|
|
|
|
|
|
|
|
+---------------Host2---------------+
|
|
|
|
|
|
|
|
|
eth0+----------------+
|
|
|
|
|
|
mgmt +--------+ |
|
|
|
| |
|
|
|
+-----------+
| |
|
|
|
external+----+ br-prv
+----+eth1+---------------------+
|
+-----------+
| |
|
|
| |
|
|
storage +-----+ |
|
|
|
|
+-----------------------------------+
|
|
IPMI+-----------+
+-----------------------------------+
The following figure shows the interfaces and nics of JumpHost and deployment nodes in intel-pod8 network
configuration(openstack networks are seperated by multiple NICs).
+-------------+JumpHost+------------+
|
|
|
+-+Compass+-+
|
|
|
+
+--------+
|
External-network
|
|
eth2+---+br-ext +-+eth0+----------------------+
|
|
+
+--------+
|
|
|
|
|
|
|
|
|
|
|
|
|
|
+
+--------+
|
Install-network
|
|
|
eth1+---+install +-+eth1+-----------------+
|
|
|
+
+--------+
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
+
|
IPMI-network
|
|
|
|
eth0
eth2+-----------+
|
|
|
|
+
|
|
|
|
|
+---+VM+----+
|
|
|
|
+-----------------------------------+
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
+--------------+Host1+--------------+
|
|
|
|
|
|
|
|
|
eth0+----------------+
|
|
|
|
|
|
|
+--------+
|
|
|
|
|
external+----+br-prv +-+eth1+---------------------+
|
+--------+
|
|
|
|
|
storage +---------------+eth2+-------------------------+
|
|
|
|
|
|
|
Mgmt
+---------------+eth3+----------------------------+
|
|
|
|
|
| |
|
|
|
|
|
| |
392
Chapter 3. Developer
OPNFV Documentation, Release Danube
+-----------------------------------+
|
|
|
| |
|
IPMI+-----------+
|
|
| |
+-----------------------------------+
|
|
|
| |
|
|
|
| |
|
|
|
| |
|
|
|
| |
|
|
|
| |
+--------------+Host2+--------------+
|
|
|
| |
|
|
|
|
|
| |
|
eth0+----------------+
|
| |
|
|
|
|
| |
|
+--------+
|
|
|
| |
|
external+----+br-prv +-+eth1+---------------------+
| |
|
+--------+
|
|
| |
|
storage +---------------+eth2+-------------------------+ |
|
|
| storage-network |
|
Mgmt
+---------------+eth3+----------------------------+
|
|
| mgmt-network
|
|
|
+-----------------------------------+
|
|
IPMI+-----------+
+-----------------------------------+
Start Deployment (Bare Metal Deployment)
1. Edit deploy.sh
1.1. Set OS version for deployment nodes. Compass4nfv supports ubuntu and centos based openstack newton.
E.g.
# Set OS version for target hosts
# Ubuntu16.04 or CentOS7
export OS_VERSION=xenial
or
export OS_VERSION=centos7
1.2. Set ISO image corresponding to your code
E.g.
# Set ISO image corresponding to your code
export ISO_URL=file:///home/compass/compass4nfv.iso
1.3. Set hardware deploy jumpserver PXE NIC. (set eth1 E.g.) You do not need to set it when virtual deploy.
E.g.
# Set hardware deploy jumpserver PXE NIC
# you need to comment out it when virtual deploy
export INSTALL_NIC=eth1
1.4. Set scenario that you want to deploy
E.g.
nosdn-nofeature scenario deploy sample
# DHA is your dha.yml's path
export DHA=./deploy/conf/hardware_environment/huawei-pod1/os-nosdn-nofeature-ha.yml
3.2. OPNFV Projects
393
OPNFV Documentation, Release Danube
# NETWORK is your network.yml's path
export NETWORK=./deploy/conf/hardware_environment/huawei-pod1/network.yml
ocl-nofeature scenario deploy sample
# DHA is your dha.yml's path
export DHA=./deploy/conf/hardware_environment/huawei-pod1/os-ocl-nofeature-ha.yml
# NETWORK is your network.yml's path
export NETWORK=./deploy/conf/hardware_environment/huawei-pod1/network_ocl.yml
odl_l2-moon scenario deploy sample
# DHA is your dha.yml's path
export DHA=./deploy/conf/hardware_environment/huawei-pod1/os-odl_l2-moon-ha.yml
# NETWORK is your network.yml's path
export NETWORK=./deploy/conf/hardware_environment/huawei-pod1/network.yml
odl_l2-nofeature scenario deploy sample
# DHA is your dha.yml's path
export DHA=./deploy/conf/hardware_environment/huawei-pod1/os-odl_l2-nofeature-ha.yml
# NETWORK is your network.yml's path
export NETWORK=./deploy/conf/hardware_environment/huawei-pod1/network.yml
odl_l3-nofeature scenario deploy sample
# DHA is your dha.yml's path
export DHA=./deploy/conf/hardware_environment/huawei-pod1/os-odl_l3-nofeature-ha.yml
# NETWORK is your network.yml's path
export NETWORK=./deploy/conf/hardware_environment/huawei-pod1/network.yml
onos-nofeature scenario deploy sample
# DHA is your dha.yml's path
export DHA=./deploy/conf/hardware_environment/huawei-pod1/os-onos-nofeature-ha.yml
# NETWORK is your network.yml's path
export NETWORK=./deploy/conf/hardware_environment/huawei-pod1/network_onos.yml
onos-sfc deploy scenario sample
# DHA is your dha.yml's path
export DHA=./deploy/conf/hardware_environment/huawei-pod1/os-onos-sfc-ha.yml
# NETWORK is your network.yml's path
export NETWORK=./deploy/conf/hardware_environment/huawei-pod1/network_onos.yml
2. Run deploy.sh
./deploy.sh
Installation on virtual machines
Nodes Configuration (Virtual Deployment)
394
Chapter 3. Developer
OPNFV Documentation, Release Danube
virtual machine setting
• VIRT_NUMBER – the number of nodes for virtual deployment.
• VIRT_CPUS – the number of CPUs allocated per virtual machine.
• VIRT_MEM – the memory size(MB) allocated per virtual machine.
• VIRT_DISK – the disk size allocated per virtual machine.
export
export
export
export
VIRT_NUMBER=${VIRT_NUMBER:-5}
VIRT_CPUS=${VIRT_CPU:-4}
VIRT_MEM=${VIRT_MEM:-16384}
VIRT_DISK=${VIRT_DISK:-200G}
roles setting The below file is the inventory template of deployment nodes:
”./deploy/conf/vm_environment/huawei-virtual1/dha.yml”
The “dha.yml” is a collectively name for “os-nosdn-nofeature-ha.yml os-ocl-nofeature-ha.yml os-odl_l2-moon-ha.yml
etc”.
You can write your own address/roles reference to it.
• name – Host name for deployment node after installation.
• roles – Components deployed.
Set TYPE and FLAVOR
E.g.
TYPE: virtual
FLAVOR: cluster
Assignment of different roles to servers
E.g. Openstack only deployment roles setting
hosts:
- name: host1
roles:
- controller
- ha
- name: host2
roles:
- compute
NOTE: IF YOU SELECT MUTIPLE NODES AS CONTROLLER, THE ‘ha’ role MUST BE SELECT, TOO.
E.g. Openstack and ceph deployment roles setting
hosts:
- name: host1
roles:
- controller
- ha
- ceph-adm
- ceph-mon
- name: host2
roles:
3.2. OPNFV Projects
395
OPNFV Documentation, Release Danube
- compute
- ceph-osd
E.g. Openstack and ODL deployment roles setting
hosts:
- name: host1
roles:
- controller
- ha
- odl
- name: host2
roles:
- compute
E.g. Openstack and ONOS deployment roles setting
hosts:
- name: host1
roles:
- controller
- ha
- onos
- name: host2
roles:
- compute
Network Configuration (Virtual Deployment) The same with Baremetal Deployment.
Start Deployment (Virtual Deployment)
1. Edit deploy.sh
1.1. Set OS version for deployment nodes. Compass4nfv supports ubuntu and centos based openstack newton.
E.g.
# Set OS version for target hosts
# Ubuntu16.04 or CentOS7
export OS_VERSION=xenial
or
export OS_VERSION=centos7
1.2. Set ISO image corresponding to your code
E.g.
# Set ISO image corresponding to your code
export ISO_URL=file:///home/compass/compass4nfv.iso
1.3. Set scenario that you want to deploy
E.g.
nosdn-nofeature scenario deploy sample
396
Chapter 3. Developer
OPNFV Documentation, Release Danube
# DHA is your dha.yml's path
export DHA=./deploy/conf/vm_environment/os-nosdn-nofeature-ha.yml
# NETWORK is your network.yml's path
export NETWORK=./deploy/conf/vm_environment/huawei-virtual1/network.yml
ocl-nofeature scenario deploy sample
# DHA is your dha.yml's path
export DHA=./deploy/conf/vm_environment/os-ocl-nofeature-ha.yml
# NETWORK is your network.yml's path
export NETWORK=./deploy/conf/vm_environment/huawei-virtual1/network_ocl.yml
odl_l2-moon scenario deploy sample
# DHA is your dha.yml's path
export DHA=./deploy/conf/vm_environment/os-odl_l2-moon-ha.yml
# NETWORK is your network.yml's path
export NETWORK=./deploy/conf/vm_environment/huawei-virtual1/network.yml
odl_l2-nofeature scenario deploy sample
# DHA is your dha.yml's path
export DHA=./deploy/conf/vm_environment/os-odl_l2-nofeature-ha.yml
# NETWORK is your network.yml's path
export NETWORK=./deploy/conf/vm_environment/huawei-virtual1/network.yml
odl_l3-nofeature scenario deploy sample
# DHA is your dha.yml's path
export DHA=./deploy/conf/vm_environment/os-odl_l3-nofeature-ha.yml
# NETWORK is your network.yml's path
export NETWORK=./deploy/conf/vm_environment/huawei-virtual1/network.yml
onos-nofeature scenario deploy sample
# DHA is your dha.yml's path
export DHA=./deploy/conf/vm_environment/os-onos-nofeature-ha.yml
# NETWORK is your network.yml's path
export NETWORK=./deploy/conf/vm_environment/huawei-virtual1/network_onos.yml
onos-sfc deploy scenario sample
# DHA is your dha.yml's path
export DHA=./deploy/conf/vm_environment/os-onos-sfc-ha.yml
# NETWORK is your network.yml's path
export NETWORK=./deploy/conf/vm_environment/huawei-virtual1/network_onos.yml
2. Run deploy.sh
./deploy.sh
3.2. OPNFV Projects
397
OPNFV Documentation, Release Danube
Offline Deploy
Compass4nfv uses offline approach to deploy cluster and support complete offline deployment on a jumphost without
access internet. Here is the offline deployment instruction:
Preparation for offline deploy
1. Download compass.iso from OPNFV artifacts repository (Search compass4nfv in http://artifacts.opnfv.org/ and
download an appropriate ISO. ISO can also be generated by script build.sh in compass4nfv root directory.)
2. Download the Jumphost preparation package from our httpserver. (Download the jumphost environment package from here. It should be awared that currently we only support ubuntu trusty as offline jumphost OS.)
3. Clone the compass4nfv code repository.
Steps of offline deploy
1. Copy the compass.iso, jh_env_package.tar.gz and the compass4nfv code repository to your jumphost.
2. Export the local path of the compass.iso and jh_env_package.tar.gz on jumphost. Then you can perform deployment on a offline jumphost.
E.g.
Export the compass4nfv.iso and jh_env_package.tar.gz path
# ISO_URL and JHPKG_URL should be absolute path
export ISO_URL=file:///home/compass/compass4nfv.iso
export JHPKG_URL=file:///home/compass/jh_env_package.tar.gz
Run deploy.sh
./deploy.sh
Expansion Guide
Edit NETWORK File The below file is the inventory template of deployment nodes:
”./deploy/conf/hardware_environment/huawei-pod1/network.yml”
You need to edit the network.yml which you had edited the first deployment.
NOTE: External subnet’s ip_range should exclude the IPs those have already been used.
Edit DHA File The below file is the inventory template of deployment nodes:
”./deploy/conf/hardware_environment/expansion-sample/hardware_cluster_expansion.yml”
You can write your own IPMI IP/User/Password/Mac address/roles reference to it.
• name – Host name for deployment node after installation.
• ipmiIP – IPMI IP address for deployment node. Make sure it can access from Jumphost.
• ipmiUser – IPMI Username for deployment node.
• ipmiPass – IPMI Password for deployment node.
• mac – MAC Address of deployment node PXE NIC .
398
Chapter 3. Developer
OPNFV Documentation, Release Danube
Set TYPE/FLAVOR and POWER TOOL
E.g.
TYPE: baremetal
FLAVOR: cluster
POWER_TOOL: ipmitool
Set ipmiUser/ipmiPass and ipmiVer
E.g.
ipmiUser: USER
ipmiPass: PASSWORD
ipmiVer: '2.0'
Assignment of roles to servers
E.g. Only increase one compute node
hosts:
- name: host6
mac: 'E8:4D:D0:BA:60:45'
interfaces:
- eth1: '08:4D:D0:BA:60:44'
ipmiIp: 172.16.131.23
roles:
- compute
E.g. Increase two compute nodes
hosts:
- name: host6
mac: 'E8:4D:D0:BA:60:45'
interfaces:
- eth1: '08:4D:D0:BA:60:44'
ipmiIp: 172.16.131.23
roles:
- compute
- name: host6
mac: 'E8:4D:D0:BA:60:78'
interfaces:
- eth1: '08:4D:56:BA:60:83'
ipmiIp: 172.16.131.23
roles:
- compute
Start Expansion
1. Edit network.yml and dha.yml file
You need to Edit network.yml and virtual_cluster_expansion.yml or hardware_cluster_expansion.yml. Edit the
DHA and NETWORK envionment variables. External subnet’s ip_range and management ip should be changed
as the first 6 IPs are already taken by the first deployment.
E.g.
--- network.yml
2017-02-16 20:07:10.097878150 +0800
+++ network_expansion.yml
2017-02-17 11:40:08.734480478 +0800
@@ -56,7 +56,7 @@
3.2. OPNFV Projects
399
OPNFV Documentation, Release Danube
- name: external
ip_ranges:
- - "192.168.116.201"
- - "192.168.116.206"
- "192.168.116.221"
cidr: "192.168.116.0/24"
gw: "192.168.116.1"
+
2. Edit deploy.sh
2.1. Set EXPANSION and VIRT_NUMBER. VIRT_NUMBER decide how many virtual machines needs to expand
when virtual expansion
E.g.
export
export
export
export
EXPANSION="true"
MANAGEMENT_IP_START="10.1.0.55"
VIRT_NUMBER=1
DEPLOY_FIRST_TIME="false"
2.2. Set scenario that you need to expansion
E.g.
# DHA is your dha.yml's path
export DHA=./deploy/conf/hardware_environment/expansion-sample/hardware_cluster_expansion.yml
# NETWORK is your network.yml's path
export NETWORK=./deploy/conf/hardware_environment/huawei-pod1/network.yml
Note: Other environment variable shoud be same as your first deployment. Please check the environment variable before you run deploy.sh.
2. Run deploy.sh
./deploy.sh
References
OPNFV OPNFV Compass4nfv project page
Tutoring videos
OpenStack OpenStack Newton Release artifacts
OpenDaylight OpenDaylight artifacts
ONOS ONOS artifacts
Compass Compass Home Page
400
Chapter 3. Developer
OPNFV Documentation, Release Danube
Compass4nfv Design Guide
How to integrate a feature into compass4nfv
This document describes how to integrate a feature (e.g. sdn, moon, kvm, sfc) into compass installer. Follow the steps
below, you can achieve the goal.
Create a role for the feature Currently Ansible is the main packages installation plugin in the adapters of Compass4nfv, which is used to deploy all the roles listed in the playbooks. (More details about ansible and playbook
can be achieved according to the Reference.) The mostly used playbook in compass4nfv is named “HA-ansiblemultinodes.yml” located in “your_path_to_compass4nfv/compass4nfv/deploy/ adapters/ansible/openstack/”.
Before you add your role into the playbook, create your role under the directory of
“your_path_to_compass4nfv/compass4nfv/deploy/adapters/ansible/roles/”. For example Fig 1 shows some roles
currently existed in compass4nfv.
Fig. 3.4: Fig 1. Existed roles in compass4nfv
Let’s take a look at “moon” and understand the construction of a role. Fig 2 below presents the tree of “moon”.
There are five directories in moon, which are files, handlers, tasks, templates and vars. Almost every role has such five
directories.
For “files”, it is used to store the files you want to copy to the hosts without any modification. These files can be
configuration files, code files and etc. Here in moon’s files directory, there are two python files and one configuration
file. All of the three files will be copied to controller nodes for some purposes.
For “handlers”, it is used to store some operations frequently used in your tasks. For example, restart the service
daemon.
For “tasks”, it is used to store the task yaml files. You need to add the yaml files including the tasks you write to deploy
your role on the hosts. Please attention that a main.yml should be existed as the entrance of running tasks. In Fig 2,
you can find that there are four yaml files in the tasks directory of moon. The main.yml is the entrance which will call
the other three yaml files.
For “templates”, it is used to store the files that you want to replace some variables in them before copying to hosts.
These variables are usually defined in “vars” directory. This can avoid hard coding.
For “vars”, it is used to store the yaml files in which the packages and variables are defined. The packages defined
here are some generic debian or rpm packages. The script of making repo will scan the packages names here and
download them into related PPA. For some special packages, section “Build packages for the feature” will introduce
how to handle with special packages. The variables defined here are used in the files in “templates” and “tasks”.
Note: you can get the special packages in the tasks like this:
- name: get the special packages' http server
shell: awk -F'=' '/compass_server/ {print $2}' /etc/compass.conf
register: http_server
3.2. OPNFV Projects
401
OPNFV Documentation, Release Danube
Fig. 3.5: Fig 2. Tree of moon role
402
Chapter 3. Developer
OPNFV Documentation, Release Danube
- name: download odl package
get_url:
url: "http://{{ http_server.stdout_lines[0] }}/packages/odl/{{ odl_pkg_url }}"
dest: /opt/
Build packages for the feature In the previous section, we have explained how to build the generic packages for
your feature. In this section, we will talk about how to build the special packages used by your feature.
Fig. 3.6: Fig 3. Features building directory in compass4nfv
Fig 3 shows the tree of “your_path_to_compass4nfv/compass4nfv/repo/features/”. Dockerfile is used to start a docker
container to run the scripts in scripts directory. These scripts will download the special feature related packages into
the container. What you need to do is to write a shell script to download or build the package you want. And then put
the script into “your_path_to_compass4nfv/compass4nfv/repo/features/scripts/”. Attention that, you need to make a
directory under /pkg. Take opendaylight as an example:
mkdir -p /pkg/odl
After downloading or building your feature packages, please copy all of your packages into the directory you made,
e.g. /pkg/odl.
Note: If you have specail requirements for the container OS or kernel vesion, etc. Please contact us.
After all of these, come back to your_path_to_compass4nfv/compass4nfv/ directory, and run the command below:
./repo/make_repo.sh feature # To get special packages
./repo/make_repo.sh openstack # To get generic packages
When
execution
finished,
you
will
get
a
tar
package
named
packages.tar.gz
under
“your_path_to_compass4nfv/compass4nfv/work/repo/”. Your feature related packages have been archived in
this tar package. And you will also get the PPA packages which includes the generic packages you defined
in the role directory. The PPA packages are xenial-newton-ppa.tar.gz and centos7-newton-ppa.tar.gz, also in
“your_path_to_compass4nfv/compass4nfv/work/repo/”.
Build compass ISO including the feature Before you deploy a cluster with your feature installed, you need an ISO
with feature packages, generic packages and role included. This section introduces how to build the ISO you want.
What you need to do are two simple things:
Configure the build configuration file
The build configuration file is located in “your_path_to_compass4nfv/compass4nfv/build/”. There are lines in the file
like this:
3.2. OPNFV Projects
403
OPNFV Documentation, Release Danube
export APP_PACKAGE=${APP_PACKAGE:-$FEATURE_URL/packages.tar.gz}
export XENIAL_NEWTON_PPA=${XENIAL_NEWTON_PPA:-$PPA_URL/xenial-newton-ppa.tar.gz}
export CENTOS7_NEWTON_PPA=${CENTOS7_NEWTON_PPA:-$PPA_URL/centos7-newton-ppa.tar.gz}
Just replace the $FEATURE_URL and $PPA_URL to the directory where your packages.tar.gz located in. For example:
export APP_PACKAGE=${APP_PACKAGE:-file:///home/opnfv/compass4nfv/work/repo/packages.tar.gz}
export XENIAL_NEWTON_PPA=${XENIAL_NEWTON_PPA:-file:///home/opnfv/compass4nfv/work/repo/xenial-newton-
export CENTOS7_NEWTON_PPA=${CENTOS7_NEWTON_PPA:-file:///home/opnfv/compass4nfv/work/repo/centos7-newt
Build the ISO
After the configuration, just run the command below to build the ISO you want for deployment.
./build.sh
References Ansible documentation: http://docs.ansible.com/ansible/index.html>
Copper
OPNFV Copper Danube Overview
OPNFV Danube Copper Overview
• Introduction
• Overall Goals for Configuration Policy
Introduction The OPNFV Copper project aims to help ensure that virtualized infrastructure and application deployments comply with goals of the NFV service provider or the VNF designer/user.
This is the third (“Danube”) release of the Copper project. The documentation provided here focuses on the overall
goals of the Copper project and the specific features supported in the Colorado release.
Overall Goals for Configuration Policy As focused on by Copper, configuration policy helps ensure that the NFV
service environment meets the requirements of the variety of stakeholders which will provide or use NFV platforms.
These requirements can be expressed as an intent of the stakeholder, in specific terms or more abstractly, but at the
highest level they express:
• what I want
• what I don’t want
Using road-based transportation as an analogy, some examples of this are shown below:
404
Chapter 3. Developer
OPNFV Documentation, Release Danube
Table 3.1: Configuration Intent Example
Who I Am
user
road provider
public safety
What I Want
a van, wheelchair-accessible, electric powered
keep drivers moving at an optimum safe speed
shoulder warning strips, center media barriers
What I Don’t Want
someone driving off with my van
four-way stops
speeding, tractors on the freeway
According to their role, service providers may apply more specific configuration requirements than users, since service
providers are more likely to be managing specific types of infrastructure capabilities.
Developers and users may also express their requirements more specifically, based upon the type of application or how
the user intends to use it.
For users, a high-level intent can be also translated into a more or less specific configuration capability by the service
provider, taking into consideration aspects such as the type of application or its constraints.
Examples of such translation are:
Table 3.2: Intent Translation into Configuration Capability
Intent
network security
compute/storage security
high availability
disaster recovery
high compute/storage performance
high network performance
resource reclamation
Configuration Capability
firewall, DPI, private subnets
vulnerability monitoring, resource access controls
clustering, auto-scaling, anti-affinity, live migration
geo-diverse anti-affinity
clustering, affinity
data plane acceleration
low-usage monitoring
Although such intent-to-capability translation is conceptually useful, it is unclear how it can address the variety of
aspects that may affect the choice of an applicable configuration capability.
For that reason, the Copper project will initially focus on more specific configuration requirements as fulfilled by
specific configuration capabilities, as well as how those requirements and capabilities are expressed in VNF and
service design and packaging or as generic policies for the NFV Infrastructure.
OPNFV Copper Danube Requirements
Requirements
This section outlines general requirements for configuration policies, per the two main aspects in the Copper project
scope:
• Ensuring resource requirements of VNFs and services are applied per VNF designer, service, and tenant intent
• Ensuring that generic policies are not violated, e.g. networks connected to VMs must either be public or owned
by the VM owner
Resource Requirements Resource requirements describe the characteristics of virtual resources (compute, storage,
network) that are needed for VNFs and services, and how those resources should be managed over the lifecycle of a
VNF/service. Upstream projects already include multiple ways in which resource requirements can be expressed and
fulfilled, e.g.:
• OpenStack Nova
– the Image feature, enabling “VM templates” to be defined for NFs and referenced by name as a specific
NF version to be used
3.2. OPNFV Projects
405
OPNFV Documentation, Release Danube
– the Flavor feature, addressing basic compute and storage requirements, with extensibility for custom attributes
• OpenStack Heat
– the Heat Orchestration Template feature, enabling a variety of VM aspects to be defined and managed by
Heat throughout the VM lifecycle, notably
* alarm handling (requires Ceilometer)
* attached volumes (requires Cinder)
* domain name assignment (requires Designate)
* images (requires Glance)
* autoscaling
* software configuration associated with VM lifecycle hooks (CREATE, UPDATE, SUSPEND, RESUME, DELETE)
* wait conditions and signaling for sequencing orchestration steps
* orchestration service user management (requires Keystone)
* shared storage (requires Manila)
* load balancing (requires Neutron LBaaS)
* firewalls (requires Neutron FWaaS)
* various Neutron-based network and security configuration items
* Nova flavors
* Nova server attributes including access control
* Nova server group affinity and anti-affinity
* “Data-intensive application clustering” (requires Sahara)
* DBaaS (requires Trove)
* “multi-tenant cloud messaging and notification service” (requires Zaqar)
• OpenStack Group-Based Policy
– API-based grouping of endpoints with associated contractual expectations for data flow processing and
service chaining
• OpenStack Tacker
– “a fully functional ETSI MANO based general purpose NFV Orchestrator and VNF Manager for OpenStack”
• OpenDaylight Group-Based Policy
– model-based grouping of endpoints with associated contractual expectations for data flow processing
• OpenDaylight Service Function Chaining (SFC)
– model-based management of “service chains” and the infrastucture that enables them
• Additional projects that are commonly used for configuration management, implemented as client-server frameworks using model-based, declarative, or scripted configuration management data.
– Puppet
– Chef
406
Chapter 3. Developer
OPNFV Documentation, Release Danube
– Ansible
– Salt
Generic Policy Requirements Generic policy requirements address conditions related to resource state and events
which need to be monitored for, and optionally responded to or prevented. These conditions are typically expected
to be VNF/service-independent, as VNF/service-dependent condition handling (e.g. scale in/out) are considered to be
addressed by VNFM/NFVO/VIM functions as described under Resource Requirements or as FCAPS related functions.
However the general capabilities below can be applied to VNF/service-specific policy handling as well, or in particular
to invocation of VNF/service-specific management/orchestration actions. The high-level required capabilities include:
• Polled monitoring: Exposure of state via request-response APIs.
• Notifications: Exposure of state via pub-sub APIs.
• Realtime/near-realtime notifications: Notifications that occur in actual or near realtime.
• Delegated policy: CRUD operations on policies that are distributed to specific components for local handling,
including one/more of monitoring, violation reporting, and enforcement.
• Violation reporting: Reporting of conditions that represent a policy violation.
• Reactive enforcement: Enforcement actions taken in response to policy violation events.
• Proactive enforcement: Enforcement actions taken in advance of policy violation events, e.g. blocking actions
that could result in a policy violation.
• Compliance auditing: Periodic auditing of state against policies.
Upstream projects already include multiple ways in which configuration conditions can be monitored and responded
to:
• OpenStack Congress provides a table-based mechanism for state monitoring and proactive/reactive policy enforcement, including data obtained from internal databases of OpenStack core and optional services. The
Congress design approach is also extensible to other VIMs (e.g. SDNCs) through development of data source
drivers for the new monitored state information.
• OpenStack Aodh provides means to trigger alarms upon a wide variety of conditions derived from its monitored
OpenStack analytics.
• Nagios “offers complete monitoring and alerting for servers, switches, applications, and services”.
Requirements Validation Approach The Copper project will assess the completeness of the upstream project solutions for requirements in scope though a process of:
• developing configuration policy use cases to focus solution assessment tests
• integrating the projects into the OPNFV platform for testing
• executing functional and performance tests for the solutions
• assessing overall requirements coverage and gaps in the most complete upstream solutions
Depending upon the priority of discovered gaps, new requirements will be submitted to upstream projects for the next
available release cycle.
OPNFV Copper Installation
OPNFV Copper Installation Guide
This document describes how to install Copper, its dependencies and required system resources.
3.2. OPNFV Projects
407
OPNFV Documentation, Release Danube
• Version History
• Introduction
• Manual Installation
Version History
Date
2017 Feb 7
Ver.
1.0
Author
Bryan Sullivan
Comment
Introduction The Congress service is automatically configured as required by the JOID and Apex installers, including creation of datasources per the installed datasource drivers. This release includes default support for the following
datasource drivers:
• nova
• neutronv2
• ceilometer
• cinder
• glancev2
• keystone
For JOID, Congress is installed through a JuJu Charm, and for Apex through a Puppet Module. Both the Charm and
Module are being upstreamed to OpenStack for future maintenance.
Other project installer support (e.g. Doctor) may install additional datasource drivers once Congress is installed.
Manual Installation NOTE: This section describes a manual install procedure that had been tested under the JOID
and Apex base installs prior to the integration of native installer support through JuJu (JOID) and Puppet (Apex). This
procedure is being maintained as a basis for additional installer support in future releases. However, since Congress is
pre-installed for JOID and Apex, this procedure is not necessary and not recommended for use if Congress is already
installed.
Copper provides a set of bash scripts to automatically install Congress based upon a JOID or Apex install which does
not already have Congress installed. These scripts are in the Copper repo at:
• components/congress/install/bash/install_congress_1.sh
• components/congress/install/bash/install_congress_2.sh
Prerequisites to using these scripts:
• OPFNV installed via JOID or Apex
• For Apex installs, on the jumphost, ssh to the undercloud VM and “su stack”.
• For JOID installs, admin-openrc.sh saved from Horizon to ~/admin-openrc.sh
• Retrieve the copper install script as below, optionally specifying the branch to use as a URL parameter, e.g.
?h=stable%2Fbrahmaputra
To invoke the procedure, enter the following shell commands, optionally specifying the branch identifier to use for
OpenStack.
cd ~
wget https://git.opnfv.org/cgit/copper/plain/components/congress/install/bash/install_congress_1.sh
wget https://git.opnfv.org/cgit/copper/plain/components/congress/install/bash/install_congress_2.sh
bash install_congress_1.sh [openstack-branch]
408
Chapter 3. Developer
OPNFV Documentation, Release Danube
OPNFV Copper Configuration Guide
Copper Configuration
This release includes installer support for the OpenStack Congress service under JOID and Apex installers. Congress
is installed by default for all JOID and Apex scenarios. Support for other OPNFV installer deployed environments is
planned for the next release.
• Pre-configuration Activities
• Hardware Configuration
Pre-configuration Activities None required.
Hardware Configuration There is no specific hardware configuration required for the Copper project.
Copper Post Installation Procedures
This section describes optional procedures for verifying that the Congress service is operational.
Copper Functional Tests This release includes the following test cases which are integrated into OPNFV Functest
for the JOID and Apex installers:
• DMZ Placement: dmz.sh
• SMTP Ingress: smtp_ingress.sh
• Reserved Subnet: reserved_subnet.sh
These scripts, related scripts that clean up the OpenStack environment afterward, and a combined test runner (run.sh)
are in the Copper repo under the “tests” folder. Instructions for using the tests are provided as script comments.
Further description of the tests is provided on the Copper wiki at https://wiki.opnfv.org/display/copper/testing.
Congress Test Webapp This release also provides a webapp that can be automatically installed in a Docker container
on the OPNFV jumphost. This script is in the Copper repo at:
• components/congress/test-webapp/setup/install_congress_testserver.sh
Prerequisites for using this script:
• OPFNV installed per JOID or Apex installer
• For Apex installs, on the jumphost, ssh to the undercloud VM and “su stack”
To invoke the procedure, enter the following shell commands, optionally specifying the branch identifier to use for
Copper:
wget https://git.opnfv.org/cgit/copper/plain/components/congress/test-webapp/setup/install_congress_t
bash install_congress_testserver.sh [copper-branch]
Using the Test Webapp Browse to the webapp IP address provided at the end of the install procedure.
Interactive options are meant to be self-explanatory given a basic familiarity with the Congress service and data model.
3.2. OPNFV Projects
409
OPNFV Documentation, Release Danube
Removing the Test Webapp The webapp can be removed by running this script from the Copper repo:
• components/congress/test-webapp/setup/clean_congress_testserver.sh
OPNFV Copper User Guide
OPNFV Copper User Guide
This release focuses on use of the OpenStack Congress service for managing configuration policy. See the Congress
intro guide for general information on the capabilities and usage of Congress.
Examples of Congress API usage can be found in the Copper tests as described on the OPNFV wiki at
https://wiki.opnfv.org/display/copper/testing.
OPNFV Copper Danube Design
Definitions
Table 3.3: Definitions
Term Meaning
State Information that can be used to convey or imply the state of something, e.g. an application, resource,
entity, etc. This can include data held inside OPNFV components, “events” that have occurred (e.g. “policy
violation”), etc.
Event An item of significance to the policy engine, for which the engine has become aware through some method
of discovery e.g. polling or notification.
Abbreviations
Table 3.4: Abbreviations
Term
CRUD
FCAPS
NF
SFC
VNF
NFVI
Meaning
Create, Read, Update, Delete (database operation types)
Fault, Configuration, Accounting, Performance, Security
Network Function
Service Function Chaining
Virtual Network Function
Network Function Virtualization Infrastructure
Architecture
Architectural Concept The following example diagram illustrates a “relationship diagram” type view of an NFVI
platform, in which the roles of components focused on policy management, services, and infrastructure are shown.
This view illustrates that a large-scale deployment of NFVI may leverage multiple components of the same “type” (e.g.
SDN Controller), which fulfill specific purposes for which they are optimized. For example, a global SDN controller
and cloud orchestrator can act as directed by a service orchestrator in the provisioning of VNFs per intent, while
various components at a local and global level handle policy-related events directly and/or feed them back through a
closed-loop policy design that responds as needed, directly or through the service orchestrator.
410
Chapter 3. Developer
OPNFV Documentation, Release Danube
(source of the diagram above: https://git.opnfv.org/cgit/copper/plain/design_docs/images/policy_architecture.pptx)
Architectural Aspects
• Policies are reflected in two high-level goals
– Ensure resource requirements of VNFs and services are applied per VNF designer, service, and tenant
intent
– Ensure that generic policies are not violated, e.g. networks connected to VMs must either be public or
owned by the VM owner
• Policies are distributed through two main means
– As part of VNF packages, customized if needed by Service Design tools, expressing intent of the VNF
designer and service provider, and possibly customized or supplemented by service orchestrators per the
intent of specific tenants
– As generic policies provisioned into VIMs (SDN controllers and cloud orchestrators), expressing intent of
the service provider re what states/events need to be policy-governed independently of specific VNFs
• Policies are applied locally and in closed-loop systems per the capabilities of the local policy enforcer and the
impact of the related state/event conditions
– VIMs should be able to execute most policies locally
– VIMs may need to pass policy-related state/events to a closed-loop system, where those events are relevant
to other components in the architecture (e.g. service orchestrator), or some additional data/arbitration is
needed to resolve the state/event condition
3.2. OPNFV Projects
411
OPNFV Documentation, Release Danube
• Policies are localized as they are distributed/delegated
– High-level policies (e.g. expressing “intent”) can be translated into VNF package elements or generic
policies, perhaps using distinct syntaxes
– Delegated policy syntaxes are likely VIM-specific, e.g. Datalog (Congress)
• Closed-loop policy and VNF-lifecycle event handling are //somewhat// distinct
– Closed-loop policy is mostly about resolving conditions that can’t be handled locally, but as above in some
cases the conditions may be of relevance and either delivered directly or forwarded to service orchestrators
– VNF-lifecycle events that can’t be handled by the VIM locally are delivered directly to the service orchestrator
• Some events/analytics need to be collected into a more “open-loop” system which can enable other actions, e.g.
– audits and manual interventions
– machine-learning focused optimizations of policies (largely a future objective)
Issues to be investigated as part of establishing an overall cohesive/adaptive policy architecture:
• For the various components which may fulfill a specific purpose, what capabilities (e.g. APIs) do they have/need
to
– handle events locally
– enable closed-loop policy handling components to subscribe/optimize policy-related events that are of
interest
• For global controllers and cloud orchestrators
– How do they support correlation of events impacting resources in different scopes (network and cloud)
– What event/response flows apply to various policy use cases
• What specific policy use cases can/should fall into each overall class
– locally handled by NFVI components
– handled by a closed-loop policy system, either VNF/service-specific or VNF-independent
Doctor
Doctor Development Guide
Testing Doctor
You have two options to test Doctor functions with the script developed for doctor CI.
You need to install OpenStack and other OPNFV components except Doctor Sample Inspector, Sample Monitor and
Sample Consumer, as these will be launched in this script. You are encouraged to use OPNFV offcial installers, but
you can also deploy all components with other installers such as devstack or manual operation. In those cases, the
versions of all components shall be matched with the versions of them in OPNFV specific release.
Run Test Script Doctor project has own testing script under doctor/tests. This test script can be used for functional
testing agained an OPNFV deployment.
Before running this script, make sure OpenStack env parameters are set properly following OpenStack CLI manual,
so that Doctor Inspector can operate OpenStack services.
412
Chapter 3. Developer
OPNFV Documentation, Release Danube
Then, you can run the script as follows:
git clone https://gerrit.opnfv.org/gerrit/doctor
cd doctor/tests
export INSTALLER_TYPE=local
export INSPECTOR_TYPE=sample
./run.sh
INSTALLER_TYPE can be ‘apex’, ‘fuel’, ‘joid’ and ‘local’(default). If you are not using OPNFV installers in this
option, chose ‘local’. INSPECTOR_TYPE can be specified either ‘sample’(default) or ‘congress’.
For testing with stable version, checkout stable branch of doctor repo before ‘./run.sh’.
Run Functest Suite Functest supports Doctor testing by triggering the test script above in a Functest container. You
can run the Doctor test with the following steps:
DOCKER_TAG=latest
docker pull opnfv/functest:${DOCKER_TAG}
docker run --privileged=true -id \
-e INSTALLER_TYPE=${INSTALLER_TYPE} \
-e INSTALLER_IP=${INSTALLER_IP} \
-e INSPECTOR_TYPE=sample \
opnfv/functest:${DOCKER_TAG} /bin/bash
docker exec <container_id> python /home/opnfv/repos/functest/functest/ci/prepare_env.py start
docker exec <container_id> functest testcase run doctor
See Functest Userguide for more information.
For testing with stable version, change DOCKER_TAG to ‘stable’ or other release tag identifier.
Tips
Doctor: Fault Management and Maintenance
Project Doctor, https://wiki.opnfv.org/doctor
Editors Ashiq Khan (NTT DOCOMO), Gerald Kunzmann (NTT DOCOMO)
Authors Ryota Mibu (NEC), Carlos Goncalves (NEC), Tomi Juvonen (Nokia), Tommy Lindgren (Ericsson), Bertrand Souville (NTT DOCOMO), Balazs Gibizer (Ericsson), Ildiko Vancsa (Ericsson) and
others.
Abstract Doctor is an OPNFV requirement project [DOCT]. Its scope is NFVI fault management, and
maintenance and it aims at developing and realizing the consequent implementation for the OPNFV
reference platform.
This deliverable is introducing the use cases and operational scenarios for Fault Management considered in the Doctor project. From the general features, a high level architecture describing logical
building blocks and interfaces is derived. Finally, a detailed implementation is introduced, based
on available open source components, and a related gap analysis is done as part of this project.
The implementation plan finally discusses an initial realization for a NFVI fault management and
maintenance solution in open source software.
3.2. OPNFV Projects
413
OPNFV Documentation, Release Danube
Definition of terms
Different SDOs and communities use different terminology related to NFV/Cloud/SDN. This list tries to define an
OPNFV terminology, mapping/translating the OPNFV terms to terminology used in other contexts.
ACT-STBY configuration Failover configuration common in Telco deployments. It enables the operator to use a
standby (STBY) instance to take over the functionality of a failed active (ACT) instance.
Administrator Administrator of the system, e.g. OAM in Telco context.
Consumer User-side Manager; consumer of the interfaces produced by the VIM; VNFM, NFVO, or Orchestrator in
ETSI NFV [ENFV] terminology.
EPC Evolved Packet Core, the main component of the core network architecture of 3GPP’s LTE communication
standard.
MME Mobility Management Entity, an entity in the EPC dedicated to mobility management.
NFV Network Function Virtualization
NFVI Network Function Virtualization Infrastructure; totality of all hardware and software components which build
up the environment in which VNFs are deployed.
S/P-GW Serving/PDN-Gateway, two entities in the EPC dedicated to routing user data packets and providing connectivity from the UE to external packet data networks (PDN), respectively.
Physical resource Actual resources in NFVI; not visible to Consumer.
VNFM Virtualized Network Function Manager; functional block that is responsible for the lifecycle management of
VNF.
NFVO Network Functions Virtualization Orchestrator; functional block that manages the Network Service (NS)
lifecycle and coordinates the management of NS lifecycle, VNF lifecycle (supported by the VNFM) and NFVI
resources (supported by the VIM) to ensure an optimized allocation of the necessary resources and connectivity.
VIM Virtualized Infrastructure Manager; functional block that is responsible for controlling and managing the NFVI
compute, storage and network resources, usually within one operator’s Infrastructure Domain, e.g. NFVI Point
of Presence (NFVI-PoP).
Virtual Machine (VM)
puter/server.
Virtualized computation environment that behaves very much like a physical com-
Virtual network Virtual network routes information among the network interfaces of VM instances and physical
network interfaces, providing the necessary connectivity.
Virtual resource A Virtual Machine (VM), a virtual network, or virtualized storage; Offered resources to “Consumer” as result of infrastructure virtualization; visible to Consumer.
Virtual Storage Virtualized non-volatile storage allocated to a VM.
VNF Virtualized Network Function. Implementation of a Network Function that can be deployed on a Network
Function Virtualization Infrastructure (NFVI).
Introduction
The goal of this project is to build an NFVI fault management and maintenance framework supporting high availability of the Network Services on top of the virtualized infrastructure. The key feature is immediate notification of
unavailability of virtualized resources from VIM, to support failure recovery, or failure avoidance of VNFs running on
them. Requirement survey and development of missing features in NFVI and VIM are in scope of this project in order
to fulfil requirements for fault management and maintenance in NFV.
414
Chapter 3. Developer
OPNFV Documentation, Release Danube
The purpose of this requirement project is to clarify the necessary features of NFVI fault management, and maintenance, identify missing features in the current OpenSource implementations, provide a potential implementation architecture and plan, provide implementation guidelines in relevant upstream projects to realize those missing features,
and define the VIM northbound interfaces necessary to perform the task of NFVI fault management, and maintenance
in alignment with ETSI NFV [ENFV].
Problem description A Virtualized Infrastructure Manager (VIM), e.g. OpenStack [OPSK], cannot detect certain
Network Functions Virtualization Infrastructure (NFVI) faults. This feature is necessary to detect the faults and notify
the Consumer in order to ensure the proper functioning of EPC VNFs like MME and S/P-GW.
• EPC VNFs are often in active standby (ACT-STBY) configuration and need to switch from STBY mode to ACT
mode as soon as relevant faults are detected in the active (ACT) VNF.
• NFVI encompasses all elements building up the environment in which VNFs are deployed, e.g., Physical Machines, Hypervisors, Storage, and Network elements.
In addition, VIM, e.g. OpenStack, needs to receive maintenance instructions from the Consumer, i.e. the operator/administrator of the VNF.
• Change the state of certain Physical Machines (PMs), e.g. empty the PM, so that maintenance work can be
performed at these machines.
Note: Although fault management and maintenance are different operations in NFV, both are considered as part of this
project as – except for the trigger – they share a very similar work and message flow. Hence, from implementation
perspective, these two are kept together in the Doctor project because of this high degree of similarity.
Use cases and scenarios
Telecom services often have very high requirements on service performance. As a consequence they often utilize
redundancy and high availability (HA) mechanisms for both the service and the platform. The HA support may be
built-in or provided by the platform. In any case, the HA support typically has a very fast detection and reaction
time to minimize service impact. The main changes proposed in this document are about making a clear distinction
between fault management and recovery a) within the VIM/NFVI and b) High Availability support for VNFs on the
other, claiming that HA support within a VNF or as a service from the platform is outside the scope of Doctor and is
discussed in the High Availability for OPNFV project. Doctor should focus on detecting and remediating faults in the
NFVI. This will ensure that applications come back to a fully redundant configuration faster than before.
As an example, Telecom services can come with an Active-Standby (ACT-STBY) configuration which is a (1+1)
redundancy scheme. ACT and STBY nodes (aka Physical Network Function (PNF) in ETSI NFV terminology) are in
a hot standby configuration. If an ACT node is unable to function properly due to fault or any other reason, the STBY
node is instantly made ACT, and affected services can be provided without any service interruption.
The ACT-STBY configuration needs to be maintained. This means, when a STBY node is made ACT, either the
previously ACT node, after recovery, shall be made STBY, or, a new STBY node needs to be configured. The actual
operations to instantiate/configure a new STBY are similar to instantiating a new VNF and therefore are outside the
scope of this project.
The NFVI fault management and maintenance requirements aim at providing fast failure detection of physical and
virtualized resources and remediation of the virtualized resources provided to Consumers according to their predefined
request to enable applications to recover to a fully redundant mode of operation.
1. Fault management/recovery using ACT-STBY configuration (Triggered by critical error)
2. Preventive actions based on fault prediction (Preventing service stop by handling warnings)
3. VM Retirement (Managing service during NFVI maintenance, i.e. H/W, Hypervisor, Host OS, maintenance)
3.2. OPNFV Projects
415
OPNFV Documentation, Release Danube
Faults
Fault management using ACT-STBY configuration In figure1, a system-wide view of relevant functional
blocks is presented. OpenStack is considered as the VIM implementation (aka Controller) which has interfaces with
the NFVI and the Consumers. The VNF implementation is represented as different virtual resources marked by different colors. Consumers (VNFM or NFVO in ETSI NFV terminology) own/manage the respective virtual resources
(VMs in this example) shown with the same colors.
The first requirement in this use case is that the Controller needs to detect faults in the NFVI (“1. Fault Notification”
in figure1) affecting the proper functioning of the virtual resources (labelled as VM-x) running on top of it. It
should be possible to configure which relevant fault items should be detected. The VIM (e.g. OpenStack) itself could
be extended to detect such faults. Alternatively, a third party fault monitoring tool could be used which then informs
the VIM about such faults; this third party fault monitoring element can be considered as a component of VIM from
an architectural point of view.
Once such fault is detected, the VIM shall find out which virtual resources are affected by this fault. In the example
in figure1, VM-4 is affected by a fault in the Hardware Server-3. Such mapping shall be maintained in the VIM,
depicted as the “Server-VM info” table inside the VIM.
Once the VIM has identified which virtual resources are affected by the fault, it needs to find out who is the Consumer
(i.e. the owner/manager) of the affected virtual resources (Step 2). In the example shown in figure1, the VIM
knows that for the red VM-4, the manager is the red Consumer through an Ownership info table. The VIM then
notifies (Step 3 “Fault Notification”) the red Consumer about this fault, preferably with sufficient abstraction rather
than detailed physical fault information.
Fig. 3.7: Fault management/recovery use case
The Consumer then switches to STBY configuration by switching the STBY node to ACT state (Step 4). It further
initiates a process to instantiate/configure a new STBY. However, switching to STBY mode and creating a new STBY
machine is a VNFM/NFVO level operation and therefore outside the scope of this project. Doctor project does not
416
Chapter 3. Developer
OPNFV Documentation, Release Danube
create interfaces for such VNFM level configuration operations. Yet, since the total failover time of a consumer service
depends on both the delay of such processes as well as the reaction time of Doctor components, minimizing Doctor’s
reaction time is a necessary basic ingredient to fast failover times in general.
Once the Consumer has switched to STBY configuration, it notifies (Step 5 “Instruction” in figure1) the VIM. The
VIM can then take necessary (e.g. pre-determined by the involved network operator) actions on how to clean up the
fault affected VMs (Step 6 “Execute Instruction”).
The key issue in this use case is that a VIM (OpenStack in this context) shall not take a standalone fault recovery
action (e.g. migration of the affected VMs) before the ACT-STBY switching is complete, as that might violate the
ACT-STBY configuration and render the node out of service.
As an extension of the 1+1 ACT-STBY resilience pattern, a STBY instance can act as backup to N ACT nodes (N+1).
In this case, the basic information flow remains the same, i.e., the consumer is informed of a failure in order to activate
the STBY node. However, in this case it might be useful for the failure notification to cover a number of failed
instances due to the same fault (e.g., more than one instance might be affected by a switch failure). The reaction of the
consumer might depend on whether only one active instance has failed (similar to the ACT-STBY case), or if more
active instances are needed as well.
Preventive actions based on fault prediction The fault management scenario explained in Fault management using
ACT-STBY configuration can also be performed based on fault prediction. In such cases, in VIM, there is an intelligent fault prediction module which, based on its NFVI monitoring information, can predict an imminent fault in the
elements of NFVI. A simple example is raising temperature of a Hardware Server which might trigger a pre-emptive
recovery action. The requirements of such fault prediction in the VIM are investigated in the OPNFV project “Data
Collection for Failure Prediction” [PRED].
This use case is very similar to Fault management using ACT-STBY configuration. Instead of a fault detection (Step 1
“Fault Notification in” figure1), the trigger comes from a fault prediction module in the VIM, or from a third party
module which notifies the VIM about an imminent fault. From Step 2~5, the work flow is the same as in the “Fault
management using ACT-STBY configuration” use case, except in this case, the Consumer of a VM/VNF switches to
STBY configuration based on a predicted fault, rather than an occurred fault.
NFVI Maintenance
VM Retirement All network operators perform maintenance of their network infrastructure, both regularly and
irregularly. Besides the hardware, virtualization is expected to increase the number of elements subject to such maintenance as NFVI holds new elements like the hypervisor and host OS. Maintenance of a particular resource element
e.g. hardware, hypervisor etc. may render a particular server hardware unusable until the maintenance procedure is
complete.
However, the Consumer of VMs needs to know that such resources will be unavailable because of NFVI maintenance.
The following use case is again to ensure that the ACT-STBY configuration is not violated. A stand-alone action (e.g.
live migration) from VIM/OpenStack to empty a physical machine so that consequent maintenance procedure could be
performed may not only violate the ACT-STBY configuration, but also have impact on real-time processing scenarios
where dedicated resources to virtual resources (e.g. VMs) are necessary and a pause in operation (e.g. vCPU) is not
allowed. The Consumer is in a position to safely perform the switch between ACT and STBY nodes, or switch to an
alternative VNF forwarding graph so the hardware servers hosting the ACT nodes can be emptied for the upcoming
maintenance operation. Once the target hardware servers are emptied (i.e. no virtual resources are running on top),
the VIM can mark them with an appropriate flag (i.e. “maintenance” state) such that these servers are not considered
for hosting of virtual machines until the maintenance flag is cleared (i.e. nodes are back in “normal” status).
A high-level view of the maintenance procedure is presented in figure2. VIM/OpenStack, through its northbound
interface, receives a maintenance notification (Step 1 “Maintenance Request”) from the Administrator (e.g. a network
operator) including information about which hardware is subject to maintenance. Maintenance operations include
replacement/upgrade of hardware, update/upgrade of the hypervisor/host OS, etc.
3.2. OPNFV Projects
417
OPNFV Documentation, Release Danube
The consequent steps to enable the Consumer to perform ACT-STBY switching are very similar to the fault management scenario. From VIM/OpenStack’s internal database, it finds out which virtual resources (VM-x) are running
on those particular Hardware Servers and who are the managers of those virtual resources (Step 2). The VIM then
informs the respective Consumer (VNFMs or NFVO) in Step 3 “Maintenance Notification”. Based on this, the Consumer takes necessary actions (Step 4, e.g. switch to STBY configuration or switch VNF forwarding graphs) and then
notifies (Step 5 “Instruction”) the VIM. Upon receiving such notification, the VIM takes necessary actions (Step 6
“Execute Instruction” to empty the Hardware Servers so that consequent maintenance operations could be performed.
Due to the similarity for Steps 2~6, the maintenance procedure and the fault management procedure are investigated
in the same project.
Fig. 3.8: Maintenance use case
High level architecture and general features
Functional overview The Doctor project circles around two distinct use cases: 1) management of failures of virtualized resources and 2) planned maintenance, e.g. migration, of virtualized resources. Both of them may affect
a VNF/application and the network service it provides, but there is a difference in frequency and how they can be
handled.
Failures are spontaneous events that may or may not have an impact on the virtual resources. The Consumer should
as soon as possible react to the failure, e.g., by switching to the STBY node. The Consumer will then instruct the
VIM on how to clean up or repair the lost virtual resources, i.e. restore the VM, VLAN or virtualized storage. How
much the applications are affected varies. Applications with built-in HA support might experience a short decrease in
retainability (e.g. an ongoing session might be lost) while keeping availability (establishment or re-establishment of
sessions are not affected), whereas the impact on applications without built-in HA may be more serious. How much
the network service is impacted depends on how the service is implemented. With sufficient network redundancy the
service may be unaffected even when a specific resource fails.
418
Chapter 3. Developer
OPNFV Documentation, Release Danube
On the other hand, planned maintenance impacting virtualized resources are events that are known in advance. This
group includes e.g. migration due to software upgrades of OS and hypervisor on a compute host. Some of these
might have been requested by the application or its management solution, but there is also a need for coordination on
the actual operations on the virtual resources. There may be an impact on the applications and the service, but since
they are not spontaneous events there is room for planning and coordination between the application management
organization and the infrastructure management organization, including performing whatever actions that would be
required to minimize the problems.
Failure prediction is the process of pro-actively identifying situations that may lead to a failure in the future unless
acted on by means of maintenance activities. From applications’ point of view, failure prediction may impact them in
two ways: either the warning time is so short that the application or its management solution does not have time to
react, in which case it is equal to the failure scenario, or there is sufficient time to avoid the consequences by means of
maintenance activities, in which case it is similar to planned maintenance.
Architecture Overview NFV and the Cloud platform provide virtual resources and related control functionality
to users and administrators. figure3 shows the high level architecture of NFV focusing on the NFVI, i.e., the
virtualized infrastructure. The NFVI provides virtual resources, such as virtual machines (VM) and virtual networks.
Those virtual resources are used to run applications, i.e. VNFs, which could be components of a network service
which is managed by the consumer of the NFVI. The VIM provides functionalities of controlling and viewing virtual
resources on hardware (physical) resources to the consumers, i.e., users and administrators. OpenStack is a prominent
candidate for this VIM. The administrator may also directly control the NFVI without using the VIM.
Although OpenStack is the target upstream project where the new functional elements (Controller, Notifier, Monitor,
and Inspector) are expected to be implemented, a particular implementation method is not assumed. Some of these
elements may sit outside of OpenStack and offer a northbound interface to OpenStack.
General Features and Requirements The following features are required for the VIM to achieve high availability
of applications (e.g., MME, S/P-GW) and the Network Services:
1. Monitoring: Monitor physical and virtual resources.
2. Detection: Detect unavailability of physical resources.
3. Correlation and Cognition: Correlate faults and identify affected virtual resources.
4. Notification: Notify unavailable virtual resources to their Consumer(s).
5. Fencing: Shut down or isolate a faulty resource.
6. Recovery action: Execute actions to process fault recovery and maintenance.
The time interval between the instant that an event is detected by the monitoring system and the Consumer notification
of unavailable resources shall be < 1 second (e.g., Step 1 to Step 4 in figure4).
Monitoring The VIM shall monitor physical and virtual resources for unavailability and suspicious behavior.
Detection The VIM shall detect unavailability and failures of physical resources that might cause errors/faults in
virtual resources running on top of them. Unavailability of physical resource is detected by various monitoring and
managing tools for hardware and software components. This may include also predicting upcoming faults. Note, fault
prediction is out of scope of this project and is investigated in the OPNFV “Data Collection for Failure Prediction”
project [PRED].
The fault items/events to be detected shall be configurable.
The configuration shall enable Failure Selection and Aggregation. Failure aggregation means the VIM determines
unavailability of physical resource from more than two non-critical failures related to the same resource.
3.2. OPNFV Projects
419
OPNFV Documentation, Release Danube
Fig. 3.9: High level architecture
420
Chapter 3. Developer
OPNFV Documentation, Release Danube
There are two types of unavailability - immediate and future:
• Immediate unavailability can be detected by setting traps of raw failures on hardware monitoring tools.
• Future unavailability can be found by receiving maintenance instructions issued by the administrator of the
NFVI or by failure prediction mechanisms.
Correlation and Cognition The VIM shall correlate each fault to the impacted virtual resource, i.e., the VIM shall
identify unavailability of virtualized resources that are or will be affected by failures on the physical resources under
them. Unavailability of a virtualized resource is determined by referring to the mapping of physical and virtualized
resources.
VIM shall allow configuration of fault correlation between physical and virtual resources. VIM shall support correlating faults:
• between a physical resource and another physical resource
• between a physical resource and a virtual resource
• between a virtual resource and another virtual resource
Failure aggregation is also required in this feature, e.g., a user may request to be only notified if failures on more than
two standby VMs in an (N+M) deployment model occurred.
Notification The VIM shall notify the alarm, i.e., unavailability of virtual resource(s), to the Consumer owning it
over the northbound interface, such that the Consumers impacted by the failure can take appropriate actions to recover
from the failure.
The VIM shall also notify the unavailability of physical resources to its Administrator.
All notifications shall be transferred immediately in order to minimize the stalling time of the network service and to
avoid over assignment caused by delay of capability updates.
There may be multiple consumers, so the VIM has to find out the owner of a faulty resource. Moreover, there may
be a large number of virtual and physical resources in a real deployment, so polling the state of all resources to the
VIM would lead to heavy signaling traffic. Thus, a publication/subscription messaging model is better suited for these
notifications, as notifications are only sent to subscribed consumers.
Notifications will be send out along with the configuration by the consumer. The configuration includes endpoint(s) in
which the consumers can specify multiple targets for the notification subscription, so that various and multiple receiver
functions can consume the notification message. Also, the conditions for notifications shall be configurable, such that
the consumer can set according policies, e.g. whether it wants to receive fault notifications or not.
Note: the VIM should only accept notification subscriptions for each resource by its owner or administrator. Notifications to the Consumer about the unavailability of virtualized resources will include a description of the fault, preferably
with sufficient abstraction rather than detailed physical fault information.
Fencing Recovery actions, e.g. safe VM evacuation, have to be preceded by fencing the failed host. Fencing hereby
means to isolate or shut down a faulty resource. Without fencing – when the perceived disconnection is due to some
transient or partial failure – the evacuation might lead into two identical instances running together and having a
dangerous conflict.
There is a cross-project definition in OpenStack of how to implement fencing, but there has not been any progress. The
general description is available here: https://wiki.openstack.org/wiki/Fencing_Instances_of_an_Unreachable_Host
OpenStack provides some mechanisms that allow fencing of faulty resources. Some are automatically invoked by
the platform itself (e.g. Nova disables the compute service when libvirtd stops running, preventing new VMs to be
scheduled to that node), while other mechanisms are consumer trigger-based actions (e.g. Neutron port admin-stateup). For other fencing actions not supported by OpenStack, the Doctor project may suggest ways to address the gap
3.2. OPNFV Projects
421
OPNFV Documentation, Release Danube
(e.g. through means of resourcing to external tools and orchestration methods), or documenting or implementing them
upstream.
The Doctor Inspector component will be responsible of marking resources down in the OpenStack and back up if
necessary.
Recovery Action In the basic Fault management using ACT-STBY configuration use case, no automatic actions will
be taken by the VIM, but all recovery actions executed by the VIM and the NFVI will be instructed and coordinated
by the Consumer.
In a more advanced use case, the VIM may be able to recover the failed virtual resources according to a pre-defined
behavior for that resource. In principle this means that the owner of the resource (i.e., its consumer or administrator)
can define which recovery actions shall be taken by the VIM. Examples are a restart of the VM or migration/evacuation
of the VM.
High level northbound interface specification
Fault Management This interface allows the Consumer to subscribe to fault notification from the VIM. Using a
filter, the Consumer can narrow down which faults should be notified. A fault notification may trigger the Consumer
to switch from ACT to STBY configuration and initiate fault recovery actions. A fault query request/response message
exchange allows the Consumer to find out about active alarms at the VIM. A filter can be used to narrow down the
alarms returned in the response message.
The high level message flow for the fault management use case is shown in figure4. It consists of the following
steps:
1. The VIM monitors the physical and virtual resources and the fault management workflow is triggered by a
monitored fault event.
2. Event correlation, fault detection and aggregation in VIM. Note: this may also happen after Step 3.
3. Database lookup to find the virtual resources affected by the detected fault.
4. Fault notification to Consumer.
5. The Consumer switches to standby configuration (STBY).
6. Instructions to VIM requesting certain actions to be performed on the affected resources, for example migrate/update/terminate specific resource(s). After reception of such instructions, the VIM is executing the requested action, e.g., it will migrate or terminate a virtual resource.
NFVI Maintenance The NFVI maintenance interface allows the Administrator to notify the VIM about a planned
maintenance operation on the NFVI. A maintenance operation may for example be an update of the server firmware
or the hypervisor. The MaintenanceRequest message contains instructions to change the state of the physical resource
from ‘enabled’ to ‘going-to-maintenance’ and a timeout 1 . After receiving the MaintenanceRequest,the VIM decides
on the actions to be taken based on maintenance policies predefined by the affected Consumer(s).
The high level message flow for the NFVI maintenance policy enforcement is shown in figure5a. It consists of the
following steps:
1. Maintenance trigger received from Administrator.
2. VIM switches the affected physical resources to “going-to-maintenance” state e.g. so that no new VM will be
scheduled on the physical servers.
3. Database lookup to find the Consumer(s) and virtual resources affected by the maintenance operation.
1
Timeout is set by the Administrator and corresponds to the maximum time to empty the physical resources.
422
Chapter 3. Developer
OPNFV Documentation, Release Danube
Fig. 3.10: High-level message flow for fault management
3.2. OPNFV Projects
423
OPNFV Documentation, Release Danube
Fig. 3.11: High-level message flow for maintenance policy enforcement
424
Chapter 3. Developer
OPNFV Documentation, Release Danube
4. Maintenance policies are enforced in the VIM, e.g. affected VM(s) are shut down on the physical server(s), or
affected Consumer(s) are notified about the planned maintenance operation (steps 4a/4b).
Once the affected Consumer(s) have been notified, they take specific actions (e.g. switch to standby (STBY) configuration, request to terminate the virtual resource(s)) to allow the maintenance action to be executed. After the
physical resources have been emptied, the VIM puts the physical resources in “in-maintenance” state and sends a
MaintenanceResponse back to the Administrator.
Fig. 3.12: Successful NFVI maintenance
The high level message flow for a successful NFVI maintenance is show in figure5b. It consists of the following
steps:
5. The Consumer C3 switches to standby configuration (STBY).
6. Instructions from Consumers C2/C3 are shared to VIM requesting certain actions to be performed (steps 6a,
6b). After receiving such instructions, the VIM executes the requested action in order to empty the physical
resources (step 6c) and informs the Consumer about the result of the actions (steps 6d, 6e).
7. The VIM switches the physical resources to “in-maintenance” state
8. Maintenance response is sent from VIM to inform the Administrator that the physical servers have been emptied.
9. The Administrator is coordinating and executing the maintenance operation/work on the NFVI. Note: this step
is out of scope of Doctor project.
3.2. OPNFV Projects
425
OPNFV Documentation, Release Danube
The requested actions to empty the physical resources may not be successful (e.g. migration fails or takes too long)
and in such a case, the VIM puts the physical resources back to ‘enabled’ and informs the Administrator about the
problem.
Fig. 3.13: Example of failed NFVI maintenance
An example of a high level message flow to cover the failed NFVI maintenance case is shown in figure5c. It
consists of the following steps:
5. The Consumer C3 switches to standby configuration (STBY).
6. Instructions from Consumers C2/C3 are shared to VIM requesting certain actions to be performed (steps 6a, 6b).
The VIM executes the requested actions and sends back a NACK to consumer C2 (step 6d) as the migration of
the virtual resource(s) is not completed by the given timeout.
7. The VIM switches the physical resources to “enabled” state.
8. MaintenanceNotification is sent from VIM to inform the Administrator that the maintenance action cannot start.
Gap analysis in upstream projects
This section presents the findings of gaps on existing VIM platforms. The focus was to identify gaps based on the
features and requirements specified in Section 3.3. The analysis work determined gaps that are presented here.
426
Chapter 3. Developer
OPNFV Documentation, Release Danube
VIM Northbound Interface
Immediate Notification
• Type: ‘deficiency in performance’
• Description
– To-be
* VIM has to notify unavailability of virtual resource (fault) to VIM user immediately.
* Notification should be passed in ‘1 second’ after fault detected/notified by VIM.
* Also, the following conditions/requirement have to be met:
· Only the owning user can receive notification of fault related to owned virtual resource(s).
– As-is
* OpenStack Metering ‘Ceilometer’ can notify unavailability of virtual resource (fault) to the owner of
virtual resource based on alarm configuration by the user.
· Ceilometer Alarm API: http://docs.openstack.org/developer/ceilometer/webapi/v2.html#alarms
* Alarm notifications are triggered by alarm evaluator instead of notification agents that might receive
faults
· Ceilometer Architecture: http://docs.openstack.org/developer/ceilometer/architecture.html#id1
* Evaluation interval should be equal to or larger than configured pipeline interval for collection of
underlying metrics.
· https://github.com/openstack/ceilometer/blob/stable/juno/ceilometer/alarm/service.py#L38-42
* The interval for collection has to be set large enough which depends on the size of the deployment
and the number of metrics to be collected.
* The interval may not be less than one second in even small deployments. The default value is 60
seconds.
* Alternative: OpenStack has a message bus to publish system events. The operator can allow the user
to connect this, but there are no functions to filter out other events that should not be passed to the
user or which were not requested by the user.
– Gap
* Fault notifications cannot be received immediately by Ceilometer.
• Solved by
– Event Alarm Evaluator:
alarm-evaluator.html
https://specs.openstack.org/openstack/ceilometer-specs/specs/liberty/event-
– New OpenStack alarms and notifications project AODH: http://docs.openstack.org/developer/aodh/
Maintenance Notification
• Type: ‘missing’
• Description
– To-be
* VIM has to notify unavailability of virtual resource triggered by NFVI maintenance to VIM user.
3.2. OPNFV Projects
427
OPNFV Documentation, Release Danube
* Also, the following conditions/requirements have to be met:
· VIM should accept maintenance message from administrator and mark target physical resource
“in maintenance”.
· Only the owner of virtual resource hosted by target physical resource can receive the notification
that can trigger some process for applications which are running on the virtual resource (e.g. cut
off VM).
– As-is
* OpenStack: None
* AWS (just for study)
· AWS provides API and CLI to view status of resource (VM) and to create instance
status and system status alarms to notify you when an instance has a failed status check. http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/monitoring-instances-statuscheck_sched.html
· AWS provides API and CLI to view scheduled events, such as a reboot or retirement, for your instances.
Also, those events will be notified via e-mail.
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/monitoring-system-instance-statuscheck.html
– Gap
* VIM user cannot receive maintenance notifications.
• Solved by
– https://blueprints.launchpad.net/nova/+spec/service-status-notification
VIM Southbound interface
Normalization of data collection models
• Type: ‘missing’
• Description
– To-be
* A normalized data format needs to be created to cope with the many data models from different
monitoring solutions.
– As-is
* Data can be collected from many places (e.g. Zabbix, Nagios, Cacti, Zenoss). Although each solution
establishes its own data models, no common data abstraction models exist in OpenStack.
– Gap
* Normalized data format does not exist.
• Solved by
– Specification in Section Detailed southbound interface specification.
OpenStack
428
Chapter 3. Developer
OPNFV Documentation, Release Danube
Ceilometer OpenStack offers a telemetry service, Ceilometer, for collecting measurements of the utilization of physical and virtual resources [CEIL]. Ceilometer can collect a number of metrics across multiple OpenStack components
and watch for variations and trigger alarms based upon the collected data.
Scalability of fault aggregation
• Type: ‘scalability issue’
• Description
– To-be
* Be able to scale to a large deployment, where thousands of monitoring events per second need to be
analyzed.
– As-is
* Performance issue when scaling to medium-sized deployments.
– Gap
* Ceilometer seems to be unsuitable for monitoring medium and large scale NFVI deployments.
• Solved by
– Usage of Zabbix for fault aggregation [ZABB]. Zabbix can support a much higher number of
fault events (up to 15 thousand events per second, but obviously also has some upper bound:
http://blog.zabbix.com/scalable-zabbix-lessons-on-hitting-9400-nvps/2615/
– Decentralized/hierarchical deployment with multiple instances, where one instance is only responsible for
a small NFVI.
Monitoring of hardware and software
• Type: ‘missing (lack of functionality)’
• Description
– To-be
* OpenStack (as VIM) should monitor various hardware and software in NFVI to handle faults on them
by Ceilometer.
* OpenStack may have monitoring functionality in itself and can be integrated with third party monitoring tools.
* OpenStack need to be able to detect the faults listed in the Annex.
– As-is
* For each deployment of OpenStack, an operator has responsibility to configure monitoring tools with
relevant scripts or plugins in order to monitor hardware and software.
* OpenStack Ceilometer does not monitor hardware and software to capture faults.
– Gap
* Ceilometer is not able to detect and handle all faults listed in the Annex.
• Solved by
– Use of dedicated monitoring tools like Zabbix or Monasca. See Annex: NFVI Faults.
3.2. OPNFV Projects
429
OPNFV Documentation, Release Danube
Nova OpenStack Nova [NOVA] is a mature and widely known and used component in OpenStack cloud deployments. It is the main part of an “infrastructure-as-a-service” system providing a cloud computing fabric controller,
supporting a wide diversity of virtualization and container technologies.
Nova has proven throughout these past years to be highly available and fault-tolerant. Featuring its own API, it also
provides a compatibility API with Amazon EC2 APIs.
Correct states when compute host is down
• Type: ‘missing (lack of functionality)’
• Description
– To-be
* The API shall support to change VM power state in case host has failed.
* The API shall support to change nova-compute state.
* There could be single API to change different VM states for all VMs belonging to a specific host.
* Support external systems that are monitoring the infrastructure and resources that are able to call the
API fast and reliable.
* Resource states are reliable such that correlation actions can be fast and automated.
* User shall be able to read states from OpenStack and trust they are correct.
– As-is
* When a VM goes down due to a host HW, host OS or hypervisor failure, nothing happens in OpenStack. The VMs of a crashed host/hypervisor are reported to be live and OK through the OpenStack
API.
* nova-compute state might change too slowly or the state is not reliable if expecting also VMs to be
down. This leads to ability to schedule VMs to a failed host and slowness blocks evacuation.
– Gap
* OpenStack does not change its states fast and reliably enough.
* The API does not support to have an external system to change states and to trust the states are reliable
(external system has fenced failed host).
* User cannot read all the states from OpenStack nor trust they are right.
• Solved by
– https://blueprints.launchpad.net/nova/+spec/mark-host-down
– https://blueprints.launchpad.net/python-novaclient/+spec/support-force-down-service
Evacuate VMs in Maintenance mode
• Type: ‘missing’
• Description
– To-be
* When maintenance mode for a compute host is set, trigger VM evacuation to available compute nodes
before bringing the host down for maintenance.
– As-is
430
Chapter 3. Developer
OPNFV Documentation, Release Danube
* If setting a compute node to a maintenance mode, OpenStack only schedules evacuation of all VMs
to available compute nodes if in-maintenance compute node runs the XenAPI and VMware ESX
hypervisors. Other hypervisors (e.g. KVM) are not supported and, hence, guest VMs will likely stop
running due to maintenance actions administrator may perform (e.g. hardware upgrades, OS updates).
– Gap
* Nova libvirt hypervisor driver does not implement automatic guest VMs evacuation when compute nodes are set to maintenance mode ($ nova host-update --maintenance enable
<hostname>).
Monasca Monasca is an open-source monitoring-as-a-service (MONaaS) solution that integrates with OpenStack.
Even though it is still in its early days, it is the interest of the community that the platform be multi-tenant, highly
scalable, performant and fault-tolerant. It provides a streaming alarm engine, a notification engine, and a northbound
REST API users can use to interact with Monasca. Hundreds of thousands of metrics per second can be processed
[MONA].
Anomaly detection
• Type: ‘missing (lack of functionality)’
• Description
– To-be
* Detect the failure and perform a root cause analysis to filter out other alarms that may be triggered
due to their cascading relation.
– As-is
* A mechanism to detect root causes of failures is not available.
– Gap
* Certain failures can trigger many alarms due to their dependency on the underlying root cause of
failure. Knowing the root cause can help filter out unnecessary and overwhelming alarms.
• Status
– Monasca as of now lacks this feature, although the community is aware and working toward supporting it.
Sensor monitoring
• Type: ‘missing (lack of functionality)’
• Description
– To-be
* It should support monitoring sensor data retrieval, for instance, from IPMI.
– As-is
* Monasca does not monitor sensor data
– Gap
* Sensor monitoring is very important. It provides operators status on the state of the physical infrastructure (e.g. temperature, fans).
• Addressed by
3.2. OPNFV Projects
431
OPNFV Documentation, Release Danube
– Monasca can be configured to use third-party monitoring solutions (e.g. Nagios, Cacti) for retrieving
additional data.
Hardware monitoring tools
Zabbix Zabbix is an open-source solution for monitoring availability and performance of infrastructure components
(i.e. servers and network devices), as well as applications [ZABB]. It can be customized for use with OpenStack. It is
a mature tool and has been proven to be able to scale to large systems with 100,000s of devices.
Delay in execution of actions
• Type: ‘deficiency in performance’
• Description
– To-be
* After detecting a fault, the monitoring tool should immediately execute the appropriate action, e.g.
inform the manager through the NB I/F
– As-is
* A delay of around 10 seconds was measured in two independent testbed deployments
– Gap
* Cause of the delay is a periodic evaluation and notification.
configured as 30s default value and can be reduced to 5s
https://github.com/zabbix/zabbix/blob/trunk/conf/zabbix_server.conf#L329
but
Periodicity is
not below.
Detailed architecture and interface specification
This section describes a detailed implementation plan, which is based on the high level architecture introduced in
Section 3. Section 5.1 describes the functional blocks of the Doctor architecture, which is followed by a high level
message flow in Section 5.2. Section 5.3 provides a mapping of selected existing open source components to the
building blocks of the Doctor architecture. Thereby, the selection of components is based on their maturity and the
gap analysis executed in Section 4. Sections 5.4 and 5.5 detail the specification of the related northbound interface
and the related information elements. Finally, Section 5.6 provides a first set of blueprints to address selected gaps
required for the realization functionalities of the Doctor project.
Functional Blocks This section introduces the functional blocks to form the VIM. OpenStack was selected as the
candidate for implementation. Inside the VIM, 4 different building blocks are defined (see figure6).
Monitor The Monitor module has the responsibility for monitoring the virtualized infrastructure. There are already
many existing tools and services (e.g. Zabbix) to monitor different aspects of hardware and software resources which
can be used for this purpose.
Inspector The Inspector module has the ability a) to receive various failure notifications regarding physical resource(s) from Monitor module(s), b) to find the affected virtual resource(s) by querying the resource map in the
Controller, and c) to update the state of the virtual resource (and physical resource).
The Inspector has drivers for different types of events and resources to integrate any type of Monitor and Controller
modules. It also uses a failure policy database to decide on the failure selection and aggregation from raw events. This
failure policy database is configured by the Administrator.
432
Chapter 3. Developer
OPNFV Documentation, Release Danube
Fig. 3.14: Functional blocks
3.2. OPNFV Projects
433
OPNFV Documentation, Release Danube
The reason for separation of the Inspector and Controller modules is to make the Controller focus on simple operations
by avoiding a tight integration of various health check mechanisms into the Controller.
Controller The Controller is responsible for maintaining the resource map (i.e. the mapping from physical resources
to virtual resources), accepting update requests for the resource state(s) (exposing as provider API), and sending all
failure events regarding virtual resources to the Notifier. Optionally, the Controller has the ability to force the state
of a given physical resource to down in the resource mapping when it receives failure notifications from the Inspector
for that given physical resource. The Controller also re-calculates the capacity of the NVFI when receiving a failure
notification for a physical resource.
In a real-world deployment, the VIM may have several controllers, one for each resource type, such as Nova, Neutron
and Cinder in OpenStack. Each controller maintains a database of virtual and physical resources which shall be the
master source for resource information inside the VIM.
Notifier The focus of the Notifier is on selecting and aggregating failure events received from the controller based
on policies mandated by the Consumer. Therefore, it allows the Consumer to subscribe for alarms regarding virtual
resources using a method such as API endpoint. After receiving a fault event from a Controller, it will notify the fault
to the Consumer by referring to the alarm configuration which was defined by the Consumer earlier on.
To reduce complexity of the Controller, it is a good approach for the Controllers to emit all notifications without any
filtering mechanism and have another service (i.e. Notifier) handle those notifications properly. This is the general
philosophy of notifications in OpenStack. Note that a fault message consumed by the Notifier is different from the
fault message received by the Inspector; the former message is related to virtual resources which are visible to users
with relevant ownership, whereas the latter is related to raw devices or small entities which should be handled with an
administrator privilege.
The northbound interface between the Notifier and the Consumer/Administrator is specified in Detailed northbound
interface specification.
Sequence
Fault Management The detailed work flow for fault management is as follows (see also figure7):
1. Request to subscribe to monitor specific virtual resources. A query filter can be used to narrow down the alarms
the Consumer wants to be informed about.
2. Each subscription request is acknowledged with a subscribe response message. The response message contains
information about the subscribed virtual resources, in particular if a subscribed virtual resource is in “alarm”
state.
3. The NFVI sends monitoring events for resources the VIM has been subscribed to. Note: this subscription
message exchange between the VIM and NFVI is not shown in this message flow.
4. Event correlation, fault detection and aggregation in VIM.
5. Database lookup to find the virtual resources affected by the detected fault.
6. Fault notification to Consumer.
7. The Consumer switches to standby configuration (STBY)
8. Instructions to VIM requesting certain actions to be performed on the affected resources, for example migrate/update/terminate specific resource(s). After reception of such instructions, the VIM is executing the requested action, e.g. it will migrate or terminate a virtual resource.
1. Query request from Consumer to VIM to get information about the current status of a resource.
434
Chapter 3. Developer
OPNFV Documentation, Release Danube
2. Response to the query request with information about the current status of the queried resource. In case the
resource is in “fault” state, information about the related fault(s) is returned.
In order to allow for quick reaction to failures, the time interval between fault detection in step 3 and the corresponding
recovery actions in step 7 and 8 shall be less than 1 second.
figure8 shows a more detailed message flow (Steps 4 to 6) between the 4 building blocks introduced in Functional
Blocks.
4. The Monitor observed a fault in the NFVI and reports the raw fault to the Inspector. The Inspector filters and
aggregates the faults using pre-configured failure policies.
5. a) The Inspector queries the Resource Map to find the virtual resources affected by the raw fault in the NFVI. b)
The Inspector updates the state of the affected virtual resources in the Resource Map. c) The Controller observes
a change of the virtual resource state and informs the Notifier about the state change and the related alarm(s).
Alternatively, the Inspector may directly inform the Notifier about it.
6. The Notifier is performing another filtering and aggregation of the changes and alarms based on the preconfigured alarm configuration. Finally, a fault notification is sent to northbound to the Consumer.
NFVI Maintenance The detailed work flow for NFVI maintenance is shown in figure9 and has the following
steps. Note that steps 1, 2, and 5 to 8a in the NFVI maintenance work flow are very similar to the steps in the fault
management work flow and share a similar implementation plan in Release 1.
1. Subscribe to fault/maintenance notifications.
2. Response to subscribe request.
3. Maintenance trigger received from administrator.
4. VIM switches NFVI resources to “maintenance” state. This, e.g., means they should not be used for further
allocation/migration requests
5. Database lookup to find the virtual resources affected by the detected maintenance operation.
6. Maintenance notification to Consumer.
7. The Consumer switches to standby configuration (STBY)
8. Instructions from Consumer to VIM requesting certain recovery actions to be performed (step 8a). After reception of such instructions, the VIM is executing the requested action in order to empty the physical resources
(step 8b).
9. Maintenance response from VIM to inform the Administrator that the physical machines have been emptied (or
the operation resulted in an error state).
10. Administrator is coordinating and executing the maintenance operation/work on the NFVI.
1. Query request from Administrator to VIM to get information about the current state of a resource.
2. Response to the query request with information about the current state of the queried resource(s). In case the
resource is in “maintenance” state, information about the related maintenance operation is returned.
figure10 shows a more detailed message flow (Steps 3 to 6 and 9) between the 4 building blocks introduced in
Section 5.1..
3. The Administrator is sending a StateChange request to the Controller residing in the VIM.
4. The Controller queries the Resource Map to find the virtual resources affected by the planned maintenance
operation.
5. a) The Controller updates the state of the affected virtual resources in the Resource Map database.
3.2. OPNFV Projects
435
OPNFV Documentation, Release Danube
Fig. 3.15: Fault management work flow
436
Chapter 3. Developer
OPNFV Documentation, Release Danube
Fig. 3.16: Fault management scenario
b) The Controller informs the Notifier about the virtual resources that will be affected by the maintenance
operation.
6. A maintenance notification is sent to northbound to the Consumer.
...
9. The Controller informs the Administrator after the physical resources have been freed.
Information elements This section introduces all attributes and information elements used in the messages exchange
on the northbound interfaces between the VIM and the VNFO and VNFM.
Note: The information elements will be aligned with current work in ETSI NFV IFA working group.
Simple information elements:
• SubscriptionID (Identifier): identifies a subscription to receive fault or maintenance notifications.
• NotificationID (Identifier): identifies a fault or maintenance notification.
• VirtualResourceID (Identifier): identifies a virtual resource affected by a fault or a maintenance action of the
underlying physical resource.
• PhysicalResourceID (Identifier): identifies a physical resource affected by a fault or maintenance action.
• VirtualResourceState (String): state of a virtual resource, e.g. “normal”, “maintenance”, “down”, “error”.
• PhysicalResourceState (String): state of a physical resource, e.g. “normal”, “maintenance”, “down”, “error”.
• VirtualResourceType (String): type of the virtual resource, e.g. “virtual machine”, “virtual memory”, “virtual
storage”, “virtual CPU”, or “virtual NIC”.
• FaultID (Identifier): identifies the related fault in the underlying physical resource. This can be used to correlate
different fault notifications caused by the same fault in the physical resource.
3.2. OPNFV Projects
437
OPNFV Documentation, Release Danube
Fig. 3.17: NFVI maintenance work flow
438
Chapter 3. Developer
OPNFV Documentation, Release Danube
Fig. 3.18: NFVI Maintenance scenario
• FaultType (String): Type of the fault. The allowed values for this parameter depend on the type of the related
physical resource. For example, a resource of type “compute hardware” may have faults of type “CPU failure”,
“memory failure”, “network card failure”, etc.
• Severity (Integer): value expressing the severity of the fault. The higher the value, the more severe the fault.
• MinSeverity (Integer): value used in filter information elements. Only faults with a severity higher than the
MinSeverity value will be notified to the Consumer.
• EventTime (Datetime): Time when the fault was observed.
• EventStartTime and EventEndTime (Datetime): Datetime range that can be used in a FaultQueryFilter to narrow
down the faults to be queried.
• ProbableCause (String): information about the probable cause of the fault.
• CorrelatedFaultID (Integer): list of other faults correlated to this fault.
• isRootCause (Boolean): Parameter indicating if this fault is the root for other correlated faults. If TRUE, then
the faults listed in the parameter CorrelatedFaultID are caused by this fault.
• FaultDetails (Key-value pair): provides additional information about the fault, e.g. information about the threshold, monitored attributes, indication of the trend of the monitored parameter.
• FirmwareVersion (String): current version of the firmware of a physical resource.
• HypervisorVersion (String): current version of a hypervisor.
• ZoneID (Identifier): Identifier of the resource zone. A resource zone is the logical separation of physical and
software resources in an NFVI deployment for physical isolation, redundancy, or administrative designation.
• Metadata (Key-value pair): provides additional information of a physical resource in maintenance/error state.
Complex information elements (see also UML diagrams in figure13 and figure14):
3.2. OPNFV Projects
439
OPNFV Documentation, Release Danube
• VirtualResourceInfoClass:
– VirtualResourceID [1] (Identifier)
– VirtualResourceState [1] (String)
– Faults [0..*] (FaultClass): For each resource, all faults including detailed information about the faults are
provided.
• FaultClass: The parameters of the FaultClass are partially based on ETSI TS 132 111-2 (V12.1.0) 2 , which is
specifying fault management in 3GPP, in particular describing the information elements used for alarm notifications.
– FaultID [1] (Identifier)
– FaultType [1] (String)
– Severity [1] (Integer)
– EventTime [1] (Datetime)
– ProbableCause [1] (String)
– CorrelatedFaultID [0..*] (Identifier)
– FaultDetails [0..*] (Key-value pair)
• SubscribeFilterClass
– VirtualResourceType [0..*] (String)
– VirtualResourceID [0..*] (Identifier)
– FaultType [0..*] (String)
– MinSeverity [0..1] (Integer)
• FaultQueryFilterClass: narrows down the FaultQueryRequest, for example it limits the query to certain physical
resources, a certain zone, a given fault type/severity/cause, or a specific FaultID.
– VirtualResourceType [0..*] (String)
– VirtualResourceID [0..*] (Identifier)
– FaultType [0..*] (String)
– MinSeverity [0..1] (Integer)
– EventStartTime [0..1] (Datetime)
– EventEndTime [0..1] (Datetime)
• PhysicalResourceStateClass:
– PhysicalResourceID [1] (Identifier)
– PhysicalResourceState [1] (String): mandates the new state of the physical resource.
– Metadata [0..*] (Key-value pair)
• PhysicalResourceInfoClass:
– PhysicalResourceID [1] (Identifier)
– PhysicalResourceState [1] (String)
– FirmwareVersion [0..1] (String)
2
http://www.etsi.org/deliver/etsi_ts/132100_132199/13211102/12.01.00_60/ts_13211102v120100p.pdf
440
Chapter 3. Developer
OPNFV Documentation, Release Danube
– HypervisorVersion [0..1] (String)
– ZoneID [0..1] (Identifier)
– Metadata [0..*] (Key-value pair)
• StateQueryFilterClass: narrows down a StateQueryRequest, for example it limits the query to certain physical
resources, a certain zone, or a given resource state (e.g., only resources in “maintenance” state).
– PhysicalResourceID [1] (Identifier)
– PhysicalResourceState [1] (String)
– ZoneID [0..1] (Identifier)
Detailed northbound interface specification This section is specifying the northbound interfaces for fault management and NFVI maintenance between the VIM on the one end and the Consumer and the Administrator on the other
ends. For each interface all messages and related information elements are provided.
Note: The interface definition will be aligned with current work in ETSI NFV IFA working group .
All of the interfaces described below are produced by the VIM and consumed by the Consumer or Administrator.
Fault management interface This interface allows the VIM to notify the Consumer about a virtual resource that
is affected by a fault, either within the virtual resource itself or by the underlying virtualization infrastructure. The
messages on this interface are shown in figure13 and explained in detail in the following subsections.
Note: The information elements used in this section are described in detail in Section 5.4.
SubscribeRequest (Consumer -> VIM) Subscription from Consumer to VIM to be notified about faults of specific
resources. The faults to be notified about can be narrowed down using a subscribe filter.
Parameters:
• SubscribeFilter [1] (SubscribeFilterClass): Optional information to narrow down the faults that shall be notified
to the Consumer, for example limit to specific VirtualResourceID(s), severity, or cause of the alarm.
SubscribeResponse (VIM -> Consumer) Response to a subscribe request message including information about the
subscribed resources, in particular if they are in “fault/error” state.
Parameters:
• SubscriptionID [1] (Identifier): Unique identifier for the subscription. It can be used to delete or update the
subscription.
• VirtualResourceInfo [0..*] (VirtualResourceInfoClass): Provides additional information about the subscribed
resources, i.e., a list of the related resources, the current state of the resources, etc.
FaultNotification (VIM -> Consumer) Notification about a virtual resource that is affected by a fault, either within
the virtual resource itself or by the underlying virtualization infrastructure. After reception of this request, the Consumer will decide on the optimal action to resolve the fault. This includes actions like switching to a hot standby
virtual resource, migration of the fault virtual resource to another physical machine, termination of the faulty virtual
resource and instantiation of a new virtual resource in order to provide a new hot standby resource. In some use cases
the Consumer can leave virtual resources on failed host to be booted up again after fault is recovered. Existing resource
management interfaces and messages between the Consumer and the VIM can be used for those actions, and there is
no need to define additional actions on the Fault Management Interface.
Parameters:
3.2. OPNFV Projects
441
OPNFV Documentation, Release Danube
Fig. 3.19: Fault management NB I/F messages
442
Chapter 3. Developer
OPNFV Documentation, Release Danube
• NotificationID [1] (Identifier): Unique identifier for the notification.
• VirtualResourceInfo [1..*] (VirtualResourceInfoClass): List of faulty resources with detailed information about
the faults.
FaultQueryRequest (Consumer -> VIM) Request to find out about active alarms at the VIM. A FaultQueryFilter
can be used to narrow down the alarms returned in the response message.
Parameters:
• FaultQueryFilter [1] (FaultQueryFilterClass): narrows down the FaultQueryRequest, for example it limits the
query to certain physical resources, a certain zone, a given fault type/severity/cause, or a specific FaultID.
FaultQueryResponse (VIM -> Consumer)
fied in the FaultQueryRequest.
List of active alarms at the VIM matching the FaultQueryFilter speci-
Parameters:
• VirtualResourceInfo [0..*] (VirtualResourceInfoClass): List of faulty resources. For each resource all faults
including detailed information about the faults are provided.
NFVI maintenance The NFVI maintenance interfaces Consumer-VIM allows the Consumer to subscribe to maintenance notifications provided by the VIM. The related maintenance interface Administrator-VIM allows the Administrator to issue maintenance requests to the VIM, i.e. requesting the VIM to take appropriate actions to empty physical
machine(s) in order to execute maintenance operations on them. The interface also allows the Administrator to query
the state of physical machines, e.g., in order to get details in the current status of the maintenance operation like a
firmware update.
The messages defined in these northbound interfaces are shown in figure14 and described in detail in the following
subsections.
SubscribeRequest (Consumer -> VIM) Subscription from Consumer to VIM to be notified about maintenance
operations for specific virtual resources. The resources to be informed about can be narrowed down using a subscribe
filter.
Parameters:
• SubscribeFilter [1] (SubscribeFilterClass): Information to narrow down the faults that shall be notified to the
Consumer, for example limit to specific virtual resource type(s).
SubscribeResponse (VIM -> Consumer) Response to a subscribe request message, including information about
the subscribed virtual resources, in particular if they are in “maintenance” state.
Parameters:
• SubscriptionID [1] (Identifier): Unique identifier for the subscription. It can be used to delete or update the
subscription.
• VirtualResourceInfo [0..*] (VirtalResourceInfoClass): Provides additional information about the subscribed virtual resource(s), e.g., the ID, type and current state of the resource(s).
MaintenanceNotification (VIM -> Consumer) Notification about a physical resource switched to “maintenance”
state. After reception of this request, the Consumer will decide on the optimal action to address this request, e.g., to
switch to the standby (STBY) configuration.
Parameters:
3.2. OPNFV Projects
443
OPNFV Documentation, Release Danube
444
Chapter 3. Developer
OPNFV Documentation, Release Danube
• VirtualResourceInfo [1..*] (VirtualResourceInfoClass): List of virtual resources where the state has been
changed to maintenance.
StateChangeRequest (Administrator -> VIM) Request to change the state of a list of physical resources, e.g. to
“maintenance” state, in order to prepare them for a planned maintenance operation.
Parameters:
• PhysicalResourceState [1..*] (PhysicalResourceStateClass)
StateChangeResponse (VIM -> Administrator) Response message to inform the Administrator that the requested
resources are now in maintenance state (or the operation resulted in an error) and the maintenance operation(s) can be
executed.
Parameters:
• PhysicalResourceInfo [1..*] (PhysicalResourceInfoClass)
StateQueryRequest (Administrator -> VIM) In this procedure, the Administrator would like to get the information
about physical machine(s), e.g. their state (“normal”, “maintenance”), firmware version, hypervisor version, update
status of firmware and hypervisor, etc. It can be used to check the progress during firmware update and the confirmation
after update. A filter can be used to narrow down the resources returned in the response message.
Parameters:
• StateQueryFilter [1] (StateQueryFilterClass): narrows down the StateQueryRequest, for example it limits the
query to certain physical resources, a certain zone, or a given resource state.
StateQueryResponse (VIM -> Administrator) List of physical resources matching the filter specified in the StateQueryRequest.
Parameters:
• PhysicalResourceInfo [0..*] (PhysicalResourceInfoClass): List of physical resources. For each resource, information about the current state, the firmware version, etc. is provided.
NFV IFA, OPNFV Doctor and AODH alarms This section compares the alarm interfaces of ETSI NFV IFA with
the specifications of this document and the alarm class of AODH.
ETSI NFV specifies an interface for alarms from virtualised resources in ETSI GS NFV-IFA 005 [ENFV]. The interface specifies an Alarm class and two notifications plus operations to query alarm instances and to subscribe to the
alarm notifications.
The specification in this document has a structure that is very similar to the ETSI NFV specifications. The notifications
differ in that an alarm notification in the NFV interface defines a single fault for a single resource while the notification
specified in this document can contain multiple faults for multiple resources. The Doctor specification is lacking the
detailed time stamps of the NFV specification essential for synchronizaion of the alarm list using the query operation.
The detailed time stamps are also of value in the event and alarm history DBs.
AODH defines a base class for alarms, not the notifications. This means that some of the dynamic attributes of the
ETSI NFV alarm type, like alarmRaisedTime, are not applicable to the AODH alarm class but are attributes of in
the actual notifications. (Description of these attributes will be added later.) The AODH alarm class is lacking some
attributes present in the NFV specification, fault details and correlated alarms. Instead the AODH alarm class has
attributes for actions, rules and user and project id.
3.2. OPNFV Projects
445
OPNFV Documentation, Release Danube
ETSI NFV
Type
alarmId
-
Alarm
OPNFV Doctor Requirement Specs
FaultId
-
AODH Event Alarm
Notification
alarm_id
alarm_name
Description / Comment
Identifier of an alarm.
Human
readable
alarm name.
Identifier of the affected virtual resource
is part of the AODH
reason parameter.
User and project identifiers.
Timestamp
when
alarm was raised.
managedObjectId
VirtualResourceId
(reason)
-
-
user_id, project_id
alarmRaisedTime
-
-
alarmChangedTime
-
-
alarmClearedTime
-
-
eventTime
-
-
-
EventTime
generated
state: E.g. Fired, Updated Cleared
VirtualResourceState:
E.g. normal, down
maintenance, error
current: ok, alarm, insufficient_data
ETSI
NFV
IFA
005/006 lists example
alarm states.
perceivedSeverity:
E.g. Critical, Major,
Minor, Warning, Indeterminate, Cleared
Severity (Integer)
Severity: low (default), moderate, critical
ETSI
NFV
IFA
005/006 lists example
perceived
severity
values.
Timestamp
when alarm was
changed/updated.
Timestamp
when
alarm was cleared.
Timestamp
when
alarm was first observed by the Monitor.
Timestamp of the Notification.
Recommendations
May be added in ETSI
NFV Stage 3.
-
May be added in ETSI
NFV Stage 3.
To be added to Doctor and AODH. May
be derived (e.g. in
a shimlayer) from the
AODH alarm history.
see above
see above
see above
Update
parameter
name in Doctor spec.
May be added in ETSI
NFV Stage 3.
Maintenance state is
missing in AODH.
List of alarm states
will be specified in
ETSI NFV Stage 3.
List of alarm states
will be specified in
ETSI NFV Stage 3.
OPNFV: Severity (Integer):
• update
OPNFV
Doctor
specification to
Enum
perceivedSeverity=Indetermin
446
• remove
value
Indetermined
in
IFA
and map
undefined
values to
“minor”
severity,
Chapter 3. Developer
or
• add value
indetermined in
OPNFV Documentation, Release Danube
Table: Comparison of alarm attributes
The primary area of improvement should be alignment of the perceived severity. This is important for a quick and
accurate evaluation of the alarm. AODH thus should support also the X.733 values Critical, Major, Minor, Warning
and Indeterminate.
The detailed time stamps (raised, changed, cleared) which are essential for synchronizing the alarm list using a query
operation should be added to the Doctor specification.
Other areas that need alignment is the so called alarm state in NFV. Here we must however consider what can be
attributes of the notification vs. what should be a property of the alarm instance. This will be analyzed later.
Detailed southbound interface specification This section is specifying the southbound interfaces for fault management between the Monitors and the Inspector. Although southbound interfaces should be flexible to handle various
events from different types of Monitors, we define unified event API in order to improve interoperability between the
Monitors and the Inspector. This is not limiting implementation of Monitor and Inspector as these could be extended
in order to support failures from intelligent inspection like prediction.
Note: The interface definition will be aligned with current work in ETSI NFV IFA working group.
Fault event interface This interface allows the Monitors to notify the Inspector about an event which was captured
by the Monitor and may effect resources managed in the VIM.
EventNotification Event notification including fault description. The entity of this notification is event, and not
fault or error specifically. This allows us to use generic event format or framework build out of Doctor project. The
parameters below shall be mandatory, but keys in ‘Details’ can be optional.
Parameters:
• Time [1]: Datetime when the fault was observed in the Monitor.
• Type [1]: Type of event that will be used to process correlation in Inspector.
• Details [0..1]: Details containing additional information with Key-value pair style. Keys shall be defined depending on the Type of the event.
E.g.:
{
'event': {
'time': '2016-04-12T08:00:00',
'type': 'compute.host.down',
'details': {
'hostname': 'compute-1',
'source': 'sample_monitor',
'cause': 'link-down',
'severity': 'critical',
'status': 'down',
'monitor_id': 'monitor-1',
'monitor_event_id': '123',
}
}
}
Optional parameters in ‘Details’:
• Hostname: the hostname on which the event occurred.
3.2. OPNFV Projects
447
OPNFV Documentation, Release Danube
• Source: the display name of reporter of this event. This is not limited to monitor, other entity can be specified
such as ‘KVM’.
• Cause: description of the cause of this event which could be different from the type of this event.
• Severity: the severity of this event set by the monitor.
• Status: the status of target object in which error occurred.
• MonitorID: the ID of the monitor sending this event.
• MonitorEventID: the ID of the event in the monitor. This can be used by operator while tracking the monitor
log.
• RelatedTo: the array of IDs which related to this event.
Also, we can have bulk API to receive multiple events in a single HTTP POST message by using the ‘events’ wrapper
as follows:
{
'events': [
'event': {
'time': '2016-04-12T08:00:00',
'type': 'compute.host.down',
'details': {},
},
'event': {
'time': '2016-04-12T08:00:00',
'type': 'compute.host.nic.error',
'details': {},
}
]
}
Blueprints This section is listing a first set of blueprints that have been proposed by the Doctor project to the open
source community. Further blueprints addressing other gaps identified in Section 4 will be submitted at a later stage
of the OPNFV. In this section the following definitions are used:
• “Event” is a message emitted by other OpenStack services such as Nova and Neutron and is consumed by the
“Notification Agents” in Ceilometer.
• “Notification” is a message generated by a “Notification Agent” in Ceilometer based on an “event” and is delivered to the “Collectors” in Ceilometer that store those notifications (as “sample”) to the Ceilometer “Databases”.
Instance State Notification (Ceilometer) 3 The Doctor project is planning to handle “events” and “notifications”
regarding Resource Status; Instance State, Port State, Host State, etc. Currently, Ceilometer already receives “events”
to identify the state of those resources, but it does not handle and store them yet. This is why we also need a new event
definition to capture those resource states from “events” created by other services.
This BP proposes to add a new compute notification state to handle events from an instance (server) from nova. It also
creates a new meter “instance.state” in OpenStack.
Event Publisher for Alarm (Ceilometer) 4
Problem statement:
The existing “Alarm Evaluator” in OpenStack Ceilometer is periodically querying/polling the databases in order to
check all alarms independently from other processes. This is adding additional delay to the fault notification send to
the Consumer, whereas one requirement of Doctor is to react on faults as fast as possible.
3
4
https://etherpad.opnfv.org/p/doctor_bps
https://etherpad.opnfv.org/p/doctor_bps
448
Chapter 3. Developer
OPNFV Documentation, Release Danube
The existing message flow is shown in figure12: after receiving an “event”, a “notification agent” (i.e. “event
publisher”) will send a “notification” to a “Collector”. The “collector” is collecting the notifications and is updating the
Ceilometer “Meter” database that is storing information about the “sample” which is capured from original “event”.
The “Alarm Evaluator” is periodically polling this databases then querying “Meter” database based on each alarm
configuration.
Fig. 3.21: Implementation plan in Ceilometer architecture
In the current Ceilometer implementation, there is no possibility to directly trigger the “Alarm Evaluator” when a new
“event” was received, but the “Alarm Evaluator” will only find out that requires firing new notification to the Consumer
when polling the database.
Change/feature request:
This BP proposes to add a new “event publisher for alarm”, which is bypassing several steps in Ceilometer in order to avoid the polling-based approach of the existing Alarm Evaluator that makes notification slow to users. See
figure12.
After receiving an “(alarm) event” by listening on the Ceilometer message queue (“notification bus”), the new “event
publisher for alarm” immediately hands a “notification” about this event to a new Ceilometer component “Notificationdriven alarm evaluator” proposed in the other BP (see Section 5.6.3).
Note, the term “publisher” refers to an entity in the Ceilometer architecture (it is a “notification agent”). It offers the
capability to provide notifications to other services outside of Ceilometer, but it is also used to deliver notifications to
other Ceilometer components (e.g. the “Collectors”) via the Ceilometer “notification bus”.
Implementation detail
• “Event publisher for alarm” is part of Ceilometer
3.2. OPNFV Projects
449
OPNFV Documentation, Release Danube
• The standard AMQP message queue is used with a new topic string.
• No new interfaces have to be added to Ceilometer.
• “Event publisher for Alarm” can be configured by the Administrator of Ceilometer to be used as “Notification
Agent” in addition to the existing “Notifier”
• Existing alarm mechanisms of Ceilometer can be used allowing users to configure how to distribute the “notifications” transformed from “events”, e.g. there is an option whether an ongoing alarm is re-issued or not
(“repeat_actions”).
Notification-driven alarm evaluator (Ceilometer) 5
Problem statement:
The existing “Alarm Evaluator” in OpenStack Ceilometer is periodically querying/polling the databases in order to
check all alarms independently from other processes. This is adding additional delay to the fault notification send to
the Consumer, whereas one requirement of Doctor is to react on faults as fast as possible.
Change/feature request:
This BP is proposing to add an alternative “Notification-driven Alarm Evaluator” for Ceilometer that is receiving
“notifications” sent by the “Event Publisher for Alarm” described in the other BP. Once this new “Notification-driven
Alarm Evaluator” received “notification”, it finds the “alarm” configurations which may relate to the “notification” by
querying the “alarm” database with some keys i.e. resource ID, then it will evaluate each alarm with the information
in that “notification”.
After the alarm evaluation, it will perform the same way as the existing “alarm evaluator” does for firing alarm notification to the Consumer. Similar to the existing Alarm Evaluator, this new “Notification-driven Alarm Evaluator” is
aggregating and correlating different alarms which are then provided northbound to the Consumer via the OpenStack
“Alarm Notifier”. The user/administrator can register the alarm configuration via existing Ceilometer API 6 . Thereby,
he can configure whether to set an alarm or not and where to send the alarms to.
Implementation detail
• The new “Notification-driven Alarm Evaluator” is part of Ceilometer.
• Most of the existing source code of the “Alarm Evaluator” can be re-used to implement this BP
• No additional application logic is needed
• It will access the Ceilometer Databases just like the existing “Alarm evaluator”
• Only the polling-based approach will be replaced by a listener for “notifications” provided by the “Event Publisher for Alarm” on the Ceilometer “notification bus”.
• No new interfaces have to be added to Ceilometer.
Report host fault to update server state immediately (Nova) 7
Problem statement:
• Nova state change for failed or unreachable host is slow and does not reliably state host is down or not. This
might cause same server instance to run twice if action taken to evacuate instance to another host.
• Nova state for server(s) on failed host will not change, but remains active and running. This gives the user false
information about server state.
• VIM northbound interface notification of host faults towards VNFM and NFVO should be in line with OpenStack state. This fault notification is a Telco requirement defined in ETSI and will be implemented by OPNFV
Doctor project.
5
6
7
https://etherpad.opnfv.org/p/doctor_bps
https://wiki.openstack.org/wiki/Ceilometer/Alerting
https://blueprints.launchpad.net/nova/+spec/update-server-state-immediately
450
Chapter 3. Developer
OPNFV Documentation, Release Danube
• Openstack user cannot make HA actions fast and reliably by trusting server state and host state.
Proposed change:
There needs to be a new API for Admin to state host is down. This API is used to mark services running in host down
to reflect the real situation.
Example on compute node is:
• When compute node is up and running::
vm_state: activeand power_state: running
nova-compute state: up status: enabled
• When compute node goes down and new API is called to state host is down::
vm_state: stopped power_state: shutdown
nova-compute state: down status: enabled
Alternatives:
There is no attractive alternative to detect all different host faults than to have an external tool to detect different host
faults. For this kind of tool to exist there needs to be new API in Nova to report fault. Currently there must be some
kind of workarounds implemented as cannot trust or get the states from OpenStack fast enough.
Other related BPs This section lists some BPs related to Doctor, but proposed by drafters outside the OPNFV
community.
pacemaker-servicegroup-driver 8 This BP will detect and report host down quite fast to OpenStack. This however
might not work properly for example when management network has some problem and host reported faulty while
VM still running there. This might lead to launching same VM instance twice causing problems. Also NB IF message
needs fault reason and for that the source needs to be a tool that detects different kind of faults as Doctor will be doing.
Also this BP might need enhancement to change server and service states correctly.
Summary and conclusion
The Doctor project aimed at detailing NFVI fault management and NFVI maintenance requirements. These are indispensable operations for an Operator, and extremely necessary to realize telco-grade high availability. High availability
is a large topic; the objective of Doctor is not to realize a complete high availability architecture and implementation.
Instead, Doctor limited itself to addressing the fault events in NFVI, and proposes enhancements necessary in VIM,
e.g. OpenStack, to ensure VNFs availability in such fault events, taking a Telco VNFs application level management
system into account.
The Doctor project performed a robust analysis of the requirements from NFVI fault management and NFVI maintenance operation, concretely found out gaps in between such requirements and the current implementation of OpenStack. A detailed architecture and interface specification has been described in this document and work to realize
Doctor features and fill out the identified gaps in upstream communities is in the final stages of development.
Annex: NFVI Faults
Faults in the listed elements need to be immediately notified to the Consumer in order to perform an immediate
action like live migration or switch to a hot standby entity. In addition, the Administrator of the host should trigger a
maintenance action to, e.g., reboot the server or replace a defective hardware element.
8
https://blueprints.launchpad.net/nova/+spec/pacemaker-servicegroup-driver
3.2. OPNFV Projects
451
OPNFV Documentation, Release Danube
Faults can be of different severity, i.e., critical, warning, or info. Critical faults require immediate action as a severe
degradation of the system has happened or is expected. Warnings indicate that the system performance is going down:
related actions include closer (e.g. more frequent) monitoring of that part of the system or preparation for a cold
migration to a backup VM. Info messages do not require any action. We also consider a type “maintenance”, which is
no real fault, but may trigger maintenance actions like a re-boot of the server or replacement of a faulty, but redundant
HW.
Faults can be gathered by, e.g., enabling SNMP and installing some open source tools to catch and poll SNMP. When
using for example Zabbix one can also put an agent running on the hosts to catch any other fault. In any case of failure,
the Administrator should be notified. The following tables provide a list of high level faults that are considered within
the scope of the Doctor project requiring immediate action by the Consumer.
Compute/Storage
Fault
Processor/CPU failure, CPU condition
not ok
Memory failure/ Memory condition not
ok
Network card failure, e.g. network
adapter connectivity lost
Disk crash
Storage controller
PDU/power failure, power off, server
reset
Power degration, power redundancy lost,
power threshold exceeded
Chassis problem (e.g. fan
degraded/failed, chassis power
degraded), CPU fan problem,
temperature/ thermal condition not ok
Mainboard failure
OS crash (e.g. kernel panic)
Sever-How
Comment
ity
to detect?
Crit- Zabibix
cal
Crit- Zabibix
cal (IPMI)
Crit- Zabibix/
cal Ceilometer
Info RAID Network storage is very
moni- redundant (e.g. RAID
toring system) and can guarantee
high availability
Crit- Zabibix
cal (IPMI)
Crit- Zabibix/
cal Ceilometer
Warn-SNMP
ing
Warn-SNMP
ing
Critical
Critical
Zabbix
(IPMI)
Zabbix
Immediate action
to recover
Switch to hot
standby
Switch to hot
standby
Switch to hot
standby
Inform OAM
Live migration if
storage is still
accessible; otherwise
hot standby
Switch to hot
standby
Live migration
Live migration
e.g. PCIe, SAS link failure
Switch to hot
standby
Switch to hot
standby
Hypervisor
452
Chapter 3. Developer
OPNFV Documentation, Release Danube
Fault
Severity
System has restarted
Hypervisor failure
Critical
Warning/
Critical
Warning
Hypervisor status not retrievable
after certain period
How to
detect?
Zabbix
Zabbix/
Ceilometer
Alarming
service
Comment
Zabbix/ Ceilometer
unreachable
Immediate action to
recover
Switch to hot standby
Evacuation/switch to
hot standby
Rebuild VM
Network
Fault
SDN/OpenFlow switch,
controller
degraded/failed
Hardware failure of
physical switch/router
Sever- How to
ity
detect?
Crit- Ceiloical
meter
Comment
Warn- SNMP
ing
Redundancy of physical
infrastructure is reduced or no
longer available
Immediate action to
recover
Switch to hot standby or
reconfigure virtual network
topology
Live migration if possible
otherwise evacuation
References and bibliography
Doctor Installation Guide
Doctor Configuration
OPNFV installers install most components of Doctor framework including OpenStack Nova, Neutron and Cinder
(Doctor Controller) and OpenStack Ceilometer and Aodh (Doctor Notifier) except Doctor Monitor.
After major components of OPNFV are deployed, you can setup Doctor functions by following instructions in this
section. You can also learn detailed steps in setup_installer() under doctor/tests.
Doctor Inspector You need to configure one of Doctor Inspector below.
Doctor Sample Inspector
Sample Inspector is intended to show minimum functions of Doctor Inspector.
Doctor Sample Inspector suggested to be placed in one of the controller nodes, but it can be put on any host where
Doctor Monitor can reach and access the OpenStack Controller (Nova).
Make sure OpenStack env parameters are set properly, so that Doctor Inspector can issue admin actions such as
compute host force-down and state update of VM.
Then, you can configure Doctor Inspector as follows:
git clone https://gerrit.opnfv.org/gerrit/doctor -b stable/danube
cd doctor/tests
INSPECTOR_PORT=12345
python inspector.py $INSPECTOR_PORT > inspector.log 2>&1 &
Congress
OpenStack Congress is a Governance as a Service (previously Policy as a Service). Congress implements Doctor
Inspector as it can inspect a fault situation and propagate errors onto other entities.
Congress is deployed by OPNFV installers. You need to enable doctor datasource driver and set policy rules. By the
example configuration below, Congress will force down nova compute service when it received a fault event of that
compute host. Also, Congress will set the state of all VMs running on that host from ACTIVE to ERROR state.
3.2. OPNFV Projects
453
OPNFV Documentation, Release Danube
openstack congress datasource create doctor doctor
openstack congress policy rule create \
--name host_down classification \
'host_down(host) :doctor:events(hostname=host, type="compute.host.down", status="down")'
openstack congress policy rule create \
--name active_instance_in_host classification \
'active_instance_in_host(vmid, host) :nova:servers(id=vmid, host_name=host, status="ACTIVE")'
openstack congress policy rule create \
--name host_force_down classification \
'execute[nova:services.force_down(host, "nova-compute", "True")] :host_down(host)'
openstack congress policy rule create \
--name error_vm_states classification \
'execute[nova:servers.reset_state(vmid, "error")] :host_down(host),
active_instance_in_host(vmid, host)'
Doctor Monitor
Doctor Sample Monitor
Doctor Monitors are suggested to be placed in one of the controller nodes, but those can be put on any host which
is reachable to target compute host and accessible by the Doctor Inspector. You need to configure Monitors for all
compute hosts one by one.
Make sure OpenStack env parameters are set properly, so that Doctor Inspector can issue admin actions such as
compute host force-down and state update of VM.
Then, you can configure the Doctor Monitor as follows (Example for Apex deployment):
git clone https://gerrit.opnfv.org/gerrit/doctor -b stable/danube
cd doctor/tests
INSPECTOR_PORT=12345
COMPUTE_HOST='overcloud-novacompute-1.localdomain.com'
COMPUTE_IP=192.30.9.5
sudo python monitor.py "$COMPUTE_HOST" "$COMPUTE_IP" \
"http://127.0.0.1:$INSPECTOR_PORT/events" > monitor.log 2>&1 &
Doctor User Guide
Doctor capabilities and usage
figure1 shows the currently implemented and tested architecture of Doctor. The implementation is based on OpenStack and related components. The Monitor can be realized by a sample Python-based implementation provided in the
Doctor code repository. The Controller is realized by OpenStack Nova, Neutron and Cinder for compute, network and
storage, respectively. The Inspector can be realized by OpenStack Congress or a sample Python-based implementation
also available in the code repository of Doctor. The Notifier is realized by OpenStack Aodh.
Immediate Notification Immediate notification can be used by creating ‘event’ type alarm via OpenStack Alarming
(Aodh) API with relevant internal components support.
454
Chapter 3. Developer
OPNFV Documentation, Release Danube
Fig. 3.22: Implemented and tested architecture
See, upstream spec document:
evaluator.html
http://specs.openstack.org/openstack/ceilometer-specs/specs/liberty/event-alarm-
An example of a consumer of this notification can be found in the Doctor repository. It can be executed as follows:
git clone https://gerrit.opnfv.org/gerrit/doctor -b stable/danube
cd doctor/tests
CONSUMER_PORT=12346
python consumer.py "$CONSUMER_PORT" > consumer.log 2>&1 &
Consistent resource state awareness Resource state of compute host can be changed/updated according to a trigger
from a monitor running outside of OpenStack Compute (Nova) by using force-down API.
See http://artifacts.opnfv.org/doctor/danube/manuals/mark-host-down_manual.html for more detail.
Valid compute host status given to VM owner The resource state of a compute host can be retrieved by a user with
the OpenStack Compute (Nova) servers API.
See http://artifacts.opnfv.org/doctor/danube/manuals/get-valid-server-state.html for more detail.
Design Documents
This is the directory to store design documents which may include draft versions of blueprints written before proposing
to upstream OSS communities such as OpenStack, in order to keep the original blueprint as reviewed in OPNFV. That
means there could be out-dated blueprints as result of further refinements in the upstream OSS community. Please
refer to the link in each document to find the latest version of the blueprint and status of development in the relevant
OSS community.
3.2. OPNFV Projects
455
OPNFV Documentation, Release Danube
See also https://wiki.opnfv.org/requirements_projects .
Note: This is a specification draft of a blueprint proposed for OpenStack Nova Liberty. It was written by project
member(s) and agreed within the project before submitting it upstream. No further changes to its content will be made
here anymore; please follow it upstream:
• Current version upstream: https://review.openstack.org/#/c/169836/
• Development activity: https://blueprints.launchpad.net/nova/+spec/mark-host-down
Original draft is as follow:
Report host fault to update server state immediately
https://blueprints.launchpad.net/nova/+spec/update-server-state-immediately
A new API is needed to report a host fault to change the state of the instances and compute node immediately. This
allows usage of evacuate API without a delay. The new API provides the possibility for external monitoring system to
detect any kind of host failure fast and reliably and inform OpenStack about it. Nova updates the compute node state
and states of the instances. This way the states in the Nova DB will be in sync with the real state of the system.
Problem description
• Nova state change for failed or unreachable host is slow and does not reliably state compute node is down or
not. This might cause same instance to run twice if action taken to evacuate instance to another host.
• Nova state for instances on failed compute node will not change, but remains active and running. This gives user
a false information about instance state. Currently one would need to call “nova reset-state” for each instance to
have them in error state.
• OpenStack user cannot make HA actions fast and reliably by trusting instance state and compute node state.
• As compute node state changes slowly one cannot evacuate instances.
Use Cases Use case in general is that in case there is a host fault one should change compute node state fast and
reliably when using DB servicegroup backend. On top of this here is the use cases that are not covered currently
to have instance states changed correctly: * Management network connectivity lost between controller and compute
node. * Host HW failed.
Generic use case flow:
• The external monitoring system detects a host fault.
• The external monitoring system fences the host if not down already.
• The external system calls the new Nova API to force the failed compute node into down state as well as instances
running on it.
• Nova updates the compute node state and state of the effected instances to Nova DB.
Currently nova-compute state will be changing “down”, but it takes a long time. Server state keeps as “vm_state:
active” and “power_state: running”, which is not correct. By having external tool to detect host faults fast, fence
host by powering down and then report host down to OpenStack, all these states would reflect to actual situation.
Also if OpenStack will not implement automatic actions for fault correlation, external tool can do that. This could be
configured for example in server instance METADATA easily and be read by external tool.
Project Priority Liberty priorities have not yet been defined.
456
Chapter 3. Developer
OPNFV Documentation, Release Danube
Proposed change There needs to be a new API for Admin to state host is down. This API is used to mark compute
node and instances running on it down to reflect the real situation.
Example on compute node is:
• When compute node is up and running: vm_state: active and power_state: running nova-compute state: up
status: enabled
• When compute node goes down and new API is called to state host is down: vm_state: stopped power_state:
shutdown nova-compute state: down status: enabled
vm_state values: soft-delete, deleted, resized and error should not be touched. task_state effect needs to be worked
out if needs to be touched.
Alternatives There is no attractive alternatives to detect all different host faults than to have a external tool to detect
different host faults. For this kind of tool to exist there needs to be new API in Nova to report fault. Currently there
must have been some kind of workarounds implemented as cannot trust or get the states from OpenStack fast enough.
Data model impact None
REST API impact
• Update CLI to report host is down
nova host-update command
usage: nova host-update [–status <enable|disable>] [–maintenance <enable|disable>] [–report-host-down]
<hostname>
Update host settings.
Positional arguments
<hostname> Name of host.
Optional arguments
–status <enable|disable> Either enable or disable a host.
–maintenance <enable|disable> Either put or resume host to/from maintenance.
–down Report host down to update instance and compute node state in db.
• Update Compute API to report host is down:
/v2.1/{tenant_id}/os-hosts/{host_name}
Normal response codes: 200 Request parameters
Parameter Style Type Description host_name URI xsd:string The name of the host of interest to you.
{
“host”: { “status”: “enable”, “maintenance_mode”: “enable” “host_down_reported”: “true”
}
}
{
“host”: { “host”: “65c5d5b7e3bd44308e67fc50f362aee6”, “maintenance_mode”: “enabled”, “status”:
“enabled” “host_down_reported”: “true”
3.2. OPNFV Projects
457
OPNFV Documentation, Release Danube
}
}
• New method to nova.compute.api module HostAPI class to have a to mark host related instances and compute
node down: set_host_down(context, host_name)
• class novaclient.v2.hosts.HostManager(api) method update(host, values) Needs to handle reporting host down.
• Schema does not need changes as in db only service and server states are to be changed.
Security impact API call needs admin privileges (in the default policy configuration).
Notifications impact None
Other end user impact None
Performance Impact Only impact is that user can get information faster about instance and compute node state.
This also gives possibility to evacuate faster. No impact that would slow down. Host down should be rare occurrence.
Other deployer impact
Developer can make use of any external tool to detect host fault and report it to OpenStack.
Developer impact None
Implementation
Assignee(s) Primary assignee: Tomi Juvonen Other contributors: Ryota Mibu
Work Items
• Test cases.
• API changes.
• Documentation.
Dependencies None
Testing Test cases that exists for enabling or putting host to maintenance should be altered or similar new cases
made test new functionality.
Documentation Impact New API needs to be documented:
• Compute API extensions documentation. http://developer.openstack.org/api-ref-compute-v2.1.html
• Nova
commands
documentation.
admin/content/novaclient_commands.html
• Compute
command-line
client
reference/content/novaclient_commands.html
documentation.
http://docs.openstack.org/user-guidehttp://docs.openstack.org/cli-
• nova.compute.api documentation. http://docs.openstack.org/developer/nova/api/nova.compute.api.html
458
Chapter 3. Developer
OPNFV Documentation, Release Danube
• High Availability guide might have page to tell external tool could provide ability to provide faster HA as able
to update states by new API. http://docs.openstack.org/high-availability-guide/content/index.html
References
• OPNFV Doctor project: https://wiki.opnfv.org/doctor
• OpenStack Instance HA Proposal: http://blog.russellbryant.net/2014/10/15/openstack-instance-ha-proposal/
• The Different Facets of OpenStack HA: http://blog.russellbryant.net/2015/03/10/ the-different-facets-ofopenstack-ha/
Notification Alarm Evaluator
Note:
This is spec draft of blueprint for OpenStack Ceilomter Liberty.
see
current
version:
https://review.openstack.org/172893
To
track
development
https://blueprints.launchpad.net/ceilometer/+spec/notification-alarm-evaluator
To
activity:
https://blueprints.launchpad.net/ceilometer/+spec/notification-alarm-evaluator
This blueprint proposes to add a new alarm evaluator for handling alarms on events passed from other OpenStack
services, that provides event-driven alarm evaluation which makes new sequence in Ceilometer instead of the pollingbased approach of the existing Alarm Evaluator, and realizes immediate alarm notification to end users.
Problem description As an end user, I need to receive alarm notification immediately once Ceilometer captured
an event which would make alarm fired, so that I can perform recovery actions promptly to shorten downtime of my
service. The typical use case is that an end user set alarm on “compute.instance.update” in order to trigger recovery
actions once the instance status has changed to ‘shutdown’ or ‘error’. It should be nice that an end user can receive
notification within 1 second after fault observed as the same as other helth- check mechanisms can do in some cases.
The existing Alarm Evaluator is periodically querying/polling the databases in order to check all alarms independently
from other processes. This is good approach for evaluating an alarm on samples stored in a certain period. However,
this is not efficient to evaluate an alarm on events which are emitted by other OpenStack servers once in a while.
The periodical evaluation leads delay on sending alarm notification to users. The default period of evaluation cycle
is 60 seconds. It is recommended that an operator set longer interval than configured pipeline interval for underlying
metrics, and also longer enough to evaluate all defined alarms in certain period while taking into account the number
of resources, users and alarms.
Proposed change The proposal is to add a new event-driven alarm evaluator which receives messages from Notification Agent and finds related Alarms, then evaluates each alarms;
• New alarm evaluator could receive event notification from Notification Agent by which adding a dedicated
notifier as a publisher in pipeline.yaml (e.g. notifier://?topic=event_eval).
• When new alarm evaluator received event notification, it queries alarm database by Project ID and Resource ID
written in the event notification.
• Found alarms are evaluated by referring event notification.
• Depending on the result of evaluation, those alarms would be fired through Alarm Notifier as the same as existing
Alarm Evaluator does.
This proposal also adds new alarm type “notification” and “notification_rule”. This enables users to create alarms
on events. The separation from other alarm types (such as “threshold” type) is intended to show different timing of
3.2. OPNFV Projects
459
OPNFV Documentation, Release Danube
evaluation and different format of condition, since the new evaluator will check each event notification once it received
whereas “threshold” alarm can evaluate average of values in certain period calculated from multiple samples.
The new alarm evaluator handles Notification type alarms, so we have to change existing alarm evaluator to exclude
“notification” type alarms from evaluation targets.
Alternatives There was similar blueprint proposal “Alarm type based on notification”, but the approach is different.
The old proposal was to adding new step (alarm evaluations) in Notification Agent every time it received event from
other OpenStack services, whereas this proposal intends to execute alarm evaluation in another component which can
minimize impact to existing pipeline processing.
Another approach is enhancement of existing alarm evaluator by adding notification listener. However, there are two
issues; 1) this approach could cause stall of periodical evaluations when it receives bulk of notifications, and 2) this
could break the alarm portioning i.e. when alarm evaluator received notification, it might have to evaluate some alarms
which are not assign to it.
Data model impact Resource ID will be added to Alarm model as an optional attribute. This would help the new
alarm evaluator to filter out non-related alarms while querying alarms, otherwise it have to evaluate all alarms in the
project.
REST API impact Alarm API will be extended as follows;
• Add “notification” type into alarm type list
• Add “resource_id” to “alarm”
• Add “notification_rule” to “alarm”
Sample data of Notification-type alarm:
{
"alarm_actions": [
"http://site:8000/alarm"
],
"alarm_id": null,
"description": "An alarm",
"enabled": true,
"insufficient_data_actions": [
"http://site:8000/nodata"
],
"name": "InstanceStatusAlarm",
"notification_rule": {
"event_type": "compute.instance.update",
"query" : [
{
"field" : "traits.state",
"type" : "string",
"value" : "error",
"op" : "eq",
},
]
},
"ok_actions": [],
"project_id": "c96c887c216949acbdfbd8b494863567",
"repeat_actions": false,
"resource_id": "153462d0-a9b8-4b5b-8175-9e4b05e9b856",
"severity": "moderate",
460
Chapter 3. Developer
OPNFV Documentation, Release Danube
"state": "ok",
"state_timestamp": "2015-04-03T17:49:38.406845",
"timestamp": "2015-04-03T17:49:38.406839",
"type": "notification",
"user_id": "c96c887c216949acbdfbd8b494863567"
}
“resource_id” will be refered to query alarm and will not be check permission and belonging of project.
Security impact None
Pipeline impact None
Other end user impact None
Performance/Scalability Impacts When Ceilomter received a number of events from other OpenStack services in
short period, this alarm evaluator can keep working since events are queued in a messaging queue system, but it can
cause delay of alarm notification to users and increase the number of read and write access to alarm database.
“resource_id” can be optional, but restricting it to mandatory could be reduce performance impact. If user create
“notification” alarm without “resource_id”, those alarms will be evaluated every time event occurred in the project.
That may lead new evaluator heavy.
Other deployer impact New service process have to be run.
Developer impact Developers should be aware that events could be notified to end users and avoid passing raw infra
information to end users, while defining events and traits.
Implementation
Assignee(s)
Primary assignee: r-mibu
Other contributors: None
Ongoing maintainer: None
Work Items
• New event-driven alarm evaluator
• Add new alarm type “notification” as well as AlarmNotificationRule
• Add “resource_id” to Alarm model
• Modify existing alarm evaluator to filter out “notification” alarms
• Add new config parameter for alarm request check whether accepting alarms without specifying “resource_id”
or not
3.2. OPNFV Projects
461
OPNFV Documentation, Release Danube
Future lifecycle This proposal is key feature to provide information of cloud resources to end users in real-time
that enables efficient integration with user-side manager or Orchestrator, whereas currently those information are
considered to be consumed by admin side tool or service. Based on this change, we will seek orchestrating scenarios
including fault recovery and add useful event definition as well as additional traits.
Dependencies None
Testing New unit/scenario tests are required for this change.
Documentation Impact
• Proposed evaluator will be described in the developer document.
• New alarm type and how to use will be explained in user guide.
References
• OPNFV Doctor project: https://wiki.opnfv.org/doctor
• Blueprint “Alarm type based on notification”: https://blueprints.launchpad.net/ceilometer/+spec/alarm-onnotification
Neutron Port Status Update
Note: This document represents a Neutron RFE reviewed in the Doctor project before submitting upstream to Launchpad Neutron space. The document is not intended to follow a blueprint format or to be an extensive document. For
more information, please visit http://docs.openstack.org/developer/neutron/policies/blueprints.html
The
RFE
was
submitted
to
Neutron.
https://bugs.launchpad.net/neutron/+bug/1598081
You
can
follow
the
discussions
in
Neutron port status field represents the current status of a port in the cloud infrastructure. The field can take one of the
following values: ‘ACTIVE’, ‘DOWN’, ‘BUILD’ and ‘ERROR’.
At present, if a network event occurs in the data-plane (e.g. virtual or physical switch fails or one of its ports, cable
gets pulled unintentionally, infrastructure topology changes, etc.), connectivity to logical ports may be affected and
tenants’ services interrupted. When tenants/cloud administrators are looking up their resources’ status (e.g. Nova
instances and services running in them, network ports, etc.), they will wrongly see everything looks fine. The problem
is that Neutron will continue reporting port ‘status’ as ‘ACTIVE’.
Many SDN Controllers managing network elements have the ability to detect and report network events to upper
layers. This allows SDN Controllers’ users to be notified of changes and react accordingly. Such information could be
consumed by Neutron so that Neutron could update the ‘status’ field of those logical ports, and additionally generate
a notification message to the message bus.
However, Neutron misses a way to be able to receive such information through e.g. ML2 driver or the REST API
(‘status’ field is read-only). There are pros and cons on both of these approaches as well as other possible approaches.
This RFE intends to trigger a discussion on how Neutron could be improved to receive fault/change events from SDN
Controllers or even also from 3rd parties not in charge of controlling the network (e.g. monitoring systems, human
admins).
462
Chapter 3. Developer
OPNFV Documentation, Release Danube
Port data plane status
https://bugs.launchpad.net/neutron/+bug/1598081
Neutron does not detect data plane failures affecting its logical resources. This spec addresses that issue by means of
allowing external tools to report to Neutron about faults in the data plane that are affecting the ports. A new REST
API field is proposed to that end.
Problem Description An initial description of the problem was introduced in bug #159801 [1]. This spec focuses
on capturing one (main) part of the problem there described, i.e. extending Neutron’s REST API to cover the scenario
of allowing external tools to report network failures to Neutron. Out of scope of this spec are works to enable port
status changes to be received and managed by mechanism drivers.
This spec also tries to address bug #1575146 [2]. Specifically, and argued by the Neutron driver team in [3]:
• Neutron should not shut down the port completly upon detection of physnet failure; connectivity between instances on the same node may still be reachable. Externals tools may or may not want to trigger a status change
on the port based on their own logic and orchestration.
• Port down is not detected when an uplink of a switch is down;
• The physnet bridge may have multiple physical interfaces plugged; shutting down the logical port may not be
needed in case network redundancy is in place.
Proposed Change A couple of possible approaches were proposed in [1] (comment #3). This spec proposes tackling
the problema via a new extension API to the port resource. The extension adds a new attribute ‘dp-down’ (data plane
down) to represent the status of the data plane. The field should be read-only by tenants and read-write by admins.
Neutron should send out an event to the message bus upon toggling the data plane status value. The event is relevant
for e.g. auditing.
Data Model Impact A new attribute as extension will be added to the ‘ports’ table.
Attribute Name
Type
Access
Default Value
Validation/ Conversion
dp_down
boolean
RO, tenant RW,
admin
False
True/False
Description
REST API Impact A new API extension to the ports resource is going to be introduced.
EXTENDED_ATTRIBUTES_2_0 = {
'ports': {
'dp_down': {'allow_post': False, 'allow_put': True,
'default': False, 'convert_to': convert_to_boolean,
'is_visible': True},
},
}
Examples Updating port data plane status to down:
PUT /v2.0/ports/<port-uuid>
Accept: application/json
{
"port": {
"dp_down": true
3.2. OPNFV Projects
463
OPNFV Documentation, Release Danube
}
}
Command Line Client Impact
neutron port-update [--dp-down <True/False>] <port>
openstack port set [--dp-down <True/False>] <port>
Argument –dp-down is optional. Defaults to False.
Security Impact None
Notifications Impact A notification (event) upon toggling the data plane status (i.e. ‘dp-down’ attribute) value
should be sent to the message bus. Such events do not happen with high frequency and thus no negative impact on the
notification bus is expected.
Performance Impact None
IPv6 Impact None
Other Deployer Impact None
Developer Impact None
Implementation
Assignee(s)
• cgoncalves
Work Items
• New ‘dp-down’ attribute in ‘ports’ database table
• API extension to introduce new field to port
• Client changes to allow for data plane status (i.e. ‘dp-down’ attribute’) being set
• Policy (tenants read-only; admins read-write)
Documentation Impact Documentation for both administrators and end users will have to be contemplated. Administrators will need to know how to set/unset the data plane status field.
References
464
Chapter 3. Developer
OPNFV Documentation, Release Danube
Inspector Design Guideline
Note: This is spec draft of design guideline for inspector component. JIRA ticket to track the update and collect
comments: DOCTOR-73.
This document summarize the best practise in designing a high performance inspector to meet the requirements in
OPNFV Doctor project.
Problem Description Some pitfalls has be detected during the development of sample inspector, e.g. we suffered a
significant performance degrading in listing VMs in a host.
A patch set for caching the list has been committed to solve issue. When a new inspector is integrated, it would be
nice to have an evaluation of existing design and give recommendations for improvements.
This document can be treated as a source of related blueprints in inspector projects.
Guidelines
Host specific VMs list TBD, see DOCTOR-76.
Parallel execution TBD, see discussion in mailing list.
Performance Profiler
https://goo.gl/98Osig
This blueprint proposes to create a performance profiler for doctor scenarios.
Problem Description
such as
In the verification job for notification time, we have encountered some performance issues,
1. In environment deployed by APEX, it meets the criteria while in the one by Fuel, the performance is much more
poor. 2. Signification performance degradation was spotted when we increase the total number of VMs
It takes time to dig the log and analyse the reason. People have to collect timestamp at each checkpoints manually to
find out the bottleneck. A performance profiler will make this process automatic.
Proposed Change Current Doctor scenario covers the inspector and notifier in the whole fault management cycle:
start
end
+
+
+
+
+
+
|
|
|
|
|
|
|monitor|inspector|notifier|manager|controller|
+------>+
|
|
|
|
occurred +-------->+
|
|
|
|
detected
+------->+
|
|
|
|
identified
+-------+
|
|
|
notified
+--------->+
|
|
|
processed resolved
|
|
|
|
|
+<-----doctor----->+
|
|
|
3.2. OPNFV Projects
465
OPNFV Documentation, Release Danube
|
|
+<---------------fault management------------>+
The notification time can be split into several parts and visualized as a timeline:
start
end
0----5---10---15---20---25---30---35---40---45--> (x 10ms)
+
+
+
+
+
+
+
+
+
+
+
0-hostdown |
|
|
|
|
|
|
|
|
+--->+
|
|
|
|
|
|
|
|
|
| 1-raw failure |
|
|
|
|
|
|
|
+-->+
|
|
|
|
|
|
|
|
|
| 2-found affected
|
|
|
|
|
|
|
+-->+
|
|
|
|
|
|
|
|
|
3-marked host down|
|
|
|
|
|
|
+-->+
|
|
|
|
|
|
|
|
4-set VM error|
|
|
|
|
|
|
+--->+
|
|
|
|
|
|
|
| 5-notified VM error |
|
|
|
|
+----->|
|
|
|
|
|
|
|
|
6-transformed event
|
|
|
|
+-->+
|
|
|
|
|
|
|
| 7-evaluated event
|
|
|
|
|
+-->+
|
|
|
|
|
|
|
8-fired alarm
|
|
|
|
|
+-->+
|
|
|
|
|
|
9-received alarm
|
|
|
|
|
+-->+
sample | sample
|
|
|
|10-handled alarm
monitor| inspector |nova| c/m |
aodh
|
|
|
+<-----------------doctor--------------->+
Note: c/m = ceilometer
And a table of components sorted by time cost from most to least
Component
inspector
aodh
monitor
...
...
Time Cost
160ms
110ms
50ms
Percentage
40%
30%
14%
Note: data in the table is for demonstration only, not actual measurement
Timestamps can be collected from various sources
1. log files
2. trace point in code
The performance profiler will be integrated into the verification job to provide detail result of the test. It can also be
deployed independently to diagnose performance issue in specified environment.
Working Items
1. PoC with limited checkpoints
2. Integration with verification job
466
Chapter 3. Developer
OPNFV Documentation, Release Danube
3. Collect timestamp at all checkpoints
4. Display the profiling result in console
5. Report the profiling result to test database
6. Independent package which can be installed to specified environment
Manuals
OpenStack NOVA API for marking host down.
Related Blueprints:
https://blueprints.launchpad.net/nova/+spec/mark-host-down
novaclient/+spec/support-force-down-service
https://blueprints.launchpad.net/python-
What the API is for
This API will give external fault monitoring system a possibility of telling OpenStack Nova fast that
compute host is down. This will immediately enable calling of evacuation of any VM on host and further
enabling faster HA actions.
What this API does
In OpenStack the nova-compute service state can represent the compute host state and this new API
is used to force this service down. It is assumed that the one calling this API has made sure the host
is also fenced or powered down. This is important, so there is no chance same VM instance will appear twice in case evacuated to new compute host. When host is recovered by any means, the external system is responsible of calling the API again to disable forced_down flag and let the host novacompute service report again host being up. If network fenced host come up again it should not boot
VMs it had if figuring out they are evacuated to other compute host. The decision of deleting or booting VMs there used to be on host should be enhanced later to be more reliable by Nova blueprint:
https://blueprints.launchpad.net/nova/+spec/robustify-evacuate
REST API for forcing down:
Parameter explanations: tenant_id: Identifier of the tenant. binary: Compute service binary name. host:
Compute host name. forced_down: Compute service forced down flag. token: Token received after
successful authentication. service_host_ip: Serving controller node ip.
request: PUT /v2.1/{tenant_id}/os-services/force-down { “binary”: “nova-compute”, “host”: “compute1”, “forced_down”: true }
response: 200 OK { “service”: { “host”: “compute1”, “binary”: “nova-compute”, “forced_down”: true }
}
Example: curl -g -i -X PUT http://{service_host_ip}:8774/v2.1/{tenant_id}/os-services /force-down
-H “Content-Type: application/json” -H “Accept: application/json ” -H “X-OpenStack-Nova-APIVersion: 2.11” -H “X-Auth-Token: {token}” -d ‘{“b inary”: “nova-compute”, “host”: “compute1”,
“forced_down”: true}’
3.2. OPNFV Projects
467
OPNFV Documentation, Release Danube
CLI for forcing down:
nova service-force-down <hostname> nova-compute
Example: nova service-force-down compute1 nova-compute
REST API for disabling forced down:
Parameter explanations: tenant_id: Identifier of the tenant. binary: Compute service binary name. host:
Compute host name. forced_down: Compute service forced down flag. token: Token received after
successful authentication. service_host_ip: Serving controller node ip.
request: PUT /v2.1/{tenant_id}/os-services/force-down { “binary”: “nova-compute”, “host”: “compute1”, “forced_down”: false }
response: 200 OK { “service”: { “host”: “compute1”, “binary”: “nova-compute”, “forced_down”: false
}}
Example: curl -g -i -X PUT http://{service_host_ip}:8774/v2.1/{tenant_id}/os-services /force-down
-H “Content-Type: application/json” -H “Accept: application/json ” -H “X-OpenStack-Nova-APIVersion: 2.11” -H “X-Auth-Token: {token}” -d ‘{“b inary”: “nova-compute”, “host”: “compute1”,
“forced_down”: false}’
CLI for disabling forced down:
nova service-force-down –unset <hostname> nova-compute
Example: nova service-force-down –unset compute1 nova-compute
Get valid server state
Related Blueprints: https://blueprints.launchpad.net/nova/+spec/get-valid-server-state
Problem description Previously when the owner of a VM has queried his VMs, he has not received enough state
information, states have not changed fast enough in the VIM and they have not been accurate in some scenarios. With
this change this gap is now closed.
A typical case is that, in case of a fault of a host, the user of a high availability service running on top of that host,
needs to make an immediate switch over from the faulty host to an active standby host. Now, if the compute host is
forced down [1] as a result of that fault, the user has to be notified about this state change such that the user can react
accordingly. Similarly, a change of the host state to “maintenance” should also be notified to the users.
What is changed A new host_status parameter is added to the /servers/{server_id} and
/servers/detail endpoints in microversion 2.16. By this new parameter user can get additional state information about the host.
host_status possible values where next value in list can override the previous:
• UP if nova-compute is up.
• UNKNOWN if nova-compute status was not reported by servicegroup driver within configured time period. Default is within 60 seconds, but can be changed with service_down_time in nova.conf.
• DOWN if nova-compute was forced down.
• MAINTENANCE if nova-compute was disabled. MAINTENANCE in API directly means nova-compute service
is disabled. Different wording is used to avoid the impression that the whole host is down, as only scheduling
of new VMs is disabled.
468
Chapter 3. Developer
OPNFV Documentation, Release Danube
• Empty string indicates there is no host for server.
host_status is returned in the response in case the policy permits. By default the policy is for admin only in Nova
policy.json:
"os_compute_api:servers:show:host_status": "rule:admin_api"
For an NFV use case this has to also be enabled for the owner of the VM:
"os_compute_api:servers:show:host_status": "rule:admin_or_owner"
REST API examples: Case where nova-compute is enabled and reporting normally:
GET /v2.1/{tenant_id}/servers/{server_id}
200 OK
{
"server": {
"host_status": "UP",
...
}
}
Case where nova-compute is enabled, but not reporting normally:
GET /v2.1/{tenant_id}/servers/{server_id}
200 OK
{
"server": {
"host_status": "UNKNOWN",
...
}
}
Case where nova-compute is enabled, but forced_down:
GET /v2.1/{tenant_id}/servers/{server_id}
200 OK
{
"server": {
"host_status": "DOWN",
...
}
}
Case where nova-compute is disabled:
GET /v2.1/{tenant_id}/servers/{server_id}
200 OK
{
"server": {
"host_status": "MAINTENANCE",
...
}
}
Host Status is also visible in python-novaclient:
3.2. OPNFV Projects
469
OPNFV Documentation, Release Danube
+-------+------+--------+------------+-------------+----------+-------------+
| ID
| Name | Status | Task State | Power State | Networks | Host Status |
+-------+------+--------+------------+-------------+----------+-------------+
| 9a... | vm1 | ACTIVE | | RUNNING
| xnet=... | UP
|
+-------+------+--------+------------+-------------+----------+-------------+
Links: [1]
Manual
for
OpenStack
NOVA
API
http://artifacts.opnfv.org/doctor/docs/manuals/mark-host-down_manual.html
for
marking
host
down
[2] OpenStack compute manual page http://developer.openstack.org/api-ref-compute-v2.1.html#compute-v2.1
Domino
Domino Project Overview
Domino Description
Domino provides a distribution service for Network Service Descriptors (NSDs) and Virtual Network Function Descriptors (VNFDs) that are composed using Tosca Simple Profile for Network Functions Virtualization
(http://docs.oasis-open.org/tosca/tosca-nfv/v1.0/tosca-nfv-v1.0.html). Domino service is targeted towards supporting
many Software Defined Network (SDN) controllers, Service Orchestrators (SOs), VNF Managers (VNFMs), Virtual
Infastructure Managers (VIMs), Operation and Business Support Systems that produce and/or consume NSDs and
VNFDs.
Producers of NSDs and VNFDs use Domino Service through Service Access Points (SAPs) or End Points (EPs) to
publish these descriptors. Consumers of NSDs and VNFDs subscribe with the Domino Service through the same
SAPs/EPs and declare their resource capabilities to onboard and perform Life Cycle Management (LCM) for Network
Services (NSs) and Virtual Network Functions (VNFs). Thus, Domino acts as a service broker for NSs and VNFs
modeled in a Tosca template.
Domino Capabilities and Usage
Labels in Domino Domino’s pub/sub architecture is based on labels (see Fig. 1 below). Each Template Producer
and Template Consumer is expected to run a local Domino Client to publish templates and subscribe for labels.
Domino Service does not interpret what the labels mean. Domino derives labels directly from the normative definitions
in TOSCA Simple YAML Profile for NFV. Domino parses the policy rules included in the NSD/VNFD, form “policy”
labels, and determine which resources are associated with which set of labels. Domino identifies which Domino
Clients can host which resource based on the label subscriptions by these clients. Once mapping of resources to the
clients are done, new NSDs/VNFDs are created based on the mapping. These new NSDs/VNFDs are translated and
delivered to the clients.
Label Format and Examples Domino supports policy labels in the following form:
<policytype>:properties:<key:value>
Orchestrators, controllers, and managers use Domino service to announce their capabilities by defining labels in this
form and subscribing for these labels with the Domino Server.
For instance a particular VIM that is capable of performing an affinity based VNF or VDU placement at host machine
granularity can specify a label in the form:
470
Chapter 3. Developer
OPNFV Documentation, Release Danube
Fig. 3.23: Domino provides a pub/sub server for NSDs and VNFDs
3.2. OPNFV Projects
471
OPNFV Documentation, Release Danube
tosca.policies.Placement.affinity:properties:granularity:hostlevel
When the VIM registers with the Domino Service and subscribed for that label, Domino views this VIM as a candidate
location that can host a VNF or VDU requesting affinity based placement policy at host machine granularity.
Another use case is the announcement of lifecycle management capabilities for VNFs and VNF Forwarding Graphs
(VNFFG) by different SDN Controllers (SDN-Cs), VNFMs, or VIMs. For instance
tosca.policies.Scaling.VNFFG:properties:session_continuity:true
can be used as a label to indicate that when a scaling operation on a VNFFG (e.g., add more VNFs into the graph) is
requested, existing session can still be enforced to go through the same chain of VNF instances.
To utilize Domino’s domain mapping services for virtual network resources (e.g., VNF, VDU, VNFFG, CP, VL, etc.),
a network service or network function request must include policy rules that are composed of policy types and property
values that match to the label announcements made by these domains. For instance, when a TOSCA template includes
a policy rule with type “tosca.policies.Scaling.VNFFG” and property field “session_continuity” set as “true” targeting
one or more VNFFGs, this serves as the hint for the Domino Server to identify all the Domain Clients that subscribed
the label “tosca.policies.Scaling.VNFFG:properties:session_continuity:true”.
Template Example for Label Extraction Consider the following NSD TOSCA template:
tosca_definitions_version: tosca_simple_profile_for_nfv_1_0_0
description: Template for deploying a single server with predefined properties.
metadata:
template_name: TOSCA NFV Sample Template
policy_types:
tosca.policies.Placement.Geolocation:
description: Geolocation policy
derived_from: tosca.policies.Placement
topology_template:
node_templates:
VNF1:
type: tosca.nodes.nfv.VNF
properties:
id: vnf1
vendor: acmetelco
version: 1.0
VNF2:
type: tosca.nodes.nfv.VNF
properties:
id: vnf2
vendor: ericsson
version: 1.0
VNF3:
type: tosca.nodes.nfv.VNF
properties:
id: vnf3
vendor: huawei
version: 1.0
policies:
- rule1:
type: tosca.policies.Placement.Geolocation
targets: [ VNF1 ]
properties:
region: [ us-west-1 ]
- rule2:
type: tosca.policies.Placement.Geolocation
472
Chapter 3. Developer
OPNFV Documentation, Release Danube
targets: [ VNF2, VNF3 ]
properties:
region: [ us-west-1 , us-west-2 ]
Domino Server extracts all possible policy labels by exhaustively concatenating key-value pairs under the properties
section of the policy rules to the policy type of these rules:
tosca.policies.Placement.Geolocation:properties:region:us-west-1
tosca.policies.Placement.Geolocation:properties:region:us-west-2
Furthermore, Domino Server iterates over the targets specified under policy rules to generate a set of labels for each
target node:
required_labels['VNF1'] = { tosca.policies.Placement.Geolocation:properties:region:us-west-1 }
required_labels['VNF2'] = { tosca.policies.Placement.Geolocation:properties:region:us-west-1 , tosca.
required_labels['VNF3'] = { tosca.policies.Placement.Geolocation:properties:region:us-west-1 , tosca.
When a Template Consuming site (e.g., VNFM or VIM) registers with the Domino Server using Domino Client, it
becomes an eligible candidate for template distribution with an initially empty set of label subscriptions. Suppose
three different Domino Clients register with the Domino Server and subscribe for some or none of the policy labels
such that the Domino Server has the current subscription state as follows:
subscribed_labels[site-1] = { } #this is empty set
subscribed_labels[site-2] = { tosca.policies.Placement.Geolocation:properties:region:us-west-1 }
subscribed_labels[site-3] = { tosca.policies.Placement.Geolocation:properties:region:us-west-1 ,
Based on the TOSCA example and hypothetical label subscriptions above, Domino Server identifies all the VNFs can
be hosted by Site-3, while VNF1 can be hosted by both Site-2 and Site-3. Note that Site-1 cannot host any of the
VNFs listed in the TOSCA file. When a VNF can be hosted by multiple sites, Domino Server picks the site that can
host the most number of VNFs. When not all VNFs can be hosted on the same site, the TOSCA file is partitioned into
multiple files, one for each site. These files share a common part (e.g, meta-data, policy-types, version, description,
virtual resources that are not targeted by any policy rule, etc.). Each site specific file has also a non-common part that
only appears in that file (i.e., virtual resources explicitly assigned to that site and the policy rules that accompany those
virtual resources.
In the current Domino convention, if a VNF (or any virtual resource) does not have a policy rule (i.e., it is not specified
as a target in any of the policy rules) and it also is not dependent on any VNF (or any virtual resource) that is assigned to
another site, that resource is wild carded by default and treated as part of the “common part”. Also note that currently
Domino does not support all or nothing semantics: if some of the virtual resources are not mappable to any domain
because they are targets of policy rules that are not supported by any site, these portions will be excluded while the
remaining virtual resources will be still be part of one or more template files to be distributed to hosting sites. When
NSDs and VNFDs are prepared, these conventions must be kept in mind. In the future releases, these conventions can
change based on the new use cases.
For the example above, no partitioning would occur as all VNFs are mapped onto site-3; Domino Server simply
delivers the Tosca file to Domino Client hosted on site-3. When TOSCA cannot be consumed by a particular site
directly, Domino Server can utilize existing translators (e.g., heat-translator) to first translate the template before
delivery.
Internal Processing Pipeline at Domino Server Fig. 2 shows the block diagram for the processing stages of a
published TOSCA template. Domino Client issues an RPC call publish(tosca file). Domino Server passes the received
tosca file to Label Extractor that outputs resource labels. Domain Mapper uses the extracted labels and tosca file
to find mappings from resources to domains as well as the resource dependencies. Resource to domain mappings
and resource dependencies are utilized to partition the orchestration template into individual resource orchestration
templates (one for each domain). If a translation is required (e.g., TOSCA to HOT), individual resource orchestration
templates are first translated and then placed on a template distribution workflow based on resource dependencies.
3.2. OPNFV Projects
473
tos
OPNFV Documentation, Release Danube
Message Sender block in the server takes one distribution task at a time from the workflow generator and pushes the
orchestration template to the corresponding Domino Client.
Fig. 3.24: Domino Service Processing Pipeline
Resource Scheduling Domino Service currently supports maximum packing strategy when a virtual resource type
can be hosted on multiple candidate sites. Initially, Domino Scheduler identifies virtual resources that has only one
feasible site for hosting. Each such virtual resource is trivially assigned to its only feasible site. The remaining
virtual resources with multiple candidate locations are sequentially allocated to one of their candidate locations that
has the most virtual resource assignments so far. Note that wildcarded resources are assigned to all sites. To prevent
wildcarding within the current release, (i) all sites must subscribed to a base policy with a dummy key-value pair
defined under the properties tab and (ii) all the independent resources must be specified as target of that policy in NSD
or VNFD file.
Domino Installation Instruction
Domino Installation
Note: The steps below are tested for Ubuntu (16.04, 14.04) and OS X El Capitan.
Prerequisites
• git
474
Chapter 3. Developer
OPNFV Documentation, Release Danube
• python-pip
• python (version =2.7)
• tosca-parser (version >=0.4.0)
• heat-translator (version >=0.5.0)
Installation Steps (Single Node)
• Step-0: Prepare Environment
> $sudo pip install tosca-parser
> $sudo pip install heat-translator
> $sudo pip install requests
• Step-1: Get the Domino code
git clone https://gerrit.opnfv.org/gerrit/domino -b stable/danube
• Step-2: Go to the main domino directory
cd domino
You should see DominoClient.py, DominoServer.py, and domino-cli.py as executables.
Installation Steps (Multiple Node) Repeat the installation steps for single node on each node. The script
run_on_remotenodes.sh under ./domino/tests directory deploys the Domino Code on three hosts from a deployment
node and tests RPC calls. In the script, the private key location and remote host IP addresses must be manually entered.
IS_IPandKEY_CONFIGURED should be set true, i.e., IS_IPandKEY_CONFIGURED=true.
Domino Configuration Guide
Domino Configuration
Domino Server and Clients can be configured via (i) passing command line options (see API documentation) and (ii)
the configuration file “domino_conf.py” under the main directory.
• The default file for logging is set as none and log level set as “WARNING”.
Domino Server
• The default server unique user ID is set as 0 in the configuration file.
• The default TCP port for RPC calls is set as 9090 in the configuration file.
• The default database file for Domino Server is set as “dominoserver.db” under the main directory
• The default folder for keeping published TOSCA files and pushed parts is set as “toscafiles” in the configuration
file via variable TOSCADIR.
• The default log level (WARNING) can be changed by passing the flags –log or -l followed by a log level, e.g.,
ERROR, WARNING, INFO, or DEBUG.
3.2. OPNFV Projects
475
OPNFV Documentation, Release Danube
Domino Client
• The default mode of CLI is non-interactive (i.e., Domino CLI Utility is used). Passing –iac=TRUE would begin
the client in interactive mode.
• The default log level (WARNING) can be changed by passing the flags –log or -l followed by a log level, e.g.,
ERROR, WARNING, INFO, or DEBUG.
• The default Domino Server IP is set as “localhost”. This can be overwritten at the time of launching DominoClient via the option flags -i or –ipaddr followed by the IP address of the actual server hosting the Domino
Server.
• The default Domino Client TCP port for RPC calls is set as 9091 in the configuration file. It can be overwritten
when the DominoClient is launched by passing the flags –port or -p followed by the port number.
• The default folder for keeping preceived TOSCA files is set as “toscafiles” in the configuration file via variable
TOSCA_RX_DIR.
Domino User Guide
Domino API Usage Guidelines and Examples
Using domino-cli Client Prerequisites:
1. Make sure that domino-cli.py is in +x mode.
2. Change directory to where domino-cli.py, DominoClient.py and DominoServer.py are located or include file
path in the PATH environment variable.
3. Start the Domino Server:
./DominoServer.py --log=debug
4. Start the Domino Client:
./DominoClient.py -p <portnumber> --cliport <cli-portnumber> --log=debug
Note1: The default log level is WARNING and omitting –log option will lead to minimal/no logging on the console
Note2: domino_conf.py file includes most of the default values
• Registration Command
Command line input:
./domino-cli.py <cli-portnumber> register
This message has the following fields that are automatically filled in.
Message Type (= REGISTER)
DESIRED UDID (= if not allocated, this will be assigned as Unique Domino ID)
Sequence Number (=incremented after each RPC call)
IP ADDR (= IP address of DOMINO Client to be used by DOMINO Server for future RPC Calls to this clien
TCP PORT (= TCP port of DOMINO Client to be used by DOMINO Server for future RPC Calls to this client
Supported Templates (= Null, this field not used currently)
• Heart Beat Command
Command line input:
./domino-cli.py <cli-portnumber> heartbeat
476
Chapter 3. Developer
OPNFV Documentation, Release Danube
This message has the following fields that are automatically filled in.
Message Type (= HEART_BEAT)
UDID (= Unique Domino ID assigned during registration)
Sequence Number (=incremented after each RPC call)
• Label and Template Type Subscription Command
./domino-cli.py <cli-portnumber> subscribe -l <labelname> -t <templatetype>
Note that -l can be substituted by –label and -t can be substituted by –ttype.
More than one label or template type can be subscribed within the same command line as comma separated labels or
template types
./domino-cli.py <cli-portnumber> subscribe -l <label1>,<label2>,<labeln> -t <ttype1>,<ttype2>,<ttypen
To subscribe more than one label or template type, one can also repeat the options -l and -t, e.g.:
./domino-cli.py <cli-portnumber> subscribe -l <label1> -l <label2> -l <labeln> -t <ttype1> -t <ttype2
It is safe to call subscribe command multiple times with duplicate labels.
This message has the following fields that are automatically filled in.
Message Type (= SUBSCRIBE)
UDID (= Unique Domino ID assigned during registration)
Sequence Number (=incremented after each RPC call)
Template Operation (= APPEND)
Label Operation (= APPEND)
The following fields are filled in based on arguments passed on via -l/–label and -t/–ttype flags
Subscribe RPC also supports options for label using –lop=APPEND/DELETE/OVERWRITE
and for supported template types using –top=APPEND/DELETE/OVERWRITE.
When unspecified, the default is APPEND. DELETE deletes existing labels (template types) specified in the current
call via key -l/–label (-t/–ttype). OVERWRITE removes the current set of labels (template types) and sets it to the new
set of values passed in the same RPC call.
By default, no translation service is provided. Currently, only TOSCA to Heat Orchestration Template (HOT) translation is supported using OpenStack heat-translator library. A domain that requires HOT files must subscribe HOT
template type using
./domino-cli.py <cli-portnumber> subscribe -t hot
• Template Publishing Command
./domino-cli.py <cli-portnumber> publish -t <toscafile>
Note that -t can be substituted with –tosca-file.
If -t or –tosca-file flag is used multiple times, the last tosca file passed as input will be used. This usage is not
recommended as undefined/unintended results may emerge as the Domino client will continue to publish.
This message has the following fields that are automatically filled in.
Message Type (= PUBLISH)
UDID (= Unique Domino ID assigned during registration)
Sequence Number (=incremented after each RPC call)
Template Type (= TOSCA)
Template File
3.2. OPNFV Projects
477
OPNFV Documentation, Release Danube
Since Danube release, Domino Server supports stateful updates for template publishing. The following command can
be used to update the service template for an existing Template Unique ID (TUID):
./domino-cli.py <cli-portnumber> publish -t <toscafile> -k <TUID>
Note that -k can be substituted with –tuid. When Domino Server receives this command, it verifies whether the client
previously published the provided TUID. If such TUID does not exist, then Domino Server returns FAILED response
back to the client. If such TUID exists, Domino Server recomputes which resources are mapped onto which domains
and updates each domain with the new VNF and NS descriptors. If a previously utilized domain is no longer targeted,
it is updated with a null descriptor.
• Template Listing Command
./domino-cli.py <cli-portnumber> list-tuids
Queries all the Template Unique IDs (TUIDs) published by the Domino Client from the Domino Server.
Interactive CLI mode To enter this mode, start Domino Client with interactive console option set as true, i.e.,
–iac=true:
./DominoClient -p <portnumber> --iax=true --log=DEBUG
The rest of the API calls are the same as in the case of using domino-cli.py except that at the prompt there is no need
to write “domino-cli.py <cli-portnumber>, e.g.,:
>>register
>>heartbeat
>>subscribe -l <label1> -t <ttype1>
>>publish -t <toscafile>
The interactive CLI mode is mainly supported for manual testing.
IPV6
IPv6 Installation Procedure
Abstract
This document provides the users with the Installation Procedure to install OPNFV Danube Release on IPv6-only
Infrastructure.
Install OPNFV on IPv6-Only Infrastructure
This section provides instructions to install OPNFV on IPv6-only Infrastructure. All underlay networks and API
endpoints will be IPv6-only except:
1. “admin” network in underlay/undercloud still has to be IPv4, due to lack of support of IPMI over IPv6 or PXE
over IPv6.
2. OVS VxLAN (or GRE) tunnel endpoint is still IPv4 only, although IPv6 traffic can be encapsulated within the
tunnel.
3. Metadata server is still IPv4 only.
Except the limitations above, the use case scenario of the IPv6-only infrastructure includes:
1. Support OPNFV deployment on an IPv6 only infrastructure.
478
Chapter 3. Developer
OPNFV Documentation, Release Danube
2. Horizon/ODL-DLUX access using IPv6 address from an external host.
3. OpenStack API access using IPv6 addresses from various python-clients.
4. Ability to create Neutron Routers, IPv6 subnets (e.g. SLAAC/DHCPv6-Stateful/ DHCPv6-Stateless) to support
North-South traffic.
5. Inter VM communication (East-West routing) when VMs are spread across two compute nodes.
6. VNC access into a VM using IPv6 addresses.
Install OPNFV in OpenStack-Only Environment
Apex Installer:
# HA, Virtual deployment in OpenStack-only environment
./opnfv-deploy -v -d /etc/opnfv-apex/os-nosdn-nofeature-ha.yaml \
-n /etc/opnfv-apex/network_settings_v6.yaml
# HA, Bare Metal deployment in OpenStack-only environment
./opnfv-deploy -d /etc/opnfv-apex/os-nosdn-nofeature-ha.yaml \
-i <inventory file> -n /etc/opnfv-apex/network_settings_v6.yaml
# Non-HA, Virtual deployment in OpenStack-only environment
./opnfv-deploy -v -d /etc/opnfv-apex/os-nosdn-nofeature-noha.yaml \
-n /etc/opnfv-apex/network_settings_v6.yaml
# Non-HA, Bare Metal deployment in OpenStack-only environment
./opnfv-deploy -d /etc/opnfv-apex/os-nosdn-nofeature-noha.yaml \
-i <inventory file> -n /etc/opnfv-apex/network_settings_v6.yaml
#
#
#
#
#
#
Note:
1. Parameter ""-v" is mandatory for Virtual deployment
2. Parameter "-i <inventory file>" is mandatory for Bare Metal deployment
2.1 Refer to https://git.opnfv.org/cgit/apex/tree/config/inventory for examples of inventory file
3. You can use "-n /etc/opnfv-apex/network_settings.yaml" for deployment in IPv4 infrastructure
Please NOTE that:
• You need to refer to installer’s documentation for other necessary parameters applicable to your deployment.
• You need to refer to Release Notes and installer’s documentation if there is any issue in installation.
Install OPNFV in OpenStack with ODL-L3 Environment
Apex Installer:
# HA, Virtual deployment in OpenStack with Open Daylight L3 environment
./opnfv-deploy -v -d /etc/opnfv-apex/os-odl_l3-nofeature-ha.yaml \
-n /etc/opnfv-apex/network_settings_v6.yaml
# HA, Bare Metal deployment in OpenStack with Open Daylight L3 environment
./opnfv-deploy -d /etc/opnfv-apex/os-odl_l3-nofeature-ha.yaml \
-i <inventory file> -n /etc/opnfv-apex/network_settings_v6.yaml
# Non-HA, Virtual deployment in OpenStack with Open Daylight L3 environment
./opnfv-deploy -v -d /etc/opnfv-apex/os-odl_l3-nofeature-noha.yaml \
-n /etc/opnfv-apex/network_settings_v6.yaml
# Non-HA, Bare Metal deployment in OpenStack with Open Daylight L3 environment
./opnfv-deploy -d /etc/opnfv-apex/os-odl_l3-nofeature-noha.yaml \
-i <inventory file> -n /etc/opnfv-apex/network_settings_v6.yaml
3.2. OPNFV Projects
479
OPNFV Documentation, Release Danube
#
#
#
#
#
#
Note:
1. Parameter ""-v" is mandatory for Virtual deployment
2. Parameter "-i <inventory file>" is mandatory for Bare Metal deployment
2.1 Refer to https://git.opnfv.org/cgit/apex/tree/config/inventory for examples of inventory file
3. You can use "-n /etc/opnfv-apex/network_settings.yaml" for deployment in IPv4 infrastructure
Please NOTE that:
• You need to refer to installer’s documentation for other necessary parameters applicable to your deployment.
• You need to refer to Release Notes and installer’s documentation if there is any issue in installation.
Testing Methodology There are 2 levels of testing to validate the deployment.
Underlay Testing for OpenStack API Endpoints Underlay Testing is to validate that API endpoints are listening
on IPv6 addresses. Currently, we are only considering the Underlay Testing for OpenStack API endpoints. The
Underlay Testing for Open Daylight API endpoints is for future release.
The Underlay Testing for OpenStack API endpoints can be as simple as validating Keystone service, and as complete
as validating each API endpoint. It is important to reuse Tempest API testing. Currently:
• Apex Installer will change OS_AUTH_URL in overcloudrc during installation process. For example:
export OS_AUTH_URL=http://[2001:db8::15]:5000/v2.0. OS_AUTH_URL points to Keystone and Keystone catalog.
• When FuncTest runs Tempest for the first time, the OS_AUTH_URL is taken from the environment and placed
automatically in Tempest.conf.
• Under this circumstance, openstack catalog list will return IPv6 URL endpoints for all the services
in catalog, including Nova, Neutron, etc, and covering public URLs, private URLs and admin URLs.
• Thus, as long as the IPv6 URL is given in the overclourc, all the tests will use that (including Tempest).
Therefore Tempest API testing is reused to validate API endpoints are listening on IPv6 addresses as stated above.
They are part of OpenStack default Smoke Tests, run in FuncTest and integrated into OPNFV’s CI/CD environment.
Overlay Testing Overlay Testing is to validate that IPv6 is supported in tenant networks, subnets and routers. Both
Tempest API testing and Tempest Scenario testing are used in our Overlay Testing.
Tempest API testing validates that the Neutron API supports the creation of IPv6 networks, subnets, routers, etc:
tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_network
tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_port
tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_subnet
tempest.api.network.test_networks.NetworksIpV6Test.test_create_update_delete_network_subnet
tempest.api.network.test_networks.NetworksIpV6Test.test_external_network_visibility
tempest.api.network.test_networks.NetworksIpV6Test.test_list_networks
tempest.api.network.test_networks.NetworksIpV6Test.test_list_subnets
tempest.api.network.test_networks.NetworksIpV6Test.test_show_network
tempest.api.network.test_networks.NetworksIpV6Test.test_show_subnet
tempest.api.network.test_networks.NetworksIpV6TestAttrs.test_create_update_delete_network_subnet
tempest.api.network.test_networks.NetworksIpV6TestAttrs.test_external_network_visibility
tempest.api.network.test_networks.NetworksIpV6TestAttrs.test_list_networks
tempest.api.network.test_networks.NetworksIpV6TestAttrs.test_list_subnets
tempest.api.network.test_networks.NetworksIpV6TestAttrs.test_show_network
tempest.api.network.test_networks.NetworksIpV6TestAttrs.test_show_subnet
tempest.api.network.test_ports.PortsIpV6TestJSON.test_create_port_in_allowed_allocation_pools
480
Chapter 3. Developer
OPNFV Documentation, Release Danube
tempest.api.network.test_ports.PortsIpV6TestJSON.test_create_port_with_no_securitygroups
tempest.api.network.test_ports.PortsIpV6TestJSON.test_create_update_delete_port
tempest.api.network.test_ports.PortsIpV6TestJSON.test_list_ports
tempest.api.network.test_ports.PortsIpV6TestJSON.test_show_port
tempest.api.network.test_routers.RoutersIpV6Test.test_add_multiple_router_interfaces
tempest.api.network.test_routers.RoutersIpV6Test.test_add_remove_router_interface_with_port_id
tempest.api.network.test_routers.RoutersIpV6Test.test_add_remove_router_interface_with_subnet_id
tempest.api.network.test_routers.RoutersIpV6Test.test_create_show_list_update_delete_router
tempest.api.network.test_security_groups.SecGroupIPv6Test.test_create_list_update_show_delete_securit
tempest.api.network.test_security_groups.SecGroupIPv6Test.test_create_show_delete_security_group_rule
tempest.api.network.test_security_groups.SecGroupIPv6Test.test_list_security_groups
Tempest Scenario testing validates some specific overlay IPv6 scenarios (i.e. use cases) as follows:
tempest.scenario.test_network_v6.TestGettingAddress.test_dhcp6_stateless_from_os
tempest.scenario.test_network_v6.TestGettingAddress.test_dualnet_dhcp6_stateless_from_os
tempest.scenario.test_network_v6.TestGettingAddress.test_dualnet_multi_prefix_dhcpv6_stateless
tempest.scenario.test_network_v6.TestGettingAddress.test_dualnet_multi_prefix_slaac
tempest.scenario.test_network_v6.TestGettingAddress.test_dualnet_slaac_from_os
tempest.scenario.test_network_v6.TestGettingAddress.test_multi_prefix_dhcpv6_stateless
tempest.scenario.test_network_v6.TestGettingAddress.test_multi_prefix_slaac
tempest.scenario.test_network_v6.TestGettingAddress.test_slaac_from_os
The above Tempest API testing and Scenario testing are quite comprehensive to validate overlay IPv6 tenant networks.
They are part of OpenStack default Smoke Tests, run in FuncTest and integrated into OPNFV’s CI/CD environment.
IPv6 Configuration Guide
Abstract
This document provides the users with the Configuration Guide to set up a service VM as an IPv6 vRouter using
OPNFV Danube Release.
IPv6 Configuration - Setting Up a Service VM as an IPv6 vRouter
This section provides instructions to set up a service VM as an IPv6 vRouter using OPNFV Danube Release installers.
The environment may be pure OpenStack option or Open Daylight L2-only option. The deployment model may be
HA or non-HA. The infrastructure may be bare metal or virtual environment.
Pre-configuration Activities The configuration will work in 2 environments:
1. OpenStack-only environment
2. OpenStack with Open Daylight L2-only environment
Depending on which installer will be used to deploy OPNFV, each environment may be deployed on bare metal or
virtualized infrastructure. Each deployment may be HA or non-HA.
Refer to the previous installer configuration chapters, installations guide and release notes.
Setup Manual in OpenStack-Only Environment If you intend to set up a service VM as an IPv6 vRouter in
OpenStack-only environment of OPNFV Danube Release, please NOTE that:
• Because the anti-spoofing rules of Security Group feature in OpenStack prevents a VM from forwarding packets,
we need to disable Security Group feature in the OpenStack-only environment.
3.2. OPNFV Projects
481
OPNFV Documentation, Release Danube
• The hostnames, IP addresses, and username are for exemplary purpose in instructions. Please change as needed
to fit your environment.
• The instructions apply to both deployment model of single controller node and HA (High Availability) deployment model where multiple controller nodes are used.
Install OPNFV and Preparation
NFV Danube Release:
OPNFV-NATIVE-INSTALL-1: To install OpenStack-only environment of OP-
A