Cisco Elastic Services Controller Install and Upgrade Guide
Oglas
Oglas
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
First Published: 2020-09-25
Last Modified: 2020-10-15
Last Modified: 2021-06-22
Americas Headquarters
Cisco Systems, Inc.
170 West Tasman Drive
San Jose, CA 95134-1706
USA http://www.cisco.com
Tel: 408 526-4000
800 553-NETS (6387)
Fax: 408 527-0883
THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS,
INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.
THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH
THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY,
CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.
The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB's public domain version of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California.
NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS" WITH ALL FAULTS.
CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE.
IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT
LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS
HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone numbers. Any examples, command display output, network topology diagrams, and other figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses or phone numbers in illustrative content is unintentional and coincidental.
Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To view a list of Cisco trademarks, go to this URL: http://www.cisco.com/go/trademarks . Third-party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (1110R)
© 2020 Cisco Systems, Inc. All rights reserved.
C O N T E N T S
P R E F A C E
C H A P T E R 1
P A R T I
C H A P T E R 2
C H A P T E R 3
Elastic Services Controller Overview 1
Elastic Services Controller Overview 1
Installing Cisco Elastic Services Controller on OpenStack 3
Virtual Resource Requirements 5
Preparing for the Installation 6
Installing Cisco Elastic Services Controller on OpenStack 7
Main Components of Cisco Elastic Services Controller Setup 8
Installing Cisco Elastic Services Controller Using the QCOW Image 8
Additional Installation Options 11
Managing Root Certificates in Cisco Elastic Services Controller 15
Enabling/Disabling the Root Certificate Validation 15
Removing a Root Certificate 16
Managing Root Certificates During the Upgrade 17
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide iii
Contents
C H A P T E R 4
C H A P T E R 5
C H A P T E R 6
C H A P T E R 7
C H A P T E R 8
Managing Keystore in Cisco Elastic Services Controller 17
Using a Bootable Volume in ESC Installation 19
Installing High Availability Active/Standby 21
High Availability Active/Standby Overview 21
ESC Active/Standby Architecture 22
How High Availability Works 22
Deploying ESC High Availability Active/Standby 23
Deploying ESC in High Availability Active/Standby Mode on Internal Storage 24
Deploying ESC in High Availability Active/Standby Mode on Replicate External Storage 25
Configuring the Northbound Interface Access 26
Configuring ESC HA Active/Standby with Multiple Interfaces 26
Configuring the ESC HA Active/Standby Virtual IP Address 27
Configuring the ESC L3 HA Active/Standby With BGP 28
Troubleshooting High Availability Active/Standby 32
Cisco Elastic Services Controller Active/Active High Availability Overview 33
Cisco Elastic Services Controller Active/Active HA Overview 33
ESC Active/Active Architecture 34
Installing Active/Active High Availability Cluster 35
Installing Active/Active High Availability Cluster 35
Setting up the User Configuration 35
Validating Active/Active High Availability Cluster Post Installation 37
Adding Default VIM Connector to the Active/Active High Available Cluster 38
Adding BGP in an Active/Active Cluster 38
Managing Cluster in ESC Active/Active High Availability 41
Managing Cluster in ESC Active/Active High Availability 41
Configuring GEO in Active/Active High Availability 43
iv
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Contents
C H A P T E R 9
C H A P T E R 1 0
P A R T I I
C H A P T E R 1 1
C H A P T E R 1 2
C H A P T E R 1 3
Configuring GEO in Active/Active High Availability 43
Active/Active GEO HA Failure Injection Limitations 46
DRBD Encryption for ESC Active/Standby and Active/Active HA Data Replication 49
DRBD Encryption for ESC HA Data Replication 49
ESC HA with DRBD Encryption 49
Upgrading ESC Active/Active High Availability 51
Upgrading ESC Active/Active High Availability 51
Installing a New ESC Active/Active VM 52
Installing Cisco Elastic Services Controller on VMware vCenter 55
Virtual Resource and Hypervisor Requirements 57 vCenter Resources 57
Installing Cisco Elastic Services Controller on VMware vCenter 61
Installing Cisco Elastic Services Controller on VMware vCenter 61
Preparing to Install Cisco Elastic Services Controller 61
Installing the Elastic Services Controller Using the OVA Image 62
Installing Elastic Services Controller Using OVF Tool 64
Powering on Cisco Elastic Services Controller Virtual Machine 66
Next Steps: Cisco Elastic Services Controller Virtual Machine 67
Logging in to Cisco Elastic Services Controller Portal 67
Configuring the Virtual Machine to Automatically Power Up 67
Installing High Availability 69
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide v
Contents
P A R T I I I
C H A P T E R 1 4
P A R T I V
C H A P T E R 1 5
P A R T V
C H A P T E R 1 6
P A R T V I
High Availability Active/Standby Overview 69
How High Availability Active/Standby Works 70
Deploying ESC High Availability Active/Standby with User Data (HA Active/Standby Pair) 70
Deploying ESC High Availability Active/Standby (Standalone Instances) 74
Important Notes for ESC HA Active/Standby 75
Troubleshooting High Availability Active/Standby 75
Installing Cisco Elastic Services Controller on VMware vCloud Director 77
Installing Cisco Elastic Services Controller on VMware vCloud Director 79
Uploading the ESC OVA file into a Catalog in VMware vCloud Director 79
Deploying ESC vApp on vCD Web Client 79
Adding Self-Signed Certificate 80
Installing Cisco Elastic Services Controller on a Kernel-based Virtual Machine (KVM) 81
Installing Cisco Elastic Services Controller on a Kernel-based Virtual Machine 83
Installing Cisco Elastic Services Controller in a Kernel-based Virtual Machine 83
Preparing to Install Cisco Elastic Services Controller on a Kernel-based Virtual Machine 83
Installing Elastic Services Controller on a Kernel-Based Virtual Machine 84
Next Steps: Cisco Elastic Services Controller Kernel-based Virtual Machine 85
Logging in to Cisco Elastic Services Controller Portal 85
Verifying ESC installation for a Kernel-based Virtual Machine (KVM) 85
Installing Cisco Elastic Services Controller on Amazon Web Services (AWS) 87
Installing Cisco Elastic Services Controller on Amazon Web Services 89
Installing the Elastic Services Controller Instance in AWS 90
Installing Cisco Elastic Services Controller on Cisco Cloud Services Platform 2100 93
vi
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Contents
C H A P T E R 1 7
P A R T V I I
C H A P T E R 1 8
Installing Cisco Elastic Services Controller on Cisco Cloud Services Platform 2100 95
Installing the Elastic Services Controller Instance in CSP 2100 95
ESC HA Active/Standby Installation 100
List of Variables Used in CSP 2100 Sample Files 104
Changing the ConfD Netconf/CLI Administrator Password Using the Command Line Interface
Creating Readonly User Group for ConfD in ESC 110
Changing Linux Account Password 112
Changing the ESC Portal Password 112
Configuring Pluggable Authentication Module (PAM) Support for Cisco Elastic Services Controller
Adding PAM User to an ESC Service/Component 114
Configuring Cisco Elastic Services Controller as Identity Management Client 114
Configuring Cisco Elastic Services Controller as the Identity Policy and Audit Client 115
Authenticating REST Requests 116
Enabling ETSI REST Authentication 117
Changing the REST Interface Password 117
Changing the ETSI REST Interface Password 118
Sending an Authorized REST Request 118
Sending an Authorized ETSI REST Request 119
Configuring Openstack Credentials 119
Reconfiguring ESC Virtual Machine 125
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide vii
Contents
P A R T V I I I
C H A P T E R 1 9
C H A P T E R 2 0
Verifying ESC Configurations and Other Post-Install Operations 128
Logging in to the ESC Portal 130
Upgrading Cisco Elastic Services Controller 133
Setting ESC in a Maintenance Mode 135
Setting ESC in an Operation Mode 136
Backup the Database from the ESC Standalone Instances 136
Backup the Database from the ESC HA Active/Standby Instances 138
Upgrading Cisco Elastic Services Controller 141
Upgrading Standalone ESC Instance 142
Deploy the ESC for Upgrade 143
Restoring the ESC Database 143
Upgrading ESC HA Active/Standby Instances 144
Deploying the ESC HA Active/Standby nodes for Upgrade 144
Restoring the ESC Database on New Master and Standby ESC Instances 144
Upgrading VNF Monitoring Rules 146
In-Service Upgrade of the ESC HA Active/Standby Nodes in OpenStack 146
In-Service upgrade in OpenStack using ESC RPM packages 146
In-Service upgrade in OpenStack using ESC qcow2 Image 147
In-Service Upgrade of the ESC H AActive/Standby Nodes in Kernel-Based Virtual Machine (KVM)
In-Service Upgrade in KVM using ESC RPM packages 150
In-Service Upgrade in KVM using ESC OVA Image 151
In-Service Upgrade of the ESC HA Active/Standby Nodes in VMware 154
In-Service upgrade in VMware using ESC RPM packages 154
In-Service upgrade in VMware using ESC qcow2 Image 155
In-Service Upgrade of the ESC HA Active/Standby Nodes in CSP 158
viii
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
P A R T I X
C H A P T E R 2 1
A P P E N D I X A
A P P E N D I X B
Troubleshooting Cisco Elastic Services Controller Installation 161
Troubleshooting ESC Issues 163
General Installation Errors 168
Cisco Elastic Services Controller Installer Arguments 173
List of Variables Used in CSP 2100 Sample Files 185
Contents
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide ix
Contents x
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
About This Guide
This guide describes the installation requirements, procedures, and upgrade procedures for Cisco Elastic
Services Controller.
•
Audience
This guide is designed for network administrators responsible for provisioning, configuring, and monitoring
VNFs. Cisco Elastic Services Controller (ESC) and the VNFs whose lifecycle it manages are deployed in a
Virtual Infrastructure Manager (VIM). Currently OpenStack, VMware vCenter, VMware vCloud Director,
CSP 2100 / 5000, and Amazon Web Services (AWS) are the supported VIMs. The administrator must be familiar with the VIM layer, vCenter, OpenStack and AWS resources, and the commands used.
Cisco ESC is targeted for Service Providers (SPs) and Large Enterprises. ESC helps SPs reduce cost of operating the networks by providing effective and optimal resource usage. For Large Enterprises, ESC automates provisioning, configuring and monitoring of network functions.
Terms and Definitions
The below table defines the terms used in this guide.
Table 1: Terms and Definitions
Terms
AWS
ESC
ETSI
Definitions
Amazon Web Services (AWS) is a secure cloud services platform, offering compute, database storage, content delivery and other functionalities.
Elastic Services Controller (ESC) is a Virtual Network Function Manager (VNFM), performing lifecycle management of Virtual Network Functions.
European Telecommunications Standards Institute (ETSI) is an independent standardization organization that has been instrumental in developing standards for information and communications technologies (ICT) within Europe.
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide xi
About This Guide
About This Guide
Terms Definitions
ETSI
Deployment
Flavour
A deployment flavour definition contains information about affinity relationships, scaling, min/max VDU instances, and other policies and constraints to be applied to the VNF instance.
The deployment flavour defined in the VNF Descriptor (VNFD) must be selected by passing the flavour_id attribute in the InstantiateVNFRequest payload during the instantiate VNF
LCM operation.
HA
KPI
MSX
NFV
ESC High Availability (HA) is a solution for preventing single points of ESC failure and achieving minimum ESC downtime.
Key Performance Indicator (KPI) measures performance management. KPIs specify what, how and when parameters are measured. KPI incorporates information about source, definitions, measures, calculations for specific parameters.
Cisco Managed Services Accelerator (MSX) is a service creation and delivery platform that enables fast deployment of cloud-based networking services for both Enterprises and Service
Providers customers.
Network Function Virtualization (NFV) is the principle of separating network functions from the hardware they run on by using virtual hardware abstraction.
NFVO
NSO
OpenStack
Compute
Flavor
Service
VDU
VIM
VM
VNF
NFV Orchestrator (NFVO) is a functional block that manages the Network Service (NS) lifecycle and coordinates the management of NS lifecycle, VNF lifecycle (supported by the
VNFM) and NFVI resources (supported by the VIM) to ensure an optimized allocation of the necessary resources and connectivity.
Cisco Network Services Orchestrator (NSO) is an orchestrator for service activation which supports pure physical networks, hybrid networks (physical and virtual) and NFV use cases.
Flavors define the compute, memory, and storage capacity of nova computing instances. A flavor is an available hardware configuration for a server. It defines the size of a virtual server that can be launched.
A service consists of a single or multiple VNFs.
The Virtualisation Deployment Unit (VDU) is a construct that can be used in an information model, supporting the description of the deployment and operational behaviour of a subset of a VNF, or the entire VNF if it was not componentized in subsets.
The Virtualized Infrastructure Manager (VIM) adds a management layer for the data center hardware. Its northbound APIs are consumed by other layers to manage the physical and virtual resources for instantiation, termination, scale in and out procedures, and fault & performance alarms.
A Virtual Machine (VM) is an operating system OS or an application installed on a software, which imitates a dedicated hardware. The end user has the same experience on a virtual machine as they would have on dedicated hardware.
A Virtual Network Function (VNF) consists of a single or a group of VMs with different software and processes that can be deployed on a Network Function Virtualization (NFV)
Infrastructure.
xii
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
About This Guide
Related Documentation
Terms
VNFC
VNFM
Definitions
A Virtual Network Function Component is (VNFC) a composite part of the VNF, synonymous with a VDU, which could be implemented as a VM or a container.
Virtual Network Function Manager (VNFM) manages the life cycle of a VNF.
Related Documentation
The Cisco ESC doc set comprises of the following guides to help you perform installation, configuration; the lifecycle management operations, healing, scaling, monitoring and maintenance of the VNFs using different
APIs.
Guide Information Provided in This Guide
Cisco Elastic Services Controller Release
Notes
Includes new features and bugs, known issues.
Cisco Elastic Services Controller Install and Upgrade Guide
Includes procedure for new installation and upgrade scenarios, pre and post installation tasks, and procedure for ESC High
Availability (HA) deployment.
Cisco Elastic Services Controller User
Guide
Cisco Elastic Services Controller ETSI
NFV MANO User Guide
Includes lifecycle management operations, monitoring, healing and scaling of the VNFs.
Includes lifecycle management operations, monitoring, healing and scaling of the VNFs using the ETSI APIs.
Cisco Elastic Services Controller
Administration Guide
Cisco Elastic Services Controller
NETCONF API Guide
Cisco Elastic Services Controller REST
API Guide
Cisco Elastic Services Controller ETSI
REST API Guide
Cisco Elastic Services Controller
Deployment Attributes
Cisco Elastic Services Controller Open
Source
Includes maintenance, monitoring the health of ESC, and information on system logs generated by ESC.
Information on the Cisco Elastic Services Controller NETCONF northbound API, and how to use them.
Information on the Cisco Elastic Services Controller RESTful northbound API, and how to use them.
Includes information on the Cisco Elastic Services Controller
ETSI APIs, and how to use them.
Includes information about deployment attributes used in a deployment datamodel.
Includes information on licenses and notices for open source software used in Cisco Elastic Services Controller.
Obtaining Documentation Request
For information on obtaining documentation, using the Cisco Bug Search Tool (BST), submitting a service request, and gathering additional information, see What's New in Cisco Product Documentation , at: http://www.cisco.com/c/en/us/td/docs/general/whatsnew/whatsnew.html
.
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide xiii
About This Guide
About This Guide
Subscribe to What's New in Cisco Product Documentation , which lists all new and revised Cisco technical documentation, as an RSS feed and deliver content directly to your desktop using a reader application. The
RSS feeds are a free service.
xiv
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
C H A P T E R
1
Elastic Services Controller Overview
•
•
Elastic Services Controller Overview, on page 1
Change History
Table 2: Change History for ESC 5.3
Date
October 15, 2020
Revision Location
Updated the Virtual Requirements' table.
Virtual Resource Requirements, on page 5
Elastic Services Controller Overview
Cisco Elastic Services Controller (ESC) is a Virtual Network Functions Manager (VNFM), which performs life cycle management of Virtual Network Functions (VNFs). ESC provides agent-less and multi vendor VNF management by provisioning virtual services, and monitoring their health and load. ESC provides the flexibility to define monitoring rules, and associate actions to be triggered based on the outcome of these rules. As a
VNFM, in addition to the typical lifecycle management operations, ESC also supports automatic VM recovery when a VM fails and performs automatic scaling in and out functions. ESC fully integrates with Cisco and other third party applications.
• As part of the Cisco Orchestration Suite, ESC is packaged with Cisco Network Services Orchestrator
(NSO), and available within Cisco Solution, Managed Services Accelerator Solution (MSX).
• As a Specialized Virtual Network Function Manager (SVNFM), ESC tightly integrates with the Cisco
Mobility VNFs.
• ESC can also be utilized as a Generic Virtual Network Function Manager (GVNFM) to provide lifecycle management for both Cisco and third-party VNFs.
ESC is deployed in a virtual machine within OpenStack, VMware vCenter, KVM or AWS and manages its
VNFs in a Virtual Infrastructure Manager (VIM).
Elastic Services Controller as a VNF Manager targets the virtually managed services and all service provider
NFV deployments such as virtual video, WiFi, authentication and others.
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
1
Elastic Services Controller Overview
Elastic Services Controller Overview
ESC can manage both basic and complex VNFs . Basic VNFs include a single VM such as a vFW, vRouter and others.
Complex VNFs include multiple VMs that are orchestrated as a single entity with dependencies between them.
IPv6 Support
Elastic Services Controller provides IPv6 support on OpenStack for:
• VNF Management
• HA— ESC manages VNFs on IPv4 and IPv6 (OpenStack and KVM only).
Elastic Services Controller provides IPv6 support for northbound interface (for example, NFVO to VNFM), and southbound interface (for example, VNFM to VNF). In order to support both northbound and southbound
IPv6 concurrently, the following pre-requisites must be met:
• OpenStack cloud computing is set up and configured for ipv6, including the endpoints (that are ipv6 based).
• The OpenStack cloud computing must contain a Controller, endpoints, and a few Compute hosts, with an ipv6 management and os_api based networks.
• The ESC default security group rules support the IPv6 traffic.
Note When you are deploying a VM, you can attach an out-of-band port of an IPv6 subnet to a VM. However, if you are deleting this VM, you cannot attach the same IPv6 address to another VM due to a known OpenStack issue.
2
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
P A R T
I
Installing Cisco Elastic Services Controller on
OpenStack
•
•
Installing Cisco Elastic Services Controller on OpenStack, on page 7
•
Installing High Availability Active/Standby, on page 21
•
Cisco Elastic Services Controller Active/Active High Availability Overview, on page 33
•
Installing Active/Active High Availability Cluster, on page 35
•
Managing Cluster in ESC Active/Active High Availability, on page 41
•
Configuring GEO in Active/Active High Availability, on page 43
•
DRBD Encryption for ESC Active/Standby and Active/Active HA Data Replication , on page 49
•
Upgrading ESC Active/Active High Availability, on page 51
C H A P T E R
2
Prerequisites
The following sections detail the prerequisites for installing Cisco Elastic Services Controller:
•
Virtual Resource Requirements, on page 5
•
Software Requirements, on page 6
•
Preparing for the Installation, on page 6
Virtual Resource Requirements
The following table lists the virtual resource requirements for Cisco Elastic Services Controller:
ESC
Deployment
ESC
Standalone
Number of
ESC VMs
1
Virtual CPU per ESC VM
4
Virtual
Memory (GB) per ESC VM
Virtual Hard
Disk (GB)per
ESC VM
Number of total VMs
Supported
8 40 1000
VIM
All supported
VIMs
ESC
Active/Standby
HA
2
ESC
Active/Active
HA
3
ESC GEO
HA
6
4
8
8
8
16
16
40
40
*
40
*
2500
5000
5000
All supported
VIMs
OpenStack
Only
OpenStack
Only
Note
*
The Active/Active HA and Geo HA deployment can accommodate up to 60 GB of hard disk while provisioning the NFVI resources.
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
5
Installing Cisco Elastic Services Controller on OpenStack
Software Requirements
Software Requirements
The following table lists the software requirements on OpenStack:
Requirements Description
Supported Browsers
OpenStack Version
Google Chrome 44.x or later
Any of the following:
• Train
• Ocata (V2 and V3)
• Queens (Keystone V3)
Preparing for the Installation
Before you perform the installation, ensure that you are prepared by reviewing this checklist:
Requirements Your Information/ Notes
For Pre-installation
Configuration
QCOW image location
QCOW image
VM per Instance
IP address
Subnet mask
Hostname
Domain name
Gateway IP address
Admin password
ESC Release Package
ESC.qcow2
bootvm.py
An image file for booting up the ESC instance
The installation script is compatible with python 2.7.6 and Python 3.4
6
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
C H A P T E R
3
Installing Cisco Elastic Services Controller on
OpenStack
This chapter describes how to install Cisco Elastic Services Controller on OpenStack and includes the following sections:
•
Installation Scenarios, on page 7
•
Main Components of Cisco Elastic Services Controller Setup, on page 8
•
Installing Cisco Elastic Services Controller Using the QCOW Image, on page 8
•
Managing Root Certificates in Cisco Elastic Services Controller, on page 15
•
Managing Keystore in Cisco Elastic Services Controller, on page 17
•
Using a Bootable Volume in ESC Installation, on page 19
Installation Scenarios
The following sections briefly describe some of the common deployment scenarios that are addressed by ESC.
Cisco Elastic Services Controller can be installed in different modes as per the requirement. These different modes are configured during installation. The following sections briefly describe some of the common deployment scenarios that are addressed by ESC.
ESC Standalone
In the standalone scenario, a single active VM is deployed for ESC.
ESC with HA
ESC supports High Availability (HA) in the form of Active/Standby and Active/Active models. For
Active/Standby model, two ESC instances are deployed in the network to prevent ESC failure and to provide
ESC service with minimum interruption. If the primary ESC instance fails, the standby instance automatically takes over the ESC services. ESC HA resolves the following single point failures of ESC:
• Network failures
• Power failures
• Dead VM instance
• Scheduled downtime
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
7
Installing Cisco Elastic Services Controller on OpenStack
Main Components of Cisco Elastic Services Controller Setup
• Hardware issues
• Internal application failures
Note From ESC 5.0, the name, Active/Passive model is changed to Active/Standby model.
Note Installing or upgrading software on the ESC VM is not supported. For further assistance, contact Cisco TAC.
For more information on deploying ESC HA Active/Standby, see 'Configuring High-Availability
Active/Standby' in Installing ESC on OpenStack and Installing ESC on VMware chapters.
For more information on the Active/Active model, see ESC HA Active/Active Overview Chapter.
Main Components of Cisco Elastic Services Controller Setup
The Cisco Elastic Service Controller (ESC) setup has the following components:
• Virtual Infrastructure Manager —Elastic Services Controller (ESC) and its VNFs are deployed in a
Virtual Infrastructure Manager (VIM). This might have one or more underlying physical nodes.
• ESC Virtual Machine —The ESC VM is a VM that contains all the services and processes used to register and deploy services. This includes the ESC Manager and all other services. ESC provides Netconf
API, REST API, and Portal as north bound interfaces to communicate with ESC. ESC VM contains CLI to interact with ESC VM. There are two CLI, one uses the REST API and the other uses Netconf API.
Installing Cisco Elastic Services Controller Using the QCOW
Image
You can install Cisco Elastic Services Controller (ESC) on OpenStack by using a QCOW image. ESC will be deployed in the OpenStack as running VM instance to manage VNFs. Therefore, ESC need OpenStack environment parameters to be installed in OpenStack. The installation time varies from 10 to 20 minutes, depending on the host and the storage area network load. This procedure describes how to create ESC Virtual
Machine (VM) in OpenStack.
Before you begin
• All system requirements are met as specified in
.
• You have the information identified in
Preparing for the Installation, on page 6 .
• Copy the ESC image file on the system where you want to install ESC.
• This system must be accessible by OpenStack.
8
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Installing Cisco Elastic Services Controller on OpenStack
Installing Cisco Elastic Services Controller Using the QCOW Image
Procedure
Step 1
Step 2
Step 3
Log in to the system where you want to install ESC.
Check the compatibility of the bootvm.py and the ESC image:
./bootvm.py --version
For more information on the ESC installer arguments, see Appendix: A Cisco Elastic Services Controller
Installer Arguments .
In a text editor, create a file named PROJECT-openrc.sh file and add the following authentication information.
The following example shows the information for a project called admin, where the OpenStack username is also admin, and the identity host is located at controller node.
Note To set the required environment variables for the OpenStack command-line clients, you must create an environment file called an OpenStack rc file, or openrc.sh file. This project-specific environment file contains the credentials that all OpenStack services use. The ESC installation script requires these OpenStack environment parameters to perform authentication and installation on OpenStack.
If all the OpenStack credentials are passed through its own arguments, the bootvm.py script doesn't require these parameters.
Step 4
Step 5 export OS_NO_CACHE=true export OS_TENANT_NAME=admin export OS_USERNAME=admin export OS_PASSWORD=admin_pass export OS_AUTH_URL= http://controller node:35357/v2.0
The other OpenStack parameters required for installation are: --os_auth_url, --os_username, --os_password
, -os_tenant_name, --bs_os_user_domain_name, --bs_os_project_domain_name, --bs_os_identity_api_version,
--bs_os_auth_url, --bs_os_username, --bs_os_password , -bs_os_tenant_name, --bs_os_user_domain_name,
--bs_os_project_domain_name, --bs_os_identity_api_version.
For OpenStack V2 API, you need following items to be defined in your global environment variables:
--os_password, --os_auth_url, --os_username, --os_tenant_name.
For OpenStack V3 API, set --os_identity_api_version=3 . Other parameters required for OpenStack V3 API are: --os_user_domain_name, --os_project_domain_name, --os_project_name, --os_password, --os_auth_url,
--os_username, --os_identity_api_version, --os_ca_cert, --requests_ ca_bundle.
Note The arguments, --os_tenant_name, --os_username, --os_password, --os_auth_url will also by default configure the VIM connector. If you want to skip configuring the VIM connector, pass the parameter
(--no_vim_credentials) with the bootvm.py. When no_vim_credentials parameter is provided, the bootvm.py arguments (os_tenant_name, os_username, os_password,os_auth_url) are ignored. For more information on configuring VIM connectors after installation, and managing VIM connectors, see Managing VIM Connectors in the Cisco Elastic Services Controller User Guide .
Note --os_ca_cert and --requests_ca_bundle arguments are only required for https connection.
On any shell from which you want to run OpenStack commands, source the PROJECT-openrc.sh
file for the respective project. In this example, you source the admin-openrc.sh
file for the admin project .
$ source admin-openrc.sh
Check the environment variables.
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
9
Installing Cisco Elastic Services Controller on OpenStack
Installing Cisco Elastic Services Controller Using the QCOW Image
Step 6
Step 7
Step 8
$ env | grep OS_
Register ESC image file in the OpenStack image using the glance command:
$ glance image-create \
--name <image_name> \
--is-public=<true or false> or --visibility public or private\
--disk-format <disk_format> \
--container-format <container_format> \
--file <file> \
--progress
An example configuration is shown below:
$ glance image-create \
--name esc-1_0_01_11_2011-01-01 \
--is-public=<true or false> or --visibility public or private \
--disk-format qcow2 \
--container-format bare \
--file esc-1_0_01_11_2011-01-01.qcow2 \
--progress
The glance image-create command is used to create a new image. The command takes the following arguments:
Note The 'is-public' argument is applicable only for OpenStack Kilo release.
Arguments name is-public disk-format container-format file progress
Description
Name of the image.
(Optional) Makes the image accessible to the public.
Disk format of the image. ESC uses a qcow2 disk format.
Container format of image. ESC uses a bare container format.
Local file that contains disk image to be uploaded during creation.
(Optional) Shows upload progress bar.
To verify whether the image has been registered successfully: a) Using OpenStack Controller dashboard:
• Log into OpenStack using your credentials.
• Navigate to Admin > Image .
Verify if the image appears in the list.
b) Using nova CLI:
$ nova image-show <image_name>
The standard resource requirement of ESC is 4vCPU, 8G RAM, and 40GB disk space. ESC installation script takes the pre-defined "m1.large" flavor which has the definition of 4vCPU, 8G RAM and 80G disk space. To use 40GB disk space, create a flavor with the minimum disk space requirement.
$ nova flavor-create ESC_FLAVOR_NAME ESC_FLAVOR_ID 8192 40 4
To deploy ESC VM, do the following:
10
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Installing Cisco Elastic Services Controller on OpenStack
Additional Installation Options a.
Ensure that the existing network have connectivity to OpenStack controller. To verify the network connectivity using the nova CLI use:
$ nova net-list b.
Record the ID of the network that ESC connects to boot the ESC VM by with image and flavor created earlier. The bootvm.py command requires at least one --user_pass argument to create an admin account for linux (ssh/console access) and at least one --user_confd_pass to create an admin account for ConfD
(netconf/cli access). The following are the syntax for these mandatory user credential arguments:
--user_pass admin:'PASSWORD-OR-HASH'[:OPTIONAL-PUBLIC-KEY-FILE][:OPTIONAL-ROLE]
--user_confd_pass admin:'PASSWORD-OR-HASH'[:OPTIONAL-PUBLIC-KEY-FILE]
Generating hashed password is optional. You can select a plain password as and when required.
To generate a hashed password on Ubuntu OS, use the following command: mkpasswd --method=SHA-512 --salt XyZ123 <<< <Password>
The following is an example to install ESC with an authorized public key. In the following example, single quotes are used to avoid conflict with shell reserved characters:
--user_pass admin:'$algorithm$salt$hash-of-salt-password':$HOME/.ssh/esc_rsa.pub
--user_confd_pass admin:'$algorithm$salt$hash-of-salt-password':$HOME/.ssh/esc_rsa.pub
The public key are generated as part of a key pair with, such as: ssh-keygen -t rsa -b 1024 -C "esc" -N "" -f ~/.ssh/esc_rsa
Your public key and identification key are saved in /home/username /.ssh/esc_rsa and esc_rsa.pub files.
For more examples on the user credential arguments, see Appendix A: Cisco Elastic Services Controller
Installer Arguments.
c.
To check the details of ESC VM and get the information including the IP address(es) of the ESC VM, use the following command:
$ nova show <esc_vm_name>
Additional Installation Options
• Deploying ESC in an OpenStack IPv6 environment: Before deploying ESC instance in IPv6, make sure to source openrc that supports ipv6 addresses. To deploy ESC in IPv6 environment, use the following bootvm arguments:
./bootvm.py
<esc_vm_name-ipv6> --poll --user_rest_pass <username> : <password> --image
<image_name>
--net <ipv6_network> --ipaddr <ipv6_ip_address> --enable-http-rest --user_pass
<username> : <password>
--user_confd_pass <username> : <password> --etc_hosts_file <hosts-file-name> --route
<default routing configuration>
• Deploying ESC in DHCP mode: If you use the bootvm.py argument without --ipaddr, then the ESC instance will be deployed in a DHCP mode. To deploy ESC in a DHCP network, use the following configuration:
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
11
Installing Cisco Elastic Services Controller on OpenStack
Additional Installation Options
./bootvm.py
<esc_vm_name> --image <image_name> --net <IPv6 network> <IPv4 network>
--flavor <flavor_name>
--user_pass <username> : <password>
--user_confd_pass <username> : <password>
Note By default, ESC only support DHCP in IPv4 networks. If IPv6 is used, you need to log in the ESC VM and run "dhclient -6 ethX" (ethX is the V6 interface name) manually to enable V6 DHCP.
When deploying ESC with multiple network interfaces, with one or more type of DHCP, use bootvm.py --defroute N to specify the interface index to be assigned default route and gateway IP.
By default,
N = 0
. So the first network interface on the bootvm.py
command line is default.
For Example:
--defroute 1 will assign <network2> with <default_gateway_ip_address>
./bootvm.py <esc_vm_name> --image <image_id> --net <network1>
<network2> --defroute 1 --gateway_ip <default_gateway_ip_address>
• Using a Bootable Volume in ESC Installation: You can attach a volume to an ESC instance and launch an instance from inside the volume. For more information, see the section
• Assigning Floating IP to the ESC: If you want to associate a floating IP with the ESC instance , do the following:
1.
Check for an available floating IP address and assign it to the ESC VM:
$ nova floating-ip-list
$ nova floating-ip-associate esc_vm_name <ip_address>
2.
Or create a new floating IP address and assign it to the ESC VM:
$ nova floating-ip-create <FLOATING_NETWORK - ID>
$ nova floating-ip-associate esc_vm_name <ip_address> or neutron floatingip-create FLOATING_NETWORK neutron floatingip-associate floating-ip-ID port-ID
• Deploying ESC with static IPs: To use ESC in a specific network with static IPs, for example,
192.168.0.112 at network1, specify --ipaddr and --gateway_ip to the bootvm command line , as shown below:
Note Before assigning static IP, make sure the static IP is available and is not being used on other machine.
./bootvm.py
<esc_vm_name> --image <image_id> --net <network>
--ipaddr <ip_address> --gateway_ip <default_gateway_ip_address> --user_pass
<username> : <password>
--user_confd_pass <username> : <password>
12
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Installing Cisco Elastic Services Controller on OpenStack
Additional Installation Options
• Deploying ESC with multiple network interfaces: To use multiple networks for ESC, for example,
192.168.0.112 at network1 and 10.20.0.112 at network2, specify both the IP addresses and the network names of the interfaces in the --net and --ipaddr arguments in the following command line. In addition, also choose the default gateway for ESC from the gateways of these networks. Specify the default gateway for ESC through the --gateway_ip argument.
./bootvm.py
<esc_vm_name> --image <image_id> --net <network1>
<network2> --ipaddr <ip_address1> <ip_address2> --gateway_ip <default_gateway_ip_address>
--user_pass <username> : <password>
--user_confd_pass <username> : <password>
Note If --flavor is not specified, bootvm.py will use the default flavor "m1.large" in
OpenStack.
Ensure that m1.large exist in OpenStack if decide not specified a flavor, and also ensure that the flavor suits ESC deployment requirement.
• Deploy ESC with log forwarding options: To forward ESC logs to an rsyslog server, specify the IP address of the rsyslog server while creating an ESC VM. Optionally, you can also specify the port and protocol to use.
For example, if the IP address of the rsyslog server is 172.16.0.0 the port on the server to forward logs is 514, and the protocol used is UDP, the ESC installation could be
./bootvm.py
<esc_vm_name> --image <image_id> --net network1 --rsyslog_server 172.16.0.0
--rsyslog_server_port 514 --rsyslog_server_protocol udp --user_pass <username> : <password>
--user_confd_pass <username> : <password>
• Disabling the ESC GUI: To boot up ESC VM with the graphical user interface disabled, modify the
--esc_ui_startup argument value, as shown in the command line below:
./bootvm.py
<esc_vm_name> --image <image_id> --net <network> --user_pass
<username> : <password>
--user_confd_pass <username> : <password>
--esc_portal_startup=False
• Enabling REST interface for ESC: To support the REST interface, specify --enable-https-rest argument.
You can activate REST interface on both https or http:
./bootvm.py
<esc_vm_name> --image <image_id> --net <network> --user_pass
<username> : <password>
--user_confd_pass <username> : <password> --enable-https-rest
OR
./bootvm.py
<esc_vm_name> --image <image_id> --net <network> --user_pass
<username> : <password>
--user_confd_pass <username> : <password> --enable-http-rest
•
• Enabling REST interface for ETSI: To support the ETSI REST intetrface, specify --enable-http-etsi to activate the interface over http, or --enable-https-etsi to activate the interface over https.
./bootvm.py
<esc_vm_name> --image <image_id> --net <network> --user_etsi_pass
<username> : <password> --enable-https-rest . . .
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
13
Installing Cisco Elastic Services Controller on OpenStack
Additional Installation Options
OR
./bootvm.py
<esc_vm_name> --image <image_id> --net <network> --user_etsi_pass
<username> : <password>
--user_confd_pass <username> : <password> --enable-http-etsi-rest...
Note Only the https REST and ETSI interfaces should be enabled in a production environment.
• Deploying ESC with global parameters: To set the global configurations through the esc_params_file during the installation, use the arguments as shown below. These global configurations can also be changed through REST API after the installation.
Note The default security group is applied to the tenant during tenant creation. By default, the ESC configuration parameter for the security group, openstack.DEFAULT_SECURITY_GROUP_TO_TENANT
is set to true. The configuration parameter must be set at the time of installation. You can query or update the parameter on ESC VM through the REST API. If the parameter is set to true, you can create and assign default security group during tenant creation.
If the parameter is set to false, you cannot create or assign default security group during tenant creation. For details on the parameters that can be configured through esc_params_file, see Appendix A: Cisco Elastic Services Controller Installer
Arguments.
./bootvm.py <esc_vm_name> --image <image_id> --net <network> --flavor <flavor_name>
--user_pass <username>:<password>:<public key file> --user_confd_pass
<username> : <password>
--esc_params_file <esc parameter configuration file>
• Deploying two instances of ESC to build an ESC HA (Active/Standby) pair : For more information on deploying ESC HA (Active/Standby), see Configuring High Availability in Installing ESC on
OpenStack and Installing ESC on VMware chapters.
• Adding a Dynamic Mapping File : In Cisco ESC Release 2.1 and earlier, mapping the actions and metrics defined in the datamodel to the valid actions and metrics available in the monitoring agent was enabled using the dynamic_mappings.xml
file. The file was stored in the ESC VM and was modified using a text editor. ESC 2.2 and later do not have an esc-dynamic-mapping directory and dynamic_mappings.xml
file. If you want to add an existing dynamic_mapping xml file to the ESC VM, do the following:
1.
Backup this file to a location outside of ESC, such as, your home directory.
2.
Create esc-dynamic-mapping directory on your ESC VM. Ensure that the read permissions are set.
3.
Install on your ESC VM using the following bootvm argument:
--file root:root: /opt/cisco/esc/esc-dynamic-mapping/dynamic_mappings.xml
:< path-to-local-copy-of-dynamic-mapping.xml
>
The CRUD operations for mapping the actions and the metrics is available through REST API. To update an existing mapping, delete and add a new mapping through the REST API.
14
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Installing Cisco Elastic Services Controller on OpenStack
Managing Root Certificates in Cisco Elastic Services Controller
• Changing the confd password on an ESC VM : As an administrator, you can configure the confd password through bootvm.py, during the installation time:
./bootvm.py --user_pass <username> : <password> --user_confd_pass admin:'PASSWORD-OR-HASH':OPTIONAL-PUBLIC-KEY
To reconfigure this password after the installation, execute the following commands:
$ /opt/cisco/esc/confd/bin/confd_cli -u admin
$ configure
$ set aaa authentication users user admin password <your_password>
$ commit
$ exit
Note For the ease of future upgrades, make sure that you keep a copy of all the commands and arguments that are used while installing ESC using the bootvm.py file.
Managing Root Certificates in Cisco Elastic Services Controller
Cisco Elastic Services Controller (ESC) provides a mechanism to enable verification of SSL certificates.
Currently, this feature is supported only on OpenStack. Certificate validation is enabled by default during the initial ESC boot up. However, ESC also allows you to configure these SSL certificates. This section describes how to enable/disable certificate validation, add/remove, or list the certificates for Cisco Elastic Service
Controller on OpenStack. You can add a root certificate during the ESC bootup or even after ESC bootup is completed.
Enabling/Disabling the Root Certificate Validation
Cisco Elastic Services Controller by default enables certificate validation. You can also enable or disable by modifying the parameter, DISABLE_CERT_VALIDATION, available under the Openstack category in the esc_params.conf file, or through the REST interface, or using the escadm tool.
On ESC master node, use the command, sudo escadm enable-certificate or sudo escadm disable-certificate to enable and disable the certificate validation, respectively.
Adding a Root Certificate
You can add a root certificate during the ESC bootup or even after ESC bootup is completed. Before adding certificates, ensure the OpenStack environment file, OpenStack RC file has parameters to perform authentication and installation on OpenStack. The --os_auth_url must be specified while passing the parameters. --os_auth_url specifies the secure (https) or unsecured (http) keystone URL used by OpenStack for authentication.
• Add certificate for standalone (only) during the bootup time, i.e, during the ESC VM installation:
./bootvm.py test-vm --image <image_name> --net <network> [--cert_file CERT_FILE]
[--confd_aes_key CONFD_AES_KEY]
/home/cisco/openstack.crt
--user_pass <username>:<password> --user_confd_pass <username>:<password>
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
15
Installing Cisco Elastic Services Controller on OpenStack
Removing a Root Certificate
Note Currently, ESC does not support adding a certificate for HA Active/Standby during the installation as the keepalived service is not running when a certificate is added.
• Add certificate for standalone/HA Active/Standby after booting up the ESC instance. The escadm tool has an truststore add option which has the following arguments: The --file argument refers to the CA certificate file. Using this argument you can import any file format supported by the java keytool: X.509
v1, v2, and v3 certificates, and PKCS#7. The --alias argument is unique and refers to the name this specific CA certificate is given.
1.
Copy/Transfer CA Certificate file to ESC master VM .
2.
Add certificate to ESC truststore. To do this, execute the following command: sudo escadm escadm truststore add --alias [ca cert alias] --file [file path] or sudo escadm truststore escadm truststore --alias [ca cert alias] --file [file path]
3.
Verify the certificate is added.
sudo escadm truststore show or sudo escadm truststore show
Removing a Root Certificate
The escadm tool has a 'truststore delete' option which only takes --alias argument. The --alias argument refers to the name of the CA certificate to be deleted. Use this argument on the standalone/HA Active/Standby ESC
VM:
Procedure
Step 1
Step 2
On (master) ESC use escadm to delete certificate from ESC truststore.
sudo escadm truststore delete --alias [ca cert alias] or sudo escadm truststore truststore delete --alias [ca cert alias]
Verify the certificate is removed.
sudo escadm truststore show or sudo escadm truststore show
16
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Installing Cisco Elastic Services Controller on OpenStack
Managing Root Certificates During the Upgrade
Managing Root Certificates During the Upgrade
• Image Upgrade : If you are backing up the ESC DB for upgrade, then no other action is required, the
ESC truststore will be restored once the ESC DB is restored. If you are not backing up the ESC DB for upgrade, then each CA certificate needs to be added again to the ESC truststore.
• RPM Upgrade : This upgrade method keeps the ESC truststore as is, i.e. all ca certificates in the ESC truststore should remain there after upgrade.
Managing Keystore in Cisco Elastic Services Controller
The keystore is a storage for certificates and keys that an application uses to authenticate itself with different clients.
ESC Keystore holds only one certificate, that is used by all the applications for example, ESCManager,
VIMManager, MONA, and so on.
ESCADM provides multiple commands to manage the keystore.
When deploying ESC for the first time, a self signed certificate is created and stored in the keystore by default.
important Notes:
• In an Active/Active mode, the default certificate Common Name (CN) is db.service.consul
. If you want to set a new certificate, you must set it with the same CN, otherwise, you must add the following configuration to the heat template of all the nodes to disable the certificate validation in MONA when deploying the ESC: mona: certificate_validation: false
• All the escadm keystore commands require root privileges to be executed on the ESC.
ESCADM Keystore Commands
• escadm keystore show
The escadm keystore show command displays information about the certificate currently stored in the keystore. It shows informations like, the creation date, Alias, and certificate fingerprint.
For example:
$ sudo escadm keystore show
Keystore type: PKCS12
Keystore provider: SUN
Your keystore contains 1 entry esc, Apr 13, 2020, PrivateKeyEntry,
Certificate fingerprint (SHA1):
FF:11:66:3E:93:DD:3A:0B:9A:72:40:16:35:34:D2:22:E1:25:07:80
• escadm keystore export [--out <file path>]
The export command displays the full content of the certificate. If the out option is specified, the content is saved in a file whose path is specified in the previous option.
• escadm keystore set-- file <file path>
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
17
Installing Cisco Elastic Services Controller on OpenStack
ESCADM Keystore Commands
The set command replaces the current certificate with a new one residing inside the file whose path is specified in the option file
.
Ensure that the file is in PEM format, containing both the certificate and a private key.
The following example shows how to generate a new self-signed certificate and set it to the keystore:
1.
Use the following command to generate the certificate: openssl req -newkey rsa:2048 -new -nodes -x509 -days 3650 -keyout key.pem -out cert.pem
The previous commands gives the following two files: key.pem
and cert.pem
2.
Use the following commands to combine the two files: cat key.pem > server.pem
cat cert.pem >> server.pem
3.
To set the new certificate to the keystore, use the following command:
$ sudo escadm keystore set --file server.pem
Service "keystore" successfully updated ESC keystore and will take effect once ESC services are restarted by running "sudo escadm restart"
Note Ensure that you delete any left over files containing private keys and certificate after the setting is done.
It is mandatory to restart your system for the change to take effect. Use the following command to restart in standalone mode or H/A mode: sudo escadm restart
If the ESC is running in the Active/Active mode, you must restart all the nodes in the same cluster.
However, if the ESC is running in an Active/Active GEO mode, you must stop the GEO service on all the clusters, to make sure a GEO switchover does not take place.
Only in GEO Active/Active mode, to stop GEO service, log in to any node in each cluster and use the following command:
$ sudo escadm geo stop --cluster
You can login to any node in the cluster where the certificate is originally set. Use the following command:
$ sudo escadm stop --cluster
$ sudo escadm start --cluster
If you are on a GEO Active/Active mode, once all the node are up and running in passing state, login again to any node to start the GEO service. Use the folowing command to start the service:
$ sudo escadm geo start --cluster
18
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Installing Cisco Elastic Services Controller on OpenStack
Using a Bootable Volume in ESC Installation
Using a Bootable Volume in ESC Installation
A volume in OpenStack is a detachable block storage device that can be attached to an ESC instance. You can store and also run ESC instances from a volume.
Note • Only one ESC instance can be launched from one volume at a time.
• ESC installation with a combination of bootable volume and high availability (Active/Standby and
Active/Active) on cinder is not supported.
To launch an ESC instance from a bootable volume, do the following:
Procedure
Step 1
Step 2
Create a bootable volume in OpenStack based on an ESC image or from a bootable volume. The bootable volume must be at least of 30 GB disk size. For more information, see OpenStack documentation.
Deploy ESC VM using the bootvm.py
command and choose the
--boot_volume argument instead of the
--image argument, as shown below:
./bootvm.py <esc_vm_name> --boot_volume <volume_name_or_id> --net <network> --user_pass
<username>:<password>
--user_confd_pass <username>:<password> --flavor <flavor_name>
Note • Only one of these arguments,
--image or
--boot_volume must be passed to the bootvm.py
command. The installation will fail, if both or none of the arguments are used.
• When launching an ESC instance from a bootable volume, volume disk size is considered over the flavor disk size.
• If an ESC instance is deleted, the volume attached to it will not be deleted, as the volume was created out-of-band.
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
19
Using a Bootable Volume in ESC Installation
Installing Cisco Elastic Services Controller on OpenStack
20
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
C H A P T E R
4
Installing High Availability Active/Standby
This chapter contains the following sections:
•
High Availability Active/Standby Overview, on page 21
•
How High Availability Works, on page 22
•
Deploying ESC High Availability Active/Standby, on page 23
•
Configuring the Northbound Interface Access, on page 26
•
•
Troubleshooting High Availability Active/Standby, on page 32
High Availability Active/Standby Overview
ESC supports High Availability (HA) in the form of Active/Standby and Active/Active models. For
Active/Standby model, two ESC instances are deployed in the network to prevent ESC failure and provide
ESC service with minimum service interruption. If the primary ESC instance fails, the standby instance automatically takes over the ESC services. ESC HA Active/Standby resolves the following single point failures:
• Network failures
• Power failures
• Dead VM instance
• Scheduled downtime
• Hardware issues
• Internal application failures
Note From ESC 5.0, the name, Active/Passive model is changed to Active/Standby model.
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
21
ESC Active/Standby Architecture
ESC Active/Standby Architecture
Figure 1: Cisco Elastic Services Controller Active/Standby Architecture
Installing Cisco Elastic Services Controller on OpenStack
How High Availability Works
ESC HA Active/Standby network can be either set up as a single installation of a ESC HA Active/Standby pair or deployed as two standalone ESC nodes that are converted into HA pair after re-configuring these nodes post deployment. The HA deployment consists of two ESC instances: a primary and a standby. Under normal circumstances, the primary ESC instance provides the service. The corresponding standby instance is passive.
The standby instance is in constant communication with the primary instance and monitors the primary instances' status. If the primary ESC instance fails, the standby instance automatically takes over the ESC services to provide ESC service with minimum interruption.
The standby also has a complete copy of the database of the primary, but it does not actively manage the network until the primary instance fails. The KeepAliveD service monitors both primary and standby instances activity status. When the primary instance fails, the standby takes over automatically. The standby instance takes over primary instance to manage the services while primary instance restoration is taking place.
When the failed instance is restored, if required you can manually initiate a switch-over and resume network management via the primary instance.
Both primary and standby ESC instances are connected to the northbound orchestration system through an
IPv4 or IPv6 network. For the northbound system, a unique virtual IP address is assigned to access the current primary ESC High Availability Active/Standby instance. The deployed VNFs are connected to both ESC primary and standby instances through another IPv6 network.
ESC HA Active/Standby nodes are managed by KeepAliveD and DRBD (Replication tool to keep the ESC database synchronized) sync network services. While the KeepAliveD service monitors both primary and standby instances status, the DRBD service monitors primary instance DB and sync the changes to the standby instance DB. These two services can be co-located on same VIP network or in two separate networks. VM handshake between ESC instances occurs through the KeepAliveD over the IPv4 or IPv6 network.
22
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Installing Cisco Elastic Services Controller on OpenStack
Deploying ESC High Availability Active/Standby
Deploying ESC High Availability Active/Standby
To deploy Cisco Elastic Services Controller (ESC) High Availability (HA) Active/Standby, ESC standalone instances can be installed on two separate nodes - Primary and Standby. For more information see,
Availability Works, on page 22
. You can connect the Primary and Standby instances to either a Cinder volume or Replication based volume (DRBD).
The following deployment mechanisms can be used to deploy ESC HA Active/Standby:
• Internal Storage—When ESC HA Active/Standby is configured with Internal storage, the Primary and the Standby instances have individual databases which are always synchronized. In this solution, ESC
HA Active/Standby is designed with database replication and DRBD is used as the tool for disk-level replication. The database in the Primary instance simultaneously propagates the data to the database in the Standby instance thus requiring no external storage. In the event of a Primary instance failing, the
Standby instance get assigned the role of the Primary instance along with its own synchronized database.
ESC HA Active/Standby is deployed using Internal storage, the ESC instances reply on the virtual IP address (that is kad_vip argument), and the interface of vrrp instance (that is kad_vif argument) to select the Primary ESC instance. To establish a reliable heartbeat network, it is recommended that the Primary and Standby ESC instances are on different physical hosts. The reliability of the physical links between the ESC instances (such as, network interface bonding) can also be taken into consideration.
• Replicate External-Storages — In this type of architecture, ESC HA Active/Standby is configured with
DRBD and both Primary and Standby instance store their data in two external storages (OpenStack
Cinder volumes). Each ESC node is attached by a Cinder volume and ESC data files are stored in the cinder volume. The data in two ESC node are synchronized through the database replication mechanism provided by DRBD.
The table lists the differences between the HA Active/Standby options :
Data sharing method
Installation Method
Internal Storage Based ESC HA
Active/Standby
Replicate External Storage Based ESC HA
Active/Standby
Data replication between HA
Active/Standby nodes
Post-installation Configuration
Bootvm Installation
Data replication between two external storages (cinder volume)
Bootvm Installation
VIM Support
Dependency
Advantages
OpenStack, VMware, KVM
VIM independent
• No dependency on specific
VIM components.
• Flexible to build of HA
Active/Standby clusters from commodity hardware, without the requirement for shared-storage.
OpenStack only
Rely on OpenStack cinder
• Use database replication mechanism for data synchronization
• Two cinder volumes are used as external storage and are attached to ESC node.
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
23
Installing Cisco Elastic Services Controller on OpenStack
Deploying ESC in High Availability Active/Standby Mode on Internal Storage
Limitations The data consistency may be affected in a double fault condition
(occurs when both ESC nodes have problems).
The data consistency may be affected in a double fault condition (occurs when both
ESC nodes have problems).
Deploying ESC in High Availability Active/Standby Mode on Internal Storage
When you boot ESC instances on Primary and Standby instances, you need to specify the following bootvm.py
command arguments to deploy ESC HA Active/Standby on an internal storage:
• kad_vip
Note When ESC HA Active/Standby is deployed, the kad_vip argument allows end users to access the Primary ESC instance.
• kad_vif
• ha_node_list
These arguments enable the bootvm.py
command to automatically set up the internal storage on the OpenStack.
For more information on using the bootvm.py
command arguments, see Appendix A: Cisco Elastic Services
Controller Installer Arguments.
To deploy ESC HA Active/Standby instances, use the bootvm script on both the nodes with the following arguments:
ON HA NODE 1:
$ ./bootvm.py <ESC_HA_Node1>\
--user_pass <username>:<password>\
--user_confd_pass <username>:<password>\
--gateway_ip <default gateway IP address>\
--net <network name1>\
--ipaddr <static ip address>
--image <image_name>\
--avail_zone nova:<openstack zone>\
--ha_node_list=<ESC_HA_NODE1_IP> <ESC_HA_NODE2_IP>\
--db_volume_id <cinder volume id>\
--kad_vip <virtual IP address>\
--kad_vif <VRRP_Interface_Instance>\
--ha_mode drbd
ON HA NODE 2:
$ ./bootvm.py <ESC_HA_Node2>\
--user_pass <username>:<password>\
--user_confd_pass <username>:<password>\
--gateway_ip <default gateway IP address>\
--net <network name1>\
--ipaddr <static ip addresses>\
--image <image_name>\
--avail_zone nova:<openstack zone>\
--ha_node_list=<ESC_HA_NODE1_IP> <ESC_HA_NODE2_IP>\
--db_volume_id <cinder volume id>\
--kad_vip <virtual IP address>\
24
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Installing Cisco Elastic Services Controller on OpenStack
Deploying ESC in High Availability Active/Standby Mode on Replicate External Storage
--kad_vif <VRRP_Interface_Instance>\
--ha_mode drbd
OR
You can also use escadm tool to re-configure ESC HA Active/Standby parameters on each of the standalone
ESC VMs. Three parameters "--ha_node_list , --kad_vip, --kad_vif" are all required to configure ESC HA
Active/Standby.
Note You should make sure that both the standalone ESC VMs health check is passed before running the below commands to perform HA Active/Standby configuration.
For example:
$ sudo bash
$ escadm ha set --ha_node_list='<ESC_HA_NODE1_IP> <ESC_HA_NODE2_IP>' --kad_vip <virtual IP address> --kad_vif <VRRP_Interface_Instance>
$ sudo escadm reload
$ sudo escadm restart
Deploying ESC in High Availability Active/Standby Mode on Replicate External Storage
Replicate external storage ESC HA Active/Standby requires two cinder volumes for database storage.
Before you begin
• Networks and IP addresses that both ESC instances will connect to
• Keepalived interface and virtual IP for HA Active/Standby switchover
Procedure
Step 1
Step 2
Step 3
Create two cinder volumes in OpenStack. The configured cinder volume size should be 3GB.
$ cinder create --display-name cindervolume_name_a[SIZE]
$ cinder create --display-name cindervolume_name_b[SIZE]
Check the status of the created cinder volume and find the uuids for deployment.
$ cinder list
Deploy ESC HA Active/Standby instances. Use the bootvm script on both the nodes with the following arguments:
ON HA NODE 1:
$ ./bootvm.py <ESC_HA_Node1>\
--user_pass <username>:<password>\
--user_confd_pass <username>:<password>\
--gateway_ip <default gateway IP address>\
--net <network name1>\
--ipaddr <static ip address>\
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
25
Installing Cisco Elastic Services Controller on OpenStack
Configuring the Northbound Interface Access
Step 4
--image <image_name>\
--avail_zone nova:<openstack zone>\
--kad_vip <virtual IP address>\
--kad_vif <VRRP_Interface_Instance>\
--ha_node_list=<ESC_HA_NODE1_IP> <ESC_HA_NODE2_IP>\
--db_volume_id <cinder volume id>\
--ha_mode drbd_on_cinder
ON HA NODE 2:
$ ./bootvm.py <ESC_HA_Node2>\
--user_pass <username>:<password>\
--user_confd_pass <username>:<password>\
--gateway_ip <default gateway IP address>\
--net <network name1>\
--ipaddr <static ip address>\
--image <image_name>\
--avail_zone nova:<openstack zone>\
--kad_vip <virtual IP address>\
--kad_vif <VRRP_Interface_Instance>\
--ha_node_list=<ESC_HA_NODE1_IP> <ESC_HA_NODE2_IP>\
--db_volume_id <cinder volume id>\
--ha_mode drbd_on_cinder
After both VMs are rebooted; the keepalived state on one of ESC VM should be one of ESC VM should be in MASTER state and the other one should be in BACKUP state. You can check ESC HA Active/Standby state by using following command: $ sudo escadm status --v .
Configuring the Northbound Interface Access
When you configure ESC HA Active/Standby, you can also specify a virtual Anycast IP address to the HA
Active/Standby pair. The northbound interface as well as the service portal uses virtual Anycast IP address to access the ESC Primary HA Active/Standby instance. When deploying ESC HA Active/Standby, use the following arguments with the ./bootvm.py
script.
• --ha_node_list
• --kad_vip
• --kad_vif
For more details on these arguments, see section Appendix A: Cisco Elastic Services Controller Installer
Arguments .
The following section explains how to configure ESC HA Active/Standby with multiple interfaces and to configure the virtual Anycast IP address.
Configuring ESC HA Active/Standby with Multiple Interfaces
You can configure ESC HA Active/Standby with DRDB synchronization and VRRP heartbeat broadcasting on a network interface for data synchronization and VNF monitoring. You can use an additional network interface to allocate Virtual IP for the northbound access. To configure the multiple interfaces on ESC HA
Active/Standby nodes, use --ha_node_list, --kad_vip, --kad_vif arguments to specify these multiple network interfaces configuration. For details on these arguments, see section Appendix A: Cisco Elastic Services
Controller Installer Arguments .
26
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Installing Cisco Elastic Services Controller on OpenStack
Configuring the ESC HA Active/Standby Virtual IP Address
Note KeepAlived doesn’t support single IPv4 VIP address with a IPv6 VRRP instance.
Example configuration steps are shown below:
./bootvm.py <esc_ha1> \
--user_pass <username>:<password>
--user_confd_pass <username>:<password>
--image <image_id> \
--net <net-name> \
--gateway_ip <default_gateway_ip_address> \
--ipaddr <ip_address1> <ip_address2> \
--ha_node_list < IP addresses HA nodes1> < IP addresses for HA nodes2> \
--kad_vip <keepaliveD VIP of the HA nodes and the interface for keepaliveD VIP> \ (for example: --kad_vip 192.0.2.254:eth2)
--kad_vri <virtual router id of vrrp instance>
--kad_vif <virtual IP of the HA nodes or the interface of the keepalived VRRP> \ (for example: --kad_vif eth1 )
--ha_mode <HA installation mode> \
--route <routing configuration> \ (for example:192.0.2.254/24:192.168.0.1:eth1 )
--avail_zone nova:<openstack zone> \
Similarly, a three network interface can be configured for ESC HA Active/Standby nodes. An example three interfaces configuration is shown below with the following assumptions :
• Network 1 is an IPv6 network used for northbound connection. ESC VIP is allocated in this network and the Orchestrator send requests to ESC through ESC VIP.
• Network 2 is an IPv4 network used for ESC sync traffic (DRDB synchronization) and VRRP heartbeat.
This network is also used for OpenStack connection and VNF monitoring.
• Network 3 is another IPv4 network used for management. The SA, rsyslog, etc. can use this network to manage ESC.
./bootvm.py esc-ha-0 --image ESC-2_2_x_yyy --net network-v6 network --gateway_ip 192.168.0.1 --ipaddr
2001:cc0:2020::fa 192.168.0.239 192.168.5.239 --ha_node_list 192.168.0.239 192.168.0.243 --kad_vip
[2001:cc0:2020::fc/48]:eth0 --kad_vif eth1 --ha_mode drbd --route 172.16.0.0:eth1 --avail_zone nova: zone name
./bootvm.py esc-ha-1 --image ESC-2_2_x_yyy --net network-v6 network lab-net-0 --gateway_ip 192.168.0.1
--ipaddr 2001:cc0:2020::fa 192.168.0.239 192.168.5.239 --ha_node_list 192.168.0.239 192.168.0.243 --kad_vip
[2001:cc0:2020::fc/48]:eth0 --kad_vif eth1 --ha_mode drbd --route 172.16.0.0:eth1 --avail_zone nova: zone name
Configuring the ESC HA Active/Standby Virtual IP Address
In this option, the value of kad_vip argument should be a virtual IP, which allows the service portal and the northbound to access the Primary ESC and send requests to ESC HA Active/Standby service through virtual
IP (VIP).
If northbound and both ESC HA Active/Standby nodes are located in the same network, you can connect directly through the virtual IP (VIP). If northbound doesn't sit on the same network as ESC HA Active/Standby, assign a floating IP to ESC HA Active/Standby VIP using the procedure below:
1.
Create a port with the VIP address (kad_vip) in the same network as ESC's kad_vip connects.
neutron port-create network --name network_vip --fixed-ip subnet_id=network-subnet,ip_address=192.168.0.87
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
27
Installing Cisco Elastic Services Controller on OpenStack
Configuring the ESC L3 HA Active/Standby With BGP
2.
Deploy ESC HA Active/Standby . See Configuring High-Availability Active/Standby section in Installing
ESC on OpenStack.
Note Make sure the kad_vip using the same IP address as the port created above.
3.
Associate a floating IP with the port created above. The first uuid is the floating ip id and the second one is the port id.
neutron floatingip-associate <floating IP> <port ID>
Access ESC HA Active/Standby through the floating IP and it will connect to the ESC Primary node.
4.
For the portal access, make sure the keepalive network is accessible by your browser and the virtual IP is the IP address to access the portal of the Primary node.
For example, if the VIP is 192.0.2.254, access ESC HA Active/Standby portal with https://192.0.2.254:9001/.
Configuring the ESC L3 HA Active/Standby With BGP
To configure BGP for ESC HA Active/Standby, there are two options:
1.
Directly booting ESC HA Active/Standby L3 with BGP
2.
Using post configuration from existing ESC HA Active/Standby pair
To configure BGP for ESC HA Active/Standby, the following network parameters are required:
• BGP remote IP
• IP of the interface for BGP anycast routing
• BGP local AS number for routing configuration
• BGP remote AS number for routing configuration
• BGP routing configuration
• --bgp_local_ip
• --bgp_local_router_id
Note You must configure BGP router with neighbors, and restart it. Verify that the router is able to ping the AnyCast
IP.
On the BGP router, set two neighbors. The below BGP configuration is designed for Bird router. The configuration is router specific. For each types of router, the procedure is different:
The below configurations are given according to the bootvm command : protocol bgp E3 from EXABGP { neighbor 198.18.42.222 as 65012;
}
28
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Installing Cisco Elastic Services Controller on OpenStack
Configuring the ESC L3 HA Active/Standby With BGP protocol bgp E4 from EXABGP { neighbor 198.18.61.222 as 65011;
}
Booting an ESC VM with BGP options
[admin@na-test-52-1 ~]$ health.sh
============== ESC HA (MASTER) with DRBD ================= pgsql (pgid 3701) is running vimmanager (pgid 3528) is running monitor (pgid 4219) is running mona (pgid 3352) is running drbd (pgid 2422) is master etsi (pgid 5239) is running filesystem (pgid 0) is running snmp (pgid 12698) is running bgp (pgid 4308) is running keepalived (pgid 2736) is running portal (pgid 4190) is running confd (pgid 3220) is running escmanager (pgid 3880) is running
=======================================
ESC HEALTH PASSED
And
[admin@na-test-52-2 ~]$ health.sh
============== ESC HA (BACKUP) with DRBD ================= pgsql is stopped vimmanager is stopped monitor is stopped mona is stopped drbd (pgid 2471) is backup etsi is disabled at startup filesystem is stopped snmp is disabled at startup bgp is stopped keepalived (pgid 2787) is running portal is stopped confd is stopped escmanager is stopped
=======================================
ESC HEALTH PASSED
Use below values for BGP post configuration:
./bootvm.sh <NETWORK_VM_name> \
--image <ESC_image> \
--ipaddr <static_IP_address1> <IP_address2> <IP_address_3>\
--gateway_ip <gateway IP address of NETWORK> \
--net <net_id1> <net_id2> <net_id3> \
--network_params_file <network_params_file> \
--host_mapping_file <host_mapping_file> \
--avail_zone <openStack_zone> \
--ha_node_list <IP_address_ha_node_1> <IP_address_ha_node_2> \
--user_portal_pass <user>:<password> \
--user_rest_pass <user>:<password> \
--confd_aes_key <password> \
--kad_vif <interface> \
--user_confd_pass <user>:<password> \
--user_pass <user>:<password> \
--kad_vip <vip address> \
--bgp_remote_ip <BGP_remote_IP_address> \
--bgp_local_ip <BGP_local_IP_address> \
--bgp_local_as <BGP_local_AS_#> \
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
29
Installing Cisco Elastic Services Controller on OpenStack
Configuring the ESC L3 HA Active/Standby With BGP
--bgp_remote_as <BGP_remote_AS_#>\
--bgp_local_router_id <local_BGP_reouter_id> \
--bgp_anycast_ip <BGP_anycast_IP> \
--bgp_md5 <BGP_MD5>
Where,
--ip_addr: ----> the local IP address of the ESC VM
--net: ----> the network id(s) in OpenStack that ESC will connect to.
--bgp_anycast_ip: ----> the IP address that NCS will communicate with
--bgp_remote_ip: ----> this IP address of the external router that ESC will peer with
--bgp_local_as: ----> local AS for the ESC "router"
--bgp_remote_as: ----> AS number for the external router ESC will peer with
--bgp_local_router_id: ----> id for the esc "router"
--bgp_md5: ----> optional - md5 to be used to pair with external router
Configuring BGP HA Active/Standby Post Configuration
1.
For each HA Active/Standby instance, create the network interface file:
# cat /etc/sysconfig/network-scripts/ifcfg-lo:2
IPV6INIT='no'
<------------------ bgp anycast IP IPADDR='10.0.124.124'
BROADCAST='10.0.124.255'
NETWORK='10.0.124.0'
NETMASK='255.255.255.0'
DEVICE='lo:2'
ONBOOT='yes'
NAME='loopback'
2.
For each HA Active/Standby instance:
Bring lo:2 up
# ifup lo:2
To configure BGP for ESC HA Active/Standby, use the escadm tool in ESC Virtual Machine, as shown below:
$ sudo bash
# escadm bgp set --local_ip LOCAL_IP --anycast_ip ANYCAST_IP --remote_ip REMOTE_IP --local_as
LOCAL_AS --remote_as REMOTE_AS
--local_router_id LOCAL_ROUTER_ID
# escadm reload
# reboot
Example:
[root@bgp-001 admin]# escadm bgp set --local_ip 198.18.42.124 --anycast_ip 10.0.124.124
--remote_ip 192.168.0.2 --local_as 65124 --remote_as 65000 --local_router_id 198.18.42.124
[root@bgp-002 admin]# escadm bgp set --local_ip 198.18.42.125 --anycast_ip 10.0.124.124
--remote_ip 192.168.0.2 --local_as 65114 --remote_as 65000 --local_router_id 198.18.42.125
Configuring a BGP Router
To configure a BGP router, log in to the BGP router to configure BGP Anycast routing. The required parameters are:
<Router_AS_#> same as --bgp_remote_as above
<Esc_ip_address> must be the ESC VM's IP address configured for BGP advertisement.
<ESC_AS_#> same as --bgp_local_as shown above configure router bgp <Router_AS_#>
30
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Installing Cisco Elastic Services Controller on OpenStack
Important Notes neighbor <ESC_IP_address> remote-as <ESC_AS_#> address-family ipv6 unicast route-policy anycast-in in route-policy anycast-out out route-policy anycast-in pass end-policy route-policy anycast-out drop end-policy commit
Important Notes
• ESC HA Active/Standby
• An HA Active/Standby failover takes about 2 to 5 minutes. The ESC service will not be available during the switchover time.
• When the switchover is triggered during transactions, all incomplete transactions will be dropped.
The requests should be re-sent by northbound if it does not receive any response from ESC.
• External Storage
• If the Primary ESC instance is suspended by OpenStack command, the switch over will be triggered but the cinder volume won't be attached to the new Primary ESC instance. This is not a valid use case for ESC HA Active/Standby.
• Internal Storage
• Two ESC instances have to be deployed to establish the HA Active/Standby solution. The ESC HA
Active/Standby will start to work when both ESC instances are successfully deployed and are able to connect to each other. If you just deploy one ESC instance with HA Active/Standby parameters, the ESC instance keeps Switching-to-Master state and will not be able to provide any service until it reaches its peer.
• Split-brain scenario can still happen in this ESC HA Active/Standby solution, although the chance is very low.
• ETSI-specific Notes
ESC supports ETSI MANO northbound API defined by the European Telecommunications Standards
Institute (ETSI) for NFV Management and Orchestration. The ETSI MANO API is another programmatic interface based on the REST architecture. For more information, see ETSI MANO Compliant Lifecycle
Operations in the Cisco Elastic Services Controller User Guide . Consider the following notes while enabling ETSI service on ESC which is in HA Active/Standby mode:
• The server.address value in the etsi-vnfm.properties
file must be set to a Virtual IP (VIP) address.
This IP address can be used to communicate back to the ETSI services using API callbacks. If the virtual IP address is not specified, the ETSI service startup may fail.
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
31
Installing Cisco Elastic Services Controller on OpenStack
Troubleshooting High Availability Active/Standby
• The ETSI VNFM service and the escadm script generate and maintain the security.user.name and security.user.password property values. You should not change it manually.The
security.user.password is encoded.
Troubleshooting High Availability Active/Standby
• Check for network failures. If a network problem occurs, you must check the following details:
• The IP address assigned is correct, and is based on the OpenStack configuration.
• The gateway for each network interface must be pingable.
• Check the logs for troubleshooting:
• The ESC Admin logs at /var/log/esc/escadm.log
• The ESC manager log at /var/log/esc/escmanager.log
• The AA elector log at /var/log/esc/elector-{pid}.log
• Check for DRBD (Replication based ESC HA Active/Standby) for Internal Storage solution:
• Check the DRBD configuration file at
/etc/drbd.d/esc.res
• Access the DRBD log
•
/var/log/messages|grep drbd
• To collect log files through CLI, use the following command on all ESC nodes: sudo escadm log collect
32
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
C H A P T E R
5
Cisco Elastic Services Controller Active/Active
High Availability Overview
This chapter contains the following sections:
•
Cisco Elastic Services Controller Active/Active HA Overview, on page 33
•
ESC Active/Active Architecture, on page 34
Cisco Elastic Services Controller Active/Active HA Overview
ESC supports High Availability (HA) in the form of an Active/Active model. ESC Active/Active HA has three VMs as a cluster in one datacenter.. There are two datacenters. Between the two datacenters, one datacenter acts as active, another acts as standby. ESC Active/Active HA uses Openstack heat template to deploy the three VM cluster in a datacenter.
In a datacenter, ESC service runs on each VM; however, there is only one ESC on a datacenter that runs as cluster leader. DB service runs only on the leader. The ESC services on the other two ESC VMs run as cluster follower. The DB service is only active on ESC services leader VM.
DRBD synchronises the data among ESC VMs. The ESC service on the three ESC VMs connect to the active
DB service. When a leader switchover happens, all the ESC service connects to the newly active DB service.
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
33
ESC Active/Active Architecture
ESC Active/Active Architecture
Figure 2: Cisco Elastic Services Controller Active/Active Architecture
Installing Cisco Elastic Services Controller on OpenStack
34
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
C H A P T E R
6
Installing Active/Active High Availability Cluster
This chapter contains the following sections:
•
Installing Active/Active High Availability Cluster , on page 35
•
Validating Active/Active High Availability Cluster Post Installation, on page 37
•
Adding Default VIM Connector to the Active/Active High Available Cluster, on page 38
•
Adding BGP in an Active/Active Cluster, on page 38
Installing Active/Active High Availability Cluster
To configure the Active/Active HA cluster, you can run the following commands on OpenStack, where openrc my-server is openrc for the OpenStack, and test is the stack name.
source ~/elastic-services-controller/esc-bootvm-scripts/openrc my-server-42 openstack stack create test -t aa.yaml
To check the status of the stack, use the following commands:
• openstack stack list
• openstack stack show test
• openstack stack event list test
Once the stack is in
CREATE_COMPLETE status, you can ssh into the VMs. The openstack stack show test command lists the IP addresses for the 3 VMs, you can use them to access to the VMs.
Note The Active/Active HA Cluster is deployed on OpenStack only.
Setting up the User Configuration
You can configure some of the parameters based on your settings, such as network, subnet, using either static
IP or DHCP to allocate IP address, flavor, image, password, etc. The configurable parameters are available in aa-param.yaml
, Openstack heat environment file, when the ESC cluster is instantiated via heat template
( aa.yaml
).
To instantiate ESC cluster, use the following example:
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
35
Installing Cisco Elastic Services Controller on OpenStack
Setting up the User Configuration openstack stack create name -t aa.yaml -e aa-params.yaml
The following example shows how to use a static IP address to configure the port while using environment template: sample@my-server-39:~/aa4.5/apr15$ more aa-params.yaml
parameters: network_1_name: network subnet_name: subnetwork esc_1_ip: 172.23.0.228
esc_2_ip: 172.23.0.229
esc_3_ip: 172.23.0.230
Following are the user configurable parameters available in aa.yaml : parameters: network_1_name: type: string description: Name of the image default: network subnet_name: type: string description: subnet name esc_1_ip: type: string description: static IP address of esc-1 VM.
esc_2_ip: type: string description: static IP address of esc-2 VM.
esc_3_ip: type: string description: static IP address of esc-3 VM.
resources: esc_1_port: type: OS::Neutron::Port properties: network_id: { get_param: network_1_name } fixed_ips: [ { "subnet": { get_param: subnet_name}, "ip_address": { get_param: esc_1_ip
} } ] esc_2_port: type: OS::Neutron::Port properties: network_id: { get_param: network_1_name } fixed_ips: [ { "subnet": { get_param: subnet_name}, "ip_address": { get_param: esc_2_ip
} } ] esc_3_port: type: OS::Neutron::Port properties: network_id: { get_param: network_1_name } fixed_ips: [ { "subnet": { get_param: subnet_name}, "ip_address": { get_param: esc_3_ip
} } ]
...omitting...
The following example shows how to use configurable image, flavor, and vm name prefix from environment template. Ensure that the flavor used suits the ESC deployment requirement.
sample@my-server-39:~/aa4.5/apr15$ more aa-params.yaml
parameters:
36
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Installing Cisco Elastic Services Controller on OpenStack
Validating Active/Active High Availability Cluster Post Installation nameprefix: abc image_name: ESC-5_0_DEV_4 flavor_name: m1.large
sample@my-server-39:~/aa4.5/apr15$
The following example shows how to use configurable image, flavor, and vmnameprefix from heat template: parameters: nameprefix: type: string description: Name prefix of vm default: helen image_name: type: string description: Name of the image default: ESC-5_0_DEV_4 flavor_name: type: string description: Name of the image default: m1.large
esc-1: type: OS::Nova::Server properties: name: str_replace: template: $nameprefix-esc-1 params:
$nameprefix : { get_param: nameprefix } image: { get_param: image_name } flavor: { get_param: flavor_name }
... omitting...
Validating Active/Active High Availability Cluster Post
Installation
To verify all the ESC nodes use the following commands. Here, all ESC nodes implies each VMs.
sample@my-server-39:~$ openstack --insecure server list | grep abc
| 5ea6fc79-2b2a-4064-9c6a-a83d6b06c225 | abc-test-esc-3
| ACTIVE | network=172.23.7.203
| ESC-5_0_DEV_13 | m1.large
| 10e165d9-5015-4b64-88fe-19e874e6e7c1 | abc-test-esc-1
| ACTIVE | network=172.23.7.205
| ESC-5_0_DEV_13 | m1.large
| 35f6bad1-865f-4155-8411-d37e2616e079 | abc-test-esc-2
| ACTIVE | network=172.23.7.204
| ESC-5_0_DEV_13 | m1.large
|
|
|
To find out the leader node, ssh into one of the nodes/VMs and run the following:
[admin@sample-test-esc-1 ~]$ sudo escadm elector dump
{
"[email protected]:42143": {
"state": "FOLLOWER",
"location": "[email protected]:42143",
"service": "esc_service"
},
"[email protected]:50474": {
"state": "FOLLOWER",
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
37
Installing Cisco Elastic Services Controller on OpenStack
Adding Default VIM Connector to the Active/Active High Available Cluster
}
"location": "[email protected]:50474",
"service": "esc_service"
},
"[email protected]:59514": {
"state": "LEADER",
"location": "[email protected]:59514",
"service": "esc_service"
}
Adding Default VIM Connector to the Active/Active High
Available Cluster
You can add a default vim connector to the 3 ESC VM cluster in the following two different ways:
1.
Once the 3 ESC VM cluster boots up, use netconf interface to add a default vim connector by using the following command. The vim.xml
is the default vim connector deployment file.
[admin@name-esc-1 ~]$ esc_nc_cli --host db.service.consul --user admin --password
<admin_password> edit-config vim.xml
2.
To configure a default vim connector, you must add the default vim connector configuration inside the heat template day0 file. Add the following block in the write_files section under the cloud-config in aa-day0.yaml file. Once the 3 ESC VM cluster boots up, it creates a default vim connector by its own.
The following example shows how to configure default vim connector in heat template day0 file:
- path: /opt/cisco/esc/esc-config/esc_params.conf
content: | openstack.os_auth_url=http://172.16.103.38:35357/v3 openstack.os_project_name=admin openstack.os_tenant_name=admin openstack.os_user_domain_name=default openstack.os_project_domain_name=default openstack.os_identity_api_version=3 openstack.os_image_api_version=2 openstack.os_username=admin openstack.os_password=password1
Adding BGP in an Active/Active Cluster
To initiate the BGP process, add the anycast IP to the lo device. You can configure it in sys-cfg.yaml
.
For example:
#cloud-config write_files:
- path: /etc/cloud/cloud.cfg.d/sys-cfg.yaml
content: | network: version: 1 config:
- type: physical name: lo subnets:
- type: static
38
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Installing Cisco Elastic Services Controller on OpenStack
Adding BGP in an Active/Active Cluster address: 172.23.188.188/23
You must specify the advertise IP for consul. In esc-config.yaml
, add the following: consul: advertise_addr: 172.23.1.149
Following is the example for adding the BGP section: bgp: depend_on: elector:leader anycast_ip: 172.23.188.188/23 local_as: '65001' local_ip: 192.168.1.11
local_router_id: 192.168.1.11
remote_as: '65000' remote_ip: 192.168.1.12
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
39
Adding BGP in an Active/Active Cluster
Installing Cisco Elastic Services Controller on OpenStack
40
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
C H A P T E R
7
Managing Cluster in ESC Active/Active High
Availability
This chapter contains the following section:
•
Managing Cluster in ESC Active/Active High Availability, on page 41
Managing Cluster in ESC Active/Active High Availability
To do cluster management in ESC Active/Active HA, you can call some escadm commands on any ESC node and execute it on all nodes in Active/Active cluster.
The following are the supported commands for cluster level calling:
• escadm start
• escadm stop
• escadm geo start/stop
• escadm vim show
Add the option
--cluster to execute any of the previous commands to be executed on all nodes in
Active/Active local cluster:
For example: sudo escadm geo start --v –cluster
The output of the command and exit code of each node are displayed with the IP address of that node to distinguish the execution output, and the output on the local node.
Example for escadm geo start --cluster
:
[root@name-geo-2-1 admin]# escadm geo start --cluster
192.168.1.13 # remote host exit status : 0
Starting geo service: [OK]
192.168.1.12 # remote host exit status : 0
Starting geo service: [OK]
Starting geo service: [OK] # output of the local node
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
41
Managing Cluster in ESC Active/Active High Availability
Installing Cisco Elastic Services Controller on OpenStack
42
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
C H A P T E R
8
Configuring GEO in Active/Active High
Availability
•
Configuring GEO in Active/Active High Availability, on page 43
•
Verifying GEO Services, on page 45
•
Active/Active GEO HA Failure Injection Limitations, on page 46
Configuring GEO in Active/Active High Availability
ESC Active/Active HA has three VMs as a cluster in one datacenter. The second datacenter consists of
GEO-HA.
Following are the 6 pre-defined roles in GEO:
1.
init
: initial role of geo service
2.
pre_primary
3.
primary
4.
pre_secondary
5.
secondary
6.
unknown
: used when consul is not reachable
GEO can change one role to another. Transitions are defined in esc-config.yaml
. Each transition is divided into the following 3 parts:
• from
: current role
• goto
: destination role
• condition
: when GEO changes the role
Transition Conditions
When A/A HA Geo comes up, the primary datacenter has to go through init
, pre_primary
, and primary states; meanwhile, the secondary datacenter has to go through init, pre-secondary, and secondary states. When
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
43
Installing Cisco Elastic Services Controller on OpenStack
Transition Conditions all ESC VMs' health check pass on both primary and secondary datacenters, the ESC A/A HA GEO is up and running. It is ready for use.
Condition Functions
The following are all supported condition functions:
1.
return
: do nothing but return the argument
2.
and
: return true if all arguments are true
3.
or
: return true if any of the argument is true
4.
len
: return the length of the argument
5.
equals
: return true if all arguments are equal
6.
true
: return true if args can be tested for truth value in python
7.
false
: opposite to 'true'
The following are samples for GEO configurations on primary datacenter: on_init: consul start on_primary: start on_secondary: stop on_stop: consul stop startup: manual transitions:
- condition: return: and:
- equals:
- len: service1
- 3
- equals:
- len: service2
- 3 rise: 3 service1: dc: dc1 name: consul_agent passing: true type: service service2: dc: dc2 name: geo passing: true type: service from: init goto: primary
- condition: fall: 2 return: equals:
- len: service
- 3 service: dc: dc1 name: consul_agent from: primary goto: secondary
The following are samples for GEO configurations on secondary datacenter:
44
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Installing Cisco Elastic Services Controller on OpenStack
Verifying GEO Services on_init: consul start on_primary: start on_secondary: stop on_stop: consul stop startup: manual transitions:
- condition: return: and:
- equals:
- len: service1
- 3
- equals:
- len: service2
- 3 rise: 3 service1: dc: dc1 name: consul_agent passing: true type: service service2: dc: dc2 name: geo passing: true type: service from: init goto: secondary
- condition: fall: 2 return: equals:
- len: service
- 3 service: dc: dc1 name: consul_agent from: secondary goto: primary
Verifying GEO Services
To start Active/Active GEO-HA, run the following command: escadm geo start
To verify the GEO status, use the following command:
[root@test-geo3-ha-1 esc-scripts]# escadm geo status geo (pgid 3745) is primary
The verify the GEO services in current datacenter, use the following command:
[root@test-geo3-ha-1 esc-scripts]# escadm geo dump
{
"[email protected]:44793": {
"role": "primary",
"location": "[email protected]:44793",
"service": "geo"
},
"[email protected]:52459": {
"role": "primary",
"location": "[email protected]:52459",
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
45
Installing Cisco Elastic Services Controller on OpenStack
Active/Active GEO HA Failure Injection Limitations
"service": "geo"
},
"[email protected]:38841": {
"role": "primary",
"location": "[email protected]:38841",
"service": "geo"
}
}
To verify all the GEO services in the datacenters, use the following command:
[root@test-geo4-ha-1 admin]# escadm geo dump --all
{
"[email protected]:36760": {
"role": "primary",
"location": "[email protected]:36760",
"service": "geo"
},
"[email protected]:42362": {
"role": "secondary",
"location": "[email protected]:42362",
"service": "geo"
},
"[email protected]:51936": {
"role": "primary",
"location": "[email protected]:51936",
"service": "geo"
},
"[email protected]:37604": {
"role": "secondary",
"location": "[email protected]:37604",
"service": "geo"
},
"[email protected]:44450": {
"role": "primary",
"location": "[email protected]:44450",
"service": "geo"
},
"[email protected]:34875": {
"role": "secondary",
"location": "[email protected]:34875",
"service": "geo"
}
}
Active/Active GEO HA Failure Injection Limitations
The ESC Active/Active GEO HA is enhanced to support a one-way GEO HA failover with a maintenance window to move it back to healthy state.
If a GEO failover happens and the ESC VMs move to the unhealthy state, use the following steps to bring
ESC A/A GEO HA back to the healthy state through manual intervention:
Procedure
Step 1 Resolve any issues in Datacenter1 (DC1), that may have caused the failures and enabled the GEO switch to
Datacenter2 (DC2).
46
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Installing Cisco Elastic Services Controller on OpenStack
Active/Active GEO HA Failure Injection Limitations
Step 2
Step 3
Step 4
Step 5
Ensure that the consul is running in at least 2 nodes in DC1 and DC2.
Run the sudo escadm geo replicate –all command on DC2 when DC1 has at least two nodes with consul in the running state.
Run the sudo escadm stop command on all the 6 ESC VMs.
Run the sudo escadm geo restart command on all the 6 ESC VMs.
What to do next
Note ESC does not support any operations on ESC VMs in DC1 after the GEO HA fails over to DC2.
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
47
Active/Active GEO HA Failure Injection Limitations
Installing Cisco Elastic Services Controller on OpenStack
48
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
C H A P T E R
9
DRBD Encryption for ESC Active/Standby and
Active/Active HA Data Replication
This chapter contains the following sections:
•
DRBD Encryption for ESC HA Data Replication , on page 49
•
ESC HA with DRBD Encryption, on page 49
DRBD Encryption for ESC HA Data Replication
ESC uses DRBD for data replication across different nodes in an HA cluster environment. DRBD layers logical block device over existing local block devices on cluster nodes.
The written data to the primary node is transferred to the lower-level block device and simultaneously propagated to the secondary node(s). Currently ESC mounts DRBD device directly on
/opt/cisco/esc/esc_database
.
For Example:
# df
Filesystem devtmpfs tmpfs tmpfs
…
1K-blocks
2961760
2972164
2972164
Used Available Use% Mounted on
0
4
8748
2961760
2972160
2963416
0% /dev
1% /dev/shm
1% /run tmpfs 594436 0 594436 0% /run/user/1004
/dev/mapper/esc_crypt 3028620 57212 2797848 3% /opt/cisco/esc/esc_database
Block device encryption encrypts or decrypts the data transparently as it is written/read from block devices, the underlying block device sees only encrypted data.
Security is enhanced with the dm-crypt/LUKS layer to encrypt the data in DRBD partition, between filesystem and DRBD device. LUKS (Linux Unified Key Setup) is a specification for block device encryption.
ESC HA with DRBD Encryption
The following bootvm commands boot ESC HA with DRBD encrypted:
Select DRBD LUKS encryption through bootvm.py. There are 4 variations that result in the equivalent ESC day-0 user-data/esc-config.yaml when passed to the ESC VM instance.
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
49
Installing Cisco Elastic Services Controller on OpenStack
ESC HA with DRBD Encryption bootvm.py --fs_encryption_type luks --fs_luks_key_prompt bootvm.py --fs_encryption_type luks --fs_luks_key 'LuksKeyValue'
=> injects the luks key into default file location /opt/cisco/esc/esc-config/luks_key bootvm.py --fs_encryption_type luks --file root:0400:/opt/cisco/esc/esc-config/luks_key:path-to-local-luks-key-file
=> injects a local file containing the luks key
The following command shows an advanced usage to manage the luks key file at a different path on the ESC
VM filesystem: bootvm.py --fs_encryption_type luks --fs_luks_key_file path-on-esc-vm-luks-key-file
--fs_luks_key_prompt bootvm.py --fs_encryption_type luks --fs_luks_key_file path-on-esc-vm-luks-key-file
--fs_luks_key 'LuksKeyValue'
=> injects the luks key into a different file location bootvm.py --fs_encryption_type luks --fs_luks_key_file path-on-esc-vm-luks-key-file --file root:0400:path-on-esc-vm-luks-key-file:path-to-local-luks-key-file
=> injects the luks key as read from a local file into a different file location
Use the following commands if you are installing ESC with a custom user-data, such as ESC Active/Active deployment with heat templates:
The luks key is specified as a day-0 file and an attribute under esc-config.yaml / filesystem.
Encode the luks key as base64: base64 <<<'LuksKeyValue'
THVrc0tleVZhbHVlCg==
Then, insert the previous luks key into the user-data / cloud-config file: write_files:
- path: /opt/cisco/esc/esc-config/luks_key owner: root:root permissions: '400' encoding: b64 content: THVrc0tleVZhbHVlCg==
- path: /opt/cisco/esc/esc-config/esc-config.yaml
owner: root:esc-user permissions: '0640' content: | resources: filesystem: depend_on: drbd:master encryption_type: luks luks_key_file: /opt/cisco/esc/esc-config/luks_key
50
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
C H A P T E R
10
Upgrading ESC Active/Active High Availability
This chapter contains the following sections:
•
Upgrading ESC Active/Active High Availability, on page 51
Upgrading ESC Active/Active High Availability
Cisco Elastic Service Controller Active/Active HA supports local Active/Active to Active/Active simple upgrade.
Local Active/Active to Active/Active Simple Upgrade
Procedure
Step 1
Step 2
Step 3
Step 4
Back up the database. For more information, see the section,
Backing up the Database, on page 51
.
Remove old VMs. For more information, see the section,
Removing the Old VMs, on page 52 Removing
Old VMs .
Install new ESC Active/Active VMs. For more information, see the section,
Installing a New ESC Active/Active
Restore the ESC database. For more information, see the section,
Restoring the ESC Database, on page 53 .
Backing up the Database
Before upgrade, take the back up of database using the following procedure.
Procedure
Step 1
Step 2
Put the ESC leader VM in the maintenance mode, by running the following command: escadm op_mode set --mode=maintenance
Wait until all the ESC VMs stop processing transactions. To verify, run the following command: escadm ip_trans
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
51
Installing Cisco Elastic Services Controller on OpenStack
Removing the Old VMs
Step 3
Step 4
Create a backup of database on the ESC leader by running the following command: escadm backup --file dbback.tar, scp <dbback.tar>
Collect logs from all the ESC VMs by running the following command: escadm log collect scp
Removing the Old VMs
Procedure
Step 1
Step 2
Shutdown all the ESC follower VMs and the ESC leader VM running the following command: nova stop
Remove old ESC Active/Active VMs from the OpenStack by running the following command: openstack stack delete {stack name}
Installing a New ESC Active/Active VM
After backing up the database and shutting down of the old ESC Active/Active VMs, a new/upgraded (based on new ESC package) Active/Active ESC VM must be installed.
Procedure
Step 1
Step 2
Step 3
Step 4
For OpenStack, register a new image by running the following command: glance image-create
Install the new ESC Active/Active VMs by running the following command: openstack stack create {stack name} --template {location of the template file}
Check all the ESC VMs health, and stop the escadm service in the follower VMs by running the following command: sudo escadm stop for all followers VMs
Once the escadm service is stopped in all the follower VMs, stop the escadm service in the leader VM by running the following command: sudo escadm stop
52
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Installing Cisco Elastic Services Controller on OpenStack
Restoring the ESC Database
Restoring the ESC Database
Restore the ESC database on the new ESC instance, using the following procedure:
Procedure
Step 1
Step 2
Step 3
Step 4
Step 5
Copy the backup file to the new leader by running the folowing command: scp
Restore the database on the ESC leader by running the following command: sudo escadm restore --file <dbback.tar>
After restoring, the restore process starts the escadm service in the leader VM. However, the escadm service remains to be stopped in all the follower VMs.
Verify that the ESC leader VM is running all the services without any interruption.
Put the ESC leader VM into the operation mode by running the following command: sudo escadm op_mode set --mode=operation
Start the ESC services on the follower VMs by running the following command: sudo escadm start
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
53
Restoring the ESC Database
Installing Cisco Elastic Services Controller on OpenStack
54
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
P A R T
II
Installing Cisco Elastic Services Controller on
VMware vCenter
•
•
Installing Cisco Elastic Services Controller on VMware vCenter, on page 61
•
Installing High Availability, on page 69
C H A P T E R
11
Prerequisites
The following sections detail the prerequisites for installing Cisco Elastic Services Controller:
•
Virtual Resource and Hypervisor Requirements, on page 57
•
•
Virtual Resource and Hypervisor Requirements
The following table lists the prerequisites to install Cisco Elastic Services Controller on VMware vCenter or vSphere:
See the VMware Compatibility Guide to confirm that VMware supports your hardware platform.
Requirement Description
System Requirements
Virtual CPUs
Memory
Disk Space
Hypervisor Requirements
VMWare vCenter
VMware vCloud Director (vCD)
4 VCPUs
8 GB RAM
40 GB
ESC supports VMware and vCenter versions 6.7.
ESC supports VMware vCD versions 9.7.
vCenter Resources
Resources to be created/installed on vCenter:
• Datacenters : At least one datacenter. For more details, see the Important Notes below.
• Hosts : Host configuration based on your targeted performance objectives. Each Host under the single vDS must have at least two physical Network Interface Card (NIC) connected, (one for vCenter
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
57
Installing Cisco Elastic Services Controller on VMware vCenter
Important Notes
Management Interface by default, and the other used to assign to VDS's uplink portgroup). This setup is required for data access across hosts.
• Compute Clusters : Clusters can be created to group several hosts together.
• Datastores : Shared datastore is required if user wants to leverage DRS.
• Distributed Switches : At least one distributed switch that will contains all the VNF supporting networks.
Important Notes
Keep in mind the following important notes while installing ESC on a VMware:
• A single ESC instance will only support:
• Multiple Datacenter supported deployment, network, image, subnet creation
• One vSphere Distributed Switch (VDS)
• DPM, HA Active/Standby, and vMotion must be off.
• If DRS is enabled, it has to be in the "Manual Mode".
• Fault Tolerance is not supported.
• Datastore Cluster is not supported, only flat datastore(s) structure under the cluster or under the datacenter are supported.
• ESC only supports a default resource pool. Adding and creating resource pools are not supported.
• Image (Template) created through ESC are stored under /esc-ovas folder .
• Day-0, smart license, and other supported files are packed into a ISO file, and uploaded to the same folder where the VM rest, then mount it as a CD-ROM to the VM.
• ESC/VIM does not respond for the name and file content passed in for generating ISO file. They have to be provided according to each template's requirements. e.g. for ASAv, the day-0 config has to be named as "day0-config", and smart license token has to be named as "idtoken".
• When you see the error message "Networking Configuration Operation Is Rolled Back and a Host Is
Disconnected from vCenter Server", it is due to a vCenter's limitation. See the Troubleshooting guide , page 91 to increase the timeout for rollback.
• The following VM features and operations are not supported in all versions of the Cisco CSR 1000V. If still these operations are used or performed, there may be risk of encountering dropped packets, dropped connections, and other error statistics.
1.
DRS
2.
Suspend
3.
Snapshot
4.
Resume
58
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Installing Cisco Elastic Services Controller on VMware vCenter
Important Notes
• Although deployments can be processed without shared storage, ESC does not guarantee optimized computing resource. Shared storage(s) should associate with as many as possible hosts, which will give more opportunity to DRS to balance resources.
• Every time a redeploy happens as part of recovery on VMware, VM's interface(s) will have different
MAC addresses.
• All the VM group defined in a datamodel must accompany with a "zone-host" placement policy, meaning the deployment has to be either host-targeted or cluster-targeted.
• Recovery may fail, if a VM has PCI/PCIe passthrough device(s) attached, when it's recovered to a computing-host (picked based on ESC placement algorithm) which does not have any PCI/PCIe passthrough enabled device available .
• For PCI/PCIe passthrough working, DRS has to be off .
• If you experience a PowerOn error on a VM that has PCI/PCIe passthrough device(s) attached to it, update the VM or the image (template) the VM is cloned from, using the solution described here .
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
59
Important Notes
Installing Cisco Elastic Services Controller on VMware vCenter
60
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
C H A P T E R
12
Installing Cisco Elastic Services Controller on
VMware vCenter
This chapter describes how to install Cisco Elastic Services Controller on VMware vCenter and includes the following sections:
•
Installing Cisco Elastic Services Controller on VMware vCenter, on page 61
•
Next Steps: Cisco Elastic Services Controller Virtual Machine, on page 67
Installing Cisco Elastic Services Controller on VMware vCenter
Cisco Elastic Services Controller can be installed in a VMware ESXi hypervisor and can be accessed or managed using vSphere client of VMware. You can install Cisco Elastic Services Controller in a VMware environment using an Open Virtual Appliance (OVA) package.
The VMware vSphere client can be connected directly to your ESXi installation, or it can be connected to a vCenter server which in turn is connected to your vSphere installation. Connecting through vCenter provides a number of capabilities that connecting directly to ESXi does not. If a vCenter server is available and associated with the ESXi installation, it should be used.
Preparing to Install Cisco Elastic Services Controller
In order to install Cisco Elastic Services Controller and configure its network connection, you have to answer several questions. Some of these questions concern the networking environment in which the virtual machine is being installed, and some of them concern values which are unique to the particular virtual machine being installed.
Before you perform the installation, ensure that you are prepared by reviewing this checklist:
Requirements Your Information/ Notes
OVA image location
OVA image
VSphere Web Client
Hostname
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
61
Installing Cisco Elastic Services Controller on VMware vCenter
Installing the Elastic Services Controller Using the OVA Image
Requirements
IP address
Subnet mask
Network vCenter IP vCenter Port vCenter credentials
Datacenter Name
Datastore Host
Computer Cluster Name
Your Information/ Notes
Installing the Elastic Services Controller Using the OVA Image
To install Cisco Elastic Services Controller, you must first download the correct installation file.
Using vSphere, connect directly to the ESXi installation or the vCenter server, and select the ESXi installation where the OVA is to be deployed.
This procedure describes how to deploy the Elastic Services Controller OVA image on VMware.
Before you begin
• Set your keyboard to United States English.
• Confirm that the Elastic Services Controller OVA image is available from the VMware vSphere Client.
• Make sure that all system requirements are met as specified in the Chapter 6: Prerequisites.
• Gather the information identified in Preparing to Install Cisco Elastic Services Controller.
Procedure
Step 1
Step 2
Step 3
Using the VMware vSphere Client, log in to the vCenter server.
Choose vCenterHome > Hosts and Clusters . Right click the host where you want to deploy ESC, and then choose Deploy OVF Template .
In the wizard, provide the information as described in the following table:
Screen
Select source
Action
Select the Elastic Services Controller OVA.
Review details
Select name and folder
Review OVF Template details.
Enter a name and select a folder for the VM.
62
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Installing Cisco Elastic Services Controller on VMware vCenter
Screen
Select configuration
Select a resource
Select storage
Select networks
Customize template
Bootstrap Properties
Username
Password
Host name
Network IP
Network Gateway
Enable Https Rest
Enable Portal startup
VIM Settings of vCenter Server vCenter IP vCenter Port vCenter Username vCenter Password
Installing the Elastic Services Controller Using the OVA Image
Action
Select any one of the following deployment configurations:
• Large, 1 Network
• Large, 2 Networks
• Large, 3 Networks
Select the host or cluster to run the ESC template.
Select a location to store the files for the VM and a provisioning type. The storage can be local or shared remote, such as NFS or SAN.
You can choose either Thin provisioned format or
Thick provisioned format to store the VM virtual disks.
Based on the Network configuration for the deployment selected in the Select configuration option, you can allocate the pre-configured networks in vCenter to ESC network interfaces.
Administrator username for remote login.
Administrator password.
VM Hostname.
VM IP address.
Gateway IP address.
Enable external REST interface over HTTPS on port
8443.
Enable Portal startup at port 9001 (for https).
IP address of vCenter server for VNF deployment .
Port of the vCenter server.
Username to access vCenter server.
Password to access vCenter server.
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
63
Installing Cisco Elastic Services Controller on VMware vCenter
Installing Elastic Services Controller Using OVF Tool
Step 4
Step 5
Step 6
Step 7
Screen
Datacenter Name
Datastore Name
Datastore Host
Ready to Complete
Public Key
ConfD Username
ConfD Password
ConfD Public Key
Action
Name of the Datacenter in target vCenter for VNF deployment (Default VDC after Multi-VDC supported)
The destination datastore of all the image (template) create through ESC .
The destination computing-host of all the image
(template) create through ESC.
Review the deployment settings.
Caution Any discrepancies can cause VM booting issues. Carefully review the IP address, subnet mask, and gateway information for accuracy.
Administrator authorized public key for remote login.
Administrator username for netconf and ConfD CLI.
Administrator password for netconf and ConfD CLI.
Administrator authorized public key for netconf and
ConfD CLI.
Check the Power on after deployment check box to power on the VM after deployment.
Click Finish .
A progress indicator shows the task progress until Elastic Services Controller is deployed.
After Elastic Services Controller is successfully deployed, click Close .
Power on the Elastic Services Controller VM.
Installing Elastic Services Controller Using OVF Tool
In addition to installing the Elastic Services Controller using the OVA image, you can use the VMware OVF
Tool, a command-line client, to install Elastic Services Controller on VMware vCenter or vSphere.
To install Elastic Services Controller (ESC) from the command line, do the following:
Procedure
Step 1 Use the probe mode to learn the properties of the OVA package. The probe mode allows you to investigate the contents of a source.
To invoke the probe mode, use the ovftool command with only a source and no target.
>ovftool <source locator>
The following example shows the result of probing the ESC OVA.
64
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Installing Cisco Elastic Services Controller on VMware vCenter
Installing Elastic Services Controller Using OVF Tool
Step 2
NETWORK_OVA= (Path to the OVA Package)
NETWORK_HOSTNAME="$ (User Name) "
NETWORK_GATEWAY="192.0.2.1"
NETWORK_NET1_IP="192.0.2.0.xx/24" #
NETWORK_NET2_IP="192.51.100.xx/24"
ADMIN_USERNAME=" (admin name) "
ADMIN_PASSWORD=" (password) "
HTTPS_REST="True"
VMWARE_VCENTER_PORT='80'
VMWARE_VCENTER_IP='192.0.2.0.xx'
VMWARE_DATASTORE_HOST='192.0.2.0.xx'
VMWARE_DATACENTER_NAME='DC-NETWORK-1'
VMWARE_DATASTORE_NAME='cluster-datastore1'
VMWARE_COMPUTE_CLUSTER_NAME='DC-CLUSTER-1'
VMWARE_VCENTER_USERNAME='root'
VMWARE_VCENTER_PASSWORD=' password '
VMWARE_VCENTER_FOLDER="$USER"
# All valid deployment options:
#
#
4CPU-8GB (default)
4CPU-8GB-2Net
# 4CPU-8GB-3Net
DEPLOYMENT_OPTION="4CPU-8GB-2Net"
Before you deploy the ESC OVA, configure the properties of the OVA packages. Ensure the following OVA package properties are updated for the ESC OVA: NETWORK_OVA, NETWORK_HOSTNAME,
VMWARE_VCENTER_FOLDER, NETWORK_NET1_IP, NETWORK_NET2_IP, and
VMWARE_VCENTER_FOLDER.
The OVA descriptors contain configuration properties for the OVA package. You can set only one property at a time, but you can have multiple instances of the option per command. For multiple property mappings, repeat the option, separating them with a blank, for example --prop:p1=v1 --prop:p2=v2 --prop:p3=v3.
>.ovftool/ovftool\
--powerOn \
--acceptAllEulas \
--noSSLVerify \
--datastore=$VMWARE_DATASTORE_NAME \
--diskMode=thin \
--name=$NETWORK_HOSTNAME \
--deploymentOption=$DEPLOYMENT_OPTION \
--vmFolder=$VMWARE_VCENTER_FOLDER \
--prop:admin_username=$ADMIN_USERNAME --prop:admin_password=$ADMIN_PASSWORD \
--prop:admin_username=admin \
--prop:admin_password='Strong4Security!' \
--prop:confd_admin_username=admin \
--prop:confd_admin_password='Strong4Security!' \
--prop:network_hostname=$NETWORK_HOSTNAME \
--prop:vmware_vcenter_port=$VMWARE_VCENTER_PORT \
--prop:vmware_vcenter_ip=$VMWARE_VCENTER_IP \
--prop:vmware_datastore_host=$VMWARE_DATASTORE_HOST \
--prop:vmware_datacenter_name=$VMWARE_DATACENTER_NAME \
--prop:vmware_vcenter_username=$VMWARE_VCENTER_USERNAME \
--prop:vmware_datastore_name=$VMWARE_DATASTORE_NAME \
--prop:vmware_compute_cluster_name=$VMWARE_COMPUTE_CLUSTER_NAME \
--prop:vmware_vcenter_password=$VMWARE_VCENTER_PASSWORD \
--prop:net1_ip=$NETWORK_NET1_IP \
--prop:net2_ip=$NETWORK_NET2_IP \
--prop:gateway=$NETWORK_GATEWAY \
--prop:https_rest=$HTTPS_REST \
--net:"Network1=VM Network" --net:"Network2=MgtNetwork" --net:"Network3=VNFNetwork" \
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
65
Installing Cisco Elastic Services Controller on VMware vCenter
Powering on Cisco Elastic Services Controller Virtual Machine
Step 3
$NETWORK_OVA vi://$VMWARE_VCENTER_USERNAME:$VMWARE_VCENTER_PASSWORD@$VMWARE_VCENTER_IP/$VMWARE_DATACENTER_NAME/ host/$VMWARE_COMPUTE_CLUSTER_NAME
Following are some advanced examples of passing user credentials through properties.
Advanced usage with password hash:
--prop:admin_username=admin \
--prop:admin_password='$6$wnOi$UDOQmkKm2tQtr2jDVNhoo4wS42ffYYmzxMXLDugfzTbTXmMQDw146VzpxQvMumeaa125.agYHZUqQ8L.sdM2v0'
\
--prop:confd_admin_username=admin \
--prop:confd_admin_password='$6$wnOi$UDOQmkKm2tQtr2jDVNhoo4wS42ffYYmzxMXLDugfzTbTXmMQDw146VzpxQvMumeaa125.agYHZUqQ8L.sdM2v0'
\
Advanced usage with password hash and authorized public key:
--prop:admin_username=admin \
--prop:admin_password='$6$wnOi$UDOQmkKm2tQtr2jDVNhoo4wS42ffYYmzxMXLDugfzTbTXmMQDw146VzpxQvMumeaa125.agYHZUqQ8L.sdM2v0'
\
--prop:admin_public_key='ssh-rsa
AAAAB3NzaC1yc2EAAAABIwAAAIEAu+nkTtu2pShVbTYL+mmKxtmzM5dNXFy8IeX/lH5fXsODH1EAySlzHGFXq36RT5vIG/
+c2uV8fRsWaY7xXDrdGICxfkPuEj2UQH2MQx2yFjMFcaSAT56hsqE= admin@net' \
--prop:confd_admin_username=admin \
--prop:confd_admin_password='$6$wnOi$UDOQmkKm2tQtr2jDVNhoo4wS42ffYYmzxMXLDugfzTbTXmMQDw146VzpxQvMumeaa125.agYHZUqQ8L.sdM2v0'
\
--prop:confd_admin_public_key='ssh-rsa
AAAAB3NzaC1yc2EAAAABIwAAAIEAu+nkTtu2pShVbTYL+mmKxtmzM5dNXFy8IeX+xXu6TT2sTsxkKtVy8uOAeBp1qKzkp+ c2uV8fRsWaY7xXDrdGICxfkPuEj2UQH2MQx2yFjMFcaSAT56hsqE= admin@net' \
Note You need to replace the variables (IP addresses, root password, VM names, and so on) in the examples above with values from your own system.
To deploy the OVA package with the VMware OVF Tool, use the following command syntax:
>ovftool <source locator> <target locator> where <source locator> is the path to the OVA package and <target locator> is the path target for the virtual machine, OVA package or VI. A VI location refers to any location on a VMware product, such as vSphere,
VMware Server or ESXi. For more information on the VMware OVF Tool, see the VMware OVF Tool user documentation.
The ESC VM is deployed on VMware and powered on automatically.
Powering on Cisco Elastic Services Controller Virtual Machine
To power on the Cisco Elastic Services Controller virtual machine (VM):
Note You must set the memory and CPUs based on the requirements prior to clicking the power on. Once you start the VM you cannot change the memory or CPU settings until you shut down.
66
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Installing Cisco Elastic Services Controller on VMware vCenter
Next Steps: Cisco Elastic Services Controller Virtual Machine
Procedure
Step 1
Step 2
After deploying the VM, select the virtual machine name in vSphere, right-click on it and select Open Console .
Click the Power on button ( ). During the initial boot of the newly deployed machine, you will be prompted to enter a root (system) password, which is not the Cisco Elastic Services Controller portal password.
Initialization may vary depending on the setup.
Note This is the root password for the underlying Linux operating system on which the Cisco Elastic
Services Controller portal is installed. You will be asked to enter this password twice. You will need root access to the underlying Linux operating system at various times in the future, so make sure that you remember this password.
The End User License Agreement window appears on the first boot. Read the license agreement in its entirety, and only if you understand and accept the license terms, enter y (Yes).
Next Steps: Cisco Elastic Services Controller Virtual Machine
Logging in to Cisco Elastic Services Controller Portal
To log in to the ESC Portal, see the
Logging in to the ESC Portal, on page 130
Configuring the Virtual Machine to Automatically Power Up
You can configure the ESXi hypervisor to automatically power up the ESC VM when power is restored to the ESXi hypervisor layer.
Note You must manually power up the VM.
Procedure
Step 1
Step 2
Step 3
Step 4
Step 5
In the vSphere client, select the ESXi machine to which you are connected. It is not a specific VM that you have to select but the ESXi hypervisor on which they reside.
Select the Configuration tab.
Click the Virtual Machine Startup/Shutdown link under the Software area. You should see the VM in the list shown in window.
Click the Properties...
link present at the top right corner of the page. If you do not see that, resize the window until you do.
The Virtual Machine Startup and Shutdown page is displayed.
Check the Allow virtual machines to start and stop automatically with the system check box.
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
67
Installing Cisco Elastic Services Controller on VMware vCenter
Configuring the Virtual Machine to Automatically Power Up
Step 6
Step 7
Select the virtual machine running ESC and use the Move Up button on the right to move it up into the group labeled Automatic Startup
Click OK
This ensures that whenever power is restored to the ESXi hypervisor, the ESC VM powers up automatically.
68
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
C H A P T E R
13
Installing High Availability
This chapter contains the following sections:
•
High Availability Active/Standby Overview, on page 69
•
How High Availability Active/Standby Works, on page 70
•
Deploying ESC High Availability Active/Standby with User Data (HA Active/Standby Pair), on page
•
Deploying ESC High Availability Active/Standby (Standalone Instances), on page 74
•
Important Notes for ESC HA Active/Standby, on page 75
•
Troubleshooting High Availability Active/Standby, on page 75
High Availability Active/Standby Overview
ESC supports High Availability (HA) in the form of Active/Standby and Active/Active models. For
Active/Standby model, two ESC instances are deployed in the network to prevent ESC failure and provide
ESC service with minimum service interruption. If the primary ESC instance fails, the standby instance automatically takes over the ESC services. ESC HA Active/Standby resolves the following single point failures:
• Network failures
• Power failures
• Dead VM instance
• Scheduled downtime
• Hardware issues
• Internal application failures
Note From ESC 5.0, the name, Active/Passive model is changed to Active/Standby model.
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
69
Installing Cisco Elastic Services Controller on VMware vCenter
How High Availability Active/Standby Works
How High Availability Active/Standby Works
A High Availability Active/Standby deployment consists of two ESC instances: a primary and a standby.
Under normal circumstances, the primary ESC instance provides the services. The corresponding standby instance is passive. The standby instance is in constant communication with the primary instance and monitors the primary instances' status. If the primary ESC instance fails, the standby instance automatically takes over the ESC services to provide ESC service with minimum interruption.
The standby also has a complete copy of the database of the primary, but it does not actively manage the network until the primary instance fails. When the primary instance fails, the standby takes over automatically.
Standby instance takes over primary instance to manage the services while primary instance restoration taken place.
When the failed instance is restored, failback operations can be initiated to resume network management via the original primary instance.
ESC instances are managed by using KeepAliveD service. The VM handshake between ESC instances occurs through the KeepAliveD over the IPv4 network.
Deploying ESC High Availability Active/Standby with User Data
(HA Active/Standby Pair)
Before you begin:
• Cisco Elastic Services Controller (ESC) High Availability (HA) Active/Standby requires a network to keep alive and replicate database between primary and standby nodes. Both ESC VMs must have at least one network interface connecting to the same network and must be able to communicate to each other through the network.
• Ensure the two ESC VMs are located in different hosts and datastores so that single point failures can be prevented.
You can deploy ESC HA Active/Standby on VMware vCenter or vSphere in either of two ways:
• Deploying ESC HA Active/Standby with user data as a High Availability Active/Standby pair (Supported from ESC 4.2)
• Deploying ESC HA Active/Standby as two standalone instances and then using post configuration to set them as a High Availability pair. For more information,see the section on "Deploying ESC High
Availability Active/Standby (Standalone Instances)".
To deploy ESC HA Active/Standby on VMware vCenter or vSphere with user data as a High Availability
Active/Standby pair, define the user data file for each HA Active/Standby instance and then point the user data for each instance via ovftool. The encoding of user data is done via a set of commands in the ovftool script, and the result of this is set as a variable to the –prop:user-data= property in the ovftool.
Note The admin user/password and confd user/password properties are mandatory OVF properties. These properties cannot be defined in the user-data files.
70
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Installing Cisco Elastic Services Controller on VMware vCenter
Deploying ESC High Availability Active/Standby with User Data (HA Active/Standby Pair)
• Define the two VMs for ESC HA Active/Standby.
User Data 1
#cloud-config ssh_pwauth: True write_files:
- path: /etc/cloud/cloud.cfg.d/sys-cfg.yaml
content: | network: version: 1 config:
- type: nameserver address:
- 161.44.124.122
- type: physical name: eth0 subnets:
- type: static address: 172.16.0.0
netmask: 255.255.255.0
routes:
- gateway: 172.16.0.0
network: 0.0.0.0
netmask: 0.0.0.0
- path: /opt/cisco/esc/esc-config/esc-config.yaml
content: | resources: confd: option: start-phase0 drbd: nodes:
- 172.16.0.0
- 172.16.1.0
run_forever: true esc_service: depend_on: filesystem type: group escmanager: depend_on:
- pgsql
- mona
- vimmanager etsi: depend_on: pgsql startup: false filesystem: depend_on: drbd:master keepalived: vip: 172.16.2.0
portal: depend_on: escmanager startup: false snmp: startup: false runcmd:
- [ cloud-init-per, once, escadm_ovf_merge, sh, -c, "/usr/bin/escadm ovf merge"]
- [ cloud-init-per, once, escservicestart, sh, -c, "chkconfig esc_service on && service esc_service start"]
User data 2
#cloud-config ssh_pwauth: True write_files:
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
71
Deploying ESC High Availability Active/Standby with User Data (HA Active/Standby Pair)
Installing Cisco Elastic Services Controller on VMware vCenter
- path: /etc/cloud/cloud.cfg.d/sys-cfg.yaml
content: | network: version: 1 config:
- type: nameserver address:
- 161.44.124.122
- type: physical name: eth0 subnets:
- type: static address: 172.16.1.0
netmask: 255.255.255.0
routes:
- gateway: 172.16.0.0
network: 0.0.0.0
netmask: 0.0.0.0
- path: /opt/cisco/esc/esc-config/esc-config.yaml
content: | resources: confd: option: start-phase0 drbd: nodes:
- 172.16.0.0
- 172.16.1.0
run_forever: true esc_service: depend_on: filesystem type: group escmanager: depend_on:
- pgsql
- mona
- vimmanager etsi: depend_on: pgsql startup: false filesystem: depend_on: drbd:master keepalived: vip: 172.16.2.0
portal: depend_on: escmanager startup: false snmp: startup: false runcmd:
- [ cloud-init-per, once, escadm_ovf_merge, sh, -c, "/usr/bin/escadm ovf merge"]
- [ cloud-init-per, once, escservicestart, sh, -c, "chkconfig esc_service on && service esc_service start"]
• OVFtool should be called twice - once for each VM instance. Each instance needs to provide a
"--prop:user-data" property to point to its hashed user-data.
• Here is an example to boot a pair of HA Active/Standby instances that use 172.16.0.0 and 172.16.1.0
(floating) IPs to its instances, and 172.16.2.0 as a KAD_VIP.
user_data_1=`cat ./user-data-1` user_data_2=`cat ./user-data-2` dec_user_data_1=`echo "$user_data_1" | base64 | tr -d '[:space:]'` dec_user_data_2=`echo "$user_data_2" | base64 | tr -d '[:space:]'`
# vcenter-16 is the developer lab for vmware5
72
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Installing Cisco Elastic Services Controller on VMware vCenter
Deploying ESC High Availability Active/Standby with User Data (HA Active/Standby Pair)
#
#
ESC_OVA=/scratch/BUILD-${ESC_IMAGE}/BUILD-${ESC_IMAGE}/ESC-${ESC_IMAGE}.ova
# All valid deployment options:
2CPU-4GB
4CPU-8GB (default)
#
#
4CPU-8GB-2Net
4CPU-8GB-3Net
DEPLOYMENT_OPTION="4CPU-8GB-2Net" deploy_vmware_vm1() {
/usr/bin/ovftool \
--powerOn \
--acceptAllEulas \
--noSSLVerify \
--datastore=$VM_WARE_DATASTORE_NAME \
--diskMode=thin \
--name=$INSTANCE_NAME"-0" \
--deploymentOption=$DEPLOYMENT_OPTION \
--vmFolder=$FOLDER \
--prop:admin_username=$ESC_VM_USERNAME --prop:admin_password=$ESC_VM_PASSWORD \
--prop:esc_hostname=$INSTANCE_NAME"-0" \
--prop:rest_username=$REST_USERNAME \
--prop:rest_password=$REST_PASSWORD \
--prop:portal_username=$PORTAL_USERNAME \
--prop:portal_password=$PORTAL_PASSWORD \
--prop:confd_admin_username=$CONFD_USERNAME \
--prop:confd_admin_password=$CONFD_PASSWORD \
--prop:vmware_vcenter_port=$VMWARE_VCENTER_PORT \
--prop:vmware_vcenter_ip=$VM_WARE_VCENTER_IP \
--prop:vmware_datastore_host=$VM_WARE_DATASTORE_HOST \
--prop:vmware_datacenter_name=$VM_WARE_DATACENTER_NAME \
--prop:vmware_vcenter_username=$VM_WARE_VCENTER_USERNAME \
--prop:vmware_datastore_name=$VM_WARE_DATASTORE_NAME \
--prop:vmware_vcenter_password=$VM_WARE_VCENTER_PASSWORD \
--prop:net1_ip=$NET1_IP1 \
--prop:net2_ip=$NET2_IP1 \
--prop:gateway=$ESC_GATEWAY \
--prop:https_rest=$HTTPS_REST \
--prop:user-data=$dec_user_data_1 \
--net:"Network1=VM Network" --net:"Network2=MgtNetwork" --net:"Network3=VNFNetwork" \
$ESC_OVA vi://$VM_WARE_VCENTER_USERNAME:$VM_WARE_VCENTER_PASSWORD@$VM_WARE_VCENTER_IP/
$VM_WARE_DATACENTER_NAME/host/$VM_WARE_DATASTORE_CLUSTER
} deploy_vmware_vm2() {
/usr/bin/ovftool \
--powerOn \
--acceptAllEulas \
--noSSLVerify \
--datastore=$VM_WARE_DATASTORE_NAME \
--diskMode=thin \
--name=$INSTANCE_NAME"-1" \
--deploymentOption=$DEPLOYMENT_OPTION \
--vmFolder=$FOLDER \
--prop:admin_username=$ESC_VM_USERNAME --prop:admin_password=$ESC_VM_PASSWORD \
--prop:esc_hostname=$INSTANCE_NAME"-1" \
--prop:rest_username=$REST_USERNAME \
--prop:rest_password=$REST_PASSWORD \
--prop:portal_username=$PORTAL_USERNAME \
--prop:portal_password=$PORTAL_PASSWORD \
--prop:confd_admin_username=$CONFD_USERNAME \
--prop:confd_admin_password=$CONFD_PASSWORD \
--prop:vmware_vcenter_port=$VMWARE_VCENTER_PORT \
--prop:vmware_vcenter_ip=$VM_WARE_VCENTER_IP \
--prop:vmware_datastore_host=$VM_WARE_DATASTORE_HOST \
--prop:vmware_datacenter_name=$VM_WARE_DATACENTER_NAME \
--prop:vmware_vcenter_username=$VM_WARE_VCENTER_USERNAME \
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
73
Installing Cisco Elastic Services Controller on VMware vCenter
Deploying ESC High Availability Active/Standby (Standalone Instances)
--prop:vmware_datastore_name=$VM_WARE_DATASTORE_NAME \
--prop:vmware_vcenter_password=$VM_WARE_VCENTER_PASSWORD \
--prop:net1_ip=$NET1_IP2 \
--prop:net2_ip=$NET2_IP2 \
--prop:gateway=$ESC_GATEWAY \
--prop:https_rest=$HTTPS_REST \
--prop:user-data=$dec_user_data_2 \
--net:"Network1=VM Network" --net:"Network2=MgtNetwork" --net:"Network3=VNFNetwork" \
$ESC_OVA vi://$VM_WARE_VCENTER_USERNAME:$VM_WARE_VCENTER_PASSWORD@$VM_WARE_VCENTER_IP/
$VM_WARE_DATACENTER_NAME/host/$VM_WARE_DATASTORE_CLUSTER
} deploy_vmware_vm1 deploy_vmware_vm2
• Once the VMs are deployed successfully, you can check the status of ESC HA Active/Standby. You will find that one VM instance is booted as MASTER while the other VM instance is a BACKUP.
Deploying ESC High Availability Active/Standby (Standalone
Instances)
To deploy ESC HA Active/Standby on VMware vCenter or vSphere, two separate standalone nodes need to be installed first. After the standalone ESC instances are installed, reconfigure these nodes to turn them into
Primary and Standby using the following:
• kad_vip
• kad_vif
• ha_node_list
Note • On each ESC VM, we need to run escadm tool to configure ESC HA Active/Standby parameters and then reload and restart the escadm service.
• When you are deploying ESC HA Active/Standby, the kad_vip argument allows end users to access the
Primary ESC instance.
Procedure
Step 1
Step 2
Log in to the ESC Standalone instances.
As an admin user, run the escadm tool on both the Primary and Standby instances and provide the corresponding arguments.
• kad_vip — Specifies the IP address for Keepalived VIP (virtual IP) plus the interface of Keepalived VIP
[ESC-HA Active/Standby]
• kad_vif — Specifies the interface for Keepalived virtual IP and keepalived VRRP [ESC-HA
Active/Standby]. You can also use this argument to only specify the interface for keepalived VRRP, if the VIP interface is already specified using the kad_vip argument.
74
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Installing Cisco Elastic Services Controller on VMware vCenter
Important Notes for ESC HA Active/Standby
• ha_node_list — Specifies list of IP addresses for HA Active/Standby nodes in the Primary/Standby cluster for DRDB synchronization. This argument is utilized for replication-based HA Active/Standby solution only. For ESC instances with multiple network interfaces, the IP addresses should be within the network that --kad_vif argument specifies .
Step 3
Step 4
Step 5
$ sudo escadm ha set --kad_vip= <ESC_HA_VIP> --kad_vif= <ESC_KEEPALIVE_IF> --ha_node_list=
<ESC_NODE_1_IP> <ESC_NODE_2_IP>
$ sudo escadm reload
$ sudo escadm restart
After the restart, one ESC VM should be in Primary state and the other one should be in Standby state.
Add the VIP to the allowed address pairs for both VMs so that the VIP is reachable from outside.
Verify the status of each ESC instance.
# sudo escadm status
The following table lists few other command to check the status:
Status
ESC HA Active/Standby Role
CLI Commands cat /opt/cisco/esc/keepalived_state
ESC Health
ESC Service Status sudo escadm health
If you want to see more details (such as status of the
VIM manager, SNMP, portal, ESC manager, keepalived status and so on), add '–v': sudo escadm status --v
To check the detailed status, check the
/var/log/esc/escadm.log
Important Notes for ESC HA Active/Standby
• The HA Active/Standby failover takes about 2 to 5 minutes based on the number of managed VNFs to be operational. ESC service will not be available during the switchover time.
• When the switchover is triggered during transactions, all incomplete transactions will be dropped. The requests should be re-sent by Northbound interface if it does not receive any response from ESC.
Troubleshooting High Availability Active/Standby
• Check for network failures. If a network problem occurs, you must check the following details:
• The IP address assigned is correct, and is based on the OpenStack configuration.
• The gateway for each network interface must be pinged.
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
75
Installing Cisco Elastic Services Controller on VMware vCenter
Troubleshooting High Availability Active/Standby
• Check the logs for troubleshooting:
• The ESC manager log at /var/log/esc/escmanager.log
• The KeepAliveD log at /var/log/messages by grep keepalived
• The ESC service status log at /var/log/esc/escadm.log
76
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
P A R T
III
Installing Cisco Elastic Services Controller on
VMware vCloud Director
•
Installing Cisco Elastic Services Controller on VMware vCloud Director, on page 79
C H A P T E R
14
Installing Cisco Elastic Services Controller on
VMware vCloud Director
This chapter describes how to install Cisco Elastic Services Controller on VMware vCloud Director and includes the following sections:
•
Uploading the ESC OVA file into a Catalog in VMware vCloud Director, on page 79
•
Deploying ESC vApp on vCD Web Client, on page 79
•
Adding the VIM Connectors, on page 80
•
Adding Self-Signed Certificate, on page 80
Uploading the ESC OVA file into a Catalog in VMware vCloud
Director
Procedure
Step 1
Step 2
Step 3
Step 4
Step 5
Log in to a vCD organization through a vCD web client.
From Catalogs > My Organization's Catalogs , select a catalog.
Under the vApp Templates tab, click Upload
The Upload OVF package as a vApp Template window appears.
In the OVF package field, click Local file . Click Browse and choose the ESC OVA file from the local system.
Click Upload to transfer the ESC OVA file into the catalog.
After uploading the ESC OVA file, you can access the ESC vApp template from the catalog.
Deploying ESC vApp on vCD Web Client
This section describes the deployment scenario for Elastic Services Controller (ESC) and the procedure to deploy ESC vApp on VMware vCloud Director (vCD).
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
79
Installing Cisco Elastic Services Controller on VMware vCloud Director
Adding the VIM Connectors
Resources such as organization, organization VDC, and so on must be created on vCD before deployment.
For more information, see the Managing Resources on vCloud Director (vCD).
To deploy ESC vApp on vCD web client, you must:
Procedure
Step 1
Step 2
Step 3
Step 4
Log in to a vCD organization through a vCD web client.
From Catalogs > My Organization's Catalogs, select a Catalog.
Select the ESC vApp template, right click and select Add to My Cloud .
To complete the deployment, you must specify the following in the Add to My Cloud wizard: a) Select Name and Location - Specify ESC as the name of the vApp and select its virtual datacenter.
b) Configure Resources - Specify ESC as the VM name and SharedOnly for storage policy.
c) Network Mapping - Specify interface(s) network type as MgtNetwork,SvcNetwork, and VnfNetwork .
IP allocation is Static-Manual for all types of network in an IPv4.
d) Custom Properties - Fill in the custom properties for day(0) operation.
Note : Do not fill or change anything in the VM Settings of vCenter Server section.
e) Customize Hardware - Specify CPU use, memory consumption, and other core parameters.
f) Ready to Complete - Check the Power on vApp after this wizard is finished on the bottom right corner of the wizard and click Finish.
Adding the VIM Connectors
For more information on configuring VIM connectors after installation and managing VIM connectors, see the Managing VIM Connectors chapter in the Cisco Elastic Services Controller User Guide.
Adding Self-Signed Certificate
If the vCD instance is using a self-signed certificate, use the following command to add the certificate into
ESC, so that it qualifies the SSL verification:
$ escadm truststore add --help usage: escadm truststore add [-h] [-v] --alias ALIAS --file FILE optional arguments:
-h, --help show this help message and exit
-v, --v, --verbose show verbose output
--alias ALIAS
--file FILE
80
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
P A R T
IV
Installing Cisco Elastic Services Controller on a
Kernel-based Virtual Machine (KVM)
•
Installing Cisco Elastic Services Controller on a Kernel-based Virtual Machine, on page 83
C H A P T E R
15
Installing Cisco Elastic Services Controller on a
Kernel-based Virtual Machine
This chapter describes how to install Cisco Elastic Services Controller on a Kernel-based Virtual Machine and includes the following sections:
•
Installing Cisco Elastic Services Controller in a Kernel-based Virtual Machine, on page 83
•
Next Steps: Cisco Elastic Services Controller Kernel-based Virtual Machine, on page 85
Installing Cisco Elastic Services Controller in a Kernel-based
Virtual Machine
Cisco Elastic Services Controller can be installed in a Kernel-based Virtual Machine. You can install Cisco
Elastic services controller in a Kernel-based Virtual Machine using libvirt.
Preparing to Install Cisco Elastic Services Controller on a Kernel-based Virtual
Machine
If you plan to run Cisco Elastic Services Controller on a kernel-based virtual machine, make sure the following are setup:
Python 2.7 or 3.x
python-setuptools
Notes
Installed by default on Linux
Installed by default on Linux
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
83
Installing Cisco Elastic Services Controller on a Kernel-based Virtual Machine (KVM)
Installing Elastic Services Controller on a Kernel-Based Virtual Machine pip
OpenStack clients genisoimage libvirt and virtinst
Notes
On RHEL :
# easy_install pip
Since the installation using pip compiles source files, the gcc and python development packages are also required on RHEL. To install these packages on
RHEL:
# yum install gcc python-devel
On Ubuntu : Installed by default. Since the installation using pip compiles source files, the gcc and python development packages are also required on Ubuntu.
To install these packages on Ubuntu:
# apt-get install python-dev
# pip install python-keystoneclient
# pip install python-cinderclient
# pip install python-novaclient
# pip install python-neutronclient
On RHEL :
# yum install genisoimage
On Ubuntu :
# apt-get install genisoimage
On RHEL 6.x
:
# yum install libvirt-python python-virtinst
On RHEL 7.x
:
# yum install libvirt-python virt-install
On Ubuntu :
# apt-get install libvirt-dev
# pip install libvirt-python
Note libvirt will create the default network automatically.
Installing Elastic Services Controller on a Kernel-Based Virtual Machine
To install standalone Elastic Services Controller (ESC) on a kernel-based virtual machine, do the following:
Procedure
Step 1 Load the variables from the openerc file that contains OpenStack credentials:
84
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Installing Cisco Elastic Services Controller on a Kernel-based Virtual Machine (KVM)
Next Steps: Cisco Elastic Services Controller Kernel-based Virtual Machine
Step 2
Step 3
Step 4
Step 5
Step 6 cat ./openrc.sh
export OS_TENANT_NAME='< OS tenant username >' export OS_USERNAME='< OS username >' export OS_PASSWORD='< OS password >' export OS_AUTH_URL='http://<Openstack Host>:5000/v2.0/' source ./openrc.sh
Copy the ESC qcow2 image and the bootvm.py into the kernel-based VM.
Boot ESC on a kernel-based VM on the default network that was created when libvirt was installed, use one of the following command:
./bootvm.py --user_pass <username>:<password> --user_confd_pass <username>:<password>
--libvirt --image <image_name> esc-vm --net <default network>
Boot ESC on a kernel-based VM on the default network with static IP, using the following command:
./bootvm.py --user_pass <username>:<password> --user_confd_pass <username>:<password>
--libvirt --image <image_name> esc-vm --net <network> --ipaddr <ip_address>
Get a list of used IP addresses in your network. Use IP addresses that are not in the list for both HA
Active/Standby bootvm.py command and for kad_vip. Determine the first 3 octets of your network (i.e.
192.168.122) and pass it in the below command : arp -an | grep 192.168.122
To install ESC on a kernel-based VM in high availability, use the following command twice for both the HA nodes:
Note For the second bootvm.py command, use the other HA instance name.
./bootvm.py --user_pass <username>:<password> --user_confd_pass <username>:<password>
--libvirt --image <image_name> --ha_mode drbd --gateway_ip <default_gateway_ip_address>
--ipaddr <ip_address>
--ha_node_list <ha peer ip addresses separated by comma> --kad_vip <vip address> esc-ha-1
--net <network>
Next Steps: Cisco Elastic Services Controller Kernel-based
Virtual Machine
Logging in to Cisco Elastic Services Controller Portal
To log in to the ESC Portal, see the
Logging in to the ESC Portal, on page 130
Verifying ESC installation for a Kernel-based Virtual Machine (KVM)
After deploying ESC on a Kernel-based virtual machine, use the following procedure to verify the deployment.
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
85
Installing Cisco Elastic Services Controller on a Kernel-based Virtual Machine (KVM)
Troubleshooting Tips
Procedure
Step 1
Step 2
Step 3
Check that the ESC VMs have booted using the following command:
$ virsh list
Get the IP address of the ESC VM, using the following command:
$ arp -an | grep <ip_address>
Connect to ESC using SSH and verify the processes are running:
$ ssh USERNAME@ESC_IP
Troubleshooting Tips
When SSH access is not available, due to network conditions or ESC startup failures, you can connect to ESC through console(if enabled in ESC VM image) or VNC access. To access ESC VM through VNC, do the following:
1.
Identify the vnc port.
virsh dumpxml 10 | fgrep vnc
2.
Create a ssh tunnel to the local vnc port to allow connection from your remote VNC client.
86
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
P A R T
V
Installing Cisco Elastic Services Controller on
Amazon Web Services (AWS)
•
Installing Cisco Elastic Services Controller on Amazon Web Services, on page 89
C H A P T E R
16
Installing Cisco Elastic Services Controller on
Amazon Web Services
This chapter describes how to install Cisco Elastic Services Controller on AWS and includes the following sections:
•
•
Installing the Elastic Services Controller Instance in AWS, on page 90
Prerequisites
Following are the prerequisites that you must complete before you start installing the ESC instance in AWS.
Note If the ESC AMI images are shared with your AWS account, you can ignore these prerequisites and directly use the AMI image for ESC installation.
Procedure
Step 1
Step 2
Step 3
Step 4
Configure AWS CLI . You can use pip to install AWS CLI. For more details, refer to the AWS documentation .
Configure the credentials for AWS CLI based on your account information.
Create a Amazon S3 Bucket. Use this for bucket for uploading ESC image.
Note You must have a role named vmimport that allows importing VM and you must attach an IAM policy to the role. For more information, refer to the documentation on the creation of S3 bucket in
AWS.
Extract the vmdk file from ESC ova file.
$ tar xvf ESC-<latest image file>.ova ESC-<latest image file>-disk1.vmdk
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
89
Installing Cisco Elastic Services Controller on Amazon Web Services (AWS)
Installing the Elastic Services Controller Instance in AWS
Installing the Elastic Services Controller Instance in AWS
Once you have completed the tasks specified in the prerequisites section, you can use the procedure below to deploy and launch ESC instance in AWS.
Procedure
Step 1
Step 2
Upload and register ESC image.
a) Upload the vmdk image to the S3 bucket.
aws s3 cp <esc-vmdk-file> s3://<S3 bucket name>/ b) Register the image.
aws ec2 import-image --description "<esc-vmdk-file>" --disk-containers file://containers.json
Create user data .
a) Create a user for ESC VM. Without a user, you would not be able to access the VM. It is recommended to configure 'admin' user with sudo access and ssh key.
b) Create the esc-config.yaml in user-data using write_files command.
Each instance can have up to 15 interfaces, depending on the type of instances.
Note If you want to use two interfaces, ensure that you create the two network interfaces before hand.
These interfaces on different subnets must belong to the same availability zone. Add the interface details in the 'Configure Instance Details' tab when launching the instance from AWS console.
c) Enable esc_service and start it.
Following is an example of a complete user data:
#cloud-config
# It is recommended to disable password authentication for ssh when ESC runs in public cloud such as AWS.
ssh_pwauth: False users:
- name: admin
# Put admin in 'esc-user' group, otherwise some scripts of ESC might fail when running as admin.
groups: esc-user gecos: User created by cloud-init
# This is an example of the hashed password for 'admin'.
passwd:
$6$rounds=656000$pswsUsR7Iz9NIfA4$7E1sEGV8rhDieNDhc824lYwL3cQ8Rsgp9Nds.OZBe9rG/DE56YWk0kDZoB.DsjATrj9pcBnAe.rSOpWll2r0N/
# The public key for admin user. Replace it with your public key to login.
ssh-authorized-keys:
- ssh-rsa
AAAAB3NzaC1yc2EAAAADAQABAAABAQCqGLe4EVVI/rQy4e4jZUEnc5PvYItc39x5fz9rRggZzpwYzKXSj+UnWQMgvkIai+
Mv5vTPiEYTSNZx9PrnUKayZaLr/2GiLPmnPNEgyzvjD5v77vV3Ag7eHfLXKLYbu7ausYqFKEFbNgSTGC1PWhoz2geY4zNO9hS3eMhNvxNSLpbo3ftzamQoqtWSx2aRc81M/ piy6NcBzJ3JeH4rOk9bQ+QxRAYm3bOlq/qRfuoxmrsgd68xAIXeDWyGumEThXN9MDEcQMIWO54fiPQgkqKbZWztH2EBnE9/B6rZCRBUUvdoQhQt2L/ hbCZN1k+oqQ53rlG/BjT09CGfYbgoHq2v
# false allows you to sudo with the password.
lock-passwd: false homedir: /home/admin
90
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Installing Cisco Elastic Services Controller on Amazon Web Services (AWS)
Installing the Elastic Services Controller Instance in AWS
# sudo settings sudo: ALL=(ALL) ALL write_files:
- path: /etc/cloud/cloud.cfg.d/sys-cfg.yaml
content: | network: version: 1 config:
# You must define the name server when you use the static IP address.
- type: nameserver address:
- 172.31.0.2
# Define physical network interface
- type: physical name: eth0 subnets:
# Define the static IP address
- type: static address: 172.31.5.66
netmask: 255.255.240.0
# Define the routes routes:
- gateway: 172.31.0.1
# 0.0.0.0 means the default gateway network: 0.0.0.0
netmask: 0.0.0.0
# ESC service config file
- path: /opt/cisco/esc/esc-config/esc-cfg.yaml
content: | confd:
# AAA users for ConfD init_aaa_users:
# Public key for ConfD user 'admin'
- key: c3NoLXJzYSBBQUFBQjNOemFDMXljMkVBQUFBREFRQUJBQUFCQVFDeFkwMzByaEMzSXlWekF2bStISVlmMmpkdm
RUZndTTEpCRjVPTjZoUEgvK2FBTkkzb0NCSmJndjhPdjRtVXUvYmlCYmsyS240QW52Ni9ROE1YWGducnZST241MlJuODN2ejRCWTAw
T1h2SzZrT2YrUnZkSDFtNjhscVlrWU9uZVErNEtOak5tQXRwV0huT0xCZE1mZ2pzTmF1SlF1QVJUMEtDS2VBS3k4aUVqSUZpZDhWZ3
NiSlA0aDNpTzdjcTkza0E1ZGFQb0xiNWRKRVp3ZW15WS9ENGp6ZnJUeDVKWFFuMy80SDdaQVZPaWcyNzBGUnlGVkZhNFl1VXNYcDk1d3
QveHdpc0RUREVCYTYydjkxQzdXamtaNy9rYkRlRW9VSU9OZExqdEdvbU84c2JRUUJoZHBVTTZlNXJkeUl2VzQ3YTZYOFA5N2lBR3JrQ09 qMWVHNkYgeGlhb3hpbnlAWElBT1hJTlktTS1SRVhXCg==
# Note: 'admin' is the only user supported and you cannot change the name here.
name: admin
# Hashed password for admin user.
passwd:
$6$rounds=656000$d4hZHtniblo4/b0m$fD3./1H3jcPlWAENviFlu70i5wKnH9DIasDwTkL.p70UFZlfalzD9O7utL1NdKXwudnNhxIOrvYagkBfq6AWh.
# No specific settings for esc service. Leave it empty.
esc_service: {} runcmd:
- [ cloud-init-per, once, escservicestart, sh, -c, "chkconfig esc_service on && service esc_service start"]
Following is an example to define two interfaces in user data:
- path: /etc/cloud/cloud.cfg.d/sys-cfg.yaml
content: | network: version: 1 config:
- type: physical name: eth0 subnets:
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
91
Installing Cisco Elastic Services Controller on Amazon Web Services (AWS)
Installing the Elastic Services Controller Instance in AWS
Step 3
- type: static address: 172.31.5.66
netmask: 255.255.240.0
# Define the routes routes:
- gateway: 172.31.0.1
# 0.0.0.0 means the default gateway network: 0.0.0.0
netmask: 0.0.0.0
- type: physical name: eth1 subnets:
- type: static address: 172.31.51.220
netmask: 255.255.240.0
Launch ESC VM in AWS
Launch ESC VM using one of the following method:
• From Portal : a.
Go to EC2 Management Console, IMAGES/AMIs. Select the image you imported and click Launch .
b.
Choose an instance type. Choose t2.xlarge as the instance type.
c.
Configure the Instance Details. Add details such as User Data, Storage, Tag name, and so on. While using two interfaces, create these network interfaces and them here.
d.
Configure a security group. Enable ssh only.
e.
Click Launch .
• From Command Line : Choose the image, subnet, security group and use the following command to instantiate ESC VM.
Note aws ec2 run-instances --subnet-id <subnet id> --image-id <image id> --security-group-ids
<security group id> --count 1
--instance-type <instance> --key-name <key name> --user-data <user data file location>
--associate-public-ip-address
ESC does not support HA Active/Standby installation on AWS.
What to do next
After you launch the ESC VM, check the status of the ESC service using the
$ sudo escadm status command.
92
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
P A R T
VI
Installing Cisco Elastic Services Controller on
Cisco Cloud Services Platform 2100
•
Installing Cisco Elastic Services Controller on Cisco Cloud Services Platform 2100, on page 95
C H A P T E R
17
Installing Cisco Elastic Services Controller on
Cisco Cloud Services Platform 2100
This chapter describes how to install Cisco Elastic Services Controller on CSP 2100 and includes the following sections:
•
•
Installing the Elastic Services Controller Instance in CSP 2100, on page 95
•
List of Variables Used in CSP 2100 Sample Files, on page 104
Prerequisites
Following are the prerequisites that you require before you start installing the ESC instance in CSP 2100.
• Virtual CPUs 4 (minimum)
• Memory 8 GB
• Disk size 30 GB
Installing the Elastic Services Controller Instance in CSP 2100
Once you have completed the tasks specified in the prerequisites section, you can use the following procedure to deploy and launch ESC instance in CSP 2100. Following are the three deployment alternatives available for CSP 2100.
• ESC with Single and Dual Interfaces
• ESC HA Active/Standby Installation
For list of variables used in the CSP 2100 sample files, see
List of Variables Used in CSP 2100 Sample Files, on page 104
.
ESC with Single and Dual Interface
To install ESC in CSP, you must create the user-data in the following format as the day0 configuration file:
A sample for single interface describing the day zero file as config drive and user data is as follows:
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
95
Installing the Elastic Services Controller Instance in CSP 2100
Installing Cisco Elastic Services Controller on Cisco Cloud Services Platform 2100
#cloud-config users:
- name: admin gecos: admin
# The user's login name
# The user name's real name groups: esc-user # add admin to group esc-user passwd: $6$saltsalt$9PDBehueUG4XTLEj6BFZA5MDGh/XeQ6QPbf9HYLU3RifHj1
# The hash -- not the password itself -- of the password you want
# to use for this user. You can generate a safe hash via:
# mkpasswd --method=SHA-512 --rounds=4096 lock-passwd: false # Defaults to true. Lock the password to disable password login
# Set to false if you want to password login homedir: /home/admin # Optional. Set to the local path you want to use. Defaults to
/home/<username> sudo: ALL=(ALL) ALL # Defaults to none. Set to the sudo string you want to use ssh_pwauth: True authentication for sshd.
write_files:
# Defaults to False. Set to True if you want to enable password
# ESC Configuration
- path: /opt/cisco/esc/esc-config/esc-config.yaml
content: | resources: confd: init_aaa_users:
- key: c3NoLXJzYSBBQUFBQjNOemFDMXljMkVBQUF passwd: $6$rounds=4096$adWFd7LUn2PEUPWtWPl5tCD7pO9bae672T1 option: start-phase0 escmanager: open_ports:
- '8080'
- '8443' url:
- http://0.0.0.0:8080/ESCManager
- https://0.0.0.0:8443/ESCManager esc_service: type: group
# Params
- path: /opt/cisco/esc/esc-config/esc_params.conf
content: | default.active_vim=CSP default.enable_cascade_deletion=true
# Networking
- path: /etc/sysconfig/network-scripts/ifcfg-eth0 content: |
DEVICE="eth0"
BOOTPROTO="none"
ONBOOT="yes"
TYPE="Ethernet"
USERCTL="yes"
IPADDR="VAR_NETWORK0_IPADDR"
NETMASK="VAR_NETWORK0_NETMASK"
GATEWAY="VAR_NETWORK0_GATEWAY"
DEFROUTE="yes"
NM_CONTROLLED="no"
IPV6INIT="no"
IPV4_FAILURE_FATAL="yes" bootcmd:
- [ cloud-init-per, once, disable_ipv6_eth0, sh, -c, "echo net.ipv6.conf.eth0.disable_ipv6
= 1 >> /etc/sysctl.conf"]
- [ cloud-init-per, once, update_host_name, sh, -c, "echo VAR_LOCAL_HOSTNAME >> /etc/hostname
&& hostnamectl set-hostname VAR_LOCAL_HOSTNAME"]
- [ cloud-init-per, once, update_hosts, sh, -c, "echo 127.0.0.1 VAR_LOCAL_HOSTNAME >>
/etc/hosts"]
96
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Installing Cisco Elastic Services Controller on Cisco Cloud Services Platform 2100
Installing the Elastic Services Controller Instance in CSP 2100
- [ cloud-init-per, once, add_name_server, sh, -c, "echo nameserver VAR_NAMESERVER_IP >>
/etc/resolv.conf"]
- [ cloud-init-per, once, add_ntp_server, sh, -c, "echo server VAR_NTP_SERVER iburst >>
/etc/ntp.conf"]
- [ cloud-init-per, once, enable_ecdsa-sha2-nistp521, sh, -c, "/usr/bin/ssh-keygen -f
/etc/ssh/ssh_host_ecdsa_521_key -t ecdsa -b 521 -N ''"]
- [ cloud-init-per, once, enable_ecdsa-sha2-nistp384, sh, -c, "/usr/bin/ssh-keygen -f
/etc/ssh/ssh_host_ecdsa_384_key -t ecdsa -b 384 -N ''"]
- [ cloud-init-per, once, enable_ssh_rsa, sh, -c, "sed -i '/ssh_host_rsa_key/s/^#//g'
/etc/ssh/sshd_config"] runcmd:
- [ cloud-init-per, once, apply_network_config, sh, -c, "systemctl restart network"]
- [ cloud-init-per, once, stop_chronyd, sh, -c, "systemctl stop chronyd;systemctl disable chronyd"]
- [ cloud-init-per, once, start_ntp, sh, -c, "systemctl enable ntpd;systemctl start ntpd"]
- [ cloud-init-per, once, set_timezone, sh, -c, "timedatectl set-timezone VAR_TIMEZONE"]
- [ cloud-init-per, once, confd_keygen_root, sh, -c, "/usr/bin/escadm confd keygen --user root"]
- [ cloud-init-per, once, confd_keygen_admin, sh, -c, "/usr/bin/escadm confd keygen --user admin"]
- [ cloud-init-per, once, esc_service_start, sh, -c, "chkconfig esc_service on && service esc_service start"] # You must include this line
A sample for dual interfaces describing the day zero file as config drive and user data is as follows:
You can configure an ethernet-based physical network device with a static IPv4 in ESC .
#cloud-config users:
- name: admin gecos: admin groups: esc-user
# The user's login name
# The user name's real name
# add admin to group esc-user passwd: $6$saltsalt$9PDBehueUG4XTLEj6BFZA5MDGh/XeQ6QPbf9HYLU3RifHj1
# The hash -- not the password itself -- of the password you want
# to use for this user. You can generate a safe hash via:
# mkpasswd --method=SHA-512 --rounds=4096 lock-passwd: false # Defaults to true. Lock the password to disable password login
# Set to false if you want to password login homedir: /home/admin # Optional. Set to the local path you want to use. Defaults to
/home/<username> sudo: ALL=(ALL) ALL # Defaults to none. Set to the sudo string you want to use ssh_pwauth: True authentication for sshd.
write_files:
# ESC Configuration
# Defaults to False. Set to True if you want to enable password
- path: /opt/cisco/esc/esc-config/esc-config.yaml
content: | resources: confd: init_aaa_users:
- key: c3NoLXJzYSBBQUFBQjNOemFDMXljMkVBQUF passwd: $6$rounds=4096$adWFd7LUn2PEUPWtWPl5tCD7pO9bae672T1 option: start-phase0 escmanager: open_ports:
- '8080'
- '8443' url:
- http://0.0.0.0:8080/ESCManager
- https://0.0.0.0:8443/ESCManager esc_service: type: group
# Params
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
97
Installing the Elastic Services Controller Instance in CSP 2100
Installing Cisco Elastic Services Controller on Cisco Cloud Services Platform 2100
- path: /opt/cisco/esc/esc-config/esc_params.conf
content: | default.active_vim=CSP default.enable_cascade_deletion=true
# Networking
- path: /etc/sysconfig/network-scripts/ifcfg-eth0 content: |
DEVICE="eth0"
BOOTPROTO="none"
ONBOOT="yes"
TYPE="Ethernet"
USERCTL="yes"
IPADDR="VAR_NETWORK0_IPADDR"
NETMASK="VAR_NETWORK0_NETMASK"
GATEWAY="VAR_NETWORK0_GATEWAY"
DEFROUTE="yes"
NM_CONTROLLED="no"
IPV6INIT="no"
IPV4_FAILURE_FATAL="yes"
- path: /etc/sysconfig/network-scripts/ifcfg-eth1 content: |
DEVICE="eth1"
BOOTPROTO="none"
ONBOOT="yes"
TYPE="Ethernet"
USERCTL="yes"
IPADDR="VAR_NETWORK1_IPADDR"
NETMASK="VAR_NETWORK1_NETMASK"
GATEWAY="VAR_NETWORK1_GATEWAY"
DEFROUTE="no"
NM_CONTROLLED="no"
IPV6INIT="no"
IPV4_FAILURE_FATAL="yes" bootcmd:
- [ cloud-init-per, once, disable_ipv6_eth0, sh, -c, "echo net.ipv6.conf.eth0.disable_ipv6
= 1 >> /etc/sysctl.conf"]
- [ cloud-init-per, once, update_host_name, sh, -c, "echo VAR_LOCAL_HOSTNAME >> /etc/hostname
&& hostnamectl set-hostname VAR_LOCAL_HOSTNAME"]
- [ cloud-init-per, once, update_hosts, sh, -c, "echo 127.0.0.1 VAR_LOCAL_HOSTNAME >>
/etc/hosts"]
- [ cloud-init-per, once, add_name_server, sh, -c, "echo nameserver VAR_NAMESERVER_IP >>
/etc/resolv.conf"]
- [ cloud-init-per, once, add_ntp_server, sh, -c, "echo server VAR_NTP_SERVER iburst >>
/etc/ntp.conf"]
- [ cloud-init-per, once, enable_ecdsa-sha2-nistp521, sh, -c, "/usr/bin/ssh-keygen -f
/etc/ssh/ssh_host_ecdsa_521_key -t ecdsa -b 521 -N ''"]
- [ cloud-init-per, once, enable_ecdsa-sha2-nistp384, sh, -c, "/usr/bin/ssh-keygen -f
/etc/ssh/ssh_host_ecdsa_384_key -t ecdsa -b 384 -N ''"]
- [ cloud-init-per, once, enable_ssh_rsa, sh, -c, "sed -i '/ssh_host_rsa_key/s/^#//g'
/etc/ssh/sshd_config"] runcmd:
- [ cloud-init-per, once, apply_network_config, sh, -c, "systemctl restart network"]
- [ cloud-init-per, once, stop_chronyd, sh, -c, "systemctl stop chronyd;systemctl disable chronyd"]
- [ cloud-init-per, once, start_ntp, sh, -c, "systemctl enable ntpd;systemctl start ntpd"]
- [ cloud-init-per, once, set_timezone, sh, -c, "timedatectl set-timezone VAR_TIMEZONE"]
- [ cloud-init-per, once, confd_keygen_root, sh, -c, "/usr/bin/escadm confd keygen --user root"]
- [ cloud-init-per, once, confd_keygen_admin, sh, -c, "/usr/bin/escadm confd keygen --user admin"]
- [ cloud-init-per, once, esc_service_start, sh, -c, "chkconfig esc_service on && service esc_service start"] # You must include this line
Creating ESC passwords to use in Day0 Files
98
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Installing Cisco Elastic Services Controller on Cisco Cloud Services Platform 2100
Installing the Elastic Services Controller Instance in CSP 2100
When using the Cloud-Init day0 file to deploy an ESC instance, the passwords must be passed in as a hash, and not a plain text.
To create a hashed password, use the mkpasswd tool. The following example shows how to use the mkpasswd tool to create a hashed password.
~$ mkpasswd --method=SHA-512 --rounds=4096
Password:
$6$rounds=4096$Yo11pRsFO$iT5SGMJ6z8WEnmj8TKMdInblgWeb/UChmrsQs3aspx8j.yUuuhxKk2XScOkerWWxpqD5F0sLfC5kzT5t2xGkL1
Procedure
Step 1
Step 2
Upload user-data file to CSP
To deploy ESC, the user-data file must be first uploaded to the CSP node.
Note The path to upload images and day0 files is: /osp/repository scp user-data-esc admin@<CSP_IP_ADDRESS>:/osp/repository
Deploying ESC VM
You must edit configuration to be sent to the CSP node hosting the ESC VM.
Following is the deployment datamodel for single interface. For dual interface, you have two interfaces.
<name>ESC-SA-2-IF</name>
<?xml version="1.0"?>
<services xmlns="http://www.cisco.com/ns/test/service">
<service>
<name>VAR_SERVICE_NAME</name>
<memory>8192</memory> <!-- minimum 8G -->
<numcpu>4</numcpu> <!-- minimum 4 -->
<disk_size>30.0</disk_size> <!-- minimum 30G -->
<disk-resize>true</disk-resize>
<iso_name>ESC-5_0_0_xxx</iso_name> <!-- the name of the ESC image already on the CSP
-->
<power>on</power>
<ip>172.20.117.40</ip>
<!-- add the ip for display in the CSP web/console interfaces -->
<vnc_password>password1</vnc_password>
<!-- to secure the VNC console session -->
<vnics>
<!-- This interface aligns with eth0 in the user-data file -->
<vnic>
<nic>0</nic>
<vlan>1</vlan>
<tagged>false</tagged>
<type>access</type>
<passthrough_mode>none</passthrough_mode>
<model>virtio</model>
<network_name>VAR_NETWORK0_NAME</network_name>
</vnic>
<!-- This interface aligns with eth1 in the user-data file -->
<!-- If not using 2 interfaces, this vnic block can be removed -->
<vnic>
<nic>1</nic>
<vlan>1</vlan>
<tagged>false</tagged>
<type>access</type>
<passthrough_mode>none</passthrough_mode>
<model>virtio</model>
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
99
Installing Cisco Elastic Services Controller on Cisco Cloud Services Platform 2100
ESC HA Active/Standby Installation
Step 3
Step 4
Step 5
<network_name>VAR_NETWORK1_NAME</network_name>
</vnic>
</vnics>
<disk_type>ide</disk_type>
<day0_filename>user-data-esc</day0_filename> <!-- this name MUST match the name of the file that was copied to the CSP -->
<day0-dest-filename>user-data</day0-dest-filename> <!-- mandatory value -->
<day0-volume-id>cidata</day0-volume-id> <!-- mandatory value -->
</service>
</services>
Sending Configuration
Use a netconf-console (shipped with ConfD) to deploy ESC on a CSP node.
$ netconf-console --port=2022 --host=<CSP_IP_ADDRESS> --user=CSP_ADMIN_USERNAME
--password=CSP_ADMIN_PASSWORD --edit-config=deployESCHA1.xml
If HA, repeat the command with the configuration for the second ESC.
Configuring the VIM Connector
After ESC has booted, configure the VIM Connectors.
When installing ESC in CSP, no VIM connectors are added by default. To manage VNFs, you must create the VIM connector.
Adding the VIM Connectors
For more information on configuring VIM connectors after installation, and managing VIM connectors, see
Managing VIM Connectors in the Cisco Elastic Services Controller User Guide .
ESC HA Active/Standby Installation
To install ESC in CSP, you must create the user-data in the following format as the day0 configuration file.
For HA, you must define one file for each VM.
For creating ESC passwords to use in Day0 Files, see the Creating ESC passwords to use in Day0 Files section.
A sample for ESC HA Active/Standby installation on node 1 describing the day zero file as config drive and user data is as follows: user-data sample - HA Node 1
#cloud-config users:
- name: admin gecos: admin
# The user's login name
# The user name's real name groups: esc-user # add admin to group esc-user passwd: $6$saltsalt$9PDBehueUG4XTLEj6BFZA5MDGh/XeQ6QPbf9HYLU3RifHj1
# The hash -- not the password itself -- of the password you want
# to use for this user. You can generate a safe hash via:
# mkpasswd --method=SHA-512 --rounds=4096 lock-passwd: false # Defaults to true. Lock the password to disable password login
# Set to false if you want to password login homedir: /home/admin # Optional. Set to the local path you want to use. Defaults to
/home/<username> sudo: ALL=(ALL) ALL # Defaults to none. Set to the sudo string you want to use ssh_pwauth: True # Defaults to False. Set to True if you want to enable password
100
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Installing Cisco Elastic Services Controller on Cisco Cloud Services Platform 2100
ESC HA Active/Standby Installation authentication for sshd.
write_files:
# ESC Configuration
- path: /opt/cisco/esc/esc-config/esc-cfg.yaml
content: | ha: vri: VAR_NETWORK0_KADVRI mode: drbd vip: VAR_NETWORK0_KADVIP vif: eth0 nodes:
- ipaddr: VAR_NETWORK0_IPADDR
- ipaddr: VAR_NETWORK0_IPADDR2 confd: init_aaa_users:
- name: admin passwd: $6$rounds=4096$adWFd7LUn2PEUPWtWPl5tCD7pO9bae672T1 escmanager: open_ports:
- '8080'
- '8443' url:
- http://0.0.0.0:8080/ESCManager
- https://0.0.0.0:8443/ESCManager esc_service: {}
# Params
- path: /opt/cisco/esc/esc-config/esc_params.conf
content: | default.active_vim=CSP default.enable_cascade_deletion=true
# Networking
- path: /etc/sysconfig/network-scripts/ifcfg-eth0 content: |
DEVICE="eth0"
BOOTPROTO="none"
ONBOOT="yes"
TYPE="Ethernet"
USERCTL="yes"
IPADDR="VAR_NETWORK0_IPADDR"
NETMASK="VAR_NETWORK0_NETMASK"
GATEWAY="VAR_NETWORK0_GATEWAY"
DEFROUTE="yes"
IPV6INIT="no"
IPV4_FAILURE_FATAL="yes" bootcmd:
- [ cloud-init-per, once, disable_ipv6_eth0, sh, -c, "echo net.ipv6.conf.eth0.disable_ipv6
= 1 >> /etc/sysctl.conf"]
- [ cloud-init-per, once, update_host_name, sh, -c, "echo VAR_LOCAL_HOSTNAME >> /etc/hostname
&& hostnamectl set-hostname VAR_LOCAL_HOSTNAME"]
- [ cloud-init-per, once, update_hosts, sh, -c, "echo 127.0.0.1 VAR_LOCAL_HOSTNAME >>
/etc/hosts"]
- [ cloud-init-per, once, add_name_server, sh, -c, "echo nameserver VAR_NAMESERVER_IP >>
/etc/resolv.conf"]
- [ cloud-init-per, once, add_ntp_server, sh, -c, "echo server VAR_NTP_SERVER iburst >>
/etc/ntp.conf"]
- [ cloud-init-per, once, enable_ecdsa-sha2-nistp521, sh, -c, "/usr/bin/ssh-keygen -f
/etc/ssh/ssh_host_ecdsa_521_key -t ecdsa -b 521 -N ''"]
- [ cloud-init-per, once, enable_ecdsa-sha2-nistp384, sh, -c, "/usr/bin/ssh-keygen -f
/etc/ssh/ssh_host_ecdsa_384_key -t ecdsa -b 384 -N ''"]
- [ cloud-init-per, once, enable_ssh_rsa, sh, -c, "sed -i '/ssh_host_rsa_key/s/^#//g'
/etc/ssh/sshd_config"] runcmd:
- [ cloud-init-per, once, apply_network_config, sh, -c, "systemctl restart network"]
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
101
Installing Cisco Elastic Services Controller on Cisco Cloud Services Platform 2100
ESC HA Active/Standby Installation
- [ cloud-init-per, once, stop_chronyd, sh, -c, "systemctl stop chronyd;systemctl disable chronyd"]
- [ cloud-init-per, once, start_ntp, sh, -c, "systemctl enable ntpd;systemctl start ntpd"]
- [ cloud-init-per, once, set_timezone, sh, -c, "timedatectl set-timezone VAR_TIMEZONE"]
- [ cloud-init-per, once, confd_keygen_root, sh, -c, "/usr/bin/escadm confd keygen --user root"]
- [ cloud-init-per, once, confd_keygen_admin, sh, -c, "/usr/bin/escadm confd keygen --user admin"]
- [ cloud-init-per, once, esc_service_start, sh, -c, "chkconfig esc_service on && service esc_service start"] # You must include this line
Procedure
Step 1
Step 2
Uploading user-data file to CSP
To deploy ESC, the user-data file must be first uploaded to the CSP node.
Note The path to upload images and day0 files is: /osp/repository scp user-data-esc-ha-1 CSP_ADMIN_USERNAME@<CSP_IP_ADDRESS>:/osp/repository scp user-data-esc-ha-2 CSP_ADMIN_USERNAME@<CSP_IP_ADDRESS>:/osp/repository
Deploying ESC VM
You must edit configuration to be sent to the CSP node hosting the ESC VM.
Following is the deployment datamodel for ESC HA Active/Standby on node 1 : deployESC-HA-1.xml
<?xml version="1.0"?>
<services xmlns="http://www.cisco.com/ns/test/service">
<service>
<name>VAR_SERVICE_NAME</name>
<memory>8192</memory> <!-- minimum 8G -->
<numcpu>4</numcpu> <!-- minimum 4 -->
<disk_size>30.0</disk_size> <!-- minimum 30G -->
<disk-resize>true</disk-resize>
<iso_name>ESC-5_0_0_xxx</iso_name> <!-- the name of the ESC image already on the CSP
-->
<power>on</power>
<ip>172.20.117.40</ip>
<!-- add the ip for display in the CSP web/console interfaces -->
<vnc_password>password1</vnc_password>
<!-- to secure the VNC console session -->
<vnics>
<!-- This interface aligns with eth0 in the user-data file -->
<vnic>
<nic>0</nic>
<vlan>1</vlan>
<tagged>false</tagged>
<type>access</type>
<passthrough_mode>none</passthrough_mode>
<model>virtio</model>
<network_name>VAR_NETWORK0_NAME</network_name>
</vnic>
<!-- This interface aligns with eth1 in the user-data file -->
<!-- If not using 2 interfaces, this vnic block can be removed -->
<vnic>
<nic>1</nic>
<vlan>1</vlan>
<tagged>false</tagged>
102
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Installing Cisco Elastic Services Controller on Cisco Cloud Services Platform 2100
ESC HA Active/Standby Installation
Step 3
<type>access</type>
<passthrough_mode>none</passthrough_mode>
<model>virtio</model>
<network_name>VAR_NETWORK1_NAME</network_name>
</vnic>
</vnics>
<disk_type>ide</disk_type>
<day0_filename>user-data-esc</day0_filename> <!-- this name MUST match the name of the file that was copied to the CSP -->
<day0-dest-filename>user-data</day0-dest-filename> <!-- mandatory value -->
<day0-volume-id>cidata</day0-volume-id> <!-- mandatory value -->
</service>
</services>
Following is the deployment datamodel for ESC in HA Active/Standby on node 2 : deployESC-HA-2.xml
deployESC-HA-1.xml
<?xml version="1.0"?>
<services xmlns="http://www.cisco.com/ns/test/service">
<service>
<name>VAR_SERVICE_NAME</name>
<memory>8192</memory> <!-- minimum 8G -->
<numcpu>4</numcpu> <!-- minimum 4 -->
<disk_size>30.0</disk_size> <!-- minimum 30G -->
<disk-resize>true</disk-resize>
<iso_name>ESC-5_0_0_xxx</iso_name> <!-- the name of the ESC image already on the CSP
-->
<power>on</power>
<ip>172.20.117.40</ip>
<!-- add the ip for display in the CSP web/console interfaces -->
<vnc_password>password1</vnc_password>
<!-- to secure the VNC console session -->
<vnics>
<!-- This interface aligns with eth0 in the user-data file -->
<vnic>
<nic>0</nic>
<vlan>1</vlan>
<tagged>false</tagged>
<type>access</type>
<passthrough_mode>none</passthrough_mode>
<model>virtio</model>
<network_name>VAR_NETWORK0_NAME</network_name>
</vnic>
<!-- This interface aligns with eth1 in the user-data file -->
<!-- If not using 2 interfaces, this vnic block can be removed -->
<vnic>
<nic>1</nic>
<vlan>1</vlan>
<tagged>false</tagged>
<type>access</type>
<passthrough_mode>none</passthrough_mode>
<model>virtio</model>
<network_name>VAR_NETWORK1_NAME</network_name>
</vnic>
</vnics>
<disk_type>ide</disk_type>
<day0_filename>user-data-esc</day0_filename> <!-- this name MUST match the name of the file that was copied to the CSP -->
<day0-dest-filename>user-data</day0-dest-filename> <!-- mandatory value -->
<day0-volume-id>cidata</day0-volume-id> <!-- mandatory value -->
</service>
</services>
Sending Configuration
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
103
Installing Cisco Elastic Services Controller on Cisco Cloud Services Platform 2100
List of Variables Used in CSP 2100 Sample Files
Step 4
Step 5
Use a netconf-console (shipped with ConfD) to deploy ESC on a CSP node.
$ netconf-console --port=2022 --host=<CSP_IP_ADDRESS> --user=<CSP_ADMIN_USERNAME>
--password=<CSP_ADMIN_PASSWORD> --edit-config=deployESC-HA-1.xml
$ netconf-console --port=2022 --host=<CSP_IP_ADDRESS> --user=<CSP_ADMIN_USERNAME>
--password=<CSP_ADMIN_PASSWORD> --edit-config=deployESC-HA-2.xml
Configuring the VIM Connector
After ESC has booted, configure the VIM Connectors.
When installing ESC in CSP, no VIM connectors are added by default. To manage VNFs, you must create the VIM connector.
Adding the VIM Connectors
For more information on configuring VIM connectors after installation, and managing VIM connectors, see
Managing VIM Connectors in the Cisco Elastic Services Controller User Guide .
List of Variables Used in CSP 2100 Sample Files
To create the user-data file, to configure the ESC you must have values ready for the following list of variables used in the sample files:
Table 3: List of Variables
Variable Name
VAR_TIMEZONE
VAR_SERVICE_NAME
VAR_NTP_SERVER
VAR_NETWORK1_NETMASK
VAR_NETWORK1_NAME
VAR_NETWORK1_IPADDR
VAR_NETWORK1_GATEWAY
VAR_NETWORK0_NETMASK
VAR_NETWORK0_NAME
Purpose
The timezone for the ESC clock to use
The name of the ESC service on the CSP
The IP address of an NTP server
The netmask for the eth1 interface (Dual Interface
ESC)
The name of the network on the CSP where ESC's eth1 interface exists (Dual Interface ESC)
The IP address for the eth1 interface (Dual Interface
ESC)
The gateway for the eth1 interface (Dual Interface
ESC)
The netmask for the eth0 interface
The name of the network on the CSP where ESC's eth0 interface exists
104
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Installing Cisco Elastic Services Controller on Cisco Cloud Services Platform 2100
Variable Name
VAR_NETWORK0_KADVRI
VAR_NETWORK0_KADVIP
VAR_NETWORK0_IPADDR2
VAR_NETWORK0_IPADDR
VAR_NETWORK0_GATEWAY
VAR_NAMESERVER_IP
VAR_LOCAL_HOSTNAME
CSP_IP_ADDRESS
List of Variables Used in CSP 2100 Sample Files
Purpose
The VRRP ID used for HA. Must be unique in the subnet for the HA pair and the same value used on both ESCs
Range is from 1 to 254
The VIP for the HA pair that connects to the current
Master ESC
The IP address for the other ESC's eth0 interface
The IP address for ESC (eth0 interface)
The gateway for the eth0 interface
The IP address of a DNS server
The hostname for ESC
IP address of the CSP 2100 to be used
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
105
List of Variables Used in CSP 2100 Sample Files
Installing Cisco Elastic Services Controller on Cisco Cloud Services Platform 2100
106
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
P A R T
VII
Post Installation Tasks
•
Post Installation Tasks, on page 109
Post Installation Tasks
C H A P T E R
18
This chapter contains the following sections:
Note It is recommended that you ignore the do not edit message, if you want to modify the cloud-init day-0 configuration, file.
•
Changing the ESC Password , on page 109
•
•
Configuring Cisco Elastic Services Controller as Identity Management Client, on page 114
•
Authenticating REST Requests, on page 116
•
Configuring Openstack Credentials , on page 119
•
Reconfiguring ESC Virtual Machine, on page 125
•
Verifying ESC Configurations and Other Post-Install Operations , on page 128
•
Logging in to the ESC Portal, on page 130
Changing the ESC Password
You will be forced to change the default password on the first time login. Portal will not let you bypass this step and will keep returning you to this page until you change the default password. After the first time password change, you can change your password using the procedures described in this section. Also, if the user has multiple browsers or tabs or the SAME user is logged on by 2 or more computers and one of the user changes the password then everyone will be logged off and asked to re-enter the new password. The user session has an expiry of 1 hour so if the user is inactive on the portal for an hour then portal will expire the session and the user will have to re-login. If you forgot your password, you can also update or randomly generate the password.
This section discusses how to change the passwords.
Example for REST: sudo escadm rest set --username {USERNAME} --password {PASSWORD}
Example for ETSI: sudo escadm etsi set --rest_user {USERNAME:PASSWORD}
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
109
Post Installation Tasks
Changing the ConfD Netconf/CLI Administrator Password Using the Command Line Interface
Changing the ConfD Netconf/CLI Administrator Password Using the Command Line Interface
After you install ESC, to change the Confd admin password, do the following:
Procedure
Step 1
Step 2
Step 3
Step 4
Step 5
Log in to the ESC VM.
$ ssh USERNAME@ESC_IP
Switch to the admin user.
[admin@esc-ha-0 esc]$ sudo bash
[sudo] password for admin:
Load the ConfD CLI:
$ /opt/cisco/esc/confd/bin/confd_cli -u admin
Set the new admin password:
$ configure
$ set aaa authentication users user admin password <new password>
Save the changes.
$ commit
Creating Readonly User Group for ConfD in ESC
ConfD in ESC is enhanced with the introduction of a new group named readonly. If you are a member of readonly group, you can only retrieve the information and you cannot modify the permissions.
You can use 'readonly' as the role name with bootvm. The following example shows how to create two users in ConfD. One is admin and the other is readonly:
# bootvm.py name-500-105-100 --user_confd_pass admin:admin --user_confd_pass readonly:readonly::readonly --user_pass admin:admin --image ESC-5_0_0_105 --net network
For HA A/A, you can use 'readonly' as the group name in aa-day0.yaml Following is the example: confd: init_aaa_users:
- group: readonly name: admin passwd:
$6$rounds=4096$Ps1JIjKihRTF$fo8XPBxwEHJWWfNiXDnO269rlhAxAhWBcPBfGnZxy1gM3QMxcN8jJ6guWt9Bu.ZkWdPt3hr0OghO73Wr3iDHb0
You can also create a confd readonly user after ESC vm is deployed. The following steps create a confd readonly user named 'test' with password 'test':
[root@name-500-155 admin]# /opt/cisco/esc/confd/bin/confd_cli --user admin admin connected from 127.0.0.1 using console on name-500-155 admin@name-500-155> configure
Entering configuration mode private
[ok][2019-12-06 18:17:39]
[edit]
110
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Post Installation Tasks
Creating Readonly User Group for ConfD in ESC admin@name-500-155% set aaa authentication users user test uid 9000 gid 9000 password $0$test homedir /var/confd/homes/test ssh_keydir /var/confd/homes/test/.ssh
[ok][2019-12-06 18:19:15]
[edit] admin@name-500-155% set nacm groups group readonly user-name test
[ok][2019-12-06 18:19:41]
[edit] admin@name-500-155% commit
Commit complete.
[ok][2019-12-06 18:19:47]
[edit] admin@name-500-155%
As a readonly user, you can also access ConfD remotely: name@my-server-39:~$ ssh -p 2024 [email protected]
[email protected]'s password: readonly connected from 172.16.103.46 using ssh on name-500-156 readonly@name-500-156> configure
Entering configuration mode private
[ok][2019-12-13 16:15:33]
[edit] readonly@name-500-156% show esc_datamodel tenants { tenant admin { description "Built-in Admin Tenant"; managed_resource false; vim_mapping true;
}
}
[ok][2019-12-13 16:15:38]
[edit]
ESC in ConfD sends access-denied error if you fall under readonly ConfD group, and require modify permissions. Following is the example of the access-denied error message:
$ esc_nc_cli --user readonly --password readonly edit-config dep.xml
Configure
/opt/cisco/esc/confd/bin/netconf-console --port=830 --host=127.0.0.1 --user=readonly
--password=****** --edit-config=/tmp/d.xml
<?xml version="1.0" encoding="UTF-8"?>
<rpc-reply xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="1">
<rpc-error>
<error-type>application</error-type>
<error-tag>access-denied</error-tag>
<error-severity>error</error-severity>
</rpc-error>
</rpc-reply>
If ESC is configured to use PAM/IDM. The groups in IDM servers are directly mapped to the groups in
ConfD. Hence, the readonly user must be mapped in the IDM group 'readonly'.
For example:
$ ipa group-find --all readonly
---------------
1 group matched
--------------dn: cn=readonly,cn=groups,cn=accounts,dc=linuxsysadmins,dc=local
Group name: readonly
GID: 5003
Member users: readonly ipantsecurityidentifier: S-1-5-21-2222126199-2113948134-574478857-1003
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
111
Post Installation Tasks
Changing Linux Account Password ipauniqueid: 858b8cda-0d34-11ea-bca8-525400b29c19 objectclass: top, groupofnames, nestedgroup, ipausergroup, ipaobject, posixgroup, ipantgroupattrs
----------------------------
Number of entries returned 1
----------------------------
Changing Linux Account Password
Procedure
Step 1
Step 2
Log in to ESC VM.
$ ssh USERNAME@ESC_IP
To update or generate a random password, use the following command:
/usr/bin/pwqcheck
/usr/bin/pwqgen
Changing the ESC Portal Password
The user can update or reset the default admin password.
Procedure
Step 1
Step 2
Step 3
Log in to ESC VM.
Switch to the root user.
To update the default admin password or randomly generate a password, use one of the following method:
• Using escadm utility:
To update the default admin password (admin/********):
[root@anyname-v44-52 admin]# escadm portal set --username admin --password ********
Successfully updated password for username admin
To generate a random password:
[root@anyname-v44-52 admin]# escadm portal set --username admin
Would you like to use the generated password: "Accent5omit&Wide"?[y|n]y
Successfully updated password for username admin
The --must_change variable will ask the user to change their password at the next login.
The --must_change variable is not applicable for REST users.
[root@anyname-v44-52 admin]# escadm portal set --username admin --must_change
Would you like to use the generated password: "Rainy4Dozen&Behave"?[y|n]y
Successfully reset password for username admin. User must change the password at the next login.
112
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Post Installation Tasks
Configuring Pluggable Authentication Module (PAM) Support for Cisco Elastic Services Controller
• To reset to a specific password:
[root@anyname-v44-52 admin]# escadm portal set --username admin --password P@55w0rd!
--must_change
Successfully reset password for username admin. User must change the password at the next login.
• Using the bootvm command line:
--user_portal_pass admin:<new password>
• Using the ESC Portal: a.
Log in to ESC portal using your username and password.
b.
Choose Accounts Setting on the Navigation menu.
c.
Enter the old password in the Old password field, then enter a new password in the New Password and Confirm Password fields.
d.
Click Update Password .
Configuring Pluggable Authentication Module (PAM) Support for Cisco Elastic Services Controller
You can configure ESC services to use Pluggable Authentication Modules (PAM) for user authentication in
ESC. With Cisco Elastic Services supporting PAM, you can also enable LDAP authentication in ESC. If PAM is not configured, ESC will continue to use the default authentication method for each ESC service. The following table lists the commands to enable PAM authentication for each ESC service.
Table 4: Configuring PAM for ESC Services
ESC Service/Component
ESCManager (REST interface)
ESC Monitor (Health API)
Confd
Portal
ETSI
Command sudo escadm escmanager set --auth
PAM:[:<pam_service_name>] sudo escadm monitor set --auth
PAM:[:<pam_service_name>] sudo escadm confd set --auth
PAM:[:<pam_service_name>] sudo escadm portal set --auth
PAM[:<pam_service_name>] sudo escadm etsi set --pam_service
<pam_service_name>
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
113
Post Installation Tasks
Adding PAM User to an ESC Service/Component
Note • The SSHD service that runs inside the ESC VM already uses PAM authentication by default.
• If any component sets PAM authentication without specifying the PAM service, ESC defaults to the
PAM service 'system-auth'.
Adding PAM User to an ESC Service/Component
You can add the PAM user to the following groups of ESC service:
• rest-user
• confd-user
• portal-user
• etsi-user
Perform the following steps to add the PAM user to an ESC service/component:
Procedure
Step 1
Step 2
Step 3
Step 4
Log in to the ESC VM.
Switch to the root user
Add a PAM user, by using the following command: passwd pamuser
Changing password for user pamuser.
New password:
Retype new password: passwd: all authentication tokens updated successfully.
Add a PAM user to an ESC service/component group, by using the following command: usermod -a -G <ESC Service Group> pamuser
Note The PAM user must be added to the admin or readonly group for the Confd service
Configuring Cisco Elastic Services Controller as Identity
Management Client
Prerequisites
• Ensure that the Identity Management Client (IDM) server is up and running.
114
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Post Installation Tasks
Configuring Cisco Elastic Services Controller as the Identity Policy and Audit Client
• Ensure that the DNS server in ESC is in working state. So that, ESC interacts with the IDM server using the host name.
The following example shows how ESC (esc-client-500.linuxsysadmins.local) reaches the IDM server
(idmns.linuxsysadmins.local).
[root@esc-client-500 admin]# ping idmns
PING idmns.linuxsysadmins.local (192.168.222.176) 56(84) bytes of data.
64 bytes from idmns.linuxsysadmins.local (192.168.221.176): icmp_seq=1 ttl=64 time=0.492
ms
64 bytes from idmns.linuxsysadmins.local (192.168.221.176): icmp_seq=2 ttl=64 time=0.457
ms
64 bytes from idmns.linuxsysadmins.local (192.168.221.176): icmp_seq=3 ttl=64 time=0.645
ms
You can configure IDM through sssd. ESC provides a PAM configuration file, point to
/etc/pam.d/syste-auth to system-auth-esc-sssd to start configuring services in ESC to work with the IDM server.
# cd /etc/pam.d
# ln -sf system-auth-esc-sssd system-auth
# ls -al /etc/pam.d/system-auth lrwxrwxrwx. 1 root root 20 Nov 13 00:39 /etc/pam.d/system-auth -> system-auth-esc-sssd
The following table lists the commands to enable IDM authentication for each ESC service.
Table 5: Configuring IDM for ESC Services
ESC Service/Component Command
ESCManager
ETSI
ConfD
Command
# escadm escmanager set --auth
PAM:system-auth-esc-sssd
# escadm etsi set --pam_service system-auth-esc-sssd
# escadm confd set --auth
PAM:system-auth-esc-sssd
Configuring Cisco Elastic Services Controller as the Identity Policy and Audit
Client
To Configure the ESC as an Identity Policy and Audit Client (IPA) client, run the following command: ipa-client-install
Following is the example to configure ESC as the IPA client:
[root@esc-client-500 admin]# ipa-client-install --domain linuxsysadmins.local --server idmns.linuxsysadmins.local --realm LINUXSYSADMINS.LOCAL
WARNING: ntpd time&date synchronization service will not be configured as conflicting service (chronyd) is enabled
Use --force-ntpd option to disable it and force configuration of ntpd
Autodiscovery of servers for failover cannot work with this configuration.
If you proceed with the installation, services will be configured to always access the discovered server for all operations and will not fail over to other servers in case of failure.
Proceed with fixed values and no DNS discovery? [no]: yes
Client hostname: esc-client-500.linuxsysadmins.local
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
115
Post Installation Tasks
Authenticating REST Requests
Realm: LINUXSYSADMINS.LOCAL
DNS Domain: linuxsysadmins.local
IPA Server: idmns.linuxsysadmins.local
BaseDN: dc=linuxsysadmins,dc=local
Continue to configure the system with these values? [no]: yes
Skipping synchronizing time with NTP server.
User authorized to enroll computers: admin
Password for [email protected]:
Successfully retrieved CA cert
Subject: CN=Certificate Authority,O=LINUXSYSADMINS.LOCAL
Issuer: CN=Certificate Authority,O=LINUXSYSADMINS.LOCAL
Valid From: 2019-11-12 23:23:32
Valid Until: 2039-11-12 23:23:32
Enrolled in IPA realm LINUXSYSADMINS.LOCAL
Created /etc/ipa/default.conf
Configured sudoers in /etc/nsswitch.conf
Configured /etc/sssd/sssd.conf
Configured /etc/krb5.conf for IPA realm LINUXSYSADMINS.LOCAL
trying https://idmns.linuxsysadmins.local/ipa/json
[try 1]: Forwarding 'schema' to json server 'https://idmns.linuxsysadmins.local/ipa/json' trying https://idmns.linuxsysadmins.local/ipa/session/json
[try 1]: Forwarding 'ping' to json server
'https://idmns.linuxsysadmins.local/ipa/session/json'
[try 1]: Forwarding 'ca_is_enabled' to json server
'https://idmns.linuxsysadmins.local/ipa/session/json'
Systemwide CA database updated.
Adding SSH public key from /etc/ssh/ssh_host_ecdsa_521_key.pub
Adding SSH public key from /etc/ssh/ssh_host_ecdsa_384_key.pub
Adding SSH public key from /etc/ssh/ssh_host_rsa_key.pub
Adding SSH public key from /etc/ssh/ssh_host_ed25519_key.pub
Adding SSH public key from /etc/ssh/ssh_host_ecdsa_key.pub
[try 1]: Forwarding 'host_mod' to json server
'https://idmns.linuxsysadmins.local/ipa/session/json'
Could not update DNS SSHFP records.
SSSD enabled
Configured /etc/openldap/ldap.conf
Configured /etc/ssh/ssh_config
Configured /etc/ssh/sshd_config
Configuring linuxsysadmins.local as NIS domain.
Client configuration complete.
The ipa-client-install command was successful
Authenticating REST Requests
ESC REST and ETSI REST APIs use http basic access authentication where the ESC client will have to provide a username and password when making REST requests. The user name and password will be encoded with Base64 in transit, but not encrypted or hashed. HTTPS will be used in conjunction with Basic
Authentication to provide the encryption.
This section discusses ESC REST and ETSI REST authentications, how to change the default password of the interfaces, and how to send an authorized requests from the ESC client.
REST Authentication
By default, the REST authentication is enabled. To disable REST authentication, you can pass the argument
--disable-rest-auth to bootvm. Cisco does not recommend you to use this in a production environment.
116
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Post Installation Tasks
Enabling ETSI REST Authentication
ESC also supports https communication over port 8443. ESC will generate a self-signed certificate that the client will need to trust to get the https communication going. By default, REST is enabled as HTTP and restricted to localhost.
ESC can be installed with external access to REST on HTTPS or HTTP enabled with additional bootvm.py
arguments: -enable-https-rest or --enable-http-rest .
It is recommended to use only enabled external REST API when required. When enabled, it is recommended to use bootvm.py --enable-https-rest --user_rest_pass USERNAME:PASSWORD .
Note Make sure to pass either --enable-https-rest or --enable-http-etsi-rest or both the arguments to the bootvm.py
script to enable http and https interfaces to the REST API. You must pass --user_rest_pass while using either
--enable-https-rest or --enable-http-etsi-rest when REST authentication is not disabled. To enable https or http after ESC VM is booted, use the escadm command specified below.
sudo escadm escmanager set --url http://127.0.0.1:8080/ESCManager,https://0.0.0.0:8443/ESCManager
You must change the configuration of the peer instance if ESC is in HA Active/Standby mode.
Enabling ETSI REST Authentication
If the ETSI REST http or https interfaces are enabled, then all requests to an ETSI API must contain authentication data. You can use the --enable-http-etsi-rest or --enable-https-etsi arguments respectively to enable http and https interfaces to the ESC bootvm.py installation script.
You can enable both interfaces simultaneously, but only the https interface should be enabled in a production environment.
Note To enable http or https after the ESC VM has booted, use the escadm command specified below: sudo escadm etsi enable_http_rest
OR sudo escadm etsi enable_https_rest
Then restart the ETSI service.
Changing the REST Interface Password
The REST interface has only one default username/password (admin/<default_password>). The password can be updated after the bootup using escadm tool from the ESC VM CLI. You can also update the password through the REST API.
Procedure
Step 1
Step 2
Log in to ESC VM.
To replace the existing password with a new one, use one of the below options:
• Using the escadm tool from the ESC VM CLI, you can generate a random password:
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
117
Post Installation Tasks
Changing the ETSI REST Interface Password
[root@test-v44-52 admin]# escadm rest set --help usage: escadm rest set [-h] [-v] --username USERNAME [--password PASSWORD] optional arguments:
-h, --help show this help message and exit
-v, --v, --verbose show verbose output
--username USERNAME
--password PASSWORD new password or use randomly generated password if no password provided
• Using the REST API: http://[ESCVM_IP]:8080/ESCManager/v0/authentication/setpassword?userName=admin&password=yourPassword or https://[ESCVM_IP]:8443/ESCManager/v0/authentication/setpassword?userName=admin&password=yourPassword
Changing the ETSI REST Interface Password
The ETSI REST interface has only one default username/password (admin/<default_password>). The password can be updated after the bootup using escadm tool from the ESC VM CLI.
Procedure
Step 1
Step 2
Log in to ESC VM.
To set the default ETSI REST username and password, use the following command: sudo escadm etsi set --rest_user username:password or
[admin@xyz-esc-4-4-0-59-keep ~]$ escadm etsi set --help usage: escadm etsi set [-h] [-v] [--startup {0,1,true,false,manual,auto}]
[--rest_user REST_USER] [--pam_service PAM_SERVICE] optional arguments:
-h, --help show this help message and exit
-v, --v, --verbose show verbose output
--startup {0,1,true,false,manual,auto} set to false|0|manual to disable etsi at startup.
--rest_user REST_USER
Set the user for rest. Format username:password
--pam_service PAM_SERVICE
Specify a PAM service to use for authentication. This will override the rest user. To revert to the using the rest user for authentication, supply an empty string.
Sending an Authorized REST Request
To send an authorized request, the ESC client should send the request with the following header:
Authorization: Basic YWRtaW46Y2lzY28xMjM=
118
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Post Installation Tasks
Sending an Authorized ETSI REST Request where YWRtaW46Y2lzY28xMjM= is the Base64 encoded string of the default username/password.
Most libraries and web clients have an interface for providing the username/password and the application will encode the username/password and add the HTTP Basic Auth header.
Example using the default credentials:
For HTTP: http://[ESCVM_IP]:8080/ESCManager/v0/tenants/
For HTTPS: https://[ESCVM_IP]:8443/ESCManager/v0/tenants/
Sending an Authorized ETSI REST Request
To send an authorized request, the ESC client should send the request with the following header:
Authorization: Basic YWRtaW46Y2lzY28xMjM= where YWRtaW46Y2lzY28xMjM= is the Base64 encoded string of the default username/password.
Most libraries and web clients have an interface for providing the username/password and the application will encode the username/password and add the HTTP Basic Auth header.
Example using the default credentials:
For HTTP: http://[ESCVM_IP]:8250/vnflcm/v1/vnf_lcm_op_occs
For HTTPS: http://[ESCVM_IP]:8251/vnflcm/v1/vnf_lcm_op_occs
Configuring Openstack Credentials
If ESC was deployed without passing VIM credentials, you can set VIM credentials through ESC VIM and through VIM User APIs (REST or Netconf API ).
Note ESC will accept the northbound configuration request only if the following conditions are met:
• ESC has VIM or a VIM user configured through APIs(REST/Netconf).
• ESC has VIM or a VIM user configured, and ESC is able to reach the VIM.
• ESC has VIM or a VIM user configured, and ESC is able to authenticate the user.
Configuring using Netconf API
• Passing VIM credential using Netconf :
<esc_system_config xmlns="http://www.cisco.com/esc/esc">
<vim_connectors>
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
119
Post Installation Tasks
Configuring Openstack Credentials
<!--represents a vim-->
<vim_connector>
<!--unique id for each vim-->
<id>my-server-30</id>
<!--vim type [OPENSTACK|VMWARE_VSPHERE|LIBVIRT|AWS|CSP]-->
<type>OPENSTACK</type>
<properties>
<property>
<name>os_auth_url</name>
<value>http://<os_ip:port>/v3</value>
</property>
<!-- The project name for openstack authentication and authorization -->
<property>
<name>os_project_name</name>
<value>vimProject</value>
</property>
<!-- The project domain name is needed for openstack v3 identity api -->
<property>
<name>os_project_domain_name</name>
<value>default</value>
</property>
</properties>
<users>
<user>
<id>admin</id>
<credentials>
<properties>
<property>
<name>os_password</name>
<value>********</value>
</property>
<!-- The user domain name is needed for openstack v3 identity api -->
<property>
<name>os_user_domain_name</name>
<value>default</value>
</property>
</properties>
</credentials>
</user>
</users>
</vim_connector>
</vim_connectors>
</esc_system_config>
120
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Post Installation Tasks
Configuring Openstack Credentials
Note • From ESC 3.0 onwards, multiple VIM connectors are supported but within one ESC, only one type of VIM are supported. For example, all the Vim
Connector(s) has to be for OpenStack only. One ESC VIM cannot have two
VIM connector, one points to OpenStack, one points to VMware.
• One VIM is chosen as the default VIM which supports all pre 3.0 config requests and datamodels.
• Deployments can be done on the VIM that is not the default VIM. The deployment to a non default VIM has to have all out-of-band resources
(except ephemeral volumes). No other configurations like image, flavor, network, and so on can be done on the VIM that is not the default VIM.
• The default VIM connector will be auto provisioned and does not need to be configured in the following scenarios:
• If VIM credentials have been passed during ESC boot up.
• If upgrading from 2.3.x to 3.0.
• The change in the datamodel for Openstack create VIM connector would be handled during upgrade by migration. The 'os_tenant_name' and
'os_project_domain_name' properties would be moved to the VIM Connector properties and 'os_ternant_name' will be renamed to 'os_project_name'.
• For the default VIM Connector, once it is properly authenticated, its properties cannot be updated.
• VIM user can be deleted, recreated, or its properties can be updated at anytime.
• Updating VIM Connector using Netconf:
<esc_system_config xmlns="http://www.cisco.com/esc/esc">
<vim_connectors>
<vim_connector nc:operation="replace">
<id>example_vim</id>
<type>OPENSTACK</type>
<properties>
<property>
<name>os_auth_url</name>
<value>{auth_url}</value>
</property>
<property>
<name>os_project_name</name>
<value>vimProject</value>
</property>
<!-- The project domain name is only needed for openstack v3 identity api -->
<property>
<name>os_project_domain_name</name>
<value>default</value>
</property>
<property>
<name>os_identity_api_version</name>
<value>3</value>
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
121
Post Installation Tasks
Configuring Openstack Credentials
</property>
</properties>
</vim_connector>
</vim_connectors>
</esc_system_config>
• Updating VIM user using Netconf:
<esc_system_config xmlns="http://www.cisco.com/esc/esc">
<vim_connectors>
<vim_connector>
<id>example_vim</id>
<users>
<user nc:operation="replace">
<id>my_user</id>
<credentials>
<properties>
<property>
<name>os_password</name>
<value>********</value>
</property>
<!-- The user domain name is only needed for openstack v3 identity api
-->
<property>
<name>os_user_domain_name</name>
<value>default</value>
</property>
</properties>
</credentials>
</user>
</users>
</vim_connector>
</vim_connectors>
</esc_system_config>
• Deleting VIM connector using Netconf:
<esc_system_config xmlns="http://www.cisco.com/esc/esc"> <vim_connectors>
<vim_connector nc:operation="delete">
<id>example_vim</id>
</vim_connector>
</vim_connectors>
</esc_system_config>
• Deleting VIM Connector using command:
$/esc_nc_cli delete-vim-connector <vim connector id>
• Deleting VIM user using command:
$/esc_nc_cli delete-vim-user <vim connector id> <vim user id>
Configuring using REST API
• Adding VIM using REST:
POST /ESCManager/v0/vims/
HEADER: content-type, callback
<?xml version="1.0"?>
122
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Post Installation Tasks
Configuring Openstack Credentials
<vim_connector xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
<id>example_vim</id>
<type>OPENSTACK</type>
<properties>
<property>
<name>os_auth_url</name>
<value>{auth_url}</value>
</property>
<property>
<name>os_project_name</name>
<value>vimProject</value>
</property>
<!-- The project domain name is only needed for openstack v3 identity api -->
<property>
<name>os_project_domain_name</name>
<value>default</value>
</property>
<property>
<name>os_identity_api_version</name>
<value>3</value>
</property>
</properties>
</vim_connector>
• Adding VIM user using REST:
POST /ESCManager/v0/vims/{vim_id}/vim_users
HEADER: content-type, callback
<?xml version="1.0"?>
<user xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
<id>my_user</id>
<credentials>
<properties>
<property>
<name>os_password</name>
<value>********</value>
</property>
<!-- The user domain name is only needed for openstack v3 identity api -->
<property>
<name>os_user_domain_name</name>
<value>default</value>
</property>
</properties>
</credentials>
</user>
• Update VIM using REST:
PUT /ESCManager/v0/vims/{vim_id}
HEADER: content-type, callback
<?xml version="1.0"?>
<vim_connector xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
<!--unique id for each vim-->
<id>example_vim</id>
<type>OPENSTACK</type>
<properties>
<property>
<name>os_auth_url</name>
<value>{auth_url}</value>
</property>
<property>
<name>os_project_name</name>
<value>vimProject</value>
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
123
Post Installation Tasks
Configuring Openstack Credentials
</property>
<!-- The project domain name is only needed for openstack v3 identity api -->
<property>
<name>os_project_domain_name</name>
<value>default</value>
</property>
<property>
<name>os_identity_api_version</name>
<value>3</value>
</property>
</properties>
</vim_connector>
• Update VIM user using REST:
PUT /ESCManager/v0/vims/{vim_id}/vim_users/{vim_user_id}
HEADER: content-type, callback
<?xml version="1.0"?>
<user xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
<id>my_user</id>
<credentials>
<properties>
<property>
<name>os_password</name>
<value>********</value>
</property>
<!-- The user domain name is only needed for openstack v3 identity api -->
<property>
<name>os_user_domain_name</name>
<value>default</value>
</property>
</properties>
</credentials>
</user>
• Delete VIM using REST:
DELETE /ESCManager/v0/vims/{vim_id}
• Delete VIM user using REST:
DELETE /ESCManager/v0/vims/{vim_id}/vim_users/{user_id}
• Notification example after each VIM or VIM user configuration is done :
<?xml version="1.0" encoding="UTF-8"?>
<notification xmlns="urn:ietf:params:xml:ns:netconf:notification:1.0">
<eventTime>2016-10-06T16:24:05.856+00:00</eventTime>
<escEvent xmlns="http://www.cisco.com/esc/esc">
<status>SUCCESS</status>
<status_code>200</status_code>
<status_message>Created vim connector successfully</status_message>
<vim_connector_id>my-server-30</vim_connector_id>
<event>
<type>CREATE_VIM_CONNECTOR</type>
</event>
</escEvent>
</notification>
Important Notes:
• In ESC 3.0, you can add multiple VIM Connector for Openstack VIM. Each VIM Connector can have only one VIM User.
124
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Post Installation Tasks
Reconfiguring ESC Virtual Machine
• VIM username and password can be updated at anytime. VIM endpoint will not be able to update after a resource is created through ESC.
• After VIM is connected and VIM user is authenticated, VIM can no longer be deleted or updated, only VIM user can be deleted or updated.
• The name of a VIM property or VIM user credentials property is not case sensitive, e.g.
OS_AUTH_URL and os_auth_url is the same to ESC.
Reconfiguring ESC Virtual Machine
This section covers the following:
• Reconfiguring Rsyslog
• Reconfiguring NTP
• Reconfiguring DNS
• Reconfiguring Hosts
• Reconfiguring Timezone
Reconfiguring Rsyslog
Rsyslog parameters are optional. If there is a need for customization after booting an ESC VM, you can edit the files in ESC VM (/etc/rsyslog.d/).
Procedure
Step 1 Editing the Rsyslog file :
• If you haven't specified the log forwarding configuration at the bootup time, you may create a file under
/etc/rsyslog.d/ like /etc/rsyslog.d/log-forwarding.conf.
• If you have specified the log forwarding through installation, you may just need to edit the file. The file could be /etc/rsyslog.d/20-cloud-config.conf. In this file, to forward logs to multiple rsyslog servers, edit the following line:
*.* @[server_ip]:port
Note • Use '@@' before specifying server ip address (if TCP is the protocol used to forward logs to the rsyslog server).
• Use '@' before specifying server ip address (if UDP is the protocol used to forward logs to the rsyslog server).
• server_ip can either be ipv4/ipv6 address of the rsyslog server.
• '[ ]' around the server_ip is required to separate it from ' :port# ', if an ipv6 server address is specified.
For further information on Rsyslog configuration, see the Red Hat documentation.
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
125
Post Installation Tasks
Reconfiguring NTP
Step 2
Step 3
Step 4
Configuring the ESC log file: Configure which ESC log files you want to forward to the rsyslog server: a) Navigate to /etc/rsyslog.d/ Create or modify a configuration file, such as log-esc.conf
. Make a copy of sample log-esc.conf .
b) Specify the following block for every file you want to forward to rsyslog server.
$InputFileName /var/log/esc/escmanager.log
$InputFileTag esc-manager:
$InputFileStateFile stat-esc-manager
$InputFileSeverity info
$InputRunFileMonitor
For example:
$InputFileName /var/log/esc/file1.log
$InputFileTag file1:
$InputFileStateFile stat-file1
$InputFileSeverity info
$InputRunFileMonitor
$InputFileName /var/log/esc/file2.log
$InputFileTag file2:
$InputFileStateFile stat-file2
$InputFileSeverity info
$InputRunFileMonitor
Restart the rsyslog service
# service rsyslog restart
Configure the server side to receive forwarded logs.
a) On a designated server, go to /etc/rsyslog.conf, and uncomment the lines shown below, depending on if you want to listen to logs from clients based on TCP or UDP:
#$ModLoad imudp
#$UDPServerRun 514 b) Exit the file. Run this command as the last step.
sudo service rsyslog restart
Now, the server is listening for logs on port 514, using TCP/UDP.
Reconfiguring NTP
Procedure
Step 1
Step 2
Open the NTP configuration file /etc/ntp.conf in a text editor such as vi, or create a new one if it does not already exist:
# vi /etc/ntp.conf
Add or edit the list of public NTP servers. If you don't specify the NTP server through the installation, the file should contain the following default lines, but feel free to change or expand these according to your needs:
126
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Post Installation Tasks
Reconfiguring DNS
Step 3
Step 4
Step 5 server 0.rhel.pool.ntp.org iburst server 1.rhel.pool.ntp.org iburst server 2.rhel.pool.ntp.org iburst server 3.rhel.pool.ntp.org iburst server <your_ntp_server_ip> iburst
The iburst directive at the end of each line speeds up the initial synchronization.
Once you have the list of servers complete, in the same file, set the proper permissions, giving the unrestricted access to localhost only. Make sure those lines are there in your configure file.
restrict default kod nomodify notrap nopeer noquery restrict -6 default kod nomodify notrap nopeer noquery restrict 127.0.0.1
restrict -6 ::1
Save all changes, exit the editor, and restart the NTP daemon:
# service ntpd restart
Make sure that ntpd is started at boot time:
# chkconfig ntpd on
Reconfiguring DNS
Procedure
Step 1
Step 2
Step 3
The /etc/resolv.conf file contains the configuration for the DNS client (resolver). It typically looks something like this: search domain.com
nameserver 8.8.4.4
This results in a
/etc/resolv.conf
:
Created by cloud-init on instance boot automatically, do not edit.
;
#Generated by esc-cloud domain cisco.com
search cisco.com
nameserver 8.8.4.4
Note It is recommended that you ignore the do not edit message, if you want to modify the file.
You may modify the IP address of the "nameserver" item or add new nameserver records.
search domain.com
nameserver <your_first_dns_ip> nameserver <your_second_dns_ip>
Restart Network Service.
service network restart
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
127
Post Installation Tasks
Reconfiguring Hosts
Reconfiguring Hosts
The /etc/hosts file allows you to add, edit, or remove hosts . This file contains IP addresses and their corresponding hostnames. If your network contains computers whose IP addresses are not listed in DNS, it is recommended that you add them to the /etc/hosts file.
Procedure
Step 1
Step 2
Add the IP addresses that are not listed in DNS to the /etc/hosts file.
Restart your network for the changes to take effect.
service network restart
Reconfiguring Timezone
For ESC VM, in /etc the file "localtime" is a link to or copy of a file containing information about your time zone. Access your zone information files from /usr/share/zoneinfo. To change the time zone, find your country, your city or a city in the same time zone from zone information files in /usr/share/zoneinfo and link it to the localtime in the /etc file.
$ ln \-sf /usr/share/zoneinfo/America/Los_Angeles /etc/localtime
Verifying ESC Configurations and Other Post-Install Operations
This section covers various post-install checks and operations using the escadm tool.
Verifying Existing ESC Configurations
You can use escadm dump command for displaying current ESC configurations in yaml format. The output will show the various services in ESC.
$ sudo escadm dump resources: confd: init_aaa_users:
- name: admin passwd: option: start-phase0 esc_service: group:
- confd
- mona
- vimmanager
- pgsql
- escmanager
- portal
- monitor
- snmp
128
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Post Installation Tasks
Verifying ESC Configurations and Other Post-Install Operations type: group escmanager: {} mona: {} monitor: {} pgsql: {} portal: {} snmp: run_forever: true vimmanager: {}
Verifying VIM configurations
You can use escadm vim show command to verify the vim settings are correctly populated:
$ sudo escadm vim show
[
{
"status": "CONNECTION_SUCCESSFUL",
"status_message": "Successfully connected to VIM",
"type": "OPENSTACK",
"id": "default_openstack_vim",
"properties": {
"property": [
{
"name": "os_auth_url",
"value": "http://172.16.103.143:35357/v3"
}
]
}
}
]
Troubleshooting ESC Services Startup Issues
Problem : Issues encountered while verifying ESC services status at the installation time using sudo escadm status
.
Causes : Some services take time to start or have trouble starting.
Solution :
1.
Identify the issues using one of the following method:
• Check the log /var/log/esc/escadm.log
$ cat /var/log/esc/escadm.log
2017-06-01 20:35:02,925: escadm.py(2565): INFO: promote drbd to master...
2017-06-01 20:35:02,934: escadm.py(2605): INFO: Waiting for at least one drbd to be
UptoDate...
2017-06-01 20:35:02,942: escadm.py(2616): INFO: Waiting for peer drbd node to be demoted...
2017-06-01 20:35:14,008: escadm.py(2423): INFO: mount: /dev/drbd1
/opt/cisco/esc/esc_database
2017-06-01 20:35:14,017: escadm.py(1755): INFO: Starting filesystem service: [OK]
2017-06-01 20:35:15,039: escadm.py(1755): INFO: Starting vimmanager service: [OK]
2017-06-01 20:35:16,116: escadm.py(1755): INFO: Starting monitor service: [OK]
2017-06-01 20:35:17,163: escadm.py(1755): INFO: Starting mona service: [OK]
2017-06-01 20:35:18,440: escadm.py(1755): INFO: Starting snmp service: [OK]
2017-06-01 20:35:21,397: escadm.py(1770): INFO: Starting confd service:[FAILED]
2017-06-01 20:35:28,304: escadm.py(1755): INFO: Starting pgsql service: [OK]
2017-06-01 20:35:29,331: escadm.py(1755): INFO: Starting escmanager service: [OK]
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
129
Post Installation Tasks
Logging in to the ESC Portal
2017-06-01 20:35:30,354: escadm.py(1755): INFO: Starting portal service: [OK]
2017-06-01 20:35:31,523: escadm.py(1755): INFO: Starting esc_service service: [OK]
• Add '-v' to escadm status to show the verbose output of the ESC services.
$ sudo escadm status --v
0 ESC status=0 ESC HA Master Healthy pgsql (pgid 61397) is running vimmanager (pgid 61138) is running monitor (pgid 61162) is running mona (pgid 61190) is running drbd is master snmp (pgid 61541) is running filesystem (pgid 0) is running
<<service>> is dead keepalived (pgid 60838) is running portal (pgid 61524) is running confd (pgid 61263) is running escmanager (pgid 61491) is running
2.
Confirm the status of the identified services that has issues and manually start these services.
$ sudo escadm <<service>> status// If the status is stopped or dead, manually start the services using the next command.
$ sudo escadm <<service>> start --v
Logging in to the ESC Portal
Note • The ESC portal is enabled by default. You must ensure that the ESC portal is not disabled during installation. For more information on enabling or disabling the ESC portal, see
Services Controller Using the QCOW Image, on page 8
.
• When you log in to the ESC portal for the first time you are prompted to change the default password.
To log in to the ESC portal, do the following:
Before you begin
• Register an instance of ESC. For more information on registering the ESC instance see,
Elastic Services Controller Using the QCOW Image, on page 8
• Ensure that you have the username and password.
Procedure
Step 1 Using your web browser, enter the IP address of ESC and port 8443.
Example:
For example, if the IP address of ESC is 192.0.2.254, enter:
130
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Post Installation Tasks
Logging in to the ESC Portal
Step 2
Step 3
Step 4
Step 5 https://192.0.2.254: 8443 [ login via https]
A Security Alert message is displayed.
Click Yes to accept the security certificate. The Login page is displayed.
Enter the username and password and click Login .
If you are logging in for the first time, the login page reappears, prompting you to change your password.
Enter the old password in the Old Password field, then enter a new password in the New Password and Confirm
Password fields.
Click Update Password or press Enter .
Note • If the UI becomes unresponsive, restart the UI by executing the sudo escadm portal restart from the ESC shell prompt.
• ESC Portal only supports one user.
• Currently, a pre-installed self-signed certificate supports HTTPS. The user must confirm the self-signed certificate before proceeding with the ESC Portal.
• In HTTPS communication mode, if the URL protocol type returned by OpenStack is not
HTTPS, the access to the VNF Console may be disabled. For security reasons, while running in HTTPS more non-secure communication will be rejected.
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
131
Logging in to the ESC Portal
Post Installation Tasks
132
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
P A R T
VIII
Upgrading Cisco Elastic Services Controller
•
ESC in Maintenance Mode, on page 135
•
Upgrading Cisco Elastic Services Controller, on page 141
C H A P T E R
19
ESC in Maintenance Mode
This chapter contains the following chapters:
•
Setting ESC in a Maintenance Mode, on page 135
•
Backup the Database from the ESC Standalone Instances, on page 136
•
Backup the Database from the ESC HA Active/Standby Instances, on page 138
•
Restoring ESC Database, on page 140
Setting ESC in a Maintenance Mode
ESC must be put to maintenance mode to backup and restore ESC database. To do so, use the escadm tool as specified in the below section.
Before you begin
Note From ESC release 4.4, ESC continues to be in maintenance mode after HA Active/Standby switch over or
DB restoration, if ESC was in maintenance mode before HA Active/Standby switch over or DB restoration.
During maintenance mode,
• Northbound requests are blocked by ESC and ESC responds with maintenance mode notification.
• Only REST requests receive response that ESC is unavailable temporarily. ConfD requests get the maintenance mode rejection message, or an OK message for all idempotent request such as create tenant request when the tenant already exists.
• Monitoring actions are paused.
• All ongoing requests and transactions continue to progress.
Using the escadm Tool
ESC can be put to maintenance mode using the escadm tool.
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
135
Upgrading Cisco Elastic Services Controller
Setting ESC in an Operation Mode
Procedure
Step 1
Step 2
Step 3
Put ESC to maintenance mode from the VM shell: sudo escadm op_mode set --mode=maintenance
Set mode to MAINTENANCE
Operation Mode = MAINTENANCE
If you have D-MONA deployed, do the following to each D-MONA: curl -i -k --user {REST username}:{REST password} -X POST https:/{DMONA
IP}:8443/mona/v1/rules/pause_all
To query operation mode at any time, sudo escadm op_mode show
Example:
Operation Mode = OPERATION
Set maintenance mode when there is no in-flight transaction. Using the ipt_check flag with the escadm tool, you can choose to set ESC in the maintenance mode only if there are no ongoing transactions in ESC. Set the flag to true, if you do not want ESC to set in the maintenance mode, if there are ongoing transactions in ESC.
sudo escadm op_mode set --mode=maintenace --ipt_check=true
With the ipt_check option set to true, escadm tool checks if there is any on going operation, if so, the escadm tool will not set ESC to maintenance mode.
Note ESC with D-MONA deployment, requires to do extra calls when going to maintenance mode.
Setting ESC in an Operation Mode
Put ESC in operation mode using the escadm tool: sudo escadm op_mode set --mode=operation
Response is as follows:
Set mode to OPERATION
Operation Mode = OPERATION
If you have D-MONA deployed, do the following to each D-MONA: curl -i -k --user {REST username}:{REST password} -X POST https:/{DMONA
IP}:8443/mona/v1/rules/resume_all
Verify ESC's operation mode at any time using the following command: sudo escadm op_mode show
Backup the Database from the ESC Standalone Instances
• The following assumptions should be taken into consideration :
• A third machine is required to store the database and log backups.
136
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Upgrading Cisco Elastic Services Controller
Backup the Database from the ESC Standalone Instances
• ESC does not support database schema downgrade. Restoring database to the older ESC version could cause unexpected problems.
• Before you start the backup process, ensure you have an external storage space (could be in the OpenStack controller or any system accessible by ESC). The backup/restore could be expressed in a generic format which will be used by the escadm tool: scp://<username>:<password>@<backup_ip>:<filename> . In this format, the credentials, IP address and file storage path of the third machine are required. You may also use localhost IP as the backup IP to backup database in a location of ESC VM and then copy the files to the external storage
To backup the ESC database from a standalone ESC or a HA Active/Standby (master node):
Procedure
Step 1
Step 2
Step 3
Step 4
Step 5
Step 6
Step 7
Log in to ESC VM and set it to maintenance mode, run:
$ sudo escadm op_mode set --mode=maintenance
To make sure ESC is in maintenance mode, run:
$ sudo escadm op_mode show
Backup database. Execute the commands below as a root user:
# sudo escadm backup --file /tmp/db_file_name.tar.bz2
scp://<username>:<password>@<backup_vm_ip>:<filename>
To put ESC back to operation mode, run:
$ sudo escadm op_mode set --mode=operation
$ sudo escadm op_mode show
Collect all the logs from the old ESC VM and back it up. Execute the below command as a root user.
# sudo escadm log collect
A timestamped log file will be generated in: /var/tmp/esc_log <timestamp>.tar.bz2
Note If a dynamic mapping file is used by ESC service, the dynamic mapping file should be backed up at the same time with ESC logs. The default path of the dynamic mapping file is
/opt/cisco/esc/esc-dynamic-mapping/dynamic_mappings.xml.
After a successful database back-up, shut down the old ESC VM using Horizon/Kilo or Nova commands. For
ESC VM instances based in VMware vSphere, shutdown the primary instance through VMware client dashboard. An example of shutting down a VM in OpenStack is shown below :
$ nova stop OLD_ESC_ID
Detach the old port from the old VM and rename the old ESC node. Examples of detaching and renaming the
VM in OpenStack is shown below: nova interface-detach ESC_NAME port-id-of-ESC_NAME nova rename ESC_NAME ESC_NAME.old
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
137
Upgrading Cisco Elastic Services Controller
Backup the Database from the ESC HA Active/Standby Instances
In VMWare, assign a different IP address to the old VM and then rename the old VM.
BackuptheDatabase fromtheESCHAActive/Standby Instances
• The following assumptions should be taken into consideration :
• A third machine is required to store the database and log backups.
• ESC does not support database schema downgrade. Restoring database to the older ESC version could cause unexpected problems.
• Before you start the backup process, ensure you have an external storage space available (could be in the OpenStack controller or any system accessable by ESC). The backup/restore could be expressed in a generic format which will be used by the escadm tool: scp://<username>:<password>@<backup_ip>:<filename> . In this format, the credentials, IP address and file storage path of the third machine are required. You may also use localhost IP as the backup IP to backup database in a location of ESC VM and then copy the files to the external storage.
To backup the ESC database from a standalone ESC or a HA Active/Standby (master node):
Procedure
Step 1
Step 2
Perform the following steps on the Standby ESC node.
a) Connect to the standby ESC instance using SSH:
$ ssh <username>@<backup_vm_ip> b) Verify that the ESC instance is standby and note the name of the standby ESC HA Active/Standby instance
:
$ sudo escadm status --v
If the output value shows "BACKUP", the node is the standby ESC node.
c) Change access to an admin user.
sudo bash d) Collect all the logs from the standby ESC VM and back it up.
$ sudo escadm log collect
A timestamped log file will be generated in: /var/tmp/esc_log <timestamp>.tar.bz2
e) Shutdown the standby ESC instance through OpenStack Kilo/Horizon using Nova command or VMware client. An example of shutting down the VM on OpenStack is shown below:
$ nova stop OLD_ESC_STANDBY_ID
Perform the following steps on the Master ESC node.
a) Connect to the primary ESC instance using SSH:
138
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Upgrading Cisco Elastic Services Controller
Backup the Database from the ESC HA Active/Standby Instances
Step 3
Step 4
$ ssh <username>@<master_vm_ip> b) Change access to an admin user.
$ sudo bash c) Verify that the ESC instance is Master and note the name of the Master ESC HA Active/Standby instance
$ sudo escadm status --v
If the output value shows "MASTER", the node is the Primary ESC node.
d) Back up the database files from the master node of ESC HA Active/Standby:
$ sudo escadm backup --file /tmp/db_file_name.tar.bz2
scp://<username>:<password>@<backup_vm_ip>:<filename> e) Collect the logs from the master ESC VM and back it up.
$ sudo escadm log collect
A timestamped log file will be generated in: /var/tmp/esc_log <timestamp>.tar.bz2
Note If a dynamic mapping file is used by ESC service, the dynamic mapping file should be backed up at the same time with ESC logs. The default path of the dynamic mapping file is
/opt/cisco/esc/esc-dynamic-mapping/dynamic_mappings.xml
.
Shutdown the primary ESC instance through OpenStack Kilo/Horizon using Nova command. For ESC VM instances based in VMware vSphere, shutdown the primary instance through VMware client dashboard. An example of shutting down the VM on OpenStack is shown below:
$ nova stop OLD_ESC_MASTER
Use the nova list command or the nova show OLD_ESC_STANDBY command, to verify if the ESC HA
Active/Standby instances have been successfully shut down.
(Only for OpenStack) Detach the port from the old ESC VM and rename the old VM.
If upgraded ESC VM needs to operate with same IP addresses and same instance names as old instances, detach the ports from each instance, shutdown the old ESC VMs and then rename the old ESC instances.
If you intend to use old VMware primary instance, assign a different IP address and rename the VM name.
If not, you can delete the old VM and use the same IP address for the new upgraded VMware primary instance.
After deleting the old VM, you can continue with the old instance name and the IP address.
OpenStack commands for detaching the ports and renaming the old VMs are shown below: nova interface-list ESC_NAME nova interface-detach ESC_NAME port-id-of-ESC_NAME nova rename ESC_NAME ESC_NAME.old
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
139
Upgrading Cisco Elastic Services Controller
Restoring ESC Database
Restoring ESC Database
Before you begin
To restore the database,
• In standalone ESC instance, stop ESC services. Run
# sudo escadm stop
.
• In HA Active/Standby type instances, stop escadm on the Backup first, and later on the Master ESC HA
Active/Standby instance. Run
# sudo escadm stop
.
• All the services must be stopped. To check the status, run
# sudo escadm status --v
.
Procedure
Step 1
Step 2
Step 3
Restore the database. Execute the following command from the ESC VM:
$ scp <username>@<server_ip>:/path/db.tar.bz2
$ sudo escadm restore –file /tmp/db.tar.bz2
/tmp/
Enter the ESC password in the URL, or manually enter it after executing the above command.
Restart the ESC service to complete the database restore by running the following command:
$ sudo escadm restart
Note ESC maintenance mode blocks the northbound request and VNF monitoring. However, if there are some ongoing transactions because of northbound requests before ESC entering maintenance mode, those transactions may have following restriction with backup and restore:
• ESC reports an error for the deployment, network creation, and subnet creation requests, if these transactions are interrupted by backup and restore. Northbound handles these error messages but it may cause network or subnet leakage in some cases (For example, ESC is interrupted before getting the UUID from OpenStack).
• ESC reports an error for service chain upgrade, and requires service chain undeployment and deployment (rather than downgrade and upgrade) to re-create the service.
140
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
C H A P T E R
20
Upgrading Cisco Elastic Services Controller
Cisco Elastic Service Controller supports two type of upgrades:
• Backup and Restore Upgrade : This upgrade process involves stopping the ESC keepalive daemon (for
ESC HA), backing up the database, stopping and renaming (or deleting) the ESC instances, re-installing the ESC instances, and restore database. For information on the supported ESC versions for ESC 5.3
upgrade, see the table below.
• In-service upgrade : ESC supports in-service upgrade for Active/Standby high availability nodes with a minimum downtime.
You can upgrade the ESC instance as a standalone instance or as a high availability pair. The upgrade procedure is different for standalone and high availability pair.
This chapter lists separate procedures on how to upgrade ESC standalone and ESC High Availability instance.
You must review these instructions before you decide to upgrade the ESC instance. See the
for more information on the installation scenarios.
• ESC only support direct upgrade from previous two minor releases. For example, ESC 2.3 will support direct upgrade from ESC 2.1 and ESC 2.2. For any release older than the supported versions for direct upgrade, you need to perform the staged upgrade.
Note Example of an ESC major release, ESC 3.1
Example of an ESC minor release, ESC 3.1.1
Example of an ESC patch release, ESC 3.1.0.116.
• Upgrading ESC using RPM Package (referred to as RPM Upgrade in this chapter) applies only to the
ESC upgrade between ESC patch releases with the same release number. For Example, the upgrade from
ESC 3.1.0.116 to ESC 3.1.0.150.
• If you want to upgrade ESC between minor releases (for example, upgrade from ESC 2.3.1 to ESC
2.3.2) or major releases (for example, upgrade from ESC 3.0 to ESC 4.0 ), you can upgrade through
Backup and Restore upgrade process using qcow2 image.
• For ESC upgrade, you should be familiar with ESC installation process.
• For OpenStack, refer to the OpenStack installation procedures, see Chapter 4: Installing Cisco Elastic
Services Controller in OpenStack.
• For VMware, refer to the VMware Installation installation procedures, see Chapter 7: Installing
Cisco Elastic Services Controller in VMware vCenter.
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
141
Upgrading Cisco Elastic Services Controller
Upgrading Standalone ESC Instance
• For ESC HA, please refer to the ESC HA installation procedures, see Chapter 5 : Configuring High
Availability for OpenStack and Chapter 8: Configuring High Availability for VMware.
Table 6: Supported ESC Versions for Upgrading to ESC 5.3
Virtual Infrastructure
Manager
OpenStack
VMware
CSP
Supported Versions for Backup and
Restore Upgrade
Supported Versions for In-Service Upgrade
5.2, 5.1
5.2, 5.1
5.2, 5.1
5.2, 5.1
5.2, 5.1
5.2, 5.1
IMPORTANT NOTES
• After upgrading to the new ESC version, ESC service will manage the life cycle of all VNFs deployed in the previous release. To apply any new features (with new data models) to the existing VNFs, you must undeploy and redeploy these VNFs.
• Upgrade is supported for Active/Active HA only.
Note Do not change the VIP in ESC upgrade in ETSI deployment.
If you change ETSI's REST schema, from http to https in-flight, ESC stops sending recovery notification from
ESC core for any existing deployment.
•
Upgrading Standalone ESC Instance, on page 142
•
Upgrading ESC HA Active/Standby Instances, on page 144
•
In-Service Upgrade of the ESC HA Active/Standby Nodes in OpenStack, on page 146
•
•
In-Service Upgrade of the ESC HA Active/Standby Nodes in VMware, on page 154
•
In-Service Upgrade of the ESC HA Active/Standby Nodes in CSP, on page 158
Upgrading Standalone ESC Instance
To upgrade standalone ESC instance, perform the following tasks:
1.
Back up the ESC database. For more information, see
Backup the Database from the ESC Standalone
.
Note To backup any custom scripts used by the deployments, and for restoring them before restoring the database, you must re-install the scripts on the backup as well.
2.
Redeploy the ESC instance. For more information, see the below section, Deploy the ESC for Upgrade .
142
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Upgrading Cisco Elastic Services Controller
Deploy the ESC for Upgrade
3.
Restore the ESC database on the new ESC instance. For more information, see the below section, Restoring the ESC Database .
Deploy the ESC for Upgrade
After backing up and shutting down of the old ESC VM, a new/upgraded (based on new ESC package) ESC
VM should be installed. All parameters for ESC installation should be the same as the old ESC VM deployment.
• For OpenStack, you need to register the new ESC qcow2 image using the Glance command with a new image name and then use new bootvm.py script and new image name to install ESC VM.
Note In OpenStack, if an old ESC VM was assigned with floating IP, the new ESC
VM should be associated with the same floating IP after the installation.
• For VMWare, you need to use the new ESC OVA file to install ESC VM. All other configurations and property values should be the same as the old VM.
Restoring the ESC Database
Restore the ESC database on the new ESC instance , using the following procedure:
Procedure
Step 1
Step 2
Step 3
Step 4
Step 5
Step 6
Connect to the new ESC instance using SSH.
$ ssh USERNAME@NEW_ESC_IP
Stop the ESC service.
$ sudo escadm stop
Check ESC service status to make sure all the services are stopped.
$ sudo escadm status
Restore the database files.
$ scp://<username>:<password>@<backup_ip>:<filename>
$ sudo escadm restore --file /path/where/file/scp-ed/to/db.tar.bz2
Start the ESC service:
$ sudo escadm start
After ESC service is started, the standalone ESC upgrade is complete. You can check the health of the new
ESC service by running
$ sudo escadm status in the new ESC VM.
In Openstack, after restoring the database successfully, delete the old ESC instance:
$ nova delete OLD_ESC_ID
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
143
Upgrading Cisco Elastic Services Controller
Important Notes:
Important Notes:
After upgrading to the new ESC version, ESC service will keep doing life cycle management of all VNFs deployed by the old version. However, to apply any new features (with new data models) to the VNFs deployed by the ESC with old version is not guaranteed. If you want to apply any new feature of the new ESC version to existing VNFs, you have to undeploy and redeploy those VNFs.
Upgrading ESC HA Active/Standby Instances
To upgrade ESC HA Active/Standby nodes, perform the following tasks:
1.
Back up the database from an old ESC HA Active/Standby primary instance. For more information, see
Backup the Database from the ESC HA Active/Standby Instances .
Note To backup any custom scripts used by the deployments, and for restoring them before restoring the database, you must copy the scripts on the backup as well.
2.
Deploy new ESC HA Active/Standby nodes based on new ESC version. For more information, see the below section, Deploy the ESC HA Active/Standby nodes for Upgrade .
3.
Restore the Database on Primary ESC instance (Standby ESC instance will sync with the Primary ESC instance ). For more information, see the below section, Restoring the ESC Database on New Master and Standby Instances .
Deploying the ESC HA Active/Standby nodes for Upgrade
After backing up and shutting down the two old ESC VMs, based on new ESC package install the new ESC
VMs.
• For OpenStack, you need to register the new ESC qcow2 image using the Glance command with a new image name and then to use new bootvm.py script and new image name to install ESC VM. All other bootvm.py arguments should be the same as used to setup an old VMs.
• For VMWare, there are two steps to bring up HA Active/Standby pair in VMware: 1) setup two standalone instances 2) reconfigure each instance with HA Active/Standby info. All other configurations and property values should be the same as the old VMs.
• If VIP is used for Northbound access, keep VIP the same for the new deployment as used to reconfigure the old HA Active/Standby pair.
Restoring the ESC Database on New Master and Standby ESC Instances
Procedure
Step 1
Step 2
Shut down the Standby ESC instance .
Connect to the standby ESC instance using SSH.
$ ssh USERNAME@ESC_STANDBY_IP
:
Verify that the ESC instance is standby and note the name of the standby ESC HA Active/Standby instance
144
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Upgrading Cisco Elastic Services Controller
Restoring the ESC Database on New Master and Standby ESC Instances
Step 3
Step 4
Step 5
Step 6
Step 7
Step 8
Step 9
Step 10
Step 11
$ sudo escadm status
If the output value shows "BACKUP", the node is the standby ESC node.
Note If a dynamic mapping file (dynamic_mapping.xml) is used by ESC service, the dynamic mapping file should be restored into the backup ESC VM. Before power off the standby ESC node, you need to copy the backup dynamic mapping file (dynamic_mapping.xml) to the path
/opt/cisco/esc/esc-dynamic-mapping/ .
Shutdown the standby ESC instance through OpenStack Kilo/Horizon using Nova command. For ESC VM instances based in VMware vSphere, shutdown the primary instance through VMware client dashboard. An example of shutting down the standby ESC instance in OpenStack is shown below:
$ nova stop NEW_ESC_STANDBY_ID
Restore the database on the new Master ESC instance.
Connect to the primary ESC instance using SSH.
$ ssh USERNAME@ESC_MASTER_IP
Verify that the ESC instance is primary.
$ sudo escadm status
If the output value shows 'MASTER' , the node is the master ESC node.
Stop the ESC services on the master node and verify the status to ensure the services are stopped.
$ sudo escadm stop
$ sudo escadm status
Restore the database files.
$ sudo escadm restore --file /tmp/db.tar.bz2
$ scp://<username>:<password>@<backup_ip>:<filename>
Note If a dynamic mapping file (dynamic_mapping.xml) is used by ESC service, the dynamic mapping file should be restored into the ESC VM. Before starting the ESC node, you need to copy the backup dynamic mapping file (dynamic_mapping.xml) to the path /opt/cisco/esc-dynamic-mapping/ .
Reboot the VM to restart the full ESC service:
$ sudo escadm restart
Use the
$ sudo escadm status to check the status of the ESC service.
Start the standby ESC node.
Power on the standby ESC node through OpenStack Nova/Horizon or VMware client. After starting the standby node, ESC HA Active/Standby upgrade process should be complete.
Delete the old HA Active/Standby instance through OpenStack Nova/Horizon or VMware client. An example of deleting the VM on OpenStack is shown below:
$ nova delete OLD_ESC_MASTER_RENAMED OLD_ESC_STANDBY_RENAMED
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
145
Upgrading Cisco Elastic Services Controller
Upgrading VNF Monitoring Rules
Upgrading VNF Monitoring Rules
To upgrade the VNF monitoring rules, you must back up the dynamic_mappings.xml
file and then restore the file in the upgraded ESC VM. For more information, see the backup and restore procedures. For upgrade of
HA Active/Standby instance, see
Upgrading ESC HA Active/Standby Instances
. For upgrade of the standalone instance, see
Upgrading Standalone ESC Instance
.
In-Service Upgrade of the ESC HA Active/Standby Nodes in
OpenStack
In-Service upgrade in OpenStack using ESC RPM packages
Procedure
Step 1
Step 2
Step 3
Step 4
Step 5
Backup ESC database and log files.
a) Perform ESC database backup from primary node. For more information on backing up the database, see
Backup the Database from the ESC HA Active/Standby Instances .
b) Collect and backup all logs from both primary and secondary VMs. To backup the log, use the following command:
# sudo escadm log collect
Note A timestamped file will be generated in: /var/tmp/esc_log-<timestamp>.tar.bz2
c) Copy the database backup file and logs files (generated in /tmp/esc_log-.tar.bz2)* out of ESC VMs.
Log into the ESC HA Active/Standby secondary VM and stop the escadm service.
$ sudo escadm stop
Ensure the ESC VM is in STOP state. ESC may take some time to switch to the STOP state. If ESC status turns into STOP state, please note that it won't be the part of HA Active/Standby cluster and you will lose
HA Active/Standby function temporarily.
$ sudo escadm status
Expected output:
ESC status=0 ESC HA is stopped
Copy the RPM file for upgrade to the ESC VM and execute the rpm command for upgrade.
$ sudo rpm -Uvh /home/admin/cisco-esc-3.1.0-145.x86_64.rpm
Start the escadm service.
$ sudo escadm start
146
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Upgrading Cisco Elastic Services Controller
In-Service upgrade in OpenStack using ESC qcow2 Image
Step 6
Step 7
Log into the ESC HA Active/Standby Primary VM and repeat step 2 to step 6 in Primary VM. Please note that after stop escadm service in Primary ESC VM, a failover will be triggered and the upgraded secondary
VM will take over the Primary role.
Check the ESC version on each instance to verify the version is upgraded correctly and make sure ESC service is running properly in new Primary VM.
# esc_version
# health.sh (in Primary VM)
In-Service upgrade in OpenStack using ESC qcow2 Image
Procedure
Step 1
Step 2
Backup ESC database and log files.
a) Perform ESC database backup from primary node. For more information on backing up the database, see
Backup the Database from the ESC HA Active/Standby Instances .
b) Collect and backup all logs from both primary and secondary VMs. To backup the log, use the following command:
# sudo escadm log collect
Note A timestamped file will be generated in: /var/tmp/esc_log-<timestamp>.tar.bz2
c) Copy the database backup file and logs files (generated in /tmp/esc_log-.tar.bz2)* out of ESC VMs.
Redeploy secondary instance with the new version of ESC image, and wait for the data to be synchronized.
a) Delete the secondary instance through Horizon Horizon Web UI or nova CLI. In OpenStack controller, running following command through nova client.
nova delete <secondary_vm_name> b) Register new ESC image into OpenStack Glance for redeployment usage.
glance image-create --name <image_name> --disk-format qcow2 --container-format bare
--file <esc_qcow2_file> c) Redeploy the secondary ESC VM instance based on newer image version. Re-install new the secondary instance by using the new ESC package (bootvm.py and new registered image). All other installation parameters should be the same as the former ESC VM deployment. For example, hostname, ip address, gateway_ip, ha_node_list, kad_vip, kad_vif have to use the same values. Once the new ESC instance with upgraded version is up, it will be in secondary state.
d) Log into the new instance and run the following command to check the synchronization state of the new
ESC node.
$ drbdadm status
Wait until the output of drbdadm status show both nodes are "UpToDate" like the output below. It means the new ESC instance has completed the data synchronization from the primary instance.
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
147
Upgrading Cisco Elastic Services Controller
In-Service upgrade in OpenStack using ESC qcow2 Image
Step 3
Step 4
Example for Backup/Secondary esc role:Secondary disk:UpToDate
101.1.0.119:7789 role:Primary peer-disk:UpToDate
Example for Master/Primary ESC esc role:Primary disk:UpToDate
101.1.0.120:7789 role:Secondary peer-disk:UpToDatee
Stop keepalived service on Secondary instance, Power off primary instance, and then start Secondary keepalived service.
a) To stop keepalived service, use the following command: escadm keepalived stop b) Log into the primary instance, set ESC primary node into maintenance mode.
$ sudo escadm op_mode set --mode=maintenance
Make sure there is no in-flight transaction ongoing before moving to the next step. To verify there are no in-flight transactions, use the following command:
For ESC 2.3:
$ sudo escadm ip_trans
For versions older than ESC 2.3, check escmanager log at (/var/log/esc/escmanager.log) and make sure there are no new transaction in escmanager log.
c) Log in to the upgraded secondary instance and shut down the ESC service.
$ sudo escadm stop d) Power off the primary instance through OpenStack Nova client/Horizon and make sure it is off. In
OpenStack Controller, run:
$ nova stop <primary_vm_name>
$ nova list | grep <primary_vm_name> e) Log into the previously upgraded secondary instance which is in stopped state and restart the ESC service.
The secondary ESC instance will take the primary role (switchover will be triggered) and start providing services with new version.
$ sudo escadm start
Check the ESC version on the new primary instance to verify the version is upgraded correctly.
$ sudo escadm status (check ha status)
Expected output:
0 ESC status=0 ESC Master Healthy
$ esc_version (check esc version) version : 3.x.x
148
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Upgrading Cisco Elastic Services Controller
In-Service upgrade in OpenStack using ESC qcow2 Image
Step 5
Step 6 release : xxx
Re-deploy the old primary instance with the new ESC image.
Delete the old primary instance and redeploy it by using the new ESC package (bootvm.py and new registered image).
a) Log in to the new deployed instance and check ha status. The new instance should be in secondary state:
$ sudo escadm status --v b) Run the following command to check the synchronization state of the new ESC secondary node:
$ drbdadm status
Wait until the output of drbdadm status shown as UpToDate.
c) For the new ESC secondary node, make sure the health check is passed and the ESC version are upgraded correctly.
$ sudo escadm status (check ha status)
Expected output:
0 ESC status=0 ESC Master Healthy
$ esc_version (check esc version)version : 2.x.x
release : xxx
$ health.sh
Expected output:
ESC HEALTH PASSED
Go back in to the first upgraded primary instance and check the health and keepalived state.
$ drbdadm status
Expected output:
1:esc/0 Connected Primary/Secondary UpToDate/UpToDate /opt/cisco/esc/esc_database ext4
2.9G 52M 2.7G 2%
$ sudo escadm status (check ha status)
Expected output:
0 ESC status=0 ESC Master Healthy
$ esc_version (check esc version) Expected output: version : 2.x.x
release : xxx
$ health.sh (check esc health)
Expected output:
ESC HEALTH PASSED
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
149
In-Service Upgrade of the ESC H AActive/Standby Nodes in Kernel-Based Virtual Machine (KVM)
Upgrading Cisco Elastic Services Controller
Note Quick rollback: In case of an upgrade failure, shutdown the upgraded instance and start the old primary instance to have a quick rollback.
Rollback Procedure for In-service Upgrade a.
Copy the database and log backup files to a location out of ESC VMs.
b.
Delete any remaining ESC instance and redeploy ESC HA Active/Standby VMs using qcow2 image with old version.
c.
Restore the database. Follow the procedures in the section, Upgrading ESC HA Active/Standby
Instance with Backup and Restore for HA Active/Standby database restore.
d.
After database restore, you should have ESC service back with the old version.
In-Service Upgrade of the ESC H AActive/Standby Nodes in
Kernel-Based Virtual Machine (KVM)
In-Service Upgrade in KVM using ESC RPM packages
Use this procedure to upgrade ESC high-availability nodes with a minimum service interruption on a
Kernel-based virtual machine.
Procedure
Step 1
Step 2
Step 3
Step 4
Step 5
Backup ESC database and log files.
a) Perform ESC database backup from primary node. For more information on backing up the database, see
Backup the Database from the ESC HA Active/Standby Instances .
b) Collect and backup all logs from both primary and secondary VMs. To backup the log, use the following command:
$ sudo escadm log collect
Note A timestamped log file will be generated in: /var/tmp/esc_log-<timestamp>.tar.bz2
c) Copy the database backup file and logs files (generated in /tmp/esc_log-.tar.bz2)* out of ESC VMs.
Log into the ESC HA Active/Standby secondary VM and stop the ESC service.
$ sudo escadm stop
Make sure the secondary ESC VM is in STOP state.
$ sudo escadm status --v
If ESC status=0 esc ha is stopped.
In secondary VM, execute the rpm command for upgrade:
$ sudo rpm -Uvh /home/admin/cisco-esc-<latest rpm filename>.rpm
Log into the primary instance, set ESC primary node into maintenance mode.
$ sudo escadm op_mode set --mode=maintenance
150
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Upgrading Cisco Elastic Services Controller
In-Service Upgrade in KVM using ESC OVA Image
Step 6
Step 7
Step 8
Step 9
Step 10
Make sure there are no in-flight transactions and no new transactions during the upgrade. Use following commands to check in-flight transactions.
$ sudo escadm ip_trans
For any build older than ESC 2.3, you may need to check escmanager log for transactions at
(/var/log/esc/escmanager.log).
Power off ESC primary node and make sure it is completely shut down. In KVM ESC controller, execute the following commands:
$ virsh destroy <primary_vm_name>
$ virsh list --all
Log in the upgraded ESC instance (previous secondary one), start the ESC service. The upgraded VM will take over primary role and provide ESC service.
$ sudo escadm restart
$ sudo escadm monitor start
Check the ESC version on the new primary instance to verify the upgraded version is correct. Once it is in the Primary state, make sure ESC service is running properly in the new Primary VM.
$ sudo escadm status
Expected output:
0 ESC status=0 ESC Master Healthy
$ esc_version
$ health.sh
Expected output:
ESC HEALTH PASSED
Power on the old primary instance. In KVM ESC controller, execute the following commands:
$ virsh start <primary_vm_name>
Log into the VM which is still with old ESC version and repeat step 2, 3, 4, and 7 in the VM.
In-Service Upgrade in KVM using ESC OVA Image
Procedure
Step 1 Backup ESC database and log files.
a) Perform ESC database backup from primary node. For more information on backing up the database, see
Backup the Database from the ESC HA Active/Standby Instances .
b) Collect and backup all logs from both primary and secondary VMs. To backup the log, use the following command:
$ sudo escadm log collect
Note A timestamped log file will be generated in: /var/tmp/esc_log-<timestamp>.tar.bz2
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
151
Upgrading Cisco Elastic Services Controller
In-Service Upgrade in KVM using ESC OVA Image
Step 2
Step 3
Step 4 c) Copy the database backup file and logs files (generated in /tmp/esc_log-.tar.bz2)* out of ESC VMs.
Redeploy secondary ESC instance. Register new ESC image on the secondary instance.
a) Delete the secondary instance through lib vert Virsh commands. On KVM host, run the following command:
$ Virsh destroy the <secondary_vm_name>
$ Virsh undefine --remove-all-storage <secondary_vm_name> b) Copy the new ESC image into Kvm Host for redeployment usage: sshpass -p "host Password' scp /scratch/BUILD-2_x_x_x/BUILD-2_x_x_x/ESC-2_x_x_x.qcow2
root@HOSTIP: c) Redeploy the secondary ESC VM instance based on newer image version. Re-install new the secondary instance by using the new ESC package (bootvm.py and new registered image). All other installation parameters should be the same as the former ESC VM deployment. For example, hostname, ip address, gateway_ip, ha_node_list, kad_vip, kad_vif have to use the same values. Once the new ESC instance with upgraded version is up, it will be in secondary state.
d) Log into the new instance and run the following command to check the synchronization state of the new
ESC node.
$ drbdadm status wait until the output of drbdadm status show both nodes are "UpToDate" like the output below. It means the new ESC instance has completed the data synchronization from the primary instance.
esc/0 Connected Secondary/Primary UpToDate/UpToDate
Stop keepalived service on Secondary instance, Power off primary instance, and then start Secondary keepalived service.
a) Log into the primary instance, set ESC primary node into maintenance mode.
$ sudo escadm op_mode set --mode=maintenance
Make sure there is no in-flight transaction ongoing before moving to the next step. To verify there are no in-flight transactions, use the following command:
$ sudo escadm ip_trans
Check escmanager log at (/var/log/esc/escmanager.log) and make sure there are no new transaction in escmanager log.
b) Log in to the upgraded secondary instance and shut down the keepalived service.
$ sudo escadm stop c) Power off the primary instance and make sure it has been completely turned off. In KVM ESC Controller, run:
$ virsh destroy <primary_vm_name>
$ virsh list --all d) Log into the previously upgraded secondary instance which is in stopped state and start the ESC service.
The secondary ESC instance will take the primary role (switchover will be triggered) and start providing services with new version.
$ sudo escadm restart
Check the ESC version on the new primary instance to verify the version is upgraded correctly.
152
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Upgrading Cisco Elastic Services Controller
In-Service Upgrade in KVM using ESC OVA Image
Step 5
Step 6
$ sudo escadm status (check ha status)
Expected output:
0 ESC status=0 ESC Master Healthy
$ esc_version (check esc version) version : 4.1.x
release : xxx
$ health.sh (check esc health)
Expected output:
ESC HEALTH PASSED
Re-deploy the old primary instance with the new ESC image.
Delete the old primary instance and redeploy it by using the new ESC package (bootvm.py and new registered image). All other installation parameters should be the same as the old ESC VM deployment. For example, hostname, ip address, gateway_ip, ha_node_list, kad_vip, kad_vif have to be the same values.
a) Log in to the new deployed instance and check ha status. The new instance should be in secondary state:
$ sudo escadm status b) Run the following command to check the synchronization state of the new ESC secondary node:
$ drbdadm status
Wait until the output of drbdadm status shown as UpToDate.
c) For the new ESC secondary node, make sure the health check is passed and the ESC version are upgraded correctly.
$ sudo escadm status (check ha status)
Expected output:
0 ESC status=0 ESC Master Healthy
$ esc_version (check esc version)version : 4.1.x
release : xxx
$ health.sh
Expected output:
ESC HEALTH PASSED
Go back in to the first upgraded primary instance and check the health and keepalived state.
$ drbdadm status
Expected output:
1:esc/0 Connected Primary/Secondary UpToDate/UpToDate /opt/cisco/esc/esc_database ext4
2.9G 52M 2.7G 2%
$ sudo escadm status (check ha status)
Expected output:
0 ESC status=0 ESC Master Healthy
$ esc_version (check esc version) Expected output: version : 2.x.x
release : xxx
$ health.sh (check esc health)
Expected output:
ESC HEALTH PASSED
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
153
Upgrading Cisco Elastic Services Controller
In-Service Upgrade of the ESC HA Active/Standby Nodes in VMware
Note Quick rollback: In case of an upgrade failure, shutdown the upgraded instance and start the old primary instance to have a quick rollback.
Rollback Procedure for In-service Upgrade a.
Delete any remaining ESC instance and redeploy ESC HA Active/Standby VMs using qcow2 image with old version.
b.
Restore the database. Follow the procedures in the section, Upgrading ESC HA Active/Standby
Instance with Backup and Restore for HA Active/Standby database restore.
c.
After database restore, you should have ESC service back with the old version.
In-Service Upgrade of the ESC HA Active/Standby Nodes in
VMware
In-Service upgrade in VMware using ESC RPM packages
Use this procedure to upgrade the ESC high-availability nodes one node at a time with a minimum service interruption. This process leverages the ESC HA Active/Standby replication and failover capability to smoothly move ESC service to the new upgraded node without the manual database restore.
Procedure
Step 1
Step 2
Step 3
Step 4
Step 5
Backup ESC database and log files.
a) Perform ESC database backup from primary node. For more information on backing up the database, see
Backup the Database from the ESC HA Active/Standby Instances .
b) Collect and backup all logs from both primary and secondary VMs. To backup the log, use the following command:
# sudo escadm log collect c) Copy the database backup file and logs files (generated in /tmp/esc_log-.tar.bz2)* out of ESC VMs.
Log into the ESC HA Active/Standby secondary VM and stop the keepalived service.
$ sudo escadm stop
Make sure the secondary ESC VM is in STOP state.
$ sudo escadm status --v
If ESC status=0 esc ha is stopped.
In secondary VM, execute the rpm command for upgrade:
$ sudo rpm -Uvh /home/admin/cisco-esc-2.2.9-50.rpm
Log into the primary instance, set ESC primary node into maintenance mode.
$ sudo escadm op_mode set --mode=maintenance
154
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Upgrading Cisco Elastic Services Controller
In-Service upgrade in VMware using ESC qcow2 Image
Step 6
Step 7
Step 8
Step 9
Step 10
Make sure there are no in-flight transactions and no new transactions during the upgrade. From ESC 2.3, you may use following commands to check in-flight transactions.
$ sudo escadm ip_trans
For build older than ESC 2.3, you may need to check escmanager log and make sure no new transactions are recorded in this log file. The log file can be located at (/var/log/esc/escmanager.log).
Power off ESC primary node. In VMware vSphare Client, select Home > Inventory > VMs and Templates , right click the primary instance name from the left panel, and select Power > Power Off .
Log in to the upgraded ESC instance (previous secondary one), and start the keepalived service. The upgraded
VM will take over primary role and provide ESC service.
$ sudo escamd restart
Check the ESC version on the new primary instance to verify the upgraded version is correct. Once it is in the Primary state, make sure ESC service is running properly in the new Primary VM.
$ sudo escadm status
Expected output:
0 ESC status=0 ESC Master Healthy
$ esc_version
$ health.sh
Expected output:
ESC HEALTH PASSED
Power on the old primary instance. In VMware vSphare Client, select Home > Inventory > VMs and
Templates , right click the primary instance name from the left panel, then select Power > Power On .
Log into the VM which is still with old ESC version and repeat step 2, 3, 4, and 7 in the VM.
In-Service upgrade in VMware using ESC qcow2 Image
Procedure
Step 1
Step 2
Backup ESC database and log files.
a) Perform ESC database backup from primary node. For more information on backing up the database, see
Backup the Database from the ESC HA Active/Standby Instances .
b) Collect and backup all logs from both primary and secondary VMs. To backup the log, use the following command:
# sudo escadm log collect
Note A timestamped log file will be generated in: /var/tmp/esc_log-<timestamp>.tar.bz2
c) Copy the database backup file and logs files (generated in /tmp/esc_log-.tar.bz2)* out of ESC VMs.
Redeploy secondary ESC instance. Register new ESC image on the secondary instance, and wait for the data to be synchronized.
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
155
Upgrading Cisco Elastic Services Controller
In-Service upgrade in VMware using ESC qcow2 Image
Step 3
Step 4 a) Delete the secondary instance. To delete the secondary ESC instance, you need to first "Power Off" the instance through vSphere Client and then use the Delete from Disk option. In VMware vSphare Client, select Home > Inventory > VMs and Templates , right click the instance name from the left panel, then select Power > Power Off . Now to delete the secondary instance, select Home > Inventory > VMs and
Templates , right click the instance name from the left panel, then select Delete from Disk .
b) Redeploy the secondary ESC VM instance based on newer image version. Re-install new the secondary instance by using the new ESC package (bootvm.py and new registered image). Once the new ESC instance with upgraded version is up, it will be in secondary state.
c) Log into the new instance and run the following command to check the synchronization state of the new
ESC node.
$ drbdadm status
Wait until the output of drbdadm status show both nodes are "UpToDate" like the output below. It means the new ESC instance has completed the data synchronization from the primary instance.
esc/0 Connected Secondary/Primary UpToDate/UpToDate
Stop keepalived service on Secondary instance, Power off primary instance, and then start Secondary keepalived service.
a) Log into the primary instance, set ESC primary node into maintenance mode.
$ sudo escadm op_mode set --mode=maintenance
Make sure there is no in-flight transaction ongoing before moving to the next step. To verify there are no in-flight transactions, use the following command:
For ESC 2.3:
$ sudo escadm ip_trans
For versions older than ESC 2.3, check escmanager log at (/var/log/esc/escmanager.log) and make sure there are no new transaction in escmanager log.
b) Log in to the upgraded secondary instance and shut down the keepalived service.
$ sudo escadm stop c) Power off the primary instance and make sure the primary instance has been powered off. In VMware vSphare Client, select Home > Inventory > VMs and Templates , right click the instance name from the left panel, then select Power > Power Off .
d) Log into the previously upgraded secondary instance which is in stopped state and start the keepalived service. The secondary ESC instance will take the primary role (switchover will be triggered) and start providing services with new version.
$ sudo escadm start
Check the ESC version on the new primary instance to verify the version is upgraded correctly.
$ sudo escadm status --v(check ha status)
Expected output:
0 ESC status=0 ESC Master Healthy
$ esc_version (check esc version) version : 3.x.x
release : xxx
156
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Upgrading Cisco Elastic Services Controller
In-Service upgrade in VMware using ESC qcow2 Image
Step 5
Step 6
$ health.sh (check esc health)
Expected output:
ESC HEALTH PASSED
Re-deploy the old primary instance with the new ESC image.
Delete the old primary instance and redeploy it by using the new ESC package (bootvm.py and new registered image). All other installation parameters should be the same as the old ESC VM deployment. For example, hostname, ip address, gateway_ip, ha_node_list, kad_vip, kad_vif have to be the same values. To delete, in the VMware vSphare Client, access, Home > Inventory > VMs and Templates , right click the instance name from the left panel, then select Delete from Disk .
a) Log in to the new deployed instance and check ha status. The new instance should be in secondary state:
$ sudo escadm status b) Run the following command to check the synchronization state of the new ESC secondary node:
$ drbdadm status
Wait until the output of drbdadm status shown as UpToDate.
c) For the new ESC secondary node, make sure the health check is passed and the ESC version are upgraded correctly.
$ sudo escadm status (check ha status)
Expected output:
0 ESC status=0 ESC Master Healthy
$ esc_version (check esc version)version : 3.x.x
release : xxx
$ health.sh
Expected output:
ESC HEALTH PASSED
Go back in to the first upgraded primary instance and check the health and keepalived state.
$ drbdadm status
Expected output:
1:esc/0 Connected Primary/Secondary UpToDate/UpToDate /opt/cisco/esc/esc_database ext4
2.9G 52M 2.7G 2%
$ sudo escadm status (check ha status)
Expected output:
0 ESC status=0 ESC Master Healthy
$ esc_version (check esc version) Expected output: version : 3.x.x
release : xxx
$ health.sh (check esc health)
Expected output:
ESC HEALTH PASSED
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
157
Upgrading Cisco Elastic Services Controller
In-Service Upgrade of the ESC HA Active/Standby Nodes in CSP
Note Quick rollback: In case of an upgrade failure, shutdown the upgraded instance and start the old primary instance to have a quick rollback.
Rollback Procedure for In-service Upgrade a.
Copy the database and log backup files to a location out of ESC VMs.
b.
Delete any remaining ESC instance and redeploy ESC HA Active/Standby VMs using qcow2 image with old version.
c.
Restore the database. Follow the procedures in the section, Upgrading ESC HA Active/Standby
Instance with Backup and Restore for HA Active/Standby database restore.
d.
After database restore, you should have ESC service back with the old version.
In-Service Upgrade of the ESC HA Active/Standby Nodes in CSP
Follow the steps to do the In Service upgrade of the ESC HA Active/Standby Nodes in CSP:
Before you begin
Verify that the ESC HA nodes running properly before the upgrade, by using the following command:
# escadm status
One node must be in Master state and the other node in Backup state. Verify the Master node by using the following command:
# health.sh
On a successful health check you receive the following:
ESC HEALTH PASSED
Procedure
Step 1
Step 2
Shutdown the Standby instance
Upgrade the Backup ESC VM, before power off the VM. To upgrade, follow the steps: a.
Collect the logs from the Backup ESC VM, and copy it to another machine, by using the following commands:
# collect_esc_log.sh
# scp /tmp/LOG_PACKAGE_NAME <username>@<backup_vm_ip>:<filepath> b.
Using CSP, power off the standby ESC.
Deploy the standby node again using the new ESC package
After power off the standby ESC VM, install new ESC VM for upgrading by using the new ESC package.
Except using different ESC packages from the former ESC VM, all other parameters for ESC installation must be the same as the previous ESC VM deployment.
Verify that the ESC node is in Backup state. Use the following command to check the synchronization state of the new ESC standby node:
158
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Upgrading Cisco Elastic Services Controller
In-Service Upgrade of the ESC HA Active/Standby Nodes in CSP
Step 3
Step 4
# drbdadm status
Wait until you get the following output, which shows that the new ESC VM has completed the data synchronization from the Master node and it is up to date.
[admin@esc-xyx-upgradetestha1-4-5-0-105 ~]$ drbdadm status esc role:Secondary disk:UpToDate
172.20.117.55:7789 role:Primary peer-disk:UpToDate
If a dynamic mapping file (dynamic_mapping.xm) is used by ESC service, it must be restored into the ESC
VM. Copy the Backup file to
/opt/cisco/esc-dynamic-mapping/ path.
Stop the Master node and trigger a switchover.
Power off the Master instance through CSP. Follows that, an HA switchover automatically triggers and the
Standby instance takes over the ESC service with the new ESC version. After new ESC instance becomes the
Master, verify that the new ESC Master node passes the health check.
Deploy new ESC node to replace old Master
Install the new ESC VM for upgrading the previous Master instance by using the new ESC package. Before powering off the ESC VM with the old version, collect the logs from the ESC VM, and copy it to another machine by using the following commands:
# collect_esc_log.sh
# scp /tmp/LOG_PACKAGE_NAME <username>@<backup_vm_ip>:<filepath>
Note If a dynamic mapping file is used by ESC service, the dynamic mapping file should be backed up at the same time with ESC logs. The default path of the dynamic mapping file is
/opt/cisco/esc/esc-dynamic-mapping/dynamic_mappings.xml.
Power off the ESC VM with old version through CSP.
Verify that the ESC node is in Backup state. Use the following command to check the synchronization state of the new ESC standby node:
# drbdadm status
Wait until you get the following output, which shows that the new ESC VM has completed the data synchronization from the Master node and it is up to date.
[admin@esc-xyz-upgradetestha1-4-5-0-105 ~]$ drbdadm status esc role:Secondary disk:UpToDate
172.20.117.55:7789 role:Primary peer-disk:UpToDate
If a dynamic mapping file (dynamic_mapping.xm) is used by ESC service, it must be restored into the ESC
VM. Copy the Backup file to
/opt/cisco/esc-dynamic-mapping/ path.
Note If a dynamic mapping file is used by ESC service, the dynamic mapping file should be restored into the ESC VM. Copy the backup dynamic mapping file to the default path of the dynamic mapping file
/opt/cisco/esc/esc-dynamic-mapping/dynamic_mappings.xml.
Once you sucessfully complete the in-service upgrade, you can remove the old ESC instance.
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
159
Upgrading Cisco Elastic Services Controller
In-Service Upgrade of the ESC HA Active/Standby Nodes in CSP
Note After upgrading to the new ESC version, ESC service continues life cycle management of all VNFs deployed by the old version. If you want to apply any new feature of the new ESC version to existing
VNFs, undeploy those VNFs and do a new deployment.
160
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
P A R T
IX
Troubleshooting Cisco Elastic Services
Controller Installation
•
Troubleshooting ESC Issues, on page 163
C H A P T E R
21
Troubleshooting ESC Issues
This chapter contains the following sections:
•
Viewing ESC Log Messages, on page 163
•
General Installation Errors, on page 168
•
ESC Failover Scenarios, on page 171
Viewing ESC Log Messages
Log messages are created for ESC events throughout the VNF lifecycle. These can be external messages, messages from ESC to other external systems, error messages, warnings, events, failures and so on. The log file can be found at /var/log/esc/escmanager_tagged.log
.
The log message format is as follows: date=<time-date>] [loglevel=<loglevel>] [tid=<transactionid>] [cl=<classifications>]
[tags=<tags>] [msg=<message>
Sample log is as follows: date=15:43:58,46022-Nov-2016]
[loglevel=ERROR ] [tid=0793b5c9-8255-47f3-81e6-fbb59f6571f7] [cl=OS ]
[tags=wf:create_vm,eventType:VM_DEPLOY_EVENT,tenant:CSCvd94541,depName:test-dep,vmGrpName:test-VNF, vmName:test-dep_test_0_dc3f406c-05ca-43b3-af21-0841e3b029a0]
[tags=wf:create_vm,eventType:VM_DEPLOY_EVENT,tenant:test,depName:test-dep,vmGrpName:test-VNF, vmName:test-dep_test_0_dc3f406c-05ca-43b3-af21-0841e3b029a0] [msg=sleepingfor5seconds to allow vm to become ACTIVE instance id:
162344f7-78f9-4e45-9f23-34cf87377fa7 name:test-dep_test_0_dc3f406c-05ca-43b3-af21-0841e3b029a0
When a request is received, a RequestDetails object is created which autogenerates a unique transaction id.
This value is carried forward across all threads. Classifications and tags are optional. These are prefixes added to the log messages to enhance readability, and help in debugging. With classifications and tags, the log messages can be easily parsed and filtered by the log analysis tools.
The following classifications are supported:
NBI
SBI
"com.cisco.esc.rest""com.cisco.esc.filter"(North Bound Interface -
Clientinterface)
"com.cisco.esc.rest"- source is a callback handler or"EventsResource"(South
Bound Interface - i.e. between ESC and the VIM)
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
163
Troubleshooting Cisco Elastic Services Controller Installation
Viewing ESC Log Messages
SM
MONITORING
DYNAMIC_MAPPING
"com.cisco.esc.statemachines". stands for StateMachine. This classification indicates logs in the StateMachine category.
"com.cisco.esc.monitoring""com.cisco.esc.paadaptor"(MONA related logs)
"com.cisco.esc.dynamicmapping""com.cisco.esc.db.dynamicmapping"(MONA related logs)
CONFD "com.cisco.esc.confd"
CONFD_NOTIFICATION "com.cisco.esc.confd.notif""com.cisco.esc.confd.ConfdNBIAdapter"
OS
LIBVIRT
"com.cisco.esc.vim.openstack"
"com.cisco.esc.vim.vagrant
VIM
REST_EVENT
WD
DM
"com.cisco.esc.vim"
"ESCManager_Event""com.cisco.esc.util.RestUtils". indicates REST notifications in logs.
"com.cisco.esc.watchdog"
"com.cisco.esc.datamodel""com.cisco.esc.jaxb.parameters"(Datamodel and resource objects)
DB
GW
LC
SEC
MOCONFIG
POLICY
TP
ESC
"com.cisco.esc.db"(Database related logs)
"com.cisco.esc.gateway"
"com.cisco.esc.ESCManager"(Start up related logs)
"com.cisco.esc.jaas"
"com.cisco.esc.moconfig"(MOCONFIG object related logs --this is internal for
ESC developers)
"com.cisco.esc.policy"(Service/VM Policy related logs)
"com.cisco.esc.threadpool"
"com.cisco.esc" Any other packages not mentioned above
The following tags are supported:
• Workflow [wf:] —Generated using action and resource from RequestDetails object. Example
"wf:create_network"
• Event type [eventType:] —Event that triggered the current action. Example:
"eventType:VM_DEPLOY_EVENT"
• Resource based —These values are generated based on the type of parameter used by the event. The hierarchy, that is, the tenant, the vm group and so on is added to the log.
Tenant [tenant:<tenant name>]
164
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Troubleshooting Cisco Elastic Services Controller Installation
Viewing ESC Log Messages
Network
Subnet
User
[tenant:<tenant id>, network:<network name>]
Note The tenant appears only if applicable.
[tenant:<tenant name or id>, network:<network name or id>, subnet:<subnet name>]
Note The tenant appears only if applicable.
[tenant:<tenant name>, user:<user name or id>]
Note The tenant appears only if applicable.
Image
Flavor
[image:<image name>]
[flavor:<flavor name>]
Deployment [tenant:<tenant name or id>, depName:<deployment name>]
DeploymentDetails [tenant:<tenant name or id>, depName:<deployment name>, vmGroup:<vm group name>, vmName:<vm name>]
Switch
Volume
Service
[tenant:<tenant name or id>, switch:<switch name>]
[volume:<volume name>]
[svcName:<Service Registration name>]
Further, ESC logs can also be forwarded to an rsyslog server for further analysis and log management.
Filtering Logs Using Confd APIs
You can query and retrieve logs (for example, deployment logs, or error logs ) in ESC using log filters introduced in the confd APIs. New filters for Tenant, Deployment Name, and VM Name are introduced. This enables you to query the ESC logs further for most recent error logs using the log filters in Confd APIs. You can also retrieve ESC logs related to the communication between ESC and the OS ( by setting the classification tag to "OS").
The log format to retrieve confd API logs: date=<time-date>] [loglevel=<loglevel>] [tid=<transactionid>] [cl=<classifications>]
[tags=<tags>] [msg=<message>
The sample log is as follows: date=15:43:58,46022-Nov-2016] [loglevel=ERROR ] [tid=0793b5c9-8255-47f3-81e6-fbb59f6571f7]
[cl=OS ]
[tags=wf:create_vm,eventType:VM_DEPLOY_EVENT,tenant:test,depName:test-dep,vmGrpName:test-VNF, vmName:test-dep_test_0_dc3f406c-05ca-43b3-af21-0841e3b029a0]
[msg=sleepingfor5seconds to allow vm to become ACTIVE instance id:
162344f7-78f9-4e45-9f23-34cf87377fa7 name:test-dep_test_0_dc3f406c-05ca-43b3-af21-0841e3b029a0
The parameters for log level, classification and tags are dependent on each other to retrieve the logs. You can successfully retrieve the logs with the following combination.
• log_level=ERROR, classifications=OS, tags=(depName:test-dep)
• log_level=ERROR, classifications=OS, tags=(tenant: test)
The log filter returns a value when all the following conditions are met:
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
165
Troubleshooting Cisco Elastic Services Controller Installation
Viewing ESC Log Messages
• Log level
• Classifications (if provided)
• Tags (if provided)
Note If there are more than one classification listed, it has to match at least one of the classifications. The same applies to the tags as well.
For example, the following log filter criteria does not return the log sample mentioned earlier: log_level=ERROR, classifications=VIM, tags=(depName:test-dep)
It does not return any value though the log level and tags match, the classification VIM does not match.
The data model is as follows: rpc filterLog { description "Query and filter escmanager logs using given parameters"; tailf:actionpoint escrpc; input { leaf log_level { mandatory false; description "One of DEBUG / INFO / WARNING / ERROR / TRACE / FATAL. Results will include all logs at and above the level specified"; type types:log_level_types; default ERROR;
} leaf log_count { mandatory false; description "Number of logs to return"; type uint32; default 10;
} container classifications { leaf-list classification { description "Classification values to be used for the log filtering.
For example:
'OS', 'SM'.
Logs containing any of the provided classification values will be returned."; type types:log_classification_types;
}
} container tags { list tag { key "name"; leaf name { mandatory true; description "Tag name to be used for the log filtering.
For example: 'tenant',
'depName'.
Logs containing any of the provided tag name plus the tag values will be returned."; type types:log_tag_types;
} leaf value { mandatory true; description "Tag value pairs to be used for the log filtering.
For example:
'adminTenant', 'CSRDeployment'"; type string;
}
166
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Troubleshooting Cisco Elastic Services Controller Installation
Viewing ESC Log Messages
}
}
}
} output { container filterLogResults { leaf log_level { description "Log level used to filter for the logs."; type types:log_level_types;
} list logs { container classifications { leaf-list classification { description "Classifications used to filter for the logs."; type types:log_classification_types;
}
} container tags { list tag { key "name"; leaf name { mandatory true; description "Tag name used to filter for the logs."; type types:log_tag_types;
} leaf value { mandatory true; description "Tag value used to filter for the logs."; type string;
}
}
} leaf log_date_time { description "Timestamp of the log."; type string;
} leaf log_message { description "The log message."; type string;
}
}
}
}
You can query for the confd API logs through the netconf console or esc_nc_cli
• Through the netconf-console, run the following query:
/opt/cisco/esc/confd/bin/netconf-console --port=830 --host=127.0.0.1 --user=admin
--privKeyFile=/home/admin/.ssh/confd_id_dsa --privKeyType=dsa --rpc=log.xml
• Using the esc_nc_cli, run the following query:
./esc_nc_cli filter-log log.xml
The sample log.xml is as follows:
<filterLog xmlns="https://www.cisco.com/esc/esc">
<log_level>INFO</log_level>
<log_count>1</log_count>
<classifications>
<classification>OS</classification>
<classification>SM</classification>
</classifications>
<tags>
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
167
Troubleshooting Cisco Elastic Services Controller Installation
General Installation Errors
<tag>
<name>depName</name>
<value>CSR_ap1</value>
</tag>
<tag>
<name>tenant</name>
<value>admin</value>
</tag>
</tags>
</filterLog>
The response is as follows:
<rpc-reply xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="1">
<filterLogResults xmlns="https://www.cisco.com/esc/esc">
<log_level>INFO</log_level>
<logs>
<classifications>
<classification>OS</classification>
<classification>SM</classification>
</classifications>
<tags>
<tag>
<name>depName</name>
<value>CSR_ap1</value>
</tag>
<tag>
<name>tenant</name>
<value>admin</value>
</tag>
</tags>
<log_date_time>13:06:07,575 31-Oct-2016</log_date_time>
<log_message> No pending work flow to start.</log_message>
</logs>
</filterLogResults>
</rpc-reply>
Note The logging API responses are in XML format. If the log messages contain any
XML characters, then the characters will be escaped so not to break the XML conformance.
General Installation Errors
This section cover some of the common installation problems and how to troubleshoot them.
168
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Troubleshooting Cisco Elastic Services Controller Installation
General Installation Errors
Problem/Error Possible Reason User Action
Issues encountered while verifying ESC services status at the installation time using sudo escadm status
Some services take time to start or have trouble starting.
1.
Identify the issues using one of the following method:
• Check the log /var/log/esc/escadm.log
$ cat /var/log/esc/escadm.log
• Add '-v' to escadm status to show the verbose output of the ESC services and see the services are up and running.
2.
Confirm the status of the identified services that has issues and manually start these services.
$ sudo escadm <<service>> status//
If the status is stopped or dead, manually start the services using the next command.
$ sudo escadm <<service>> start --v
Authentication related error during the ESC installation
OpenStack credential-related arguments are missing.
ESC HA-related (Active/Standby) Issues
Network Problem
Source your OpenStack RC file and verify the OpenStack clients work properly
Check for the following:
• The static IP addresses for both ESC nodes used are based on the OpenStack configuration.
• The gateway for each network interface is accessible.
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
169
Troubleshooting Cisco Elastic Services Controller Installation
General Installation Errors
Problem/Error Possible Reason User Action
ESC master node stays in the
"switching-to-master" state
This could be because of the following issues:
Check for the:
• ESC HA
Active/Standby node can not reach its peer during the initial installation.
• Connectivity between ESC master node and standby node. For initial installation, ESC master service won't be up if it can't reach the standby node. You need to make sure you have both ESC nodes successfully deployed and they can reach each other.
• ESC logs at /var/log/esc/esc_haagent.log (ESC 2.X) or /var/log/esc/escadm.log (ESC 3.X) to identify the services that has issues.
• ESC service
(tomcat) can't start properly due to database problems
(etc. database migration, database file corruption).
• Log at /var/log/esc/escmanager.log for esc_service and postgresql issues.
• Confd can't start due to the CDB file corruption.
• Postgresql can't start or init due to the file system issues.
• The connection between ESC nodes is too slow.
MTU issues for ESC
HA Active/Standby
For the ESC VMs, reduce the MTU for network interfaces from 1500 to 1450. Follow the below steps to reduce the
MTU value:
1.
Identify the interface you would like to change. Access the interface from the location,
/etc/sysconfig/network-scripts/ifcfg-ethX, where X represent the interface number you want to change.
2.
Use a text editor like VIM to add or edit the MTU items for the interface, for instance, set MTU = 1450
3.
Restart the interface.
network service restart
170
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Troubleshooting Cisco Elastic Services Controller Installation
ESC Failover Scenarios
Problem/Error
Other Issues
Possible Reason User Action
There are several useful logs to check for ESC HA
Active/Standby troubleshooting.
• ESC manager log is located at
/var/log/esc/escmanager.log
• ESC HA Active/Standby log about esc service startup/stop is located at /var/log/esc/esc_haagent.log
for ESC 2.X and /var/log/esc/escadm.log for ESC 3.X
• For Keepalived configuration:
• Check the configuration file at
/etc/keepalived/keepalived.conf
• Keepalived log is located at /var/log/messages by grep keepalived or vrrp
• For DRBD configuration:
• DRBD configuration can be verified by checking the file at /etc/drbd.d/esc.res
• The DRBD log is located at /var/log/messages by grep drbd
ESC Services Startup Issues
Service status reports
ETSI service is dead at installation time using sudo escadm status --v
If the host has multiple
IP addresses the ETSI service is unable to determine which IP address to use when generating callback
URLs.
This cause can be verified by examining the log file
/var/log/esc/etsi-vnfm/etsi-vnfm.log
for an exception including:
Configure server.host
property in configuration file.
1.
Set the server.host property: sudo escadm etsi set –server_host <IP>
2.
Start the ETSI service: sudo escadm etsi start
ESC Failover Scenarios
• When customer deploys many VNFs
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
171
ESC Failover Scenarios
Troubleshooting Cisco Elastic Services Controller Installation
• When customer does a DB backup
• 1 or more of the VMs in the VNFs is recovered via redeploy – new VM ID
• ESC Fails – It is a permanent failure
• Customer Restores DB – When there is a mismatch between 1 or more VM IDs
• Same VNF fails, but when ESC tries to recover, it fails because it is not found on the VIM by VM ID and redeploy fails because the ports are in use.
• The workaround is to delete the VM out of band and recover – if the VM is deleted it is not found by
ESC and the port is available for redeployment.
172
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
A P P E N D I X
A
Cisco Elastic Services Controller Installer
Arguments
You need to specify the following bootvm.py
script arguments to boot ESC instances.
Arguments Description esc_hostname
--image
Specifies the host name of the ESC VM instance.
Specifies the image id used in the OpenStack glance to boot up the
ESC instance.
--boot_volume
--ignore-ssl-errors
--managers
Specify the volume name or id of the external bootable volume from where you want to launch ESC instance.
It sets the "ignoreSslErrors" to "yes". It helps in deploying to a development or test environment when you don't have a root trusted certificate installed.
It is a comma delimited list of locations where SNMP traps is delivered.
It must be supplied in the following format: udp:ipv4/port" or "udp:[ipv6]/port
--net
--ipaddr
Specifies the Network IDs or names in OpenStack that ESC connects to.
(Optional) Specifies the IP addresses that ESC will be assigned in the network.
Note The IP address must correspond to the net_id in the --net argument.
--gateway_ip
--os_auth_url
--os_username
(Optional) Specifies the default gateway IP address of ESC.
(Optional) Specifies the OpenStack keystone url used by os_auth_url for authentication.
(Optional) Specifies the OpenStack keystone username used by os_username for authentication.
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
173
Cisco Elastic Services Controller Installer Arguments
Cisco Elastic Services Controller Installer Arguments
Arguments
--os_password
--os_tenant_name
--bs_os_auth_url
--bs_os_username
--bs_os_password
--bs_os_tenant_name
--flavor
--security_rules_file
--etc_hosts_file
--avail_zone
--esc_params_file
--etsi_params_file
--db_volume_id
-- ha_node_list
--kad_vip
--kad_vif
Description
(Optional) Specifies the OpenStack keystone password used by os_password for authentication.
(Optional) Specifies the OpenStack tenant name used by os_tenant_name for ESC deployment.
(Optional) Specifies the OpenStack keystone url used by bs_os_auth_url for authentication.
(Optional) Specifies the OpenStack keystone username used by bs_os_username for authentication.
(Optional) Specifies the OpenStack keystone password used by bs_os_password for authentication.
(Optional) Specifies the OpenStack tenant name used by bs_os_tenant_name for ESC deployment.
(Optional) Specifies the OpenStack flavor id to boot the ESC VM.
(Optional) Specifies the file to define security rules (IP, Port security) for ESC VM.
(Optional) Specifies the file for adding more entries to the ESC vm's hosts file (/etc/hosts).
(Optional) Specifies the OpenStack zone used for ESC deployment.
(Optional) Specifies the default parameter file for ESC deployment.
(Optional) Specifies the ETSI properties file for ESC deployment.
(Optional) Specifies the cinder volume id to mount for database storage in ESC HA Active/Standby [ESC-HA Active/Standby].
(Optional) Specifies list of IP addresses for HA Active/Standby nodes in the Active/Standby cluster. For ESC nodes with multiple network interfaces, these IPs should be the addresses in the network used for data synchronization.
Note This argument is utilized for replication-based HA
Active/Standby solution only.
(Optional) Specifies the IP address for Keepalived VIP (virtual IP) plus the interface of Keepalived VIP [ESC-HA Active/Standby].
An example format for specifying the interface of VIP is --kad_vip
192.0.2.1:eth2 or --kad_vip [2001:cc0:2020::fc]:eth2
(Optional) Specifies the interface for Keepalived virtual IP and keepalived VRRP [ESC-HA Active/Standby]. You can also use this argument to only specify the interface for Keepalived VRRP, if the
VIP interface is already specified using the kad_vip argument.
174
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Cisco Elastic Services Controller Installer Arguments
Cisco Elastic Services Controller Installer Arguments
Arguments
--kad_vri
--route
--ntp_server
--rsyslog_server
--rsyslog_server_port
--rsyslog_server_protocol
--secure
--host_mapping_file
--version
--rng_virtio
Description
Specified the virtual router id of vrrp instance. Accepted values for kad_vri are 0 to 254. ESC VMs in the same HA Active/Standby should use the same kad_vri number. If kad_vip is not used for L3 HA
Active/Standby, the kad_vir has to be used, otherwise, you can skip kad_vri argument.
Specifies the routing configuration for ESC VM.
(Optional) Specifies the NTP server address.
(Optional) Specifies the IP address of rsyslog server that ESC sends the log to
(Optional) Specifies the port of rsyslog server that ESC sends the log to.
(Optional) Specifies the protocol to be used by the ESC to forward logs to the server.
(Optional) Enables secure configuration. You can specify the following values:
• A —Root is completely locked out. You cannot login as a root even from the console.
• B —SELinux runs in the enforcing mode.
• C —IPv4/IPv6 tables are started.
• D —SSH password authentication is disabled. You need the private key to ssh into ESC vm.
• E —host keys for confd will be re-created.
(Optional) Specifies the host mapping file for VNF deployment.
(Optional) Prints the version of bootvm.py and exits.
Enables installing and deploying the ESC VM on Libvirt/KVM with the RNG Virtio device. The default values are: device=/dev/random rate_period=1000 rate_bytes=1024
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
175
Cisco Elastic Services Controller Installer Arguments
Cisco Elastic Services Controller Installer Arguments
Arguments
--user_pass
Description
This along with --user_confd_pass are mandatory arguments from 3.0
onwards.
This argument adds a user to access the ESC VM. Use this argument to specify a user without administrative privileges, i.e, a non-admin/non-root user. Use the following format: user_name:password. The bootvm.py command requires at least one
--user_pass argument to create an admin account for linux (ssh/console access) . The following is the syntax for the mandatory user credential argument:
--user_pass admin:'PASSWORD-OR-HASH'[:OPTIONAL-PUBLIC-KEY-FILE][:OPTIONAL-ROLE]
This user can only do the following:
• Login to ESC through SSH.
• Access and drive the Netconf CLI, such as, esc_nc_cli, netconf-console, and so on.
• Read ESC -related logs from /var/logs/esc
• Access REST interface through localhost
This user cannot:
• Access the ESC DB and reconfigure ESC system.
• Access the system-level logs
• Configure the system level components, such as: Rsyslog,
Keepalived, DRDB, and so on.
• Access the encryption keys and values from REST interface or
ESC logs.
Following is an example of --user_pass for admin account and stronger clear text passwords. Use single quotes to avoid conflict with shell reserved characters:
-user_pass admin:'Strong4Security!'.
Another example to install ESC using a password hash for both admin accounts. Use single quotes to avoid conflict with shell reserved characters:
--user_pass admin:'$algorithm$salt$hash-of-salt-password'.
ESC 2.1 and later, accepts the public key for this attribute. For example, the following will generate 'admin321' as the password for user 'admin' and use /tmp/abc.pub
as the key file to inject the public key for it:
--user_pass admin:admin321:/tmp/abc.pub
176
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Cisco Elastic Services Controller Installer Arguments
Cisco Elastic Services Controller Installer Arguments
Arguments
--user_confd_pass
--esc_portal_startup
--log
--esc_monitor_check_ips
--enable-https-rest
--enable-http-rest
--disable-rest-auth
--enable-snmp-agent
--ha_mode
--enable-https-etsi
Description
Used to change confd users. The bootvm.py command requires at least one --user_confd_pass to create an admin account for ConfD
(netconf/cli access). The following is the syntax for the mandatory user credential argument:
--user_confd_pass admin:'PASSWORD-OR-HASH'[:OPTIONAL-PUBLIC-KEY-FILE]
Following is an example of --user_confd_ pass for admin account and stronger clear text passwords. Use single quotes to avoid conflict with shell reserved characters:
--user_confd_ pass:'Strong4Security!'.
Another example, to install ESC using a password hash for both admin accounts. Use single quotes to avoid conflict with shell reserved characters:
--user_confd_ pass:'$algorithm$salt$hash-of-salt-password'.
ESC 2.1 and later, accepts the public key for this attribute. For example, the following will generate 'admin321' as the password for user 'admin' and use /tmp/abc.pub
as the key file to inject the public key for it:
--user_confd_ pass:admin321:/tmp/abc.pub
(Optional) Starts the ESC portal.
(Optional) Specifies the log file. By default, logs to stdout.
(Optional) Specifies the IP addresses that must be monitored by esc_monitor (for HA Active/Standby failover).
(Optional) Enables a secure REST Interface for the created ESC VM.
(Optional) Enables an unsecured REST Interface for the created ESC
VM.
(Optional) Disables REST API authentication.
Note REST authentication should not be disabled in a production environment.
(Optional) Enables automatic start-up of the SNMP service. The default value is False.
Specifies the ESC HA Active/Standby mode for HA Active/Standby installation. Specify one of the following available options for HA
Active/Standby: no_ha : No HA, cinder : Shared Cinder Volume, drbd :
Built-in DRBD, drbd_on_cinder : DRBD over Cinder Volume
(Optional) Enables a secure ETSI REST Interface for the created ESC
VM.
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
177
Cisco Elastic Services Controller Installer Arguments
Cisco Elastic Services Controller Installer Arguments
Arguments
--enable-http-etsi
--encrypt_key
--proxy
--noproxy
--kad_unicast_src_ip
--kad_unicast_peer
--placement_hint
--format {json}
--user_rest_pass
--user_portal_pass
--user_etsi_pass
--etsi_oauth2_pass
Description
(Optional) Enables an unsecured ETSI REST Interface for the created
ESC VM. Enabling this interface is not recommended in a production environment.
Specifies the key for encryption.
Uses the proxy on a given port.
Lists the hosts which do not use proxy.
Specifies the source IP address of unicast. Should be the IP address of interface that ESC VM uses for unicast (L3) VRRP communication.
Example: --kad_unicast_src_ip 10.0.0.1
Specified the peer IP addresses of unicast. Should be the ip address of interface that ESC peer VM uses for unicast (L3) VRRP communication.
Example: --kad_unicast_peer 10.0.0.1
Use this argument to specify the placement of ESC HA Active/Standby virtual machines using the server group, samehost, differenthost filters.
Example:
• --placement_hint different_host=2b299428-e7a7-4528-8566-9a4970183c6a [ID should be the VM uuid]
• --placement_hint same_host=2b299428-e7a7-4528-8566-9a4970183c6a [ID should be the VM uuid]
• --pacement_hint group=4c7758ab-e9cb-4cf0-8f02-344ec666365b
[ID should be the server group uuid]
Use this argument to capture the success and failure message in the output.
Example: $ ./bootvm.py --image ESC-2_3_0_8 --net network --format json --test-0
{ "status" : "Success" , "vm_uuid" : "UUID" }
Adds a user to access the Rest API. Format is username: password.
This option can be repeated.
Add a portal user. Format username: password. This option can be repeated.
Adds a user to access the ETSI REST API. Format username:password.
Only a single user can be supplied.
Add an OAuth2 client for access to the ETSI REST API. Format clientId:clientSecret. This option can be repeated.
178
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Cisco Elastic Services Controller Installer Arguments
Cisco Elastic Services Controller Installer Arguments
Arguments
--no_vim_credentials
Description
Use this argument to deploy ESC without passing the VIM credential.
If this argument is used, following parameters will not be passed during the installation:
• --os_auth_url
• --os_username
• --os_password
• --os_tenant_name
After the deployment is complete, the user can set these VIM credential through ESC's VIM/VIM User APIs (REST/Netconf). For more information on configuring through REST APIs and Netconf, see
Configuring VIM credentials after installing ESC in the Post Installation
Tasks chapter.
Cisco Elastic Services Controller Installer File Reference
File Description security_rules_file The file contains the following:
• Security rules to create a security group for the tenant.
• Configurations to allow traffic for the tenant.
etc_hosts_file esc_params_file
The file contains one or more entries that you want to include in the /etc/hosts file.
The file contains information to configure various parameters of ESC. For details on parameters that can be configured in the esc_params_file are described in table below.
host_mapping_file The file contains information to map a network based on the hosts.
ESC Configuration Parameters
Using this file, you can configure various ESC parameters during the installation. The parameters that can be configured are shown in the table.
Below is an example configuration using this file: openstack.endpoint=adminURL affinity.filter=ServerGroupAffinity
Table 7: ESC Configuration Parameters esc_param.conf
Type default.vm_recovery_retries_max
Int
Default Value
3
Description
Number of recovery attempts allowed per VM.
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
179
Cisco Elastic Services Controller Installer Arguments
Cisco Elastic Services Controller Installer Arguments esc_param.conf
Type openstack.endpoint
String
Default Value publicURL
Description
The parameter to set up the keystone endpoint value of
ESC. Options: adminURL, publicURL
You can change the default value using CLI or REST services.
Using CLI:
$ sudo escadm escmanager config set --key openstack.endpoint
--value publicURL
{
"category": "OPENSTACK",
"type": "STRING",
"value": "publicURL",
"key": "ENDPOINT"
}
Using REST: log.level
affinity.filter
String
String
INFO
SameHostFilter
$ curl -X PUT http://172.16.0.1:8080/ESCManager/v0/config
/openstack/endpoint/publicURL
Level of logging. Options: INFO, Trace, DEBUG
A constant string used to build PolicyEngine and initializing VM policy table.
Options: SameHostFilter, ServerGroupAffinity anti_affinity.filter
String
DifferentHostFilter
A constant string used to build PolicyEngine and initializing VM policy table.
Options: DifferentHostFilter
Note ESC uses SameHostFilter and DifferentHostFilter for ESC policy engine by default but OpenStack may not configure those filters by default. You may need to add SameHostFilter and DifferentHostFilter to the following scheduler options in the /etc/nova/nova.conf
file of the nova service in your OpenStack.
scheduler_default_filters = RetryFilter, AvailabilityZoneFilter,RamFilter, ComputeFilter,
ComputeCapabilitiesFilter,
ImagePropertiesFilter, ServerGroupAntiAffinityFilter, ServerGroupAffinityFilter,
DifferentHostFilter, SameHostFilter
ServerGroupAntiAffinityFilter for Openstack
ESC adapts to use ServerGroupAntiAffinityFilter for Openstack.
REST
PUThttp://localhost:8080/ESCManager/v0/config/anti_affinity/filter/ServerGroupAntiAffinity
PUThttp://localhost:8080/ESCManager/v0/config/affinity/filter/ServerGroupAffinity
180
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Cisco Elastic Services Controller Installer Arguments
Cisco Elastic Services Controller Installer Arguments
CLI sudo escadm escmanager config set --key ANTI_AFFINITY.FILTER --value ServerGroupAntiAffinity sudo escadm escmanager config set --key AFFINITY.FILTER --value ServerGroupAffinity
Important Points
ServerGroupAntiAffinityFilter from Openstack doesn't support inter-dep anti-affinit, scaling, and mix use of
ServerGroup and default (SameHost/DifferentHost) filter. If you are using ServerGroupAntiAffinity filter,
Intra vm group placement is not allowed. You can only use <placement_group> for the VM based placement policy, one VM per vm_group. You can not add a single vm group in two different placement_groups.
ESC Services, Ports, and Security Group Overview
Table 8: External Services
1
2
3
4
5
6
7
Service sshd
ESC Web
UI/Portal
(HTTPS)
Visibility Optional?
External
(Orchestration)
No
Interface
0.0.0.0
External
(Orchestration)
Yes
(REST and/or
Netconf can be used instead)
0.0.0.0
ESC Netconf
API
External
(Orchestration)
Yes
(REST and/or Portal can be used instead)
0.0.0.0
ESC SNMP External
(Orchestration)
Yes (only configurable through custom user-data/esc-config.yaml)
0.0.0.0
ESC DRBD
(HA
Active/Standby
Replication)
External
(Orchestration)
No. Required for HA
Active/Standby setup.
0.0.0.0
ESC ConfD
CLI
External Yes
Configurable by modifying and reloading esc_production_confd.conf
0.0.0.0
ESC REST
API (HTTPS)
External
(Orchestration)
Yes
(Portal and/or
Netconf can be used instead)
0.0.0.0
Protocol
TCP
TCP
TCP
TCP
TCP
SSH
TCP
Port
22
443
(Previously
9001)
830
2001
7789
2024
8443
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
181
Cisco Elastic Services Controller Installer Arguments
Cisco Elastic Services Controller Installer Arguments
8
9
10
11
12
13
14
15
16
17
18
19
Service
ESC Health
API
Visibility Optional?
External
(Orchestration)
No
Interface
ESC
Keepalived
ETSI-VNFM
(HTTP)
External
(Orchestration)
No. Required for HA
Active/Standby setup.
0.0.0.0
External Yes (configurable through etsi-production.properties)
0.0.0.0
ETSI-VNFM
(HTTPS)
External
ETSI-VNFM
(Prometheus)
ETSI-VNFM
(Additional
HTTPS Port for artifacts download)
External
External
Yes (configurable through etsi-production.properties)
0.0.0.0
Yes (configurable through etsi-production.properties)
0.0.0.0
Yes (configurable through etsi-production.properties)
0.0.0.0
0.0.0.0
Protocol
Multicast
VRRP
TCP
TCP
TCP
TCP
TCP
Port
N/A
8250
8251
8252
8253
For ESC 5.2
and later,
8060
For ESC 5.1
and earlier,
60000
Local only ETSI Health
API
Internal
External D-MONA
REST API
Consul
Service
External
Consul
Service
ConfD
PostgreSQL
External
External
No
No
No
No for A/A set
No for A/A set
0.0.0.0
0.0.0.0
0.0.0.0
TCP
Limited to
ESC node
IPs
TCP
Limited to
ESC node
IPs
TCP
8443
TCP
TCP and
UDP
8300, 8301,
8302
8301
8302
8600
4565
7878
182
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Cisco Elastic Services Controller Installer Arguments
Cisco Elastic Services Controller Installer Arguments
20
21
22
23
24
25
Service Visibility
ESCManager
RMI
Registry
External
ESCManager
RMI Service
External
Optional?
No for A/A set
Interface
Limited to
ESC node
IPs
Protocol
TCP
No for A/A set Limited to
ESC node
IPs
TCP
ESCManager
Prometheus
Metrics
External
MONA
Prometheus
External
VIMManager
Prometheus
Metrics
External
DHCP
Yes.
(application.properties)
Yes.
(application.properties)
Yes.
(application.properties)
External
DHCP is limited to resolving IP addresses for
ESC interfaces.
No
0.0.0.0
0.0.0.0
0.0.0.0
0.0.0.0
TCP
TCP
TCP
UDP
7
8
5
6
3
4
1
2
9
10
11
Only needed for A/A ESC set. Otherwise, the port will not listen.
Only needed for A/A ESC set. Otherwise, the port will not listen.
Introduced only since ESC 5.0
ESC A/A set (3 VMs)
Introduced only since ESC 5.0
ESC A/A set (3 VMs)
Only needed for A/A ESC set. Otherwise, the port is not listened.
Only needed for A/A ESC set. Otherwise, the port is not listened.
ESC 5.2 and later
ESC 5.2 and later
ESC 5.2 and later
Port
8679
8680
8088
8092
8097
66,67
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
183
Cisco Elastic Services Controller Installer Arguments
Cisco Elastic Services Controller Installer Arguments
184
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
A P P E N D I X
B
List of Variables Used in CSP 2100 Sample Files
To create the user-data file, to configure the ESC you must have values ready for the following list of variables used in the sample files:
Table 9: List of Variables
Variable Name
VAR_TIMEZONE
VAR_SERVICE_NAME
VAR_NTP_SERVER
VAR_NETWORK1_NETMASK
VAR_NETWORK1_NAME
VAR_NETWORK1_IPADDR
VAR_NETWORK1_GATEWAY
VAR_NETWORK0_NETMASK
VAR_NETWORK0_NAME
VAR_NETWORK0_KADVRI
VAR_NETWORK0_KADVIP
VAR_NETWORK0_IPADDR2
Purpose
The timezone for the ESC clock to use
The name of the ESC service on the CSP
The IP address of an NTP server
The netmask for the eth1 interface (Dual Interface
ESC)
The name of the network on the CSP where ESC's eth1 interface exists (Dual Interface ESC)
The IP address for the eth1 interface (Dual Interface
ESC)
The gateway for the eth1 interface (Dual Interface
ESC)
The netmask for the eth0 interface
The name of the network on the CSP where ESC's eth0 interface exists
The VRRP ID used for HA. Must be unique in the subnet for the HA pair and the same value used on both ESCs
Range is from 1 to 254
The VIP for the HA pair that connects to the current
Master ESC
The IP address for the other ESC's eth0 interface
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
185
List of Variables Used in CSP 2100 Sample Files
Variable Name
VAR_NETWORK0_IPADDR
VAR_NETWORK0_GATEWAY
VAR_NAMESERVER_IP
VAR_LOCAL_HOSTNAME
CSP_IP_ADDRESS
List of Variables Used in CSP 2100 Sample Files
Purpose
The IP address for ESC (eth0 interface)
The gateway for the eth0 interface
The IP address of a DNS server
The hostname for ESC
IP address of the CSP 2100 to be used
186
Cisco Elastic Services Controller 5.3 Install and Upgrade Guide
Prenesi
Javna povezava posodobljena
Javna povezava do vašega klepeta je bila posodobljena. You may view and manage your chat links after registration here.
Javna povezava do dokumenta
Oglas