Installing Cisco Elastic Services Controller on OpenStack. Cisco Elastic Services Controller

P A R T

I

Installing Cisco Elastic Services Controller on

OpenStack

•

Prerequisites, page 9

•

Installing Cisco Elastic Services Controller on OpenStack, page 11

•

Configuring High Availability, page 21

C H A P T E R

3

Prerequisites

The following sections detail the prerequisites for installing Cisco Elastic Services Controller:

•

Virtual Resource Requirements, page 9

•

Software Requirements, page 9

•

Preparing for the Installation, page 10

Virtual Resource Requirements

The following table lists the virtual resource requirements for Cisco Elastic Services Controller:

Requirements (for 1000 VNFs/VM management)

Virtual CPUs

Description

4 VCPUs

Memory

Disk Space

8 GB RAM

30 GB

Software Requirements

The following table lists the software requirements on OpenStack:

Requirements

Supported Browsers

Description

Any of the following:

• Mozilla Firefox 40.x or later

• Google Chrome 44.x or later

Cisco Elastic Services Controller 4.1 Install and Upgrade Guide

9

Preparing for the Installation

Requirements

OpenStack Version

Description

Any of the following:

• Mitaka

• Newton

• Ocata

Preparing for the Installation

Before you perform the installation, ensure that you are prepared by reviewing this checklist:

Requirements Your Information/ Notes

For Pre-installation

Configuration

QCOW image location

QCOW image

VM per Instance

IP address

Subnet mask

Hostname

Domain name

Gateway IP address

Admin password

ESC Release Package

ESC.qcow2

bootvm.py

An image file for booting up the ESC instance

The installation script compatible with python 2.7.6 and Python 3.4

10

Cisco Elastic Services Controller 4.1 Install and Upgrade Guide

C H A P T E R

4

Installing Cisco Elastic Services Controller on

OpenStack

This chapter describes how to install Cisco Elastic Services Controller on OpenStack and includes the following sections:

•

Installation Scenarios, page 11

•

Main Components of Cisco Elastic Services Controller Setup, page 12

•

Installing Cisco Elastic Services Controller Using the QCOW Image, page 12

•

Managing Root Certificates in Cisco Elastic Services Controller, page 18

•

Using a Bootable Volume in ESC Installation, page 19

Installation Scenarios

The following sections briefly describe some of the common deployment scenarios that are addressed by ESC.

Cisco Elastic Services Controller can be installed in different modes as per the requirement. These different modes are configured during installation. The following sections briefly describe some of the common deployment scenarios that are addressed by ESC.

ESC Standalone

In the standalone scenario, a single active VM is deployed for ESC.

ESC with HA

ESC supports High Availability (HA) in the form of a Primary and Standby model. Two ESC instances are deployed in the network to prevent ESC failure and to provide ESC service with minimum interruption. If the primary ESC instance fails, the standby instance automatically takes over the ESC services. ESC HA resolves the following single point failures of ESC:

• Network failures

• Power failures

• Dead VM instance

Cisco Elastic Services Controller 4.1 Install and Upgrade Guide

11

Main Components of Cisco Elastic Services Controller Setup

• Scheduled downtime

• Hardware issues

• Internal application failures

For more information on deploying ESC HA, see 'Configuring High-Availability' in Installing ESC on

OpenStack and Installing ESC on VMware chapters.

Main Components of Cisco Elastic Services Controller Setup

The Cisco Elastic Service Controller (ESC) setup has the following components:

• Virtual Infrastructure Manager — Elastic Services Controller (ESC) and its VNFs are deployed in a

Virtual Infrastructure Manager (VIM). This might have one or more underlying physical nodes.

• ESC Virtual Machine — The ESC VM is a VM that contains all the services and processes used to register and deploy services. This includes the ESC Manager and all other services. ESC provides Netconf

API, REST API, and Portal as north bound interfaces to communicate with ESC. ESC VM contains CLI to interact with ESC VM. There are two CLI, one uses the REST API and the other uses Netconf API.

Installing Cisco Elastic Services Controller Using the QCOW Image

You can install Cisco Elastic Services Controller (ESC) on OpenStack by using a QCOW image. ESC will be deployed in the OpenStack as running VM instance to manage VNFs. Therefore, ESC need OpenStack environment parameters to be installed in OpenStack. The installation time varies from 10 to 20 minutes, depending on the host and the storage area network load. This procedure describes how to create ESC Virtual

Machine (VM) in OpenStack.

Before You Begin

• All system requirements are met as specified in

Prerequisites .

• You have the information identified in

Preparing for the Installation .

• Copy the ESC image file on the system where you want to install ESC.

• This system must be accessible by OpenStack.

Step 1

Step 2

Step 3

Log in to the system where you want to install ESC.

Check the compatibility of the bootvm.py and the ESC image:

./bootvm.py --version

For more information on the ESC installer arguments, see Appendix: A Cisco Elastic Services Controller Installer

Arguments .

In a text editor, create a file named PROJECT-openrc.sh file and add the following authentication information. The following example shows the information for a project called admin, where the OpenStack username is also admin, and the identity host is located at controller node.

12

Cisco Elastic Services Controller 4.1 Install and Upgrade Guide

Installing Cisco Elastic Services Controller Using the QCOW Image

Step 4

Step 5

Step 6

Note To set the required environment variables for the OpenStack command-line clients, you must create an environment file called an OpenStack rc file, or openrc.sh file. This project-specific environment file contains the credentials that all OpenStack services use. The ESC installation script requires these OpenStack environment parameters to perform authentication and installation on OpenStack. If all the OpenStack credentials are passed through its own arguments, the bootvm.py script doesn't require these parameters.

export OS_NO_CACHE=true export OS_TENANT_NAME=admin export OS_USERNAME=admin export OS_PASSWORD=admin_pass export OS_AUTH_URL= http://controller node:35357/v2.0

The other OpenStack parameters required for installation are: --os_auth_url, --os_username, --os_password ,

-os_tenant_name, --bs_os_user_domain_name, --bs_os_project_domain_name, --bs_os_identity_api_version,

--bs_os_auth_url, --bs_os_username, --bs_os_password , -bs_os_tenant_name, --bs_os_user_domain_name,

--bs_os_project_domain_name, --bs_os_identity_api_version.

For OpenStack V2 API, you need following items to be defined in your global environment variables: --os_password,

--os_auth_url, --os_username, --os_tenant_name.

For OpenStack V3 API, set --os_identity_api_version=3 . Other parameters required for OpenStack V3 API are:

--os_user_domain_name, --os_project_domain_name, --os_project_name, --os_password, --os_auth_url, --os_username,

--os_identity_api_version, --os_ca_cert, --requests_ ca_bundle.

Note The arguments, --os_tenant_name, --os_username, --os_password, --os_auth_url will also by default configure

Note the VIM connector. If you want to skip configuring the VIM connector, pass the parameter (--no_vim_credentials) with the bootvm.py. When no_vim_credentials parameter is provided, the bootvm.py arguments (os_tenant_name, os_username, os_password,os_auth_url) are ignored. For more information on configuring VIM connectors after installation, and managing VIM connectors, see Managing VIM Connectors in the Cisco Elastic Services

Controller User Guide .

--os_ca_cert and --requests_ca_bundle arguments are only required for https connection.

On any shell from which you want to run OpenStack commands, source the PROJECT-openrc.sh

file for the respective project. In this example, you source the admin-openrc.sh

file for the admin project .

$ source admin-openrc.sh

Check the environment variables.

$ env | grep OS_

Register ESC image file in the OpenStack image using the glance command:

$ glance image-create \

--name <image_name> \

--is-public=<true or false> or --visibility public or private\

--disk-format <disk_format> \

--container-format <container_format> \

--file <file> \

--progress

An example configuration is shown below:

$ glance image-create \

--name esc-1_0_01_11_2011-01-01 \

--is-public=<true or false> or --visibility public or private \

--disk-format qcow2 \

--container-format bare \

--file esc-1_0_01_11_2011-01-01.qcow2 \

--progress

Cisco Elastic Services Controller 4.1 Install and Upgrade Guide

13

Installing Cisco Elastic Services Controller Using the QCOW Image

The glance image-create command is used to create a new image. The command takes the following arguments:

Note The 'is-public' argument is applicable only for OpenStack Kilo release.

Arguments name

Description

Name of the image.

is-public disk-format

(Optional) Makes the image accessible to the public.

Disk format of the image. ESC uses a qcow2 disk format.

container-format file progress

Container format of image. ESC uses a bare container format.

Local file that contains disk image to be uploaded during creation.

(Optional) Shows upload progress bar.

Step 7

Step 8

To verify whether the image has been registered successfully: a) Using OpenStack Controller dashboard:

• Log into OpenStack using your credentials.

• Navigate to Admin > Image .

Verify if the image appears in the list.

b) Using nova CLI:

$ nova image-show <image_name>

The standard resource requirement of ESC is 4vCPU, 8G RAM, and 30GB disk space. ESC installation script takes the pre-defined "m1.large" flavor which has the definition of 4vCPU, 8G RAM and 80G disc space. To use 30GB disk space, create a flavor with the minimum disk space requirement.

$ nova flavor-create ESC_FLAVOR_NAME ESC_FLAVOR_ID 8192 30 4

To deploy ESC VM, do the following:

1 Ensure that the existing network have connectivity to OpenStack controller. To verify the network connectivity using the nova CLI use:

$ nova net-list

2 Record the ID of the network that ESC connects to boot the ESC VM by with image and flavor created earlier. The bootvm.py command requires at least one --user_pass argument to create an admin account for linux (ssh/console access) and at least one --user_confd_pass to create an admin account for ConfD (netconf/cli access). The following are the syntax for these mandatory user credential arguments:

--user_pass admin:'PASSWORD-OR-HASH'[:OPTIONAL-PUBLIC-KEY-FILE][:OPTIONAL-ROLE]

--user_confd_pass admin:'PASSWORD-OR-HASH'[:OPTIONAL-PUBLIC-KEY-FILE]

To generate a password hash from a password, use the following command: mkpasswd --method=SHA-512 --salt XyZ123 <<< <Password>

14

Cisco Elastic Services Controller 4.1 Install and Upgrade Guide

Installing Cisco Elastic Services Controller Using the QCOW Image

The following is an example to install ESC with an authorized public key. In the following example, single quotes are used to avoid conflict with shell reserved characters:

--user_pass admin:'$algorithm$salt$hash-of-salt-password':$HOME/.ssh/esc_rsa.pub

--user_confd_pass admin:'$algorithm$salt$hash-of-salt-password':$HOME/.ssh/esc_rsa.pub

The public key are generated as part of a key pair with, such as: ssh-keygen -t rsa -b 1024 -C "esc" -N "" -f ~/.ssh/esc_rsa

Your public key and identification key are saved in /home/username /.ssh/esc_rsa and esc_rsa.pub files. For more examples on the user credential arguments, see Appendix A: Cisco Elastic Services Controller Installer Arguments.

3 To check the details of ESC VM and get the information including the IP address(es) of the ESC VM, use the following command:

$ nova show <esc_vm_name>

Additional Installation Options

• Deploying ESC in an OpenStack IPv6 environment: Before deploying ESC instance in IPv6, make sure to source openrc that supports ipv6 addresses. To deploy ESC in IPv6 environment, use the following bootvm arguments:

./bootvm.py

<esc_vm_name-ipv6> --poll --user_rest_pass <username> : <password> --image

<image_name>

--net <ipv6_network> --ipaddr <ipv6_ip_address> --enable-http-rest --user_pass

<username> : <password>

--user_confd_pass <username> : <password> --etc_hosts_file <hosts-file-name> --route

<default routing configuration>

• Deploying ESC in DHCP mode: If you use the bootvm.py argument without --ipaddr, then the ESC instance will be deployed in a DHCP mode. To deploy ESC in a DHCP network, use the following configuration:

./bootvm.py

<esc_vm_name> --image <image_name> --net <IPv6 network> <IPv4 network>

--flavor <flavor_name>

--user_pass <username> : <password>

--user_confd_pass <username> : <password>

Note By default, ESC only support DHCP in IPv4 networks. If IPv6 is used, you need to log in the ESC VM and run "dhclient -6 ethX" (ethX is the V6 interface name) manually to enable V6 DHCP.

• Using a Bootable Volume in ESC Installation: You can attach a volume to an ESC instance and launch an instance from inside the volume. For more information, see the section

Using a Bootable Volume in

ESC Installation .

• Assigning Floating IP to the ESC: If you want to associate a floating IP with the ESC instance , do the following:

1 Check for an available floating IP address and assign it to the ESC VM:

$ nova floating-ip-list

$ nova floating-ip-associate esc_vm_name <ip_address>

Cisco Elastic Services Controller 4.1 Install and Upgrade Guide

15

Installing Cisco Elastic Services Controller Using the QCOW Image

2 Or create a new floating IP address and assign it to the ESC VM:

$ nova floating-ip-create <FLOATING_NETWORK - ID>

$ nova floating-ip-associate esc_vm_name <ip_address> or neutron floatingip-create FLOATING_NETWORK neutron floatingip-associate floating-ip-ID port-ID

• Deploying ESC with static IPs: To use ESC in a specific network with static IPs, for example,

192.168.0.112 at esc-net1, specify --ipaddr and --gateway_ip to the bootvm command line , as shown below:

Note Before assigning static IP, make sure the static IP is available and is not being used on other machine.

./bootvm.py

<esc_vm_name> --image <image_id> --net <network>

--ipaddr <ip_address> --gateway_ip <default_gateway_ip_address> --user_pass

<username> : <password>

--user_confd_pass <username> : <password>

• Deploying ESC with multiple network interfaces: To use multiple networks for ESC, for example,

192.168.0.112 at esc-net1 and 10.20.0.112 at esc-net2, specify both the IP addresses and the network names of the interfaces in the --net and --ipaddr arguments in the following command line. In addition, also choose the default gateway for ESC from the gateways of these networks. Specify the default gateway for ESC through the --gateway_ip argument.

./bootvm.py

<esc_vm_name> --image <image_id> --net <network1>

<network2> --ipaddr <ip_address1> <ip_address2> --gateway_ip

<default_gateway_ip_address>

--user_pass <username> : <password>

--user_confd_pass <username> : <password>

Note If --flavor is not specified, bootvm.py will use the default flavor "m1.large" in

OpenStack.

• Deploy ESC with log forwarding options: To forward ESC logs to an rsyslog server, specify the IP address of the rsyslog server while creating an ESC VM. Optionally, you can also specify the port and protocol to use.

For example, if the IP address of the rsyslog server is 10.85.77.94, the port on the server to forward logs is 514, and the protocol used is UDP, the ESC installation could be

./bootvm.py

<esc_vm_name> --image <image_id> --net esc-net1 --rsyslog_server 10.85.77.94

--rsyslog_server_port 514 --rsyslog_server_protocol udp --user_pass <username> : <password>

--user_confd_pass <username> : <password>

• Disabling the ESC GUI: To boot up ESC VM with the graphical user interface disabled, modify the

--esc_ui_startup argument value, as shown in the command line below:

./bootvm.py

<esc_vm_name> --image <image_id> --net <network> --user_pass

<username> : <password>

--user_confd_pass <username> : <password>

--esc_portal_startup=False

• Enabling REST interface for ESC: To support the REST interface, specify the --enable-https-rest argument. REST interface can be activated on both https or http:

./bootvm.py

<esc_vm_name> --image <image_id> --net <network> --user_pass

16

Cisco Elastic Services Controller 4.1 Install and Upgrade Guide

Installing Cisco Elastic Services Controller Using the QCOW Image

<username> : <password>

--user_confd_pass <username> : <password> --enable-https-rest

OR

./bootvm.py

<esc_vm_name> --image <image_id> --net <network> --user_pass

<username> : <password>

--user_confd_pass <username> : <password> --enable-http-rest

• Deploying ESC with global parameters: To set the global configurations through the esc_params_file during the installation, use the arguments as shown below. These global configurations can also be changed through REST API after the installation.

Note The default security group is applied to the tenant during tenant creation. By default, the ESC configuration parameter for the security group, openstack.DEFAULT_SECURITY_GROUP_TO_TENANT

is set to true. The configuration parameter must be set at the time of installation. You can query or update the parameter on ESC VM through the REST API. If the parameter is set to true, you can create and assign default security group during tenant creation. If the parameter is set to false, you cannot create or assign default security group during tenant creation.

For details on the parameters that can be configured through esc_params_file, see

Appendix A: Cisco Elastic Services Controller Installer Arguments.

./bootvm.py <esc_vm_name> --image <image_id> --net <network> --flavor <flavor_name>

--user_pass <username>:<password>:<public key file> --user_confd_pass

<username> : <password>

--esc_params_file <esc parameter configuration file>

• Deploying two instances of ESC to build an ESC HA pair : For more information on deploying ESC

HA, see Configuring High-Availability in Installing ESC on OpenStack and Installing ESC on VMware chapters.

• Adding a Dynamic Mapping File : In Cisco ESC Release 2.1 and earlier, mapping the actions and metrics defined in the datamodel to the valid actions and metrics available in the monitoring agent was enabled using the dynamic_mappings.xml

file. The file was stored in the ESC VM and was modified using a text editor. ESC 2.2 and later do not have an esc-dynamic-mapping directory and dynamic_mappings.xml

file. If you want to add an existing dynamic_mapping xml file to the ESC VM, do the following:

1 Backup this file to a location outside of ESC, such as, your home directory.

2 Create esc-dynamic-mapping directory on your ESC VM. Ensure that the read permissions are set.

3 Install on your ESC VM using the following bootvm argument:

--file root:root: /opt/cisco/esc/esc-dynamic-mapping/dynamic_mappings.xml

:< path-to-local-copy-of-dynamic-mapping.xml

>

The CRUD operations for mapping the actions and the metrics is available through REST API. To update an existing mapping, delete and add a new mapping through the REST API.

• Changing the confd password on an ESC VM : As an administrator, you can configure the confd password through bootvm.py, during the installation time:

./bootvm.py --user_pass <username> : <password> --user_confd_pass admin:'PASSWORD-OR-HASH':OPTIONAL-PUBLIC-KEY

To reconfigure this password after the installation, execute the following commands:

$ /opt/cisco/esc/confd/bin/confd_cli -u admin

$ configure

Cisco Elastic Services Controller 4.1 Install and Upgrade Guide

17

Managing Root Certificates in Cisco Elastic Services Controller

$ set aaa authentication users user admin password <your_password>

$ commit

$ exit

Note For the ease of future upgrades, make sure that you keep a copy of all the commands and arguments that are used while installing ESC using the bootvm.py file.

Managing Root Certificates in Cisco Elastic Services Controller

Cisco Elastic Services Controller (ESC) provides a mechanism to enable verification of SSL certificates.

Currently, this feature is supported only on OpenStack. Certificate validation is enabled by default during the initial ESC boot up. However, ESC also allows you to configure these SSL certificates. This section describes how to enable/disable certificate validation, add/remove, or list the certificates for Cisco Elastic Service

Controller on OpenStack. You can add a root certificate during the ESC bootup or even after ESC bootup is completed.

Enabling/Disabling the Root Certificate Validation

Cisco Elastic Services Controller by default enables certificate validation. You can also enable or disable by modifying the parameter, DISABLE_CERT_VALIDATION, available under the Openstack category in the esc_params.conf file, or through the REST interface, or using the escadm tool.

On ESC master node, use the command, escadm enable-certificate or escadm disable-certificate to enable and disable the certificate validation, respectively.

Adding a Root Certificate

You can add a root certificate during the ESC bootup or even after ESC bootup is completed. Before adding certificates, ensure the OpenStack environment file, OpenStack RC file has parameters to perform authentication and installation on OpenStack. The --os_auth_url must be specified while passing the parameters. --os_auth_url specifies the secure (https) or unsecured (http) keystone URL used by OpenStack for authentication.

• Add certificate for standalone (only) during the bootup time, i.e, during the ESC VM installation:

./bootvm.py test-vm --image <image_name> --net <network> --cert.file

/home/cisco/mitaka.crt

--user_pass <username>:<password> --user_confd_pass <username>:<password>

Note Currently, ESC does not support adding a certificate for HA during the installation as the keepalived service is not running when a certificate is added.

• Add certificate for standalone/HA after booting up the ESC instance. The escadm tool has an add-certificate option which has the following arguments: The --file argument refers to the CA certificate file. Using this argument you can import any file format supported by the java keytool: X.509 v1, v2, and v3 certificates, and PKCS#7. The --alias argument is unique and refers to the name this specific CA certificate is given.

1 Copy/Transfer CA Certificate file to ESC master VM .

18

Cisco Elastic Services Controller 4.1 Install and Upgrade Guide

Using a Bootable Volume in ESC Installation

2 Add certificate to ESC truststore. To do this, execute the following command: escadm add-certificate --alias [ca cert alias] --file [file path]

3 Verify the certificate is added.

escadm list-certificate

Removing a Root Certificate

The escadm tool has a 'delete-certificate' option which only takes --alias argument. The --alias argument refers to the name of the CA certificate to be deleted. Use this argument on the standalone/HA ESC VM:

Step 1

Step 2

On (master) ESC use escadm to delete certificate from ESC truststore.

escadm delete-certificate --alias [ca cert alias]

Verify the certificate is removed.

escadm list-certificate

Managing Root Certificates During the Upgrade

• Image Upgrade : If you are backing up the ESC DB for upgrade, then no other action is required, the

ESC truststore will be restored once the ESC DB is restored. If you are not backing up the ESC DB for upgrade, then each CA certificate needs to be added again to the ESC truststore.

• RPM Upgrade : This upgrade method keeps the ESC truststore as is, i.e. all ca certificates in the ESC truststore should remain there after upgrade.

Using a Bootable Volume in ESC Installation

A volume in OpenStack is a detachable block storage device that can be attached to an ESC instance. You can store and also run ESC instances from a volume.

Note • Only one ESC instance can be launched from one volume at a time.

• ESC installation with a combination of bootable volume and high availability on cinder is not supported.

Cisco Elastic Services Controller 4.1 Install and Upgrade Guide

19

Using a Bootable Volume in ESC Installation

To launch an ESC instance from a bootable volume, do the following:

Step 1

Step 2

Create a bootable volume in OpenStack based on an ESC image or from a bootable volume. The bootable volume must be at least of 30 GB disk size. For more information, see OpenStack documentation.

Deploy ESC VM using the bootvm.py

command and choose the

--boot_volume argument instead of the

--image argument, as shown below:

./bootvm.py <esc_vm_name> --boot_volume <volume_name_or_id> --net <network> --user_pass

<username>:<password>

--user_confd_pass <username>:<password> --flavor <flavor_name>

Note • Only one of these arguments,

--image or

--boot_volume must be passed to the bootvm.py command. The installation will fail, if both or none of the arguments are used.

• When launching an ESC instance from a bootable volume, volume disk size is considered over the flavor disk size.

• If an ESC instance is deleted, the volume attached to it will not be deleted, as the volume was created out-of-band.

20

Cisco Elastic Services Controller 4.1 Install and Upgrade Guide

C H A P T E R

5

Configuring High Availability

This chapter contains the following sections:

•

High Availability Overview, page 21

•

How High Availability Works, page 21

•

Deploying ESC High Availability, page 22

•

Configuring the Northbound Interface Access, page 25

•

Important Notes, page 27

•

Troubleshooting High Availability, page 28

High Availability Overview

ESC supports High Availability (HA) in the form of a Primary and Standby model. Two ESC instances are deployed in the network to prevent ESC failure and provide ESC service with minimum service interruption.

If the primary ESC instance fails, the standby instance automatically takes over the ESC services. ESC HA resolves the following single point failures:

• Network failures

• Power failures

• Dead VM instance

• Scheduled downtime

• Hardware issues

• Internal application failures

How High Availability Works

ESC HA network can be either set up as a single installation of a ESC HA pair or deployed as two standalone

ESC nodes that are converted into HA pair after re-configuring these nodes post deployment. A HA deployment consists of two ESC instances: a primary and a standby. Under normal circumstances, the primary ESC

Cisco Elastic Services Controller 4.1 Install and Upgrade Guide

21

Deploying ESC High Availability instance provides the service. The corresponding standby instance is passive. The standby instance is in constant communication with the primary instance and monitors the primary instances' status. If the primary

ESC instance fails, the standby instance automatically takes over the ESC services to provide ESC service with minimum interruption.

The standby also has a complete copy of the database of the primary, but it does not actively manage the network until the primary instance fails. The KeepAliveD service monitors both primary and standby instances activity status. When the primary instance fails, the standby takes over automatically. The standby instance takes over primary instance to manage the services while primary instance restoration is taking place.

When the failed instance is restored, if required you can manually initiate a switch-over and resume network management via the primary instance.

Both primary and standby ESC instances are connected to the northbound orchestration system through an

IPv4 or IPv6 network. For the northbound system, a unique virtual IP address is assigned to access the current primary ESC High Availability instance. The deployed VNFs are connected to both ESC primary and standby instances through another IPv6 network.

ESC HA nodes are managed by KeepAliveD and DRBD (Replication tool to keep the ESC database synchronized) sync network services. While the KeepAliveD service monitors both primary and standby instances status, the DRBD service monitors primary instance DB and sync the changes to the standby instance

DB. These two services can be co-located on same VIP network or in two separate networks. VM handshake between ESC instances occurs through the KeepAliveD over the IPv4 or IPv6 network.

Deploying ESC High Availability

To deploy Cisco Elastic Services Controller (ESC) High Availability (HA), ESC standalone instances can be installed on two separate nodes - Primary and Standby. For more information see,

How High Availability

Works, on page 21

. You can connect the Primary and Standby instances to either a Cinder volume or

Replication based volume (DRBD).

The following deployment mechanisms can be used to deploy ESC HA:

• Internal Storage — When ESC HA is configured with Internal storage, the Primary and the Standby instances have individual databases which are always synchronized. In this solution, ESC HA is designed with database replication and DRBD is used as the tool for disk-level replication. The database in the

Primary instance simultaneously propagates the data to the database in the Standby instance thus requiring no external storage. In the event of a Primary instance failing, the Standby instance get assigned the role of the Primary instance along with its own synchronized database.

ESC HA is deployed using Internal storage, the ESC instances reply on the virtual IP address (that is kad_vip argument), and the interface of vrrp instance (that is kad_vif argument) to select the Primary

ESC instance. To establish a reliable heartbeat network, it is recommended that the Primary and Standby

ESC instances are on different physical hosts. The reliability of the physical links between the ESC instances (such as, network interface bonding) can also be taken into consideration.

• Replicate External-Storages — In this type of architecture, ESC HA is configured with DRBD and both

Primary and Standby instance store their data in two external storages (OpenStack Cinder volumes).

Each ESC node is attached by a Cinder volume and ESC data files are stored in the cinder volume. The data in two ESC node are synchronized through the database replication mechanism provided by DRBD.

The table lists the differences between the HA options :

Internal Storage Based ESC HA Replicate External Storage Based ESC

HA

22

Cisco Elastic Services Controller 4.1 Install and Upgrade Guide

Deploying ESC High Availability

Data sharing method

Installation Method

VIM Support

Dependency

Advantages

Data replication between HA nodes

Post-installation Configuration

Bootvm Installation

OpenStack, VMware, KVM

VIM independent

Data replication between two external storages (cinder volume)

Bootvm Installation

OpenStack only

Rely on OpenStack cinder

• No dependency on specific

VIM components.

• Flexible to build of HA clusters from commodity hardware, without the requirement for shared-storage.

• Use database replication mechanism for data synchronization

• Two cinder volumes are used as external storage and are attached to

ESC node.

Limitations The data consistency may be affected in a double fault condition

(occurs when both ESC nodes have problems).

The data consistency may be affected in a double fault condition (occurs when both

ESC nodes have problems).

Deploying ESC in High Availability Mode on Internal Storage

When you boot ESC instances on Primary and Standby instances, you need to specify the following bootvm.py

command arguments to deploy ESC HA on an internal storage:

• kad_vip

Note When ESC HA is deployed, the kad_vip argument allows end users to access the Primary

ESC instance.

• kad_vif

• ha_node_list

These arguments enable the bootvm.py

command to automatically set up the internal storage on the OpenStack.

For more information on using the bootvm.py

command arguments, see Appendix A: Cisco Elastic Services

Controller Installer Arguments.

To deploy ESC HA instances, use the bootvm script on both the nodes with the following arguments:

ON HA NODE 1:

$ ./bootvm.py <ESC_HA_Node1>\

--user_pass <username>:<password>\

--user_confd_pass <username>:<password>\

Cisco Elastic Services Controller 4.1 Install and Upgrade Guide

23

Deploying ESC High Availability

--gateway_ip <default gateway IP address>\

--net <network name1>\

--ipaddr <static ip address>

--image <image_name>\

--avail_zone nova:<openstack zone>\

--ha_node_list=<ESC_HA_NODE1_IP> <ESC_HA_NODE2_IP>\

--db_volume_id <cinder volume id>\

--kad_vip <virtual IP address>\

--kad_vif <VRRP_Interface_Instance>\

--ha_mode drbd

ON HA NODE 2:

$ ./bootvm.py <ESC_HA_Node2>\

--user_pass <username>:<password>\

--user_confd_pass <username>:<password>\

--gateway_ip <default gateway IP address>\

--net <network name1>\

--ipaddr <static ip addresses>\

--image <image_name>\

--avail_zone nova:<openstack zone>\

--ha_node_list=<ESC_HA_NODE1_IP> <ESC_HA_NODE2_IP>\

--db_volume_id <cinder volume id>\

--kad_vip <virtual IP address>\

--kad_vif <VRRP_Interface_Instance>\

--ha_mode drbd

OR

You can also use escadm tool to re-configure ESC HA parameters on each of the standalone ESC VMs. Three parameters "--ha_node_list , --kad_vip, --kad_vif" are all required to configure ESC HA. For example:

$ sudo bash

$ escadm ha set --ha_node_list='<ESC_HA_NODE1_IP> <ESC_HA_NODE2_IP>' --kad_vip <virtual IP address> --kad_vif <VRRP_Interface_Instance>

$ sudo escadm restart

Deploying ESC in High Availability Mode on Replicate External Storage

Replicate external storage ESC HA requires two cinder volumes for database storage.

Before You Begin

• Networks and IP addresses that both ESC instances will connect to

• Keepalived interface and virtual IP for HA switchover

Step 1

Step 2

Step 3

Create two cinder volumes in OpenStack. The configured cinder volume size should be 3GB.

$ cinder create --display-name cindervolume_name_a[SIZE]

$ cinder create --display-name cindervolume_name_b[SIZE]

Check the status of the created cinder volume and find the uuids for deployment.

$ cinder list

Deploy ESC HA instances. Use the bootvm script on both the nodes with the following arguments:

ON HA NODE 1:

$ ./bootvm.py <ESC_HA_Node1>\

24

Cisco Elastic Services Controller 4.1 Install and Upgrade Guide

Configuring the Northbound Interface Access

Step 4

--user_pass <username>:<password>\

--user_confd_pass <username>:<password>\

--gateway_ip <default gateway IP address>\

--net <network name1>\

--ipaddr <static ip address>\

--image <image_name>\

--avail_zone nova:<openstack zone>\

--kad_vip <virtual IP address>\

--kad_vif <VRRP_Interface_Instance>\

--ha_node_list=<ESC_HA_NODE1_IP> <ESC_HA_NODE2_IP>\

--db_volume_id <cinder volume id>\

--ha_mode drbd_on_cinder

ON HA NODE 2:

$ ./bootvm.py <ESC_HA_Node2>\

--user_pass <username>:<password>\

--user_confd_pass <username>:<password>\

--gateway_ip <default gateway IP address>\

--net <network name1>\

--ipaddr <static ip address>\

--image <image_name>\

--avail_zone nova:<openstack zone>\

--kad_vip <virtual IP address>\

--kad_vif <VRRP_Interface_Instance>\

--ha_node_list=<ESC_HA_NODE1_IP> <ESC_HA_NODE2_IP>\

--db_volume_id <cinder volume id>\

--ha_mode drbd_on_cinder

After both VMs are rebooted; the keepalived state on one of ESC VM should be one of ESC VM should be in MASTER state and the other one should be in BACKUP state. You can check ESC HA state by using following command: $ escadm status --v .

Configuring the Northbound Interface Access

When you configure ESC HA, you can also specify a virtual Anycast IP address to the HA pair. The northbound interface as well as the service portal uses virtual Anycast IP address to access the ESC Primary HA instance.

When deploying ESC HA, use the following arguments with the ./bootvm.py

script.

• --ha_node_list

• --kad_vip

• --kad_vif

For more details on these arguments, see section Appendix A: Cisco Elastic Services Controller Installer

Arguments .

The following section explains how to configure ESC HA with multiple interfaces and to configure the virtual

Anycast IP address.

Cisco Elastic Services Controller 4.1 Install and Upgrade Guide

25

Configuring the Northbound Interface Access

Configuring ESC HA with Multiple Interfaces

You can configure ESC HA with DRDB synchronization and VRRP heartbeat broadcasting on a network interface for data synchronization and VNF monitoring. You can use an additional network interface to allocate

Virtual IP for the northbound access. To configure the multiple interfaces on ESC HA nodes, use --ha_node_list,

--kad_vip, --kad_vif arguments to specify these multiple network interfaces configuration. For details on these arguments, see section Appendix A: Cisco Elastic Services Controller Installer Arguments .

Example configuration steps are shown below:

./bootvm.py <esc_ha1> \

--user_pass <username>:<password>

--user_confd_pass <username>:<password>

--image <image_id> \

--net <net-name> \

--gateway_ip <default_gateway_ip_address> \

--ipaddr <ip_address1> <ip_address2> \

--ha_node_list < IP addresses HA nodes1> < IP addresses for HA nodes2> \

--kad_vip <keepaliveD VIP of the HA nodes and the interface for keepaliveD VIP> \ (for example: --kad_vip 192.0.2.254:eth2)

--kad_vri <virtual router id of vrrp instance>

--kad_vif <virtual IP of the HA nodes or the interface of the keepalived VRRP> \ (for example: --kad_vif eth1 )

--ha_mode <HA installation mode> \

--route <routing configuration> \ (for example:192.0.2.254/24:192.168.0.1:eth1 )

--avail_zone nova:<openstack zone> \

Similarly, a three network interface can be configured for ESC HA nodes. An example three interfaces configuration is shown below with the following assumptions :

• Network 1 is an IPv6 network used for northbound connection. ESC VIP is allocated in this network and the Orchestrator send requests to ESC through ESC VIP.

• Network 2 is an IPv4 network used for ESC sync traffic (DRDB synchronization) and VRRP heartbeat.

This network is also used for OpenStack connection and VNF monitoring.

• Network 3 is another IPv4 network used for management. The SA, rsyslog, etc. can use this network to manage ESC.

./bootvm.py esc-ha-0 --image ESC-2_2_x_yyy --net esc-v6 esc-net --gateway_ip 192.168.0.1 --ipaddr

2001:cc0:2020::fa 192.168.0.239 192.168.5.239 --ha_node_list 192.168.0.239 192.168.0.243 --kad_vip

[2001:cc0:2020::fc/48]:eth0 --kad_vif eth1 --ha_mode drbd --route 10.85.103.0/24:192.168.0.1:eth1 --avail_zone nova: zone name

./bootvm.py esc-ha-1 --image ESC-2_2_x_yyy --net esc-v6 esc-net lab-net-0 --gateway_ip 192.168.0.1 --ipaddr

2001:cc0:2020::fa 192.168.0.239 192.168.5.239 --ha_node_list 192.168.0.239 192.168.0.243 --kad_vip

[2001:cc0:2020::fc/48]:eth0 --kad_vif eth1 --ha_mode drbd --route 10.85.103.0/24:192.168.0.1:eth1 --avail_zone nova: zone name

Configuring the ESC HA Virtual IP Address

In this option, the value of kad_vip argument should be a virtual IP, which allows the service portal and the northbound to access the Primary ESC and send requests to ESC HA service through virtual IP (VIP).

If northbound and both ESC HA nodes are located in the same network, you can connect directly through the virtual IP (VIP). If northbound doesn't sit on the same network as ESC HA, assign a floating IP to ESC HA

VIP using the procedure below:

26

Cisco Elastic Services Controller 4.1 Install and Upgrade Guide

Important Notes

1 Create a port with the VIP address (kad_vip) in the same network as ESC's kad_vip connects.

neutron port-create esc-net --name esc_vip --fixed-ip subnet_id=esc-subnet,ip_address=192.168.0.87

2 Deploy ESC HA . See Configuring High-Availability section in Installing ESC on OpenStack.

Note Make sure the kad_vip using the same IP address as the port created above.

3 Associate a floating IP with the port created above. The first uuid is the floating ip id and the second one is the port id.

neutron floatingip-associate <floating IP> <port ID>

Access ESC HA through the floating IP and it will connect to the ESC Primary node.

4 For the portal access, make sure the keepalive network is accessible by your browser and the virtual IP is the IP address to access the portal of the Primary node.

For example, if the VIP is 192.0.2.254, access ESC HA portal with https://192.0.2.254:9001/.

Important Notes

• ESC HA

◦ An HA failover takes about 2 to 5 minutes. The ESC service will not be available during the switchover time.

◦ When the switchover is triggered during transactions, all incomplete transactions will be dropped.

The requests should be re-sent by northbound if it does not receive any response from ESC.

• External Storage

◦ If the Primary ESC instance is suspended by OpenStack command, the switch over will be triggered but the cinder volume won't be attached to the new Primary ESC instance. This is not a valid use case for ESC HA.

• Internal Storage

◦ Two ESC instances have to be deployed to establish the HA solution. The ESC HA will start to work when both ESC instances are successfully deployed and are able to connect to each other. If you just deploy one ESC instance with HA parameters, the ESC instance keeps Switching-to-Master state and will not be able to provide any service until it reaches its peer.

◦ Split-brain scenario can still happen in this ESC HA solution, although the chance is very low.

• ETSI-specific Notes

ESC supports ETSI MANO northbound API defined by the European Telecommunications Standards

Institute (ETSI) for NFV Management and Orchestration. The ETSI MANO API is another programmatic interface based on the REST architecture. For more information, see ETSI MANO Compliant Lifecycle

Operations in the Cisco Elastic Services Controller User Guide . Consider the following notes while enabling ETSI service on ESC which is in HA mode:

Cisco Elastic Services Controller 4.1 Install and Upgrade Guide

27

Troubleshooting High Availability

• The server.address value in the etsi-vnfm.properties

file must be set to a Virtual IP (VIP) address.

This IP address can be used to communicate back to the ETSI services using API callbacks. If the virtual IP address is not specified, the ETSI service startup may fail.

• The ESC health service ( escadm health or health.sh

) will report that the ETSI service is running but not ready. You can ignore this error as the ETSI service will be running and also processing requests.

Troubleshooting High Availability

• Check for network failures. If a network problem occurs, you must check the following details:

◦ The IP address assigned is correct, and is based on the OpenStack configuration.

◦ The gateway for each network interface must be pingable.

• Check the logs for troubleshooting:

◦ The ESC Admin logs at /var/log/esc/escadm.log

◦ The ESC manager log at /var/log/esc/escmanager.log

◦ The ESC HA log at /var/log//esc/esc_haagent.log

◦ The KeepAliveD log at /var/log/messages by grep keepalived

• Check for DRBD (Replication based ESC HA) for Internal Storage solution:

◦ Check the DRBD configuration file at

/etc/drbd.d/esc.res

◦ Access the DRBD log

/var/log/messages|grep drbd

28

Cisco Elastic Services Controller 4.1 Install and Upgrade Guide