Installing Cisco Elastic Services Controller on OpenStack. Cisco Elastic Services Controller

PDF

Documento

Anúncio

P A R T

Installing Cisco Elastic Services Controller on

OpenStack

•

Prerequisites, on page 5

•

Installing Cisco Elastic Services Controller on OpenStack, on page 7

•

Installing High Availability Active/Standby, on page 21

•

Cisco Elastic Services Controller Active/Active High Availability Overview, on page 33

•

Installing Active/Active High Availability Cluster, on page 35

•

Managing Cluster in ESC Active/Active High Availability, on page 41

•

Configuring GEO in Active/Active High Availability, on page 43

•

DRBD Encryption for ESC Active/Standby and Active/Active HA Data Replication , on page 49

•

Upgrading ESC Active/Active High Availability, on page 51

C H A P T E R

Prerequisites

The following sections detail the prerequisites for installing Cisco Elastic Services Controller:

•

Virtual Resource Requirements, on page 5

•

Software Requirements, on page 6

•

Preparing for the Installation, on page 6

Virtual Resource Requirements

The following table lists the virtual resource requirements for Cisco Elastic Services Controller:

ESC

Deployment

ESC

Standalone

Number of

ESC VMs

Virtual CPU per ESC VM

Virtual

Memory (GB) per ESC VM

Virtual Hard

Disk (GB)per

ESC VM

Number of total VMs

Supported

8 40 1000

VIM

All supported

VIMs

ESC

Active/Standby

ESC

Active/Active

ESC GEO

2500

5000

All supported

VIMs

OpenStack

Only

OpenStack

Only

Note

The Active/Active HA and Geo HA deployment can accommodate up to 60 GB of hard disk while provisioning the NFVI resources.

Cisco Elastic Services Controller 5.5 Install and Upgrade Guide

Installing Cisco Elastic Services Controller on OpenStack

Software Requirements

Software Requirements

The following table lists the software requirements on OpenStack:

Requirements Description

Supported Browsers

OpenStack Version

Google Chrome 44.x or later

Any of the following:

• Train

• Ocata (V2 and V3)

• Queens (Keystone V3)

Preparing for the Installation

Before you perform the installation, ensure that you are prepared by reviewing this checklist:

Requirements Your Information/ Notes

For Pre-installation

Configuration

QCOW image location

QCOW image

VM per Instance

IP address

Subnet mask

Hostname

Domain name

Gateway IP address

Admin password

ESC Release Package

ESC.qcow2

bootvm.py

An image file for booting up the ESC instance

The installation script is compatible with python 2.7.6 and Python 3.4

Cisco Elastic Services Controller 5.5 Install and Upgrade Guide

C H A P T E R

Installing Cisco Elastic Services Controller on

OpenStack

This chapter describes how to install Cisco Elastic Services Controller on OpenStack and includes the following sections:

•

Installation Scenarios, on page 7

•

Main Components of Cisco Elastic Services Controller Setup, on page 8

•

Installing Cisco Elastic Services Controller Using the QCOW Image, on page 8

•

Managing Root Certificates in Cisco Elastic Services Controller, on page 16

•

Managing Keystore in Cisco Elastic Services Controller, on page 18

•

Using a Bootable Volume in ESC Installation, on page 20

Installation Scenarios

The following sections briefly describe some of the common deployment scenarios that are addressed by ESC.

Cisco Elastic Services Controller can be installed in different modes as per the requirement. These different modes are configured during installation. The following sections briefly describe some of the common deployment scenarios that are addressed by ESC.

ESC Standalone

In the standalone scenario, a single active VM is deployed for ESC.

ESC with HA

ESC supports High Availability (HA) in the form of Active/Standby and Active/Active models. For

Active/Standby model, two ESC instances are deployed in the network to prevent ESC failure and to provide

ESC service with minimum interruption. If the active ESC instance fails, the standby instance automatically takes over the ESC services. ESC HA resolves the following single point failures of ESC:

• Network failures

• Power failures

• Dead VM instance

• Scheduled downtime

Cisco Elastic Services Controller 5.5 Install and Upgrade Guide

Installing Cisco Elastic Services Controller on OpenStack

Main Components of Cisco Elastic Services Controller Setup

• Hardware issues

• Internal application failures

Note Installing or upgrading software on the ESC VM is not supported. For further assistance, contact Cisco TAC.

For more information on Installing ESC HA Active/Standby, see

Installing Cisco Elastic Services Controller

Using the QCOW Image, on page 8

and Installing Cisco Elastic Services Controller on VMware vCenter.

For more information on the Active/Active model, see ESC HA Active/Active Overview Chapter.

Main Components of Cisco Elastic Services Controller Setup

The Cisco Elastic Service Controller (ESC) setup has the following components:

• Virtual Infrastructure Manager —Elastic Services Controller (ESC) and its VNFs are deployed in a

Virtual Infrastructure Manager (VIM). This might have one or more underlying physical nodes.

• ESC Virtual Machine —The ESC VM is a VM that contains all the services and processes used to register and deploy services. This includes the ESC Manager and all other services. ESC provides Netconf

API, REST API, and Portal as north bound interfaces to communicate with ESC. ESC VM contains CLI to interact with ESC VM. There are two CLI, one uses the REST API and the other uses Netconf API.

Installing Cisco Elastic Services Controller Using the QCOW

Image

You can install Cisco Elastic Services Controller (ESC) on OpenStack by using a QCOW image. ESC will be deployed in the OpenStack as running VM instance to manage VNFs. Therefore, ESC need OpenStack environment parameters to be installed in OpenStack. The installation time varies from 10 to 20 minutes, depending on the host and the storage area network load. This procedure describes how to create ESC Virtual

Machine (VM) in OpenStack.

Before you begin

• All system requirements are met as specified in

Prerequisites, on page 5

• You have the information identified in

Preparing for the Installation, on page 6 .

• Copy the ESC image file on the system where you want to install ESC.

• This system must be accessible by OpenStack.

Procedure

Step 1

Step 2

Check the compatibility of the bootvm.py and the ESC image:

Cisco Elastic Services Controller 5.5 Install and Upgrade Guide

Installing Cisco Elastic Services Controller on OpenStack

Installing Cisco Elastic Services Controller Using the QCOW Image

Step 3

./bootvm.py --version

For more information on the ESC installer arguments, see Appendix: A Cisco Elastic Services Controller

Installer Arguments .

In a text editor, create a file named PROJECT-openrc.sh file and add the following authentication information.

The following example shows the information for a project called admin, where the OpenStack username is also admin, and the identity host is located at controller node.

Note To set the required environment variables for the OpenStack command-line clients, you must create an environment file called an OpenStack rc file, or openrc.sh file. This project-specific environment file contains the credentials that all OpenStack services use. The ESC installation script requires these OpenStack environment parameters to perform authentication and installation on OpenStack.

If all the OpenStack credentials are passed through its own arguments, the bootvm.py script doesn't require these parameters.

Step 4

Step 5

Step 6 export OS_NO_CACHE=true export OS_TENANT_NAME=admin export OS_USERNAME=admin export OS_PASSWORD=admin_pass export OS_AUTH_URL= http://controller node:35357/v2.0

The other OpenStack parameters required for installation are: --os_auth_url, --os_username, --os_password

, -os_tenant_name, --bs_os_user_domain_name, --bs_os_project_domain_name, --bs_os_identity_api_version,

--bs_os_auth_url, --bs_os_username, --bs_os_password , -bs_os_tenant_name, --bs_os_user_domain_name,

--bs_os_project_domain_name, --bs_os_identity_api_version.

For OpenStack V2 API, you need following items to be defined in your global environment variables:

--os_password, --os_auth_url, --os_username, --os_tenant_name.

For OpenStack V3 API, set --os_identity_api_version=3 . Other parameters required for OpenStack V3 API are: --os_user_domain_name, --os_project_domain_name, --os_project_name, --os_password, --os_auth_url,

--os_username, --os_identity_api_version, --os_ca_cert, --requests_ ca_bundle.

Note The arguments, --os_tenant_name, --os_username, --os_password, --os_auth_url will also by default configure the VIM connector. If you want to skip configuring the VIM connector, pass the parameter

(--no_vim_credentials) with the bootvm.py. When no_vim_credentials parameter is provided, the bootvm.py arguments (os_tenant_name, os_username, os_password,os_auth_url) are ignored. For more information on configuring VIM connectors after installation, and managing VIM connectors, see Managing VIM Connectors in the Cisco Elastic Services Controller User Guide .

Note --os_ca_cert and --requests_ca_bundle arguments are only required for https connection.

On any shell from which you want to run OpenStack commands, source the PROJECT-openrc.sh

file for the respective project. In this example, you source the admin-openrc.sh

file for the admin project .

$ source admin-openrc.sh

Check the environment variables.

$ env | grep OS_

$ glance image-create \

--name <image_name> \

--is-public=<true or false> or --visibility public or private\

Cisco Elastic Services Controller 5.5 Install and Upgrade Guide

Installing Cisco Elastic Services Controller on OpenStack

Installing Cisco Elastic Services Controller Using the QCOW Image

Step 7

Step 8

--disk-format <disk_format> \

--container-format <container_format> \

--file <file> \

--progress

An example configuration is shown below:

$ glance image-create \

--name esc-1_0_01_11_2011-01-01 \

--is-public=<true or false> or --visibility public or private \

--disk-format qcow2 \

--container-format bare \

--file esc-1_0_01_11_2011-01-01.qcow2 \

--progress

The glance image-create command is used to create a new image. The command takes the following arguments:

Note The 'is-public' argument is applicable only for OpenStack Kilo release.

Arguments name is-public disk-format container-format file progress

Description

Name of the image.

(Optional) Makes the image accessible to the public.

Disk format of the image. ESC uses a qcow2 disk format.

Container format of image. ESC uses a bare container format.

Local file that contains disk image to be uploaded during creation.

(Optional) Shows upload progress bar.

To verify whether the image has been registered successfully: a) Using OpenStack Controller dashboard:

• Log into OpenStack using your credentials.

• Navigate to Admin > Image .

Verify if the image appears in the list.

b) Using nova CLI:

$ nova image-show <image_name>

The standard resource requirement of ESC is 4vCPU, 8G RAM, and 40GB disk space. ESC installation script takes the pre-defined "m1.large" flavor which has the definition of 4vCPU, 8G RAM and 80G disk space. To use 40GB disk space, create a flavor with the minimum disk space requirement.

$ nova flavor-create ESC_FLAVOR_NAME ESC_FLAVOR_ID 8192 40 4

To deploy ESC VM, do the following: a.

Ensure that the existing network have connectivity to OpenStack controller. To verify the network connectivity using the nova CLI use:

$ nova net-list

Cisco Elastic Services Controller 5.5 Install and Upgrade Guide

Installing Cisco Elastic Services Controller on OpenStack

Installing Cisco Elastic Services Controller Using the QCOW Image b.

Record the ID of the network that ESC connects to boot the ESC VM by with image and flavor created earlier. The bootvm.py command requires at least one --user_pass argument to create an admin account for linux (ssh/console access) and at least one --user_confd_pass to create an admin account for ConfD

(netconf/cli access). The following are the syntax for these mandatory user credential arguments:

--user_pass admin:'PASSWORD-OR-HASH'[:OPTIONAL-PUBLIC-KEY-FILE][:OPTIONAL-ROLE]

--user_confd_pass admin:'PASSWORD-OR-HASH'[:OPTIONAL-PUBLIC-KEY-FILE]

Generating hashed password is optional. You can select a plain password as and when required.

To generate a hashed password on Ubuntu OS, use the following command: mkpasswd --method=SHA-512 --salt XyZ123 <<< <Password>

Starting ESC Release 5.4, invoking the esc_nc_cli command without username and password fails with the following message: admin@esc$ esc_nc_cli --user <username> --password <password> get esc_datamodel/opdata/status

ERROR Cannot find file /home/admin/.ssh/confd_id_rsa to support public key authentication with esc-nc-admin

* Check your arguments or installation.

* Usage without --user --password or --privKeyFile arguments requires ssh keys and configuration.

* Use the following command to generate new SSH Keys and update to authorized_keys: sudo escadm confd keygen --user admin

* NOTE: Consult with site security policies for use of public key authentication.

You must run the esc_nc_cli command with a username and password: admin@esc$ esc_nc_cli --user admin --password ADMIN-CONFD-PASSWORD get esc_datamodel/opdata/status

Operational Data

/opt/cisco/esc/confd/bin/netconf-console --port=830 --host=127.0.0.1 --user admin

--password=****** --get -x "esc_datamodel/opdata/status"

<?xml version="1.0" encoding="UTF-8"?>

<rpc-reply xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="1">

<data>

<esc_datamodel xmlns="http://www.cisco.com/esc/esc">

</opdata>

</esc_datamodel>

</data>

</rpc-reply>

The following is an example to install ESC with an authorized public key. In the following example, single quotes are used to avoid conflict with shell reserved characters:

--user_pass admin:'$algorithm$salt$hash-of-salt-password':$HOME/.ssh/esc_rsa.pub

--user_confd_pass admin:'$algorithm$salt$hash-of-salt-password':$HOME/.ssh/esc_rsa.pub

The public keys are generated as part of a key pair, such as: ssh-keygen -t rsa -b 1024 -C "esc" -N "" -f ~/.ssh/esc_rsa

Cisco Elastic Services Controller 5.5 Install and Upgrade Guide

Installing Cisco Elastic Services Controller on OpenStack

Additional Installation Options

Your public key and identification key are saved in /home/username /.ssh/esc_rsa and esc_rsa.pub files.

For more examples on the user credential arguments, see Appendix A: Cisco Elastic Services Controller

Installer Arguments.

To check the details of ESC VM and get the information including the IP address(es) of the ESC VM, use the following command:

$ nova show <esc_vm_name>

Additional Installation Options

• Creating ConfD SSH Keys: Starting ESC Release 5.4, you can invoke the following command to create

SSH keys and configure the public key authentication: sudo escadm confd keygen --user USERNAME

Run the above command on each user account on the ESC VM that requires authentication without password. In an Active/Standby deployment, run the command on both the nodes.

Enabling ConfD SSH keys using bootvm.py: Enabling the following bootvm.py command restores the functionality prior to ESC Release 5.4:

--confd_keygen

For information on ConfD cli, see Changing the ConfD NetConf CLI Administrator Password Using the

Command Line Interface.

• Deploying ESC in an OpenStack IPv6 environment: Before deploying ESC instance in IPv6, make sure to source openrc that supports ipv6 addresses. To deploy ESC in IPv6 environment, use the following bootvm arguments:

./bootvm.py

<esc_vm_name-ipv6> --poll --user_rest_pass <username> : <password> --image

<image_name>

--net <ipv6_network> --ipaddr <ipv6_ip_address> --enable-http-rest --user_pass

<username> : <password>

--user_confd_pass <username> : <password> --etc_hosts_file <hosts-file-name> --route

<default routing configuration>

• Deploying ESC in DHCP mode: If you use the bootvm.py argument without --ipaddr, then the ESC instance will be deployed in a DHCP mode. To deploy ESC in a DHCP network, use the following configuration:

./bootvm.py

<esc_vm_name> --image <image_name> --net <IPv6 network> <IPv4 network>

--flavor <flavor_name>

--user_pass <username> : <password>

--user_confd_pass <username> : <password>

Cisco Elastic Services Controller 5.5 Install and Upgrade Guide

Installing Cisco Elastic Services Controller on OpenStack

Additional Installation Options

Note By default, ESC only support DHCP in IPv4 networks. If IPv6 is used, you need to log in the ESC VM and run "dhclient -6 ethX" (ethX is the V6 interface name) manually to enable V6 DHCP.

When deploying ESC with multiple network interfaces, with one or more type of DHCP, use bootvm.py --defroute N to specify the interface index to be assigned default route and gateway IP.

By default,

N = 0

. So the first network interface on the bootvm.py

command line is default.

For Example:

--defroute 1 will assign <network2> with <default_gateway_ip_address>

./bootvm.py <esc_vm_name> --image <image_id> --net <network1>

<network2> --defroute 1 --gateway_ip <default_gateway_ip_address>

• Using a Bootable Volume in ESC Installation: You can attach a volume to an ESC instance and launch

an instance from inside the volume. For more information, see the section Using a Bootable Volume in

ESC Installation

• Assigning Floating IP to the ESC: If you want to associate a floating IP with the ESC instance , do the following:

Check for an available floating IP address and assign it to the ESC VM:

$ nova floating-ip-list

$ nova floating-ip-associate esc_vm_name <ip_address>

Or create a new floating IP address and assign it to the ESC VM:

$ nova floating-ip-create <FLOATING_NETWORK - ID>

$ nova floating-ip-associate esc_vm_name <ip_address> or neutron floatingip-create FLOATING_NETWORK neutron floatingip-associate floating-ip-ID port-ID

• Deploying ESC with static IPs: To use ESC in a specific network with static IPs, for example,

192.168.0.112 at network1, specify --ipaddr and --gateway_ip to the bootvm command line , as shown below:

Note Before assigning static IP, make sure the static IP is available and is not being used on other machine.

./bootvm.py

<esc_vm_name> --image <image_id> --net <network>

--ipaddr <ip_address> --gateway_ip <default_gateway_ip_address> --user_pass

<username> : <password>

--user_confd_pass <username> : <password>

• Deploying ESC with multiple network interfaces: To use multiple networks for ESC, for example,

192.168.0.112 at network1 and 10.20.0.112 at network2, specify both the IP addresses and the network names of the interfaces in the --net and --ipaddr arguments in the following command line. In addition,

Cisco Elastic Services Controller 5.5 Install and Upgrade Guide

Installing Cisco Elastic Services Controller on OpenStack

Additional Installation Options also choose the default gateway for ESC from the gateways of these networks. Specify the default gateway for ESC through the --gateway_ip argument.

./bootvm.py

<esc_vm_name> --image <image_id> --net <network1>

<network2> --ipaddr <ip_address1> <ip_address2> --gateway_ip <default_gateway_ip_address>

--user_pass <username> : <password>

--user_confd_pass <username> : <password>

Note If --flavor is not specified, bootvm.py will use the default flavor "m1.large" in

OpenStack.

Ensure that m1.large exist in OpenStack if decide not specified a flavor, and also ensure that the flavor suits ESC deployment requirement.

• Deploy ESC with log forwarding options: To forward ESC logs to an rsyslog server, specify the IP address of the rsyslog server while creating an ESC VM. Optionally, you can also specify the port and protocol to use.

For example, if the IP address of the rsyslog server is 172.16.0.0 the port on the server to forward logs is 514, and the protocol used is UDP, the ESC installation could be

./bootvm.py

<esc_vm_name> --image <image_id> --net network1 --rsyslog_server 172.16.0.0

--rsyslog_server_port 514 --rsyslog_server_protocol udp --user_pass <username> : <password>

--user_confd_pass <username> : <password>

• Disabling the ESC GUI: To boot up ESC VM with the graphical user interface disabled, modify the

--esc_ui_startup argument value, as shown in the command line below:

./bootvm.py

<esc_vm_name> --image <image_id> --net <network> --user_pass

<username> : <password>

--user_confd_pass <username> : <password>

--esc_portal_startup=False

• Enabling REST interface for ESC: To support the REST interface, specify --enable-https-rest argument.

You can activate REST interface on both https or http:

./bootvm.py

<esc_vm_name> --image <image_id> --net <network> --user_pass

<username> : <password>

--user_confd_pass <username> : <password> --enable-https-rest

./bootvm.py

<esc_vm_name> --image <image_id> --net <network> --user_pass

<username> : <password>

--user_confd_pass <username> : <password> --enable-http-rest

•

• Enabling REST interface for ETSI: To support the ETSI REST intetrface, specify --enable-http-etsi to activate the interface over http, or --enable-https-etsi to activate the interface over https.

./bootvm.py

<esc_vm_name> --image <image_id> --net <network> --user_etsi_pass

<username> : <password> --enable-https-etsi . . .

./bootvm.py

<esc_vm_name> --image <image_id> --net <network> --user_etsi_pass

Cisco Elastic Services Controller 5.5 Install and Upgrade Guide

Installing Cisco Elastic Services Controller on OpenStack

Additional Installation Options

<username> : <password>

--user_confd_pass <username> : <password> --enable-http-etsi...

Note Only the https REST and ETSI interfaces should be enabled in a production environment.

• Deploying ESC with ETSI OAuth2 Clients: To add ETSI OAuth2 clients during the installation, use the arguments as shown below. The OAuth2 clients can also be added using the escadm commands after the installation.

./bootvm.py <esc_vm_name> --image <image_id> --net <network> -user_etsi_pass <username>:<password> --etsi_oauth2_pass <clientId>:<clientSecret>

--enable-https-etsi ...

• Deploying ESC with global parameters: To set the global configurations through the esc_params_file during the installation, use the arguments as shown below. These global configurations can also be changed through REST API after the installation.

Note The default security group is applied to the tenant during tenant creation. By default, the ESC configuration parameter for the security group, openstack.DEFAULT_SECURITY_GROUP_TO_TENANT

is set to true. The configuration parameter must be set at the time of installation. You can query or update the parameter on ESC VM through the REST API. If the parameter is set to true, you can create and assign default security group during tenant creation.

If the parameter is set to false, you cannot create or assign default security group during tenant creation. For details on the parameters that can be configured through esc_params_file, see Appendix A: Cisco Elastic Services Controller Installer

Arguments.

./bootvm.py <esc_vm_name> --image <image_id> --net <network> --flavor <flavor_name>

--user_pass <username>:<password>:<public key file> --user_confd_pass

<username> : <password>

--esc_params_file <esc parameter configuration file>

• Deploying ESC with ETSI properties : To set the ETSI properties through the etsi_params_file during installation, use the arguments as shown below. These properties can also be changed in the etsi-production.properties

file after the installation.

Note For details on the properties that can be configured through the etsi_params_file, see the ETSI Production Properties chapter in the ETSI NFV MANO User Guide .

./bootvm.py <esc_vm_name> --image <image_id> --net <network> --flavor <flavor_name>

--user_pass <username>:<password>:<public key file> --user_confd_pass

--etsi_params_file <etsi properties file>

• Deploying two instances of ESC to build an ESC HA (Active/Standby) pair : For more information on deploying ESC HA (Active/Standby), see Configuring High Availability in Installing ESC on

OpenStack and Installing ESC on VMware chapters.

Cisco Elastic Services Controller 5.5 Install and Upgrade Guide

Installing Cisco Elastic Services Controller on OpenStack

Managing Root Certificates in Cisco Elastic Services Controller

• Adding a Dynamic Mapping File : In Cisco ESC Release 2.1 and earlier, mapping the actions and metrics defined in the datamodel to the valid actions and metrics available in the monitoring agent was enabled using the dynamic_mappings.xml

file. The file was stored in the ESC VM and was modified using a text editor. ESC 2.2 and later do not have an esc-dynamic-mapping directory and dynamic_mappings.xml

file. If you want to add an existing dynamic_mapping xml file to the ESC VM, do the following:

Backup this file to a location outside of ESC, such as, your home directory.

Create esc-dynamic-mapping directory on your ESC VM. Ensure that the read permissions are set.

Install on your ESC VM using the following bootvm argument:

--file root:root: /opt/cisco/esc/esc-dynamic-mapping/dynamic_mappings.xml

:< path-to-local-copy-of-dynamic-mapping.xml

The CRUD operations for mapping the actions and the metrics is available through REST API. To update an existing mapping, delete and add a new mapping through the REST API.

• Changing the confd password on an ESC VM : As an administrator, you can configure the confd password through bootvm.py, during the installation time:

./bootvm.py --user_pass <username> : <password> --user_confd_pass admin:'PASSWORD-OR-HASH':OPTIONAL-PUBLIC-KEY

To reconfigure this password after the installation, execute the following commands:

$ /opt/cisco/esc/confd/bin/ssh admin@localhost -p 2024

$ configure

$ set aaa authentication users user admin password <your_password>

$ commit

$ exit

Note For the ease of future upgrades, make sure that you keep a copy of all the commands and arguments that are used while installing ESC using the bootvm.py file.

Managing Root Certificates in Cisco Elastic Services Controller

Cisco Elastic Services Controller (ESC) provides a mechanism to enable verification of SSL certificates.

Currently, this feature is supported only on OpenStack. Certificate validation is enabled by default during the initial ESC boot up. However, ESC also allows you to configure these SSL certificates. This section describes how to enable/disable certificate validation, add/remove, or list the certificates for Cisco Elastic Service

Controller on OpenStack. You can add a root certificate during the ESC bootup or even after ESC bootup is completed.

Enabling/Disabling the Root Certificate Validation

Cisco Elastic Services Controller by default enables certificate validation. You can also enable or disable by modifying the parameter, DISABLE_CERT_VALIDATION, available under the Openstack category in the esc_params.conf file, or through the REST interface, or using the escadm tool.

On ESC active node, use the command, sudo escadm enable-certificate or sudo escadm disable-certificate to enable and disable the certificate validation, respectively.

Cisco Elastic Services Controller 5.5 Install and Upgrade Guide

Installing Cisco Elastic Services Controller on OpenStack

Adding a Root Certificate

Adding a Root Certificate

You can add a root certificate during the ESC bootup or even after ESC bootup is completed. Before adding certificates, ensure the OpenStack environment file, OpenStack RC file has parameters to perform authentication and installation on OpenStack. The --os_auth_url must be specified while passing the parameters. --os_auth_url specifies the secure (https) or unsecured (http) keystone URL used by OpenStack for authentication.

• Add certificate for standalone (only) during the bootup time, i.e, during the ESC VM installation:

./bootvm.py test-vm --image <image_name> --net <network> [--cert_file CERT_FILE]

[--confd_aes_key CONFD_AES_KEY]

/home/cisco/openstack.crt

--user_pass <username>:<password> --user_confd_pass <username>:<password>

Note Currently, ESC does not support adding a certificate for HA Active/Standby during the installation as the keepalived service is not running when a certificate is added.

• Add certificate for standalone/HA Active/Standby after booting up the ESC instance. The escadm tool has an truststore add option which has the following arguments: The --file argument refers to the CA certificate file. Using this argument you can import any file format supported by the java keytool: X.509

v1, v2, and v3 certificates, and PKCS#7. The --alias argument is unique and refers to the name this specific CA certificate is given.

Copy/Transfer CA Certificate file to ESC active VM .

Add certificate to ESC truststore. To do this, execute the following command: sudo escadm escadm truststore add --alias [ca cert alias] --file [file path] or sudo escadm truststore escadm truststore --alias [ca cert alias] --file [file path]

Verify the certificate is added.

sudo escadm truststore show or sudo escadm truststore show

Removing a Root Certificate

The escadm tool has a 'truststore delete' option which only takes --alias argument. The --alias argument refers to the name of the CA certificate to be deleted. Use this argument on the standalone/HA Active/Standby ESC

VM:

Procedure

Step 1 On (active)ESC use escadm to delete certificate from ESC truststore.

sudo escadm truststore delete --alias [ca cert alias]

Cisco Elastic Services Controller 5.5 Install and Upgrade Guide

Installing Cisco Elastic Services Controller on OpenStack

Managing Root Certificates During the Upgrade

Step 2 or sudo escadm truststore truststore delete --alias [ca cert alias]

Verify the certificate is removed.

sudo escadm truststore show or sudo escadm truststore show

Managing Root Certificates During the Upgrade

• Image Upgrade : If you are backing up the ESC DB for upgrade, then no other action is required, the

ESC truststore will be restored once the ESC DB is restored. If you are not backing up the ESC DB for upgrade, then each CA certificate needs to be added again to the ESC truststore.

• RPM Upgrade : This upgrade method keeps the ESC truststore as is, i.e. all ca certificates in the ESC truststore should remain there after upgrade.

Managing Keystore in Cisco Elastic Services Controller

The keystore is a storage for certificates and keys that an application uses to authenticate itself with different clients.

ESC Keystore holds only one certificate, that is used by all the applications for example, ESCManager,

VIMManager, MONA, and so on.

ESCADM provides multiple commands to manage the keystore.

When deploying ESC for the first time, a self signed certificate is created and stored in the keystore by default.

important Notes:

• In an Active/Active mode, the default certificate Common Name (CN) is db.service.consul

. If you want to set a new certificate, you must set it with the same CN, otherwise, you must add the following configuration to the heat template of all the nodes to disable the certificate validation in MONA when deploying the ESC: mona: certificate_validation: false

• All the escadm keystore commands require root privileges to be executed on the ESC.

ESCADM Keystore Commands

• escadm keystore show

The escadm keystore show command displays information about the certificate currently stored in the keystore. It shows informations like, the creation date, Alias, and certificate fingerprint.

For example:

$ sudo escadm keystore show

Keystore type: PKCS12

Keystore provider: SUN

Cisco Elastic Services Controller 5.5 Install and Upgrade Guide

Installing Cisco Elastic Services Controller on OpenStack

ESCADM Keystore Commands

Your keystore contains 1 entry esc, Apr 13, 2020, PrivateKeyEntry,

Certificate fingerprint (SHA1):

FF:11:66:3E:93:DD:3A:0B:9A:72:40:16:35:34:D2:22:E1:25:07:80

• escadm keystore export [--out <file path>]

The export command displays the full content of the certificate. If the out option is specified, the content is saved in a file whose path is specified in the previous option.

• escadm keystore set-- file <file path>

The set command replaces the current certificate with a new one residing inside the file whose path is specified in the option file

Ensure that the file is in PEM format, containing both the certificate and a private key.

The following example shows how to generate a new self-signed certificate and set it to the keystore:

Use the following command to generate the certificate: openssl req -newkey rsa:2048 -new -nodes -x509 -days 3650 -keyout key.pem -out cert.pem

The previous commands gives the following two files: key.pem

and cert.pem

Use the following commands to combine the two files: cat key.pem > server.pem

cat cert.pem >> server.pem

To set the new certificate to the keystore, use the following command:

$ sudo escadm keystore set --file server.pem

Service "keystore" successfully updated ESC keystore and will take effect once ESC services are restarted by running "sudo escadm restart"

Note Ensure that you delete any left over files containing private keys and certificate after the setting is done.

It is mandatory to restart your system for the change to take effect. Use the following command to restart in standalone mode or H/A mode: sudo escadm restart

If the ESC is running in the Active/Active mode, you must restart all the nodes in the same cluster.

However, if the ESC is running in an Active/Active GEO mode, you must stop the GEO service on all the clusters, to make sure a GEO switchover does not take place.

Only in GEO Active/Active mode, to stop GEO service, log in to any node in each cluster and use the following command:

$ sudo escadm geo stop --cluster

You can login to any node in the cluster where the certificate is originally set. Use the following command:

Cisco Elastic Services Controller 5.5 Install and Upgrade Guide

Installing Cisco Elastic Services Controller on OpenStack

Using a Bootable Volume in ESC Installation

$ sudo escadm stop --cluster

$ sudo escadm start --cluster

If you are on a GEO Active/Active mode, once all the node are up and running in passing state, login again to any node to start the GEO service. Use the folowing command to start the service:

$ sudo escadm geo start --cluster

Using a Bootable Volume in ESC Installation

A volume in OpenStack is a detachable block storage device that can be attached to an ESC instance. You can store and also run ESC instances from a volume.

Note • Only one ESC instance can be launched from one volume at a time.

• ESC installation with a combination of bootable volume and high availability (Active/Standby and

Active/Active) on cinder is not supported.

To launch an ESC instance from a bootable volume, do the following:

Procedure

Step 1

Step 2

Create a bootable volume in OpenStack based on an ESC image or from a bootable volume. The bootable volume must be at least of 30 GB disk size. For more information, see OpenStack documentation.

Deploy ESC VM using the bootvm.py

command and choose the

--boot_volume argument instead of the

--image argument, as shown below:

./bootvm.py <esc_vm_name> --boot_volume <volume_name_or_id> --net <network> --user_pass

--user_confd_pass <username>:<password> --flavor <flavor_name>

Note • Only one of these arguments,

--image or

--boot_volume must be passed to the bootvm.py

command. The installation will fail, if both or none of the arguments are used.

• When launching an ESC instance from a bootable volume, volume disk size is considered over the flavor disk size.

• If an ESC instance is deleted, the volume attached to it will not be deleted, as the volume was created out-of-band.

Cisco Elastic Services Controller 5.5 Install and Upgrade Guide

C H A P T E R

Installing High Availability Active/Standby

This chapter contains the following sections:

•

High Availability Active/Standby Overview, on page 21

•

How High Availability Works, on page 22

•

Deploying ESC High Availability Active/Standby, on page 23

•

Configuring the Northbound Interface Access, on page 26

•

Important Notes, on page 31

•

Troubleshooting High Availability Active/Standby, on page 32

High Availability Active/Standby Overview

ESC supports High Availability (HA) in the form of Active/Standby and Active/Active models. For

Active/Standby model, two ESC instances are deployed in the network to prevent ESC failure and provide

ESC service with minimum service interruption. If the active ESC instance fails, the standby instance automatically takes over the ESC services. ESC HA Active/Standby resolves the following single point failures:

• Network failures

• Power failures

• Dead VM instance

• Scheduled downtime

• Hardware issues

• Internal application failures

Cisco Elastic Services Controller 5.5 Install and Upgrade Guide

ESC Active/Standby Architecture

ESC Active/Standby Architecture

Figure 1: Cisco Elastic Services Controller Active/Standby Architecture

Installing Cisco Elastic Services Controller on OpenStack

How High Availability Works

ESC HA Active/Standby network can be either set up as a single installation of a ESC HA Active/Standby pair or deployed as two standalone ESC nodes that are converted into HA pair after re-configuring these nodes post deployment. The HA deployment consists of two ESC instances: an active and a standby. Under normal circumstances, the active ESC instance provides the service. The corresponding standby instance is passive.

The standby instance is in constant communication with the active instance and monitors the active instances' status. If the active ESC instance fails, the standby instance automatically takes over the ESC services to provide ESC service with minimum interruption.

The standby also has a complete copy of the database of the active, but it does not actively manage the network until the active instance fails. The KeepAliveD service monitors both active and standby instances activity status. When the active instance fails, the standby takes over automatically. The standby instance takes over active instance to manage the services while active instance restoration is taking place.

When the failed instance is restored, if required you can manually initiate a switch-over and resume network management via the active instance.

Both active and standby ESC instances are connected to the northbound orchestration system through an IPv4 or IPv6 network. For the northbound system, a unique virtual IP address is assigned to access the current active ESC High Availability Active/Standby instance. The deployed VNFs are connected to both ESC active and standby instances through another IPv6 network.

ESC HA Active/Standby nodes are managed by KeepAliveD and DRBD (Replication tool to keep the ESC database synchronized) sync network services. While the KeepAliveD service monitors both active and standby instances status, the DRBD service monitors active instance DB and sync the changes to the standby instance

DB. These two services can be co-located on same VIP network or in two separate networks. VM handshake between ESC instances occurs through the KeepAliveD over the IPv4 or IPv6 network.

Cisco Elastic Services Controller 5.5 Install and Upgrade Guide

Installing Cisco Elastic Services Controller on OpenStack

Deploying ESC High Availability Active/Standby

Deploying ESC High Availability Active/Standby

To deploy Cisco Elastic Services Controller (ESC) High Availability (HA) Active/Standby, ESC standalone instances can be installed on two separate nodes - Active and Standby. For more information see,

How High

Availability Works, on page 22

. You can connect the active and standby instances to either a Cinder volume or Replication based volume (DRBD).

The following deployment mechanisms can be used to deploy ESC HA Active/Standby:

• Internal Storage—When ESC HA Active/Standby is configured with Internal storage, the active and the standby instances have individual databases which are always synchronized. In this solution, ESC HA

Active/Standby is designed with database replication and DRBD is used as the tool for disk-level replication. The database in the active instance simultaneously propagates the data to the database in the standby instance thus requiring no external storage. In the event of an active instance failing, the standby instance get assigned the role of the active instance along with its own synchronized database.

ESC HA Active/Standby is deployed using internal storage, the ESC instances reply on the virtual IP address (that is kad_vip argument), and the interface of vrrp instance (that is kad_vif argument) to select the active ESC instance. To establish a reliable heartbeat network, it is recommended that the active and standby ESC instances are on different physical hosts. The reliability of the physical links between the

ESC instances (such as, network interface bonding) can also be taken into consideration.

• Replicate External-Storages — In this type of architecture, ESC HA Active/Standby is configured with

DRBD and both active and standby instances store their data in two external storages (OpenStack Cinder volumes). Each ESC node is attached by a Cinder volume and ESC data files are stored in the cinder volume. The data in two ESC nodes are synchronized through the database replication mechanism provided by DRBD.

The table lists the differences between the HA Active/Standby options :

Data sharing method

Installation Method

Internal Storage Based ESC HA

Active/Standby

Replicate External Storage Based ESC HA

Active/Standby

Data replication between HA

Active/Standby nodes

Post-installation Configuration

Bootvm Installation

Data replication between two external storages (cinder volume)

Bootvm Installation

VIM Support

Dependency

Advantages

OpenStack, VMware, KVM

VIM independent

• No dependency on specific

VIM components.

• Flexible to build of HA

Active/Standby clusters from commodity hardware, without the requirement for shared-storage.

OpenStack only

Rely on OpenStack cinder

• Use database replication mechanism for data synchronization

• Two cinder volumes are used as external storage and are attached to ESC node.

Cisco Elastic Services Controller 5.5 Install and Upgrade Guide

Installing Cisco Elastic Services Controller on OpenStack

Deploying ESC in High Availability Active/Standby Mode on Internal Storage

Limitations The data consistency may be affected in a double fault condition

(occurs when both ESC nodes have problems).

The data consistency may be affected in a double fault condition (occurs when both

ESC nodes have problems).

Deploying ESC in High Availability Active/Standby Mode on Internal Storage

When you boot ESC instances on active and standby instances, you need to specify the following bootvm.py

command arguments to deploy ESC HA Active/Standby on an internal storage:

• kad_vip

Note When ESC HA Active/Standby is deployed, the kad_vip argument allows end users to access the active ESC instance.

• kad_vif

• ha_node_list

These arguments enable the bootvm.py

command to automatically set up the internal storage on the OpenStack.

For more information on using the bootvm.py

command arguments, see Appendix A: Cisco Elastic Services

Controller Installer Arguments.

To deploy ESC HA Active/Standby instances, use the bootvm script on both the nodes with the following arguments:

ON HA NODE 1:

$ ./bootvm.py <ESC_HA_Node1>\

--user_pass <username>:<password>\

--user_confd_pass <username>:<password>\

--gateway_ip <default gateway IP address>\

--net <network name1>\

--ipaddr <static ip address>

--image <image_name>\

--avail_zone nova:<openstack zone>\

--ha_node_list=<ESC_HA_NODE1_IP> <ESC_HA_NODE2_IP>\

--db_volume_id <cinder volume id>\

--kad_vip <virtual IP address>\

--kad_vif <VRRP_Interface_Instance>\

--ha_mode drbd

ON HA NODE 2:

$ ./bootvm.py <ESC_HA_Node2>\

--user_pass <username>:<password>\

--user_confd_pass <username>:<password>\

--gateway_ip <default gateway IP address>\

--net <network name1>\

--ipaddr <static ip addresses>\

--image <image_name>\

--avail_zone nova:<openstack zone>\

--ha_node_list=<ESC_HA_NODE1_IP> <ESC_HA_NODE2_IP>\

--db_volume_id <cinder volume id>\

--kad_vip <virtual IP address>\

Cisco Elastic Services Controller 5.5 Install and Upgrade Guide

Installing Cisco Elastic Services Controller on OpenStack

Deploying ESC in High Availability Active/Standby Mode on Replicate External Storage

--kad_vif <VRRP_Interface_Instance>\

--ha_mode drbd

You can also use escadm tool to re-configure ESC HA Active/Standby parameters on each of the standalone

ESC VMs. Three parameters "--ha_node_list , --kad_vip, --kad_vif" are all required to configure ESC HA

Active/Standby.

Note You should make sure that both the standalone ESC VMs health check is passed before running the below commands to perform HA Active/Standby configuration.

For example:

$ sudo bash

$ escadm ha set --ha_node_list='<ESC_HA_NODE1_IP> <ESC_HA_NODE2_IP>' --kad_vip <virtual IP address> --kad_vif <VRRP_Interface_Instance>

$ sudo escadm reload

$ sudo escadm restart

Deploying ESC in High Availability Active/Standby Mode on Replicate External Storage

Replicate external storage ESC HA Active/Standby requires two cinder volumes for database storage.

Before you begin

• Networks and IP addresses that both ESC instances will connect to

• Keepalived interface and virtual IP for HA Active/Standby switchover

Procedure

Step 1

Step 2

Step 3

Create two cinder volumes in OpenStack. The configured cinder volume size should be 3GB.

$ cinder create --display-name cindervolume_name_a[SIZE]

$ cinder create --display-name cindervolume_name_b[SIZE]

Check the status of the created cinder volume and find the uuids for deployment.

$ cinder list

Deploy ESC HA Active/Standby instances. Use the bootvm script on both the nodes with the following arguments:

ON HA NODE 1:

$ ./bootvm.py <ESC_HA_Node1>\

--user_pass <username>:<password>\

--user_confd_pass <username>:<password>\

--gateway_ip <default gateway IP address>\

--net <network name1>\

--ipaddr <static ip address>\

Cisco Elastic Services Controller 5.5 Install and Upgrade Guide

Installing Cisco Elastic Services Controller on OpenStack

Configuring the Northbound Interface Access

Step 4

--image <image_name>\

--avail_zone nova:<openstack zone>\

--kad_vip <virtual IP address>\

--kad_vif <VRRP_Interface_Instance>\

--ha_node_list=<ESC_HA_NODE1_IP> <ESC_HA_NODE2_IP>\

--db_volume_id <cinder volume id>\

--ha_mode drbd_on_cinder

ON HA NODE 2:

$ ./bootvm.py <ESC_HA_Node2>\

--user_pass <username>:<password>\

--user_confd_pass <username>:<password>\

--gateway_ip <default gateway IP address>\

--net <network name1>\

--ipaddr <static ip address>\

--image <image_name>\

--avail_zone nova:<openstack zone>\

--kad_vip <virtual IP address>\

--kad_vif <VRRP_Interface_Instance>\

--ha_node_list=<ESC_HA_NODE1_IP> <ESC_HA_NODE2_IP>\

--db_volume_id <cinder volume id>\

--ha_mode drbd_on_cinder

After both VMs are rebooted; one of ESC VM should be in ACTIVE state and the other one should be in

STANDBY state. You can check ESC HA Active/Standby state by using following command: $ sudo escadm status --v .

Configuring the Northbound Interface Access

When you configure ESC HA Active/Standby, you can also specify a virtual Anycast IP address to the HA

Active/Standby pair. The northbound interface as well as the service portal uses virtual Anycast IP address to access the ESC active HA Active/Standby instance. When deploying ESC HA Active/Standby, use the following arguments with the ./bootvm.py

script.

• --ha_node_list

• --kad_vip

• --kad_vif

For more details on these arguments, see section Appendix A: Cisco Elastic Services Controller Installer

Arguments .

The following section explains how to configure ESC HA Active/Standby with multiple interfaces and to configure the virtual Anycast IP address.

Configuring ESC HA Active/Standby with Multiple Interfaces

You can configure ESC HA Active/Standby with DRDB synchronization and VRRP heartbeat broadcasting on a network interface for data synchronization and VNF monitoring. You can use an additional network interface to allocate Virtual IP for the northbound access. To configure the multiple interfaces on ESC HA

Active/Standby nodes, use --ha_node_list, --kad_vip, --kad_vif arguments to specify these multiple network interfaces configuration. For details on these arguments, see section Appendix A: Cisco Elastic Services

Controller Installer Arguments .

Cisco Elastic Services Controller 5.5 Install and Upgrade Guide

Installing Cisco Elastic Services Controller on OpenStack

Configuring the ESC HA Active/Standby Virtual IP Address

Note KeepAlived doesn’t support single IPv4 VIP address with a IPv6 VRRP instance.

Example configuration steps are shown below:

./bootvm.py <esc_ha1> \

--user_pass <username>:<password>

--user_confd_pass <username>:<password>

--image <image_id> \

--net <net-name> \

--gateway_ip <default_gateway_ip_address> \

--ipaddr <ip_address1> <ip_address2> \

--ha_node_list < IP addresses HA nodes1> < IP addresses for HA nodes2> \

--kad_vip <keepaliveD VIP of the HA nodes and the interface for keepaliveD VIP> \ (for example: --kad_vip 192.0.2.254:eth2)

--kad_vri <virtual router id of vrrp instance>

--kad_vif <virtual IP of the HA nodes or the interface of the keepalived VRRP> \ (for example: --kad_vif eth1 )

--ha_mode <HA installation mode> \

--route <routing configuration> \ (for example:192.0.2.254/24:192.168.0.1:eth1 )

--avail_zone nova:<openstack zone> \

Similarly, a three network interface can be configured for ESC HA Active/Standby nodes. An example three interfaces configuration is shown below with the following assumptions :

• Network 1 is an IPv6 network used for northbound connection. ESC VIP is allocated in this network and the Orchestrator send requests to ESC through ESC VIP.

• Network 2 is an IPv4 network used for ESC sync traffic (DRDB synchronization) and VRRP heartbeat.

This network is also used for OpenStack connection and VNF monitoring.

• Network 3 is another IPv4 network used for management. The SA, rsyslog, etc. can use this network to manage ESC.

./bootvm.py esc-ha-0 --image ESC-2_2_x_yyy --net network-v6 network --gateway_ip 192.168.0.1 --ipaddr

2001:cc0:2020::fa 192.168.0.239 192.168.5.239 --ha_node_list 192.168.0.239 192.168.0.243 --kad_vip

[2001:cc0:2020::fc/48]:eth0 --kad_vif eth1 --ha_mode drbd --route 172.16.0.0:eth1 --avail_zone nova: zone name

./bootvm.py esc-ha-1 --image ESC-2_2_x_yyy --net network-v6 network lab-net-0 --gateway_ip 192.168.0.1

--ipaddr 2001:cc0:2020::fa 192.168.0.239 192.168.5.239 --ha_node_list 192.168.0.239 192.168.0.243 --kad_vip

[2001:cc0:2020::fc/48]:eth0 --kad_vif eth1 --ha_mode drbd --route 172.16.0.0:eth1 --avail_zone nova: zone name

Configuring the ESC HA Active/Standby Virtual IP Address

In this option, the value of kad_vip argument should be a virtual IP, which allows the service portal and the northbound to access the active ESC and send requests to ESC HA Active/Standby service through virtual IP

(VIP).

If northbound and both ESC HA Active/Standby nodes are located in the same network, you can connect directly through the virtual IP (VIP). If northbound doesn't sit on the same network as ESC HA Active/Standby, assign a floating IP to ESC HA Active/Standby VIP using the procedure below:

Create a port with the VIP address (kad_vip) in the same network as ESC's kad_vip connects.

neutron port-create network --name network_vip --fixed-ip subnet_id=network-subnet,ip_address=192.168.0.87

Cisco Elastic Services Controller 5.5 Install and Upgrade Guide

Installing Cisco Elastic Services Controller on OpenStack

Configuring the ESC L3 HA Active/Standby With BGP

Deploy ESC HA Active/Standby . See Configuring High-Availability Active/Standby section in Installing

ESC on OpenStack.

Note Make sure the kad_vip using the same IP address as the port created above.

Associate a floating IP with the port created above. The first uuid is the floating ip id and the second one is the port id.

neutron floatingip-associate <floating IP> <port ID>

Access ESC HA Active/Standby through the floating IP and it will connect to the ESC active node.

For the portal access, make sure the keepalive network is accessible by your browser and the virtual IP is the IP address to access the portal of the active node.

For example, if the VIP is 192.0.2.254, access ESC HA Active/Standby portal with https://192.0.2.254:9001/.

Configuring the ESC L3 HA Active/Standby With BGP

To configure BGP for ESC HA Active/Standby, there are two options:

Directly booting ESC HA Active/Standby L3 with BGP

Using post configuration from existing ESC HA Active/Standby pair

To configure BGP for ESC HA Active/Standby, the following network parameters are required:

• BGP remote IP

• IP of the interface for BGP anycast routing

• BGP local AS number for routing configuration

• BGP remote AS number for routing configuration

• BGP routing configuration

• --bgp_local_ip

• --bgp_local_router_id

Note You must configure BGP router with neighbors, and restart it. Verify that the router is able to ping the AnyCast

IP.

On the BGP router, set two neighbors. The below BGP configuration is designed for Bird router. The configuration is router specific. For each types of router, the procedure is different:

The below configurations are given according to the bootvm command : protocol bgp E3 from EXABGP { neighbor 198.18.42.222 as 65012;

}

Cisco Elastic Services Controller 5.5 Install and Upgrade Guide

Installing Cisco Elastic Services Controller on OpenStack

Configuring the ESC L3 HA Active/Standby With BGP protocol bgp E4 from EXABGP { neighbor 198.18.61.222 as 65011;

}

Booting an ESC VM with BGP options

[admin@na-test-52-1 ~]$ health.sh

============== ESC HA (ACTIVE) with DRBD ================= vimmanager (pgid 41908) is running monitor (pgid 42067) is running mona (pgid 42130) is running drbd (pgid 38488) is active snmp (pgid 2121) is running etsi (pgid 43247) is running pgsql (pgid 42566) is running keepalived (pgid 40281) is running portal (pgid 43307) is running confd (pgid 25644) is running filesystem (pgid 0) is running escmanager (pgid 42775) is running

=======================================

ESC HEALTH PASSED

[admin@na-test-52-2 ~]$ health.sh

============== ESC HA (Standby) with DRBD ================= pgsql is stopped vimmanager is stopped monitor is stopped mona is stopped drbd (pgid 2471) is standby etsi is disabled at startup filesystem is stopped snmp is disabled at startup bgp is stopped keepalived (pgid 2787) is running portal is stopped confd is stopped escmanager is stopped

=======================================

ESC HEALTH PASSED

Use below values for BGP post configuration:

./bootvm.sh <NETWORK_VM_name> \

--image <ESC_image> \

--ipaddr <static_IP_address1> <IP_address2> <IP_address_3>\

--gateway_ip <gateway IP address of NETWORK> \

--net <net_id1> <net_id2> <net_id3> \

--network_params_file <network_params_file> \

--host_mapping_file <host_mapping_file> \

--avail_zone <openStack_zone> \

--ha_node_list <IP_address_ha_node_1> <IP_address_ha_node_2> \

--user_portal_pass <user>:<password> \

--user_rest_pass <user>:<password> \

--confd_aes_key <password> \

--kad_vif <interface> \

--user_confd_pass <user>:<password> \

--user_pass <user>:<password> \

--kad_vip <vip address> \

--bgp_remote_ip <BGP_remote_IP_address> \

--bgp_local_ip <BGP_local_IP_address> \

--bgp_local_as <BGP_local_AS_#> \

--bgp_remote_as <BGP_remote_AS_#>\

--bgp_local_router_id <local_BGP_reouter_id> \

--bgp_anycast_ip <BGP_anycast_IP> \

Cisco Elastic Services Controller 5.5 Install and Upgrade Guide

Installing Cisco Elastic Services Controller on OpenStack

Configuring the ESC L3 HA Active/Standby With BGP

--bgp_md5 <BGP_MD5>

Where,

--ip_addr: ----> the local IP address of the ESC VM

--net: ----> the network id(s) in OpenStack that ESC will connect to.

--bgp_anycast_ip: ----> the IP address that NCS will communicate with

--bgp_remote_ip: ----> this IP address of the external router that ESC will peer with

--bgp_local_as: ----> local AS for the ESC "router"

--bgp_remote_as: ----> AS number for the external router ESC will peer with

--bgp_local_router_id: ----> id for the esc "router"

--bgp_md5: ----> optional - md5 to be used to pair with external router

Configuring BGP HA Active/Standby Post Configuration

For each HA Active/Standby instance, create the network interface file:

# cat /etc/sysconfig/network-scripts/ifcfg-lo:2

IPV6INIT='no'

IPADDR='10.0.124.124'

BROADCAST='10.0.124.255'

<------------------ bgp anycast IP

NETWORK='10.0.124.0'

NETMASK='255.255.255.0'

DEVICE='lo:2'

ONBOOT='yes'

NAME='loopback'

For each HA Active/Standby instance:

Bring lo:2 up

# ifup lo:2

To configure BGP for ESC HA Active/Standby, use the escadm tool in ESC Virtual Machine, as shown below:

$ sudo bash

# escadm bgp set --local_ip LOCAL_IP --anycast_ip ANYCAST_IP --remote_ip REMOTE_IP --local_as

LOCAL_AS --remote_as REMOTE_AS

--local_router_id LOCAL_ROUTER_ID

# escadm reload

# reboot

Example:

[root@bgp-001 admin]# escadm bgp set --local_ip 198.18.42.124 --anycast_ip 10.0.124.124

--remote_ip 192.168.0.2 --local_as 65124 --remote_as 65000 --local_router_id 198.18.42.124

[root@bgp-002 admin]# escadm bgp set --local_ip 198.18.42.125 --anycast_ip 10.0.124.124

--remote_ip 192.168.0.2 --local_as 65114 --remote_as 65000 --local_router_id 198.18.42.125

Configuring a BGP Router

To configure a BGP router, log in to the BGP router to configure BGP Anycast routing. The required parameters are:

<Router_AS_#> same as --bgp_remote_as above

<Esc_ip_address> must be the ESC VM's IP address configured for BGP advertisement.

<ESC_AS_#> same as --bgp_local_as shown above configure router bgp <Router_AS_#> neighbor <ESC_IP_address>

Cisco Elastic Services Controller 5.5 Install and Upgrade Guide

Installing Cisco Elastic Services Controller on OpenStack

Important Notes remote-as <ESC_AS_#> address-family ipv6 unicast route-policy anycast-in in route-policy anycast-out out route-policy anycast-in pass end-policy route-policy anycast-out drop end-policy commit

Important Notes

• ESC HA Active/Standby

• An HA Active/Standby failover takes about 2 to 5 minutes. The ESC service will not be available during the switchover time.

• When the switchover is triggered during transactions, all incomplete transactions will be dropped.

The requests should be re-sent by northbound if it does not receive any response from ESC.

• External Storage

• If the active ESC instance is suspended by OpenStack command, the switch over will be triggered but the cinder volume won't be attached to the new active ESC instance. This is not a valid use case for ESC HA Active/Standby.

• Internal Storage

• Two ESC instances have to be deployed to establish the HA Active/Standby solution. The ESC HA

Active/Standby will start to work when both ESC instances are successfully deployed and are able to connect to each other. If you just deploy one ESC instance with HA Active/Standby parameters, the ESC instance keeps Switching-to-Active state and will not be able to provide any service until it reaches its peer.

• Split-brain scenario can still happen in this ESC HA Active/Standby solution, although the chance is very low.

• ETSI-specific Notes

ESC supports ETSI MANO northbound API defined by the European Telecommunications Standards

Institute (ETSI) for NFV Management and Orchestration. The ETSI MANO API is another programmatic interface based on the REST architecture. For more information, see ETSI MANO Compliant Lifecycle

Operations in the Cisco Elastic Services Controller User Guide . Consider the following notes while enabling ETSI service on ESC which is in HA Active/Standby mode:

• The server.address value in the etsi-vnfm.properties

file must be set to a Virtual IP (VIP) address.

This IP address can be used to communicate back to the ETSI services using API callbacks. If the virtual IP address is not specified, the ETSI service startup may fail.

Cisco Elastic Services Controller 5.5 Install and Upgrade Guide

Installing Cisco Elastic Services Controller on OpenStack

Troubleshooting High Availability Active/Standby

• The ETSI VNFM service and the escadm script generate and maintain the security.user.name and security.user.password property values. You should not change it manually.The

security.user.password is encoded.

Troubleshooting High Availability Active/Standby

• Check for network failures. If a network problem occurs, you must check the following details:

• The IP address assigned is correct, and is based on the OpenStack configuration.

• The gateway for each network interface must be pingable.

• Check the logs for troubleshooting:

• The ESC Admin logs at /var/log/esc/escadm.log

• The ESC manager log at /var/log/esc/escmanager.log

• The AA elector log at /var/log/esc/elector-{pid}.log

• Check for DRBD (Replication based ESC HA Active/Standby) for Internal Storage solution:

• Check the DRBD configuration file at

/etc/drbd.d/esc.res

• Access the DRBD log

•

/var/log/messages|grep drbd

• To collect log files through CLI, use the following command on all ESC nodes: sudo escadm log collect

Cisco Elastic Services Controller 5.5 Install and Upgrade Guide

C H A P T E R

Cisco Elastic Services Controller Active/Active

High Availability Overview

This chapter contains the following sections:

•

Cisco Elastic Services Controller Active/Active HA Overview, on page 33

•

ESC Active/Active Architecture, on page 34

Cisco Elastic Services Controller Active/Active HA Overview

ESC supports High Availability (HA) in the form of an Active/Active model. ESC Active/Active HA has three VMs as a cluster in one datacenter.. There are two datacenters. Between the two datacenters, one datacenter acts as active, another acts as standby. ESC Active/Active HA uses Openstack heat template to deploy the three VM cluster in a datacenter.

In a datacenter, ESC service runs on each VM; however, there is only one ESC on a datacenter that runs as cluster leader. DB service runs only on the leader. The ESC services on the other two ESC VMs run as cluster follower. The DB service is only active on ESC services leader VM.

DRBD synchronises the data among ESC VMs. The ESC service on the three ESC VMs connect to the active

DB service. When a leader switchover happens, all the ESC service connects to the newly active DB service.

Cisco Elastic Services Controller 5.5 Install and Upgrade Guide

ESC Active/Active Architecture

ESC Active/Active Architecture

Figure 2: Cisco Elastic Services Controller Active/Active Architecture

Installing Cisco Elastic Services Controller on OpenStack

Cisco Elastic Services Controller 5.5 Install and Upgrade Guide

C H A P T E R

Installing Active/Active High Availability Cluster

This chapter contains the following sections:

•

Installing Active/Active High Availability Cluster , on page 35

•

Validating Active/Active High Availability Cluster Post Installation, on page 37

•

Adding Default VIM Connector to the Active/Active High Available Cluster, on page 38

•

Adding BGP in an Active/Active Cluster, on page 38

Installing Active/Active High Availability Cluster

To configure the Active/Active HA cluster, you can run the following commands on OpenStack, where openrc my-server is openrc for the OpenStack, and test is the stack name.

source ~/elastic-services-controller/esc-bootvm-scripts/openrc my-server-42 openstack stack create test -t aa.yaml

To check the status of the stack, use the following commands:

• openstack stack list

• openstack stack show test

• openstack stack event list test

Once the stack is in

CREATE_COMPLETE status, you can ssh into the VMs. The openstack stack show test command lists the IP addresses for the 3 VMs, you can use them to access to the VMs.

Note The Active/Active HA Cluster is deployed on OpenStack only.

Setting up the User Configuration

You can configure some of the parameters based on your settings, such as network, subnet, using either static

IP or DHCP to allocate IP address, flavor, image, password, etc. The configurable parameters are available in aa-param.yaml

, Openstack heat environment file, when the ESC cluster is instantiated via heat template

( aa.yaml

To instantiate ESC cluster, use the following example:

Cisco Elastic Services Controller 5.5 Install and Upgrade Guide

Installing Cisco Elastic Services Controller on OpenStack

Setting up the User Configuration openstack stack create name -t aa.yaml -e aa-params.yaml

The following example shows how to use a static IP address to configure the port while using environment template: sample@my-server-39:~/aa4.5/apr15$ more aa-params.yaml

parameters: network_1_name: network subnet_name: subnetwork esc_1_ip: 172.23.0.228

esc_2_ip: 172.23.0.229

esc_3_ip: 172.23.0.230

Following are the user configurable parameters available in aa.yaml : parameters: network_1_name: type: string description: Name of the image default: network subnet_name: type: string description: subnet name esc_1_ip: type: string description: static IP address of esc-1 VM.

esc_2_ip: type: string description: static IP address of esc-2 VM.

esc_3_ip: type: string description: static IP address of esc-3 VM.

resources: esc_1_port: type: OS::Neutron::Port properties: network_id: { get_param: network_1_name } fixed_ips: [ { "subnet": { get_param: subnet_name}, "ip_address": { get_param: esc_1_ip

} } ] esc_2_port: type: OS::Neutron::Port properties: network_id: { get_param: network_1_name } fixed_ips: [ { "subnet": { get_param: subnet_name}, "ip_address": { get_param: esc_2_ip

} } ] esc_3_port: type: OS::Neutron::Port properties: network_id: { get_param: network_1_name } fixed_ips: [ { "subnet": { get_param: subnet_name}, "ip_address": { get_param: esc_3_ip

} } ]

...omitting...

The following example shows how to use configurable image, flavor, and vm name prefix from environment template. Ensure that the flavor used suits the ESC deployment requirement.

sample@my-server-39:~/aa4.5/apr15$ more aa-params.yaml

parameters:

Cisco Elastic Services Controller 5.5 Install and Upgrade Guide

Installing Cisco Elastic Services Controller on OpenStack

Validating Active/Active High Availability Cluster Post Installation nameprefix: abc image_name: ESC-5_0_DEV_4 flavor_name: m1.large

sample@my-server-39:~/aa4.5/apr15$

The following example shows how to use configurable image, flavor, and vmnameprefix from heat template: parameters: nameprefix: type: string description: Name prefix of vm default: helen image_name: type: string description: Name of the image default: ESC-5_0_DEV_4 flavor_name: type: string description: Name of the image default: m1.large

esc-1: type: OS::Nova::Server properties: name: str_replace: template: $nameprefix-esc-1 params:

$nameprefix : { get_param: nameprefix } image: { get_param: image_name } flavor: { get_param: flavor_name }

... omitting...

Validating Active/Active High Availability Cluster Post

Installation

To verify all the ESC nodes use the following commands. Here, all ESC nodes implies each VMs.

sample@my-server-39:~$ openstack --insecure server list | grep abc

| 5ea6fc79-2b2a-4064-9c6a-a83d6b06c225 | abc-test-esc-3

| ACTIVE | network=172.23.7.203

| ESC-5_0_DEV_13 | m1.large

| 10e165d9-5015-4b64-88fe-19e874e6e7c1 | abc-test-esc-1

| ACTIVE | network=172.23.7.205

| ESC-5_0_DEV_13 | m1.large

| 35f6bad1-865f-4155-8411-d37e2616e079 | abc-test-esc-2

| ACTIVE | network=172.23.7.204

| ESC-5_0_DEV_13 | m1.large

To find out the leader node, ssh into one of the nodes/VMs and run the following:

[admin@sample-test-esc-1 ~]$ sudo escadm elector dump

{

"[email protected]:42143": {

"state": "FOLLOWER",

"location": "[email protected]:42143",

"service": "esc_service"

"[email protected]:50474": {

"state": "FOLLOWER",

Cisco Elastic Services Controller 5.5 Install and Upgrade Guide

Installing Cisco Elastic Services Controller on OpenStack

Adding Default VIM Connector to the Active/Active High Available Cluster

}

"location": "[email protected]:50474",

"service": "esc_service"

"[email protected]:59514": {

"state": "LEADER",

"location": "[email protected]:59514",

"service": "esc_service"

}

Adding Default VIM Connector to the Active/Active High

Available Cluster

You can add a default vim connector to the 3 ESC VM cluster in the following two different ways:

Once the 3 ESC VM cluster boots up, use netconf interface to add a default vim connector by using the following command. The vim.xml

is the default vim connector deployment file.

[admin@name-esc-1 ~]$ esc_nc_cli --host db.service.consul --user admin --password

<admin_password> edit-config vim.xml

To configure a default vim connector, you must add the default vim connector configuration inside the heat template day0 file. Add the following block in the write_files section under the cloud-config in aa-day0.yaml file. Once the 3 ESC VM cluster boots up, it creates a default vim connector by its own.

The following example shows how to configure default vim connector in heat template day0 file:

- path: /opt/cisco/esc/esc-config/esc_params.conf

content: | openstack.os_auth_url=http://172.16.103.38:35357/v3 openstack.os_project_name=admin openstack.os_tenant_name=admin openstack.os_user_domain_name=default openstack.os_project_domain_name=default openstack.os_identity_api_version=3 openstack.os_image_api_version=2 openstack.os_username=admin openstack.os_password=password1

Adding BGP in an Active/Active Cluster

To initiate the BGP process, add the anycast IP to the lo device. You can configure it in sys-cfg.yaml

For example:

#cloud-config write_files:

- path: /etc/cloud/cloud.cfg.d/sys-cfg.yaml

content: | network: version: 1 config:

- type: physical name: lo subnets:

- type: static

Cisco Elastic Services Controller 5.5 Install and Upgrade Guide

Installing Cisco Elastic Services Controller on OpenStack

Adding BGP in an Active/Active Cluster address: 172.23.188.188/23

You must specify the advertise IP for consul. In esc-config.yaml

, add the following: consul: advertise_addr: 172.23.1.149

Following is the example for adding the BGP section: bgp: depend_on: elector:leader anycast_ip: 172.23.188.188/23 local_as: '65001' local_ip: 192.168.1.11

local_router_id: 192.168.1.11

remote_as: '65000' remote_ip: 192.168.1.12

Cisco Elastic Services Controller 5.5 Install and Upgrade Guide

Adding BGP in an Active/Active Cluster

Installing Cisco Elastic Services Controller on OpenStack

Cisco Elastic Services Controller 5.5 Install and Upgrade Guide

C H A P T E R

Managing Cluster in ESC Active/Active High

Availability

This chapter contains the following section:

•

Managing Cluster in ESC Active/Active High Availability, on page 41

Managing Cluster in ESC Active/Active High Availability

To do cluster management in ESC Active/Active HA, you can call some escadm commands on any ESC node and execute it on all nodes in Active/Active cluster.

The following are the supported commands for cluster level calling:

• escadm start

• escadm stop

• escadm geo start/stop

• escadm vim show

Add the option

--cluster to execute any of the previous commands to be executed on all nodes in

Active/Active local cluster:

For example: sudo escadm geo start --v –cluster

The output of the command and exit code of each node are displayed with the IP address of that node to distinguish the execution output, and the output on the local node.

Example for escadm geo start --cluster

[root@name-geo-2-1 admin]# escadm geo start --cluster

192.168.1.13 # remote host exit status : 0

Starting geo service: [OK]

192.168.1.12 # remote host exit status : 0

Starting geo service: [OK]

Starting geo service: [OK] # output of the local node

Cisco Elastic Services Controller 5.5 Install and Upgrade Guide

Managing Cluster in ESC Active/Active High Availability

Installing Cisco Elastic Services Controller on OpenStack

Cisco Elastic Services Controller 5.5 Install and Upgrade Guide

C H A P T E R

Configuring GEO in Active/Active High

Availability

•

Configuring GEO in Active/Active High Availability, on page 43

•

Verifying GEO Services, on page 45

•

Active/Active GEO HA Failure Injection Limitations, on page 46

Configuring GEO in Active/Active High Availability

ESC Active/Active HA has three VMs as a cluster in one datacenter. The second datacenter consists of

GEO-HA.

Following are the 6 pre-defined roles in GEO:

init

: initial role of geo service

pre_primary

primary

pre_secondary

secondary

unknown

: used when consul is not reachable

GEO can change one role to another. Transitions are defined in esc-config.yaml

. Each transition is divided into the following 3 parts:

• from

: current role

• goto

: destination role

• condition

: when GEO changes the role

Transition Conditions

When A/A HA Geo comes up, the primary datacenter has to go through init

, pre_primary

, and primary states; meanwhile, the secondary datacenter has to go through init, pre-secondary, and secondary states. When

Cisco Elastic Services Controller 5.5 Install and Upgrade Guide

Installing Cisco Elastic Services Controller on OpenStack

Transition Conditions all ESC VMs' health check pass on both primary and secondary datacenters, the ESC A/A HA GEO is up and running. It is ready for use.

Condition Functions

The following are all supported condition functions:

return

: do nothing but return the argument

and

: return true if all arguments are true

: return true if any of the argument is true

len

: return the length of the argument

equals

: return true if all arguments are equal

true

: return true if args can be tested for truth value in python

false

: opposite to 'true'

The following are samples for GEO configurations on primary datacenter: on_init: consul start on_primary: start on_secondary: stop on_stop: consul stop startup: manual transitions:

- condition: return: and:

- equals:

- len: service1

- 3

- equals:

- len: service2

- 3 rise: 3 service1: dc: dc1 name: consul_agent passing: true type: service service2: dc: dc2 name: geo passing: true type: service from: init goto: primary

- condition: fall: 2 return: equals:

- len: service

- 3 service: dc: dc1 name: consul_agent from: primary goto: secondary

The following are samples for GEO configurations on secondary datacenter:

Cisco Elastic Services Controller 5.5 Install and Upgrade Guide

Installing Cisco Elastic Services Controller on OpenStack

Verifying GEO Services on_init: consul start on_primary: start on_secondary: stop on_stop: consul stop startup: manual transitions:

- condition: return: and:

- equals:

- len: service1

- 3

- equals:

- len: service2

- 3 rise: 3 service1: dc: dc1 name: consul_agent passing: true type: service service2: dc: dc2 name: geo passing: true type: service from: init goto: secondary

- condition: fall: 2 return: equals:

- len: service

- 3 service: dc: dc1 name: consul_agent from: secondary goto: primary

Verifying GEO Services

To start Active/Active GEO-HA, run the following command: escadm geo start

To verify the GEO status, use the following command:

[root@test-geo3-ha-1 esc-scripts]# escadm geo status geo (pgid 3745) is primary

The verify the GEO services in current datacenter, use the following command:

[root@test-geo3-ha-1 esc-scripts]# escadm geo dump

{

"[email protected]:44793": {

"role": "primary",

"location": "[email protected]:44793",

"service": "geo"

"[email protected]:52459": {

"role": "primary",

"location": "[email protected]:52459",

Cisco Elastic Services Controller 5.5 Install and Upgrade Guide

Installing Cisco Elastic Services Controller on OpenStack

Active/Active GEO HA Failure Injection Limitations

"service": "geo"

"[email protected]:38841": {

"role": "primary",

"location": "[email protected]:38841",

"service": "geo"

}

To verify all the GEO services in the datacenters, use the following command:

[root@test-geo4-ha-1 admin]# escadm geo dump --all

{

"[email protected]:36760": {

"role": "primary",

"location": "[email protected]:36760",

"service": "geo"

"[email protected]:42362": {

"role": "secondary",

"location": "[email protected]:42362",

"service": "geo"

"[email protected]:51936": {

"role": "primary",

"location": "[email protected]:51936",

"service": "geo"

"[email protected]:37604": {

"role": "secondary",

"location": "[email protected]:37604",

"service": "geo"

"[email protected]:44450": {

"role": "primary",

"location": "[email protected]:44450",

"service": "geo"

"[email protected]:34875": {

"role": "secondary",

"location": "[email protected]:34875",

"service": "geo"

}

Active/Active GEO HA Failure Injection Limitations

The ESC Active/Active GEO HA is enhanced to support a one-way GEO HA failover with a maintenance window to move it back to healthy state.

If a GEO failover happens and the ESC VMs move to the unhealthy state, use the following steps to bring

ESC A/A GEO HA back to the healthy state through manual intervention:

Procedure

Step 1 Resolve any issues in Datacenter1 (DC1), that may have caused the failures and enabled the GEO switch to

Datacenter2 (DC2).

Cisco Elastic Services Controller 5.5 Install and Upgrade Guide

Installing Cisco Elastic Services Controller on OpenStack

Active/Active GEO HA Failure Injection Limitations

Step 2

Step 3

Step 4

Step 5

Ensure that the consul is running in at least 2 nodes in DC1 and DC2.

Run the sudo escadm geo replicate –all command on DC2 when DC1 has at least two nodes with consul in the running state.

Run the sudo escadm stop command on all the 6 ESC VMs.

Run the sudo escadm geo restart command on all the 6 ESC VMs.

What to do next

Note ESC does not support any operations on ESC VMs in DC1 after the GEO HA fails over to DC2.

Cisco Elastic Services Controller 5.5 Install and Upgrade Guide

Active/Active GEO HA Failure Injection Limitations

Installing Cisco Elastic Services Controller on OpenStack

Cisco Elastic Services Controller 5.5 Install and Upgrade Guide

C H A P T E R

DRBD Encryption for ESC Active/Standby and

Active/Active HA Data Replication

This chapter contains the following sections:

•

DRBD Encryption for ESC HA Data Replication , on page 49

•

ESC HA with DRBD Encryption, on page 49

DRBD Encryption for ESC HA Data Replication

ESC uses DRBD for data replication across different nodes in an HA cluster environment. DRBD layers logical block device over existing local block devices on cluster nodes.

The written data to the active node is transferred to the lower-level block device and simultaneously propagated to the secondary node(s). Currently ESC mounts DRBD device directly on

/opt/cisco/esc/esc_database

For Example:

# df

Filesystem devtmpfs tmpfs tmpfs

…

1K-blocks

2961760

2972164

Used Available Use% Mounted on

8748

2961760

2972160

2963416

0% /dev

1% /dev/shm

1% /run tmpfs 594436 0 594436 0% /run/user/1004

/dev/mapper/esc_crypt 3028620 57212 2797848 3% /opt/cisco/esc/esc_database

Block device encryption encrypts or decrypts the data transparently as it is written/read from block devices, the underlying block device sees only encrypted data.

Security is enhanced with the dm-crypt/LUKS layer to encrypt the data in DRBD partition, between filesystem and DRBD device. LUKS (Linux Unified Key Setup) is a specification for block device encryption.

ESC HA with DRBD Encryption

The following bootvm commands boot ESC HA with DRBD encrypted:

Select DRBD LUKS encryption through bootvm.py. There are 4 variations that result in the equivalent ESC day-0 user-data/esc-config.yaml when passed to the ESC VM instance.

Cisco Elastic Services Controller 5.5 Install and Upgrade Guide

Installing Cisco Elastic Services Controller on OpenStack

ESC HA with DRBD Encryption bootvm.py --fs_encryption_type luks --fs_luks_key_prompt bootvm.py --fs_encryption_type luks --fs_luks_key 'LuksKeyValue'

=> injects the luks key into default file location /opt/cisco/esc/esc-config/luks_key bootvm.py --fs_encryption_type luks --file root:0400:/opt/cisco/esc/esc-config/luks_key:path-to-local-luks-key-file

=> injects a local file containing the luks key

The following command shows an advanced usage to manage the luks key file at a different path on the ESC

VM filesystem: bootvm.py --fs_encryption_type luks --fs_luks_key_file path-on-esc-vm-luks-key-file

--fs_luks_key_prompt bootvm.py --fs_encryption_type luks --fs_luks_key_file path-on-esc-vm-luks-key-file

--fs_luks_key 'LuksKeyValue'

=> injects the luks key into a different file location bootvm.py --fs_encryption_type luks --fs_luks_key_file path-on-esc-vm-luks-key-file --file root:0400:path-on-esc-vm-luks-key-file:path-to-local-luks-key-file

=> injects the luks key as read from a local file into a different file location

Use the following commands if you are installing ESC with a custom user-data, such as ESC Active/Active deployment with heat templates:

The luks key is specified as a day-0 file and an attribute under esc-config.yaml / filesystem.

Encode the luks key as base64: base64 <<<'LuksKeyValue'

THVrc0tleVZhbHVlCg==

Then, insert the previous luks key into the user-data / cloud-config file: write_files:

- path: /opt/cisco/esc/esc-config/luks_key owner: root:root permissions: '400' encoding: b64 content: THVrc0tleVZhbHVlCg==

- path: /opt/cisco/esc/esc-config/esc-config.yaml

owner: root:esc-user permissions: '0640' content: | resources: filesystem: depend_on: drbd:active encryption_type: luks luks_key_file: /opt/cisco/esc/esc-config/luks_key

Cisco Elastic Services Controller 5.5 Install and Upgrade Guide

C H A P T E R

Upgrading ESC Active/Active High Availability

This chapter contains the following sections:

•

Upgrading ESC Active/Active High Availability, on page 51

Upgrading ESC Active/Active High Availability

Cisco Elastic Service Controller Active/Active HA supports local Active/Active to Active/Active simple upgrade.

Local Active/Active to Active/Active Simple Upgrade

Procedure

Step 1

Step 2

Step 3

Step 4

Back up the database. For more information, see the section,

Backing up the Database, on page 51

Remove old VMs. For more information, see the section,

Removing the Old VMs, on page 52 Removing

Old VMs .

Install new ESC Active/Active VMs. For more information, see the section,

Installing a New ESC Active/Active

VM, on page 52 .

Restore the ESC database. For more information, see the section,

Restoring the ESC Database, on page 53 .

Backing up the Database

Before upgrade, take the back up of database using the following procedure.

Procedure

Step 1

Step 2

Put the ESC leader VM in the maintenance mode, by running the following command: escadm op_mode set --mode=maintenance

Wait until all the ESC VMs stop processing transactions. To verify, run the following command: escadm ip_trans

Cisco Elastic Services Controller 5.5 Install and Upgrade Guide

Installing Cisco Elastic Services Controller on OpenStack

Removing the Old VMs

Step 3

Step 4

Create a backup of database on the ESC leader by running the following command: escadm backup --file dbback.tar, scp <dbback.tar>

Collect logs from all the ESC VMs by running the following command: escadm log collect scp

Removing the Old VMs

Procedure

Step 1

Step 2

Shutdown all the ESC follower VMs and the ESC leader VM running the following command: nova stop

Remove old ESC Active/Active VMs from the OpenStack by running the following command: openstack stack delete {stack name}

Installing a New ESC Active/Active VM

After backing up the database and shutting down of the old ESC Active/Active VMs, a new/upgraded (based on new ESC package) Active/Active ESC VM must be installed.

Procedure

Step 1

Step 2

Step 3

Step 4

For OpenStack, register a new image by running the following command: glance image-create

Install the new ESC Active/Active VMs by running the following command: openstack stack create {stack name} --template {location of the template file}

Check all the ESC VMs health, and stop the escadm service in the follower VMs by running the following command: sudo escadm stop for all followers VMs

Once the escadm service is stopped in all the follower VMs, stop the escadm service in the leader VM by running the following command: sudo escadm stop

Cisco Elastic Services Controller 5.5 Install and Upgrade Guide

Installing Cisco Elastic Services Controller on OpenStack

Restoring the ESC Database

Restoring the ESC Database

Restore the ESC database on the new ESC instance, using the following procedure:

Procedure

Step 1

Step 2

Step 3

Step 4

Step 5

Copy the backup file to the new leader by running the folowing command: scp

Restore the database on the ESC leader by running the following command: sudo escadm restore --file <dbback.tar>

After restoring, the restore process starts the escadm service in the leader VM. However, the escadm service remains to be stopped in all the follower VMs.

Verify that the ESC leader VM is running all the services without any interruption.

Put the ESC leader VM into the operation mode by running the following command: sudo escadm op_mode set --mode=operation

Start the ESC services on the follower VMs by running the following command: sudo escadm start

Cisco Elastic Services Controller 5.5 Install and Upgrade Guide

Restoring the ESC Database

Installing Cisco Elastic Services Controller on OpenStack

Cisco Elastic Services Controller 5.5 Install and Upgrade Guide

Anúncio

Installing Cisco Elastic Services Controller on

OpenStack

Prerequisites

Virtual Resource Requirements

Software Requirements

Preparing for the Installation

Installing Cisco Elastic Services Controller on

OpenStack

Installation Scenarios

Main Components of Cisco Elastic Services Controller Setup

Installing Cisco Elastic Services Controller Using the QCOW

Image

Additional Installation Options

Managing Root Certificates in Cisco Elastic Services Controller

Enabling/Disabling the Root Certificate Validation

Adding a Root Certificate

Removing a Root Certificate

Managing Root Certificates During the Upgrade

Managing Keystore in Cisco Elastic Services Controller

ESCADM Keystore Commands

Using a Bootable Volume in ESC Installation

Installing High Availability Active/Standby

High Availability Active/Standby Overview

ESC Active/Standby Architecture

How High Availability Works

Deploying ESC High Availability Active/Standby

Deploying ESC in High Availability Active/Standby Mode on Internal Storage

Deploying ESC in High Availability Active/Standby Mode on Replicate External Storage

Configuring the Northbound Interface Access

Configuring ESC HA Active/Standby with Multiple Interfaces

Configuring the ESC HA Active/Standby Virtual IP Address

Configuring the ESC L3 HA Active/Standby With BGP

Important Notes

Troubleshooting High Availability Active/Standby

Cisco Elastic Services Controller Active/Active

High Availability Overview

Cisco Elastic Services Controller Active/Active HA Overview

ESC Active/Active Architecture

Installing Active/Active High Availability Cluster

Installing Active/Active High Availability Cluster

Setting up the User Configuration

Validating Active/Active High Availability Cluster Post

Installation

Adding Default VIM Connector to the Active/Active High

Available Cluster

Adding BGP in an Active/Active Cluster

Managing Cluster in ESC Active/Active High

Availability

Managing Cluster in ESC Active/Active High Availability

Configuring GEO in Active/Active High

Availability

Configuring GEO in Active/Active High Availability

Transition Conditions

Verifying GEO Services

Active/Active GEO HA Failure Injection Limitations

DRBD Encryption for ESC Active/Standby and

Active/Active HA Data Replication

DRBD Encryption for ESC HA Data Replication

ESC HA with DRBD Encryption

Upgrading ESC Active/Active High Availability

Upgrading ESC Active/Active High Availability

Backing up the Database

Removing the Old VMs

Installing a New ESC Active/Active VM

Restoring the ESC Database

Relacionado

Cisco Elastic Services Controller Install and Upgrade Guide

Cisco Elastic Services Controller Install and Upgrade Guide

Cisco Elastic Services Controller Install and Upgrade Guide

Cisco Elastic Services Controller Install and Upgrade Guide

Cisco Elastic Services Controller Install and Upgrade Guide

Cisco Elastic Services Controller Install and Upgrade Guide

Cisco Elastic Services Controller Install and Upgrade Guide

Cisco Elastic Services Controller Install and Upgrade Guide

Cisco Elastic Services Controller Install and Upgrade Guide

Cisco Elastic Services Controller Install and Upgrade Guide

Cisco Elastic Services Controller Install and Upgrade Guide

Cisco Elastic Services Controller Install and Upgrade Guide