Troubleshooting

Troubleshooting

This chapter provides procedures for troubleshooting the problems encountered during RMS Installation.

•

Regeneration of Certificates, page 1

•

Deployment Troubleshooting , page 7

Regeneration of Certificates

Following are the scenarios that requires regeneration of certificates:

• Certificate expiry (Certificate will have a validity of one year.)

• If importing certificates are not successful.

Follow the steps to regenerate self-signed certificates:

Certificate Regeneration for DPE

To address the problems faced during the certificate generation process in Distributed Provisioning Engine

(DPE), complete the following steps:

Step 1

Remove the root.ca.cer, server.ca.cer and client.ca.cer certificates that are installed in DPE.

Enter: ssh login to Serving_Node_1

Change to root user

Navigate to the conf folder cd /rms/app/CSCObac/dpe/conf/self_signed ls -lrt

Output:

[root@rms-aio-serving self_signed]# ls -lrt total 20

-rw-r--r--. 1 bacservice bacservice 2239 Sep 23 11:08 dpe.keystore

RAN Management System Installation Guide, Release 4.1

1 May 25, 2015 HF


Step 2

-rw-r--r--. 1 bacservice bacservice 1075 Sep 23 11:08 dpe.csr

-rwxr-x---. 1 admin1 ciscorms 1742 Sep 23 11:51 server-ca.cer

-rwxr-x---. 1 admin1

-rwxr-x---. 1 admin1 ciscorms ciscorms

1182 Sep 23 11:51 root-ca.cer

1626 Sep 23 11:51 client-ca.cer

Enter: rm root-ca.cer

Output:

[root@blr-rms11-serving conf]# rm root-ca.cer

rm: remove regular file `root-ca.cer'? Y

Enter: rm server-ca.cer

Output:

[root@blr-rms11-serving conf]# rm server-ca.cer

rm: remove regular file `server-ca.cer'? Y

Enter: rm client-ca.cer

Output:

[root@blr-rms11-serving conf]# rm client-ca.cer

rm: remove regular file `client-ca.cer'? Y

Enter: ls

–lrt

Output:

[root@rms-aio-serving self_signed]# ls -lrt total 8

-rw-r--r--. 1 bacservice bacservice 2239 Sep 23 11:08 dpe.keystore

-rw-r--r--. 1 bacservice bacservice 1075 Sep 23 11:08 dpe.csr

Take a backup of old DPE Keystore and CSR:

Enter: mv /rms/app/CSCObac/dpe/conf/self_signed/dpe.keystore

/rms/app/CSCObac/dpe/conf/self_signed/dpe.keystore.bkup

Output:

System returns with command prompt

Enter: mv /rms/app/CSCObac/dpe/conf/self_signed/dpe.csr

/rms/app/CSCObac/dpe/conf/self_signed/dpe.csr.bkup

2


Troubleshooting

May 25, 2015 HF

Troubleshooting


Step 3

Step 4

Output:


Remove the existing Server and Root ca from cacerts file:

Enter:

/rms/app/CSCObac/jre/bin/keytool -delete -alias server-ca -keystore

/rms/app/CSCObac/jre/lib/security/cacerts

Note

Output:

The default password for the keystore is

"changeit".

Enter keystore password:

Enter:

/rms/app/CSCObac/jre/bin/keytool -delete -alias root-ca -keystore

/rms/app/CSCObac/jre/lib/security/cacerts

Note

Output:

The default password for the keystore is

"changeit".


Regenerate the keystore and CSR for DPE node. Ensure that CN field matches the FQDN or eth1 IP-Address of DPE).

Enter:

/rms/app/CSCObac/jre/bin/keytool -keystore /rms/app/CSCObac/dpe/conf/self_signed/dpe.keystore -alias dpe-key -genkey -keyalg RSA

Note

The values must be as specified in OVA descriptor file

Output:


Re-enter new password:

What is your first and last name?

[Unknown]: 10.5.2.217

What is the name of your organizational unit?

[Unknown]: CISCO

What is the name of your organization?

[Unknown]: CISCO

What is the name of your City or Locality?

[Unknown]: BLR

What is the name of your State or Province?

[Unknown]: KA

What is the two-letter country code for this unit?

[Unknown]: IN

Is CN=10.5.2.217, OU=CISCO, O=CISCO, L=BLR, ST=KA, C=IN correct?

[no]: yes

Enter key password for <dpe-key>

(RETURN if same as keystore password):

Re-enter new password:

May 25, 2015 HF


3

Troubleshooting

Certificate Regeneration for Upload Server

Step 5

Step 6

Step 7

Step 8

Step 9

Enter:

/rms/app/CSCObac/jre/bin/keytool -keystore /rms/app/CSCObac/dpe/conf/ self_signed /dpe.keystore

-alias dpe-key -certreq -file dpe.csr

Output:


Note

It is important to use the keytool utility provided by DPE instead of the default java keytool as per BAC documentation.

Copy the regenerated keystore and CSR to the /rms/app/CSCObac/dpe/conf/ folder.

Cp /rms/app/CSCObac/dpe/conf/self_signed/dpe.keystore /rms/app/CSCObac/dpe/conf/

Cp /rms/app/CSCObac/dpe/conf/self_signed/dpe.csr /rms/app/CSCObac/dpe/conf/

Set ownership

Enter: chown bacservice:bacservice /rms/app/CSCObac/dpe/conf/dpe.keystore

Output:


Enter: chown bacservice:bacservice /rms/app/CSCObac/dpe/conf/dpe.csr

Output:


Get the CSR signed by the signing authority and get the signed certificates and CA certificates (client-ca.cer, root-ca.cer, and server-ca.cer).

Reinstall the certificates. Follow the steps 4 and 5 in the “Installing RMS Certificates” section.

Reload the server process. Follow the step 7 in “Installing RMS Certificates" section.


Following are the Keystore regeneration steps to be performed manually if something goes wrong with the certificate generation process in LUS:

Note

Manually backup older keystores because the keystores are replaced whenever the script is executed.

Step 1

Open the generate_keystore.sh script from /opt/CSCOuls/bin/ directory as a 'root' user using the below command.

4


May 25, 2015 HF

Troubleshooting


Step 2

Step 3

Example: vi /opt/CSCOuls/bin/generate_keystore.sh

Edit the below lines as per OVA descriptor settings:

Cert_C="US"

Cert_ST="NC"

Cert_L="RTP"

Cert_O="Cisco Systems, Inc."

Cert_OU="SCTG"

Upload_SB_Fqdn="rtpfga-ova-upload"

Upload_Keystore_Password="Ch@ngeme1"

Upload_Node_Eth1_Address="10.5.2.17"

Run the script:

Enter:

./generate_keystore.sh

Output:

[root@BLR17-Upload-41N bin]# ./generate_keystore.sh

create uls keystore, private key and certificate request

Enter keystore password: Re-enter new password: Enter key password for <uls-key>

(RETURN if same as keystore password): Re-enter new password: Enter destination keystore password: Re-enter new password: Enter source keystore password: Adding UBI CA certs to uls truststore

Enter keystore password: Owner: O=Ubiquisys, CN=Co Int CA

Issuer: O=Ubiquisys, CN=Co Root CA

Serial number: 40d8ada022c1f52d

Valid from: Fri Mar 22 16:42:03 IST 2013 until: Tue Mar 16 16:42:03 IST 2038

Certificate fingerprints:

MD5: F0:F0:15:82:D3:22:A9:D7:4A:48:58:00:25:A9:E5:FC

SHA1: 38:45:74:77:61:08:A9:78:53:22:C1:29:7F:B8:8C:35:52:6F:31:79

SHA256:

DC:88:99:BE:A0:A3:BE:5F:49:11:DA:FB:85:83:05:CF:1E:A2:FA:E0:4F:4D:18:AF:0B:9B:23:3F:5F:D2:57:61

Signature algorithm name: SHA256withRSA

Version: 3

Extensions:

#1: ObjectId: 2.5.29.35 Criticality=false

AuthorityKeyIdentifier [

KeyIdentifier [

0000: 4B 49 74 B3 E2 EF 41 BF

]

]


BasicConstraints:[

CA:true

PathLen:0

]

#3: ObjectId: 2.5.29.15 Criticality=true

KIt...A.

May 25, 2015 HF


5

Troubleshooting


KeyUsage [

Key_CertSign

Crl_Sign

]


SubjectKeyIdentifier [

KeyIdentifier [

0000: 4C 29 95 49 9D 27 44 86

]

]

L).I.'D.

Trust this certificate? [no]: Certificate was added to keystore

Enter keystore password: Owner: O=Ubiquisys, CN=Co Root CA

Issuer: O=Ubiquisys, CN=Co Root CA

Serial number: 99af1d71b488d88e

Valid from: Fri Mar 22 16:12:43 IST 2013 until: Tue Mar 16 16:12:43 IST 2038

Certificate fingerprints:

MD5: FA:FA:41:EF:2E:F1:83:B8:FD:94:9F:37:A2:8E:EE:7C

SHA1: 99:B0:FA:51:C7:B2:45:5B:44:22:C0:F6:24:CD:91:3F:0F:50:DE:AB

SHA256:

1C:64:6E:CB:27:2D:23:5C:B3:01:09:6B:02:F9:3E:B6:B2:59:42:50:CD:8C:75:A6:3F:8A:66:DF:A5:18:B6:74

Signature algorithm name: SHA256withRSA

Version: 3

Extensions:

]

]


AuthorityKeyIdentifier [

KeyIdentifier [

0000: 4B 49 74 B3 E2 EF 41 BF


BasicConstraints:[

CA:true

PathLen:2147483647

]

#3: ObjectId: 2.5.29.15 Criticality=true

KeyUsage [

Key_CertSign

Crl_Sign

]

KIt...A.


SubjectKeyIdentifier [

KeyIdentifier [

0000: 4B 49 74 B3 E2 EF 41 BF

]

]

KIt...A.

Trust this certificate? [no]: Certificate was added to keystore

6


May 25, 2015 HF

Troubleshooting

Deployment Troubleshooting

Step 4

Step 5

Step 6

MAC verified OK

Changing permissions fix permissions on secure files

[root@BLR17-Upload-41N bin]#

The uls.keystore and uls.csr are regenerated in this directory: /opt/CSCOuls/conf

After getting the uls.csr file, get it signed by the signing authority to get client, server, and root certificates.

Reinstall the certificates. For more information, see the "Installing RMS Certificates” section.

Reload the server process. Follow the step 7 in “Installing RMS Certificates" section.

Deployment Troubleshooting

To address the problems faced during RMS deployment, complete the following steps.

For more details to check the status of CN, ULS and SN see RMS Installation Sanity Check .

CAR/PAR Server Not Functioning

Issue

CAR/PAR server is not functioning.

During login to aregcmd with user name 'admin' and proper password, this message is seen: "Communication with the 'radius' server failed. Unable to obtain license from server."

Cause

1

The property, "prop:Car_License_Base " is set incorrectly in the descriptor file.

or

2

CAR license has expired.

May 25, 2015 HF


7

Troubleshooting

Unable to Access BAC and DCC UI

Solution

1

Log in to Serving node as a root user.

2

Navigate to the /rms/app/CSCOar/license directory (cd

/rms/app/CSCOar/license

).

3

Edit CSCOar.lic file to vi CSCOar.lic. Either overwrite the new license in the file or comment the existing one and add the fresh license in a new line:

Overwrite:

[root@rms-aio-serving license]# vi CSCOar.lic

INCREMENT PAR-SIG-NG-TPS cisco 6.0 28-feb-2015 uncounted

VENDOR_STRING=<count>1</count>

HOSTID=ANY

NOTICE="<LicFileID>20140818221132340</LicFileID><LicLineID>1</LicLineID>

<PAK></PAK>"

SIGN=E42AA34ED7C4

Comment the existing license and add the fresh license in the new line:

[root@rms-aio-serving license]# vi CSCOar.lic

#INCREMENT PAR-SIG-NG-TPS cisco 6.0 06-sept-2014 uncounted

VENDOR_STRING=<count>1</count> HOSTID=ANY NOTICE="

<LicFileID>20140818221132340</LicFileID><LicLineID>1</LicLineID>

<PAK></PAK>"

SIGN=E42AA34ED7C4

INCREMENT PAR-SIG-NG-TPS cisco 6.0 28-feb-2015 uncounted

VENDOR_STRING=<count>1</count>

HOSTID=ANY NOTICE="<LicFileID>20140818221132340</LicFileID>

<LicLineID>1</LicLineID> <PAK></PAK>" SIGN=E42AA34ED7C4

4

Navigate to the /home directory (cd /home) and repeat the previous step on the CSCOar.lic file in this directory.

5

Go to the Serving node console and restart PAR server using the following command:

/etc/init.d/arserver stop

/etc/init.d/arserver start

After restarting the PAR server, check the status using the following command:

/rms/app/CSCOar/usrbin/arstatus

Output:

Cisco Prime AR RADIUS server running

Cisco Prime AR Server Agent running

Cisco Prime AR MCD lock manager running

Cisco Prime AR MCD server running

Cisco Prime AR GUI running

(pid: 1668)

(pid: 1655)

(pid: 1659)

(pid: 1666)

(pid: 1669)

Unable to Access BAC and DCC UI

Issue

Cause

Not able to access BAC UI and DCC UI due to expiry of certificates in browser.

Certificate added to the browser just has three months validity.

8


May 25, 2015 HF

Troubleshooting

DCC UI Shows Blank Page After Login

Solution

1

Delete the existing certificates from the browser.

Go to Tools > Options. In the Options dialog, click Advanced > Certificates

> View Certificates.

2

Select RMS setup certificate and delete.

3

Clear the browser history.

4

Access DCC UI/BAC UI again. The message "This Connection is Untrusted" appears. Click Add Exception and click Confirm Security Exception from Add

Security Exception dialog.

DCC UI Shows Blank Page After Login

Issue

Cause

Unsupported plugins installed in the Browser

Unsupported plugins cause conflicts with the DCC UI Operation

Solution

1

Remove or uninstall all unsupported/incompatible third party plugins on the browser.

Or,

2

Reinstall the Browser

DHCP Server Not Functioning

Issue

DHCP server is not functioning.

During login to nrcmd with user name 'cnradmin' and proper password, it shows groups and roles as 'superuser'; but if any command related to DHCP is entered, the following message is displayed.

"You do not have permission to perform this action."

Cause

The property, "prop:Cnr_License_IPNode" is set incorrectly in the descriptor file.

May 25, 2015 HF


9

Troubleshooting

DHCP Server Not Functioning

Solution

1

Edit the following product.license file with proper license key for PNR by logging into central node.

/rms/app/nwreg2/local/conf/product.licenses

Sample license file for reference:

INCREMENT count-dhcp cisco 8.1 permanent uncounted

VENDOR_STRING=<Count>10000</Count>

HOSTID=ANY


<PAK></PAK><CompanyName></CompanyName>" SIGN=176CCF90B694

INCREMENT base-dhcp cisco 8.1 permanent uncounted


HOSTID=ANY


<PAK></PAK><CompanyName></CompanyName>" SIGN=0F10E6FC871E

INCREMENT base-system cisco 8.1 permanent uncounted


HOSTID=ANY


<PAK></PAK><CompanyName></CompanyName>" SIGN=9242CBD0FED0

2

Log in to PNR GUI.

http://<central nb ip>:8090

User Name: cnradmin

Password: <prop:Cnradmin_Password> (Property value from the descriptor file)

3

Click Administration > Licenses from Home page.

The following three types of license keys should be present. If not present, add them using browser.

1

Base-dhcp

2

Count-dhcp

3

Base-system

4

Click Administration > Clusters.

5

Click Resynchronize.

Go to Serving Node Console and restart PNR server using the following command:

/etc/init.d/nwreglocal stop

/etc/init.d/nwreglocal start

After restarting the PNR server, check the status using the following command:

/rms/app/nwreg2/local/usrbin/cnr_status

Output:

DHCP Server running

Server Agent running

CCM Server running

WEB Server running

CNRSNMP Server running

RIC Server Running

TFTP Server is not running

DNS Server is not running

DNS Caching Server is not running

(pid: 8056)

(pid: 8050)

(pid: 8055)

(pid: 8057)

(pid: 8060)

(pid: 8058)

10


May 25, 2015 HF

Troubleshooting

DPE Processes are Not Running

DPE Processes are Not Running

Scenario 1:

DPE Installation Fails with error log:

Issue

This DPE is not licensed. Your request cannot be serviced"

Cause

Configure the property prop:Dpe_Cnrquery_Client_Socket_Address=NB IP address of serving node in the descriptor file. If other than NB IP address of serving node is given then "DPE is not licensed error" will appear in OVA first boot log.

Solution

1

Log in to DPE CLI using the command [admin1@blr-rms11-serving ~]$

2

Execute the command telnet localhost 2323.

Trying 127.0.0.1...

Connected to localhost.

Escape character is '^]'.

blr-rms11-serving BAC Device Provisioning Engine

User Access Verification

Password: blr-rms11-serving> en

Password: blr-rms11-serving# dpe cnrquery giaddr x.x.x.x

blr-rms11-serving# dpe cnrquery server-port 61610 blr-rms11-serving# dhcp reload

Scenario 2:

Issue

Cause

DPE process might not run when the password of keystore and key mismatches from the descriptor file.

The Keystore was tampered with, or password entered is incorrect resulting in a password verification failure. This occurs when the password used to generate the

Keystore file is different than the one given for the property

"prop:RMS_App_Password" in descriptor file.

May 25, 2015 HF


11

Troubleshooting

Connection to Remote Object Unsuccessful

Solution

1

Navigate to /rms/app/CSCObac/dpe/conf and execute the below command to change the password of the Keystore file.

Input:

"[root@rtpfga-s1-upload1 conf]# keytool -storepasswd

–keystore dpe.keystore"

Output:

Enter keystore password:OLD PASSWORD

New keystore password: NEW PASSWORD

Re-enter new keystore password: NEW PASSWORD

Input:

[root@rtpfga-s1-upload1 conf]# keytool -keypasswd -keystore dpe.keystore -alias dpe

–key

Output:

Enter keystore password: NEW AS PER LAST COMMAND

Enter key password for <dpe-key> : OLD PASSWORD

New key password for <dpe-key>: NEW PASSWORD

Re-enter new key password for <dpe-key>: NEW PASSWORD

Note

The new keystore password should be same as given in the descriptor file.

2

Restart the server process.

[root@rtpfga-s1-upload1 conf]# /etc/init.d/bprAgent restart dpe

[root@rtpfga-s1-upload1 conf]# /etc/init.d/bprAgent status dpe

BAC Process Watchdog is running.

Process [dpe] is running.

Broadband Access Center [BAC 3.8.1.2

(LNX_BAC3_8_1_2_20140918_1230_12)].

Connected to RDU [10.5.1.200].

Caching [3] device configs and [52] files.

188 sessions succeeded and 1 sessions failed.

6 file requests succeeded and 0 file requests failed.

68 immediate device operations succeeded, and 2 failed.

0 home PG redirections succeeded, and 0 failed.

Using signature key name [] with a validity of [3600].

Abbreviated ParamList is enabled.

Running for [4] hours [23] mins [17] secs.

Connection to Remote Object Unsuccessful

Issue

Cause

A connection to the remote object could not be made. OVF Tool does not support this server.

Completed with errors

The errors are triggered by ovftool command during ova deployment. the errors can be found in both Console and vCenter logs.

Solution

User must have Administrator privileges to VMware vCenter and ESXi.

12


May 25, 2015 HF

Troubleshooting

VLAN Not Found

VLAN Not Found

Issue

Cause

Solution

VLAN not found.

The errors are triggered by ovftool command during ova deployment. the errors can be found in both Console and vCenter logs.

Check for the appropriate "portgroup" name on virtual switch of Elastic Sky X

Integrated (ESXi) host or Distributed Virtual Switch (DVS) on VMware vCenter.

Unable to Get Live Data in DCC UI

Issue

Live Data of an AP is not coming and Connection request fails.

Cause

1

Device is offline.

2

Device is not having its radio activated/ device is registered but not activated.

Solution

1

In the Serving Node, add one more route with Destination IP as HNB-GW

SCTP IP and Gateway as Serving Node North Bound IP as in the following example:

Serving NB Gateway IP-10.5.1.1

HNBGW SCTP IP- 10.5.1.83

Add the following route in Serving node: route add -net 10.5.1.83 netmask 255.255.255.0 gw 10.5.1.1

2

Activate the device from DCC UI post registration.

3

Verify trouble shooting logs in bac.

4

Verify DPE logs and ZGTT logs from ACS simulator.

Installation Warnings about Removed Parameters

These properties have been completely removed from the 4.0 OVA installation. A warning is given by the installer, if these properties are found in the OVA descriptor file. However, installation still continues.

prop:vami.gateway.Upload-Node prop:vami.DNS.Upload-Node prop:vami.ip0.Upload-Node prop:vami.netmask0.Upload-Node prop:vami.ip1.Upload-Node prop:vami.netmask1.Upload-Node prop:vami.gateway.Central-Node prop:vami.DNS.Central-Node prop:vami.ip0.Central-Node prop:vami.netmask0.Central-Node

May 25, 2015 HF


13

Upload Server is Not Up

prop:vami.ip1.Central-Node prop:vami.netmask1.Central-Node prop:vami.gateway.Serving-Node prop:vami.DNS.Serving-Node prop:vami.ip0.Serving-Node prop:vami.netmask0.Serving-Node prop:vami.ip1.Serving-Node prop:vami.netmask1.Serving-Node prop:Debug_Mode prop:Server_Crl_Urls prop:Bacadmin_Password prop:Dccapp_Password prop:Opstools_Password prop:Dccadmin_Password prop:Postgresql_Password prop:Central_Keystore_Password prop:Upload_Stat_Password prop:Upload_Calldrop_Password prop:Upload_Demand_Password prop:Upload_Lostipsec_Password prop:Upload_Lostgwconnection_Password prop:Upload_Nwlscan_Password prop:Upload_Periodic_Password prop:Upload_Restart_Password prop:Upload_Crash_Password prop:Upload_Lowmem_Password prop:Upload_Unknown_Password prop:Serving_Keystore_Password prop:Cnradmin_Password prop:Caradmin_Password prop:Dpe_Cli_Password prop:Dpe_Enable_Password prop:Fc_Realm prop:Fc_Log_Periodic_Upload_Enable prop:Fc_Log_Periodic_Upload_Interval prop:Fc_On_Nwl_Scan_Enable prop:Fc_On_Lost_Ipsec_Enable prop:Fc_On_Crash_Upload_Enable prop:Fc_On_Call_Drop_Enable prop:Fc_On_Lost_Gw_Connection_Enable prop:Upload_Keystore_Password prop:Dpe_Keystore_Password prop:Bac_Secret prop:Admin2_Username prop:Admin2_Password prop:Admin2_Firstname prop:Admin2_Lastname prop:Admin3_Username prop:Admin3_Password prop:Admin3_Firstname prop:Admin3_Lastname prop:Upgrade_Mode prop:Asr5k_Hnbgw_Address


The upload server fails with java.lang.ExceptionInInitializerError in the following scenarios.

The errors can be seen in opt/CSCOuls/logs/uploadServer.console.log file.

Scenario 1:

Troubleshooting

14


May 25, 2015 HF

Troubleshooting


Issue

Cause

Solution

Upload Server failed with java.lang.ExceptionInInitializerError

java.lang.ExceptionInInitializerError

at com.cisco.ca.rms.upload.server.UlsSouthBoundServer.getInstance

(UlsSouthBoundServer.java:58) at com.cisco.ca.rms.upload.server.UlsServer.<init>(UlsServer.java:123) at com.cisco.ca.rms.upload.server.UlsServer.<init>(UlsServer.java:25) at com.cisco.ca.rms.upload.server.UlsServer$SingleInstanceHolder.<clinit>

(UlsServer.java:70) at com.cisco.ca.rms.upload.server.UlsServer.getInstance(UlsServer.java:82) at com.cisco.ca.rms.upload.server.UlsServer.main(UlsServer.java:55)

Caused by: org.jboss.netty.channel.ChannelException: Failed to bind to:

/10.6.22.12:8080 at org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:298) at com.cisco.ca.rms.upload.server.UlsSouthBoundServer.<init>

(UlsSouthBoundServer.java:109) at com.cisco.ca.rms.upload.server.UlsSouthBoundServer.<init>

(UlsSouthBoundServer.java:22) at com.cisco.ca.rms.upload.server.UlsSouthBoundServer$SingleInstanceHolder.<clinit>

(UlsSouthBoundServer.java:46)

... 6 more

Caused by: java.net.BindException: Cannot assign requested address at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Unknown Source) at sun.nio.ch.Net.bind(Unknown Source) at sun.nio.ch.ServerSocketChannelImpl.bind(Unknown Source) at sun.nio.ch.ServerSocketAdaptor.bind(Unknown Source) at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.bind

(NioServerSocketPipelineSink.java:140) at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleServerSocket

(NioServerSocketPipelineSink.java:90) at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk

(NioServerSocketPipelineSink.java:64) at org.jboss.netty.channel.Channels.bind(Channels.java:569) at org.jboss.netty.channel.AbstractChannel.bind(AbstractChannel.java:189) at org.jboss.netty.bootstrap.ServerBootstrap$Binder.channelOpen(

ServerBootstrap.java:343) at org.jboss.netty.channel.Channels.fireChannelOpen(Channels.java:170) at org.jboss.netty.channel.socket.nio.NioServerSocketChannel.<init>

(NioServerSocketChannel.java:80) at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.newChannel

(NioServerSocketChannelFactory.java:158) at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.newChannel

(NioServerSocketChannelFactory.java:86) at org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:277)

... 9 more

The server failed to bind to the IP /10.6.22.12:8080 because the requested address was unavailable.

Navigate to /opt/CSCOuls/conf and modify the UploadServer.properties file with proper

SB and NB IP address.

Scenario 2:

May 25, 2015 HF


15

Troubleshooting


Issue

Cause



at com.cisco.ca.rms.upload.server.security.UlsSbSslContextMgr.getInstance

(UlsSbSslContextMgr.java:65) at com.cisco.ca.rms.upload.server.UlsSouthBoundPipelineFactory.<init>

(UlsSouthBoundPipelineFactory.java:86) at com.cisco.ca.rms.upload.server.UlsSouthBoundServer.<init>

(UlsSouthBoundServer.java:102) at com.cisco.ca.rms.upload.server.UlsSouthBoundServer.<init>

(UlsSouthBoundServer.java:22) at com.cisco.ca.rms.upload.server.UlsSouthBoundServer$SingleInstanceHolder.<clinit>

(UlsSouthBoundServer.java:46) at com.cisco.ca.rms.upload.server.UlsSouthBoundServer.getInstance

(UlsSouthBoundServer.java:58) at com.cisco.ca.rms.upload.server.UlsServer.<init>(UlsServer.java:123) at com.cisco.ca.rms.upload.server.UlsServer.<init>(UlsServer.java:25) at com.cisco.ca.rms.upload.server.UlsServer$SingleInstanceHolder.<clinit>

(UlsServer.java:70) at com.cisco.ca.rms.upload.server.UlsServer.getInstance(UlsServer.java:82) at com.cisco.ca.rms.upload.server.UlsServer.main(UlsServer.java:55)

Caused by: java.lang.IllegalStateException: java.io.IOException:

Keystore was tampered with, or password was incorrect at com.cisco.ca.rms.commons.security.SslContextManager.<init>

(SslContextManager.java:79) at com.cisco.ca.rms.upload.server.security.UlsSbSslContextMgr.<init>

(UlsSbSslContextMgr.java:72) at com.cisco.ca.rms.upload.server.security.UlsSbSslContextMgr.<init>

(UlsSbSslContextMgr.java:28) at com.cisco.ca.rms.upload.server.security.UlsSbSslContextMgr$SingleInstanceHolder.<clinit>

(UlsSbSslContextMgr.java:53)

... 11 more

Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect at sun.security.provider.JavaKeyStore.engineLoad(Unknown Source) at sun.security.provider.JavaKeyStore$JKS.engineLoad(Unknown Source) at java.security.KeyStore.load(Unknown Source) at com.cisco.ca.rms.upload.server.security.UlsSbSslContextMgr.loadKeyManagers

(UlsSbSslContextMgr.java:91) at com.cisco.ca.rms.commons.security.SslContextManager.<init>(SslContextManager.java:48)

... 14 more

Caused by: java.security.UnrecoverableKeyException: Password verification failed

... 19 more

The Keystore was tampered with, or password entered is incorrect resulting in a password verification failure.

This occurs when the password used to generate the Keystore file is different than the one given for the property “Upload_Keystore_Password” in descriptor file.

16


May 25, 2015 HF

Troubleshooting


Solution

1

Navigate to /opt/CSCOuls/conf and execute the below command to change the password of the Keystore file.

"[root@rtpfga-s1-upload1 conf]# keytool -storepasswd -keystore uls.keystore"

Output: keytool -storepasswd -keystore dpe.keystore

Enter keystore password:OLD PASSWORD

New keystore password: NEW PASSWORD

Re-enter new keystore password: NEW PASSWORD

Note

The new Keystore password should be same as given in the descriptor file.

2

Run another command before restarting the server to change the key password.

keytool -keypasswd -keystore dpe.keystore -alias dpe -key

Enter keystore password: NEW AS PER LAST COMMAND

Enter key password for <dpe-key> : OLD PASSWORD

New key password for <dpe-key>: NEW PASSWORD

Re-enter new key password for <dpe-key>: NEW PASSWORD

3

Restart the server process.

[root@rtpfga-s1-upload1 conf]# service god restart

[root@rtpfga-s1-upload1 conf]# service god status

UploadServer: up

Scenario 3:

May 25, 2015 HF


17

Troubleshooting


Issue

Cause

Solution



at com.cisco.ca.rms.dcc.lus.server.LusNorthBoundServer.getInstance

(LusNorthBoundServer.java:65) at com.cisco.ca.rms.dcc.lus.server.LusServer.<init>(LusServer.java:98) at com.cisco.ca.rms.dcc.lus.server.LusServer.<init>(LusServer.java:17) at com.cisco.ca.rms.dcc.lus.server.LusServer$SingleInstanceHolder.<clinit>

(LusServer.java:45) at com.cisco.ca.rms.dcc.lus.server.LusServer.getInstance(LusServer.java:57) at com.cisco.ca.rms.dcc.lus.server.LusServer.main(LusServer.java:30)

Caused by: org.jboss.netty.channel.ChannelException: Failed to bind to:

/0.0.0.0:8082 at org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:298) at com.cisco.ca.rms.dcc.lus.server.LusNorthBoundServer.<init>

(LusNorthBoundServer.java:120) at com.cisco.ca.rms.dcc.lus.server.LusNorthBoundServer.<init>

(LusNorthBoundServer.java:30) at com.cisco.ca.rms.dcc.lus.server.LusNorthBoundServer$SingleInstanceHolder.<clinit>

(LusNorthBoundServer.java:53)

... 6 more

Caused by: java.net.BindException: Address already in use at sun.nio.ch.Net.bind(Native Method) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:137) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:77) at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.bind

(NioServerSocketPipelineSink.java:140) at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleServerSocket

(NioServerSocketPipelineSink.java:92) at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk

(NioServerSocketPipelineSink.java:66) at org.jboss.netty.channel.Channels.bind(Channels.java:462) at org.jboss.netty.channel.AbstractChannel.bind(AbstractChannel.java:186) at org.jboss.netty.bootstrap.ServerBootstrap$Binder.channelOpen

(ServerBootstrap.java:343) at org.jboss.netty.channel.Channels.fireChannelOpen(Channels.java:170) at org.jboss.netty.channel.socket.nio.NioServerSocketChannel.<init>

(NioServerSocketChannel.java:77) at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.newChannel

(NioServerSocketChannelFactory.java:137) at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.newChannel

(NioServerSocketChannelFactory.java:85) at org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:277)

... 9 more

The server failed to bind to the IP /0.0.0.0:8082 because the requested address is already in use.

Execute the command: netstat

–anp |grep <port number>

For example:

[root@rtpfga-s1-upload1 conf]# netstat -anp |grep 8082 tcp 0 0 10.6.23.16:8082 0.0.0.0:* LISTEN

Kill the particular process.

[root@rtpfga-s1-upload1 conf]# kill -9 26842

Start the server.

“[root@rtpfga-s1-upload1 conf]# service god start

[root@rtpfga-s1-upload1 conf]# service god status

UploadServer: up

”

26842/java

18


May 25, 2015 HF

Troubleshooting

OVA Installation failures

OVA Installation failures

Issue

Cause

Solution

If the OVA installer displays an error on the installation Console.

OVA Installation failures

If there are any issues during OVA installation, the ova-first-boot.log should be referred that is present in the Central node and Serving node. Validate the appropriate errors in the boot log files.

Update failures in group type, Site - DCC UI throws an error

Issue

Cause

SITE Creation Fails While Importing All Mandatory and Optional Parameters.

Invalid parameter value- FC-CSON-STATUS-HSCO-INNER with Optimised.

Solution

For FC-CSON-STATUS-HSCO-INNER parameter, allowed value is Optimized not Optimised. The spelling for Optimized should be corrected.

May 25, 2015 HF


19

Update failures in group type, Site - DCC UI throws an error

Troubleshooting

20


May 25, 2015 HF

Troubleshooting

Troubleshooting

Regeneration of Certificates

Certificate Regeneration for DPE

Certificate Regeneration for Upload Server

Deployment Troubleshooting

CAR/PAR Server Not Functioning

Unable to Access BAC and DCC UI

DCC UI Shows Blank Page After Login

DHCP Server Not Functioning

DPE Processes are Not Running

Connection to Remote Object Unsuccessful

VLAN Not Found

Unable to Get Live Data in DCC UI

Installation Warnings about Removed Parameters

Upload Server is Not Up

OVA Installation failures

Update failures in group type, Site - DCC UI throws an error

Related manuals

Cisco

Universal Small Cell RAN Management System

Cisco

Universal Small Cell RAN Management System

Cisco

Universal Small Cell RAN Management System

Cisco

Universal Small Cell RAN Management System

Cisco

Universal Small Cell RAN Management System