Troubleshooting

Troubleshooting
Troubleshooting
This chapter provides procedures for troubleshooting the problems encountered during RMS Installation.
• Regeneration of Certificates, page 1
• Deployment Troubleshooting , page 7
Regeneration of Certificates
Following are the scenarios that requires regeneration of certificates:
• Certificate expiry (Certificate will have a validity of one year.)
• If importing certificates are not successful.
Follow the steps to regenerate self-signed certificates:
Certificate Regeneration for DPE
To address the problems faced during the certificate generation process in Distributed Provisioning Engine
(DPE), complete the following steps:
Procedure
Step 1
Remove the root.ca.cer, server.ca.cer and client.ca.cer certificates that are installed in DPE.
Enter:
ssh login to Serving_Node_1
Change to root user
Navigate to the conf folder
cd /rms/app/CSCObac/dpe/conf/self_signed
ls -lrt
Output:
[root@rms-aio-serving self_signed]# ls -lrt
total 20
Cisco RAN Management System Installation Guide, Release 5.1
July 6, 2015
1
Troubleshooting
Certificate Regeneration for DPE
-rw-r--r--.
-rw-r--r--.
-rwxr-x---.
-rwxr-x---.
-rwxr-x---.
1
1
1
1
1
bacservice
bacservice
admin1
admin1
admin1
bacservice
bacservice
ciscorms
ciscorms
ciscorms
2239
1075
1742
1182
1626
Sep
Sep
Sep
Sep
Sep
23
23
23
23
23
11:08
11:08
11:51
11:51
11:51
dpe.keystore
dpe.csr
server-ca.cer
root-ca.cer
client-ca.cer
Enter:
rm root-ca.cer
Output:
[root@blr-rms11-serving conf]# rm root-ca.cer
rm: remove regular file `root-ca.cer'? Y
Enter:
rm server-ca.cer
Output:
[root@blr-rms11-serving conf]# rm server-ca.cer
rm: remove regular file `server-ca.cer'? Y
Enter:
rm client-ca.cer
Output:
[root@blr-rms11-serving conf]# rm client-ca.cer
rm: remove regular file `client-ca.cer'? Y
Enter:
ls –lrt
Output:
[root@rms-aio-serving self_signed]# ls -lrt
total 8
-rw-r--r--. 1 bacservice bacservice 2239 Sep 23 11:08 dpe.keystore
-rw-r--r--. 1 bacservice bacservice 1075 Sep 23 11:08 dpe.csr
Step 2
Take a backup of old DPE Keystore and CSR:
Enter:
mv /rms/app/CSCObac/dpe/conf/self_signed/dpe.keystore
/rms/app/CSCObac/dpe/conf/self_signed/dpe.keystore.bkup
Output:
System returns with command prompt
Enter:
mv /rms/app/CSCObac/dpe/conf/self_signed/dpe.csr
/rms/app/CSCObac/dpe/conf/self_signed/dpe.csr.bkup
Cisco RAN Management System Installation Guide, Release 5.1
2
July 6, 2015
Troubleshooting
Certificate Regeneration for DPE
Output:
System returns with command prompt
Step 3
Remove the existing Server and Root ca from cacerts file:
Enter:
/rms/app/CSCObac/jre/bin/keytool -delete -alias server-ca -keystore
/rms/app/CSCObac/jre/lib/security/cacerts
Note
Output:
The default password for the keystore is
"changeit".
Enter keystore password:
Enter:
/rms/app/CSCObac/jre/bin/keytool -delete -alias root-ca -keystore
/rms/app/CSCObac/jre/lib/security/cacerts
Note
Output:
The default password for the keystore is
"changeit".
Enter keystore password:
Step 4
Regenerate the keystore and CSR for DPE node. Ensure that CN field matches the FQDN or eth1 IP-Address
of DPE).
Enter:
/rms/app/CSCObac/jre/bin/keytool -keystore /rms/app/CSCObac/dpe/conf/self_signed/dpe.keystore
-alias dpe-key -genkey -keyalg RSA
Note
Output:
The values must be as specified in OVA descriptor
file
Enter keystore password:
Re-enter new password:
What is your first and last name?
[Unknown]: 10.5.2.217
What is the name of your organizational unit?
[Unknown]: CISCO
What is the name of your organization?
[Unknown]: CISCO
What is the name of your City or Locality?
[Unknown]: BLR
What is the name of your State or Province?
[Unknown]: KA
What is the two-letter country code for this unit?
[Unknown]: IN
Is CN=10.5.2.217, OU=CISCO, O=CISCO, L=BLR, ST=KA, C=IN correct?
[no]: yes
Enter key password for <dpe-key>
(RETURN if same as keystore password):
Re-enter new password:
Cisco RAN Management System Installation Guide, Release 5.1
July 6, 2015
3
Troubleshooting
Certificate Regeneration for Upload Server
Enter:
/rms/app/CSCObac/jre/bin/keytool -keystore /rms/app/CSCObac/dpe/conf/ self_signed
/dpe.keystore -alias dpe-key -certreq -file dpe.csr
Output:
Enter keystore password:
It is important to use the keytool utility provided by DPE instead of the default java keytool as per
BAC documentation.
Copy the regenerated keystore and CSR to the /rms/app/CSCObac/dpe/conf/ folder.
Note
Step 5
Cp /rms/app/CSCObac/dpe/conf/self_signed/dpe.keystore /rms/app/CSCObac/dpe/conf/
Cp /rms/app/CSCObac/dpe/conf/self_signed/dpe.csr /rms/app/CSCObac/dpe/conf/
Step 6
Set ownership
Enter:
chown bacservice:bacservice /rms/app/CSCObac/dpe/conf/dpe.keystore
Output:
System returns with command prompt
Enter:
chown bacservice:bacservice /rms/app/CSCObac/dpe/conf/dpe.csr
Output:
System returns with command prompt
Step 7
Get the CSR signed by the signing authority and get the signed certificates and CA certificates (client-ca.cer,
root-ca.cer, and server-ca.cer).
Step 8
Reinstall the certificates. Follow the steps 4 and 5 in the “Installing RMS Certificates” section.
Step 9
Reload the server process. Follow the step 7 in “Installing RMS Certificates" section.
Certificate Regeneration for Upload Server
Following are the Keystore regeneration steps to be performed manually if something goes wrong with the
certificate generation process in LUS:
Note
Manually backup older keystores because the keystores are replaced whenever the script is executed.
Procedure
Step 1
Open the generate_keystore.sh script from /opt/CSCOuls/bin/ directory as a 'root' user using the below
command.
Cisco RAN Management System Installation Guide, Release 5.1
4
July 6, 2015
Troubleshooting
Certificate Regeneration for Upload Server
Example:
vi /opt/CSCOuls/bin/generate_keystore.sh
Step 2
Edit the below lines as per OVA descriptor settings:
Cert_C="IN"
Cert_ST="KA"
Cert_L="BLR"
Cert_O="Cisco Systems, Inc."
Cert_OU="SCTG"
Upload_SB_Fqdn="femtolus17.testlab.in"
RMS_App_Password="Rmsuser@1"
Step 3
Run the script:
Enter:
./generate_keystore.sh
Output:
[root@BLR17-Upload-41N bin]# ./generate_keystore.sh
create uls keystore, private key and certificate request
Enter keystore password: Re-enter new password: Enter key password for <uls-key>
(RETURN if same as keystore password): Re-enter new password: Enter destination
keystore password: Re-enter new password: Enter source keystore password: Adding UBI CA
certs to uls truststore
Enter keystore password: Owner: O=Ubiquisys, CN=Co Int CA
Issuer: O=Ubiquisys, CN=Co Root CA
Serial number: 40d8ada022c1f52d
Valid from: Fri Mar 22 16:42:03 IST 2013 until: Tue Mar 16 16:42:03 IST 2038
Certificate fingerprints:
MD5: F0:F0:15:82:D3:22:A9:D7:4A:48:58:00:25:A9:E5:FC
SHA1: 38:45:74:77:61:08:A9:78:53:22:C1:29:7F:B8:8C:35:52:6F:31:79
SHA256:
DC:88:99:BE:A0:A3:BE:5F:49:11:DA:FB:85:83:05:CF:1E:A2:FA:E0:4F:4D:18:AF:0B:9B:23:3F:5F:D2:57:61
Signature algorithm name: SHA256withRSA
Version: 3
Extensions:
#1: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: 4B 49 74 B3 E2 EF 41 BF
]
]
KIt...A.
#2: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
CA:true
PathLen:0
]
#3: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
Key_CertSign
Crl_Sign
Cisco RAN Management System Installation Guide, Release 5.1
July 6, 2015
5
Troubleshooting
Certificate Regeneration for Upload Server
]
#4: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 4C 29 95 49 9D 27 44 86
]
]
L).I.'D.
Trust this certificate? [no]: Certificate was added to keystore
Enter keystore password: Owner: O=Ubiquisys, CN=Co Root CA
Issuer: O=Ubiquisys, CN=Co Root CA
Serial number: 99af1d71b488d88e
Valid from: Fri Mar 22 16:12:43 IST 2013 until: Tue Mar 16 16:12:43 IST 2038
Certificate fingerprints:
MD5: FA:FA:41:EF:2E:F1:83:B8:FD:94:9F:37:A2:8E:EE:7C
SHA1: 99:B0:FA:51:C7:B2:45:5B:44:22:C0:F6:24:CD:91:3F:0F:50:DE:AB
SHA256:
1C:64:6E:CB:27:2D:23:5C:B3:01:09:6B:02:F9:3E:B6:B2:59:42:50:CD:8C:75:A6:3F:8A:66:DF:A5:18:B6:74
Signature algorithm name: SHA256withRSA
Version: 3
Extensions:
#1: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: 4B 49 74 B3 E2 EF 41 BF
]
]
KIt...A.
#2: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
CA:true
PathLen:2147483647
]
#3: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
Key_CertSign
Crl_Sign
]
#4: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 4B 49 74 B3 E2 EF 41 BF
]
]
KIt...A.
Trust this certificate? [no]: Certificate was added to keystore
MAC verified OK
Changing permissions
fix permissions on secure files
Cisco RAN Management System Installation Guide, Release 5.1
6
July 6, 2015
Troubleshooting
Deployment Troubleshooting
[root@BLR17-Upload-41N bin]#
The uls.keystore and uls.csr are regenerated in this directory: /opt/CSCOuls/conf/self_signed
Step 4
Copy the certificates from the /self_signed directory to the /conf directory.
cp /opt/CSCOuls/conf/self_signed/openssl.cnf /opt/CSCOuls/conf/openssl.cnf
cp /opt/CSCOuls/conf/self_signed/uls.trustore /opt/CSCOuls/conf/uls.trustore
Step 5
Step 6
After getting the uls.csr file, get it signed by the signing authority to get client, server, and root certificates.
Step 7
Reload the server process. Follow the step 7 in “Installing RMS Certificates" section.
Reinstall the certificates. For more information, see the "Installing RMS Certificates” section.
Deployment Troubleshooting
To address the problems faced during RMS deployment, complete the following steps.
For more details to check the status of CN, ULS and SN see RMS Installation Sanity Check.
CAR/PAR Server Not Functioning
Issue
CAR/PAR server is not functioning.
During login to aregcmd with user name 'admin' and proper password, this message
is seen: "Communication with the 'radius' server failed. Unable to obtain license
from server."
Cause
1 The property, "prop:Car_License_Base " is set incorrectly in the descriptor file.
or
2 CAR license has expired.
Cisco RAN Management System Installation Guide, Release 5.1
July 6, 2015
7
Troubleshooting
Unable to Access BAC and DCC UI
Solution
1 Log in to Serving node as a root user.
2 Navigate to the /rms/app/CSCOar/license directory (cd
/rms/app/CSCOar/license).
3 Edit CSCOar.lic file to vi CSCOar.lic. Either overwrite the new license in the
file or comment the existing one and add the fresh license in a new line:
Overwrite:
[root@rms-aio-serving license]# vi CSCOar.lic
INCREMENT PAR-SIG-NG-TPS cisco 6.0 28-feb-2015 uncounted
VENDOR_STRING=<count>1</count>
HOSTID=ANY
NOTICE="<LicFileID>20140818221132340</LicFileID><LicLineID>1</LicLineID>
<PAK></PAK>"
SIGN=E42AA34ED7C4
Comment the existing license and add the fresh license in the new line:
[root@rms-aio-serving license]# vi CSCOar.lic
#INCREMENT PAR-SIG-NG-TPS cisco 6.0 06-sept-2014 uncounted
VENDOR_STRING=<count>1</count> HOSTID=ANY NOTICE="
<LicFileID>20140818221132340</LicFileID><LicLineID>1</LicLineID>
<PAK></PAK>"
SIGN=E42AA34ED7C4
INCREMENT PAR-SIG-NG-TPS cisco 6.0 28-feb-2015 uncounted
VENDOR_STRING=<count>1</count>
HOSTID=ANY NOTICE="<LicFileID>20140818221132340</LicFileID>
<LicLineID>1</LicLineID> <PAK></PAK>" SIGN=E42AA34ED7C4
4 Navigate to the /home directory (cd /home) and repeat the previous step on
the CSCOar.lic file in this directory.
5 Go to the Serving node console and restart PAR server using the following
command:
/etc/init.d/arserver stop
/etc/init.d/arserver start
After restarting the PAR server, check the status using the following command:
/rms/app/CSCOar/usrbin/arstatus
Output:
Cisco
Cisco
Cisco
Cisco
Cisco
Prime
Prime
Prime
Prime
Prime
AR
AR
AR
AR
AR
RADIUS server running
Server Agent running
MCD lock manager running
MCD server running
GUI running
(pid:
(pid:
(pid:
(pid:
(pid:
1668)
1655)
1659)
1666)
1669)
Unable to Access BAC and DCC UI
Issue
Not able to access BAC UI and DCC UI due to expiry of certificates in browser.
Cause
Certificate added to the browser just has three months validity.
Cisco RAN Management System Installation Guide, Release 5.1
8
July 6, 2015
Troubleshooting
DCC UI Shows Blank Page After Login
Solution
1 Delete the existing certificates from the browser.
Go to Tools > Options. In the Options dialog, click Advanced > Certificates
> View Certificates.
2 Select RMS setup certificate and delete.
3 Clear the browser history.
4 Access DCC UI/BAC UI again. The message "This Connection is Untrusted"
appears. Click Add Exception and click Confirm Security Exception from Add
Security Exception dialog.
DCC UI Shows Blank Page After Login
Issue
Unsupported plugins installed in the Browser
Cause
Unsupported plugins cause conflicts with the DCC UI Operation
Solution
1 Remove or uninstall all unsupported/incompatible third party plugins on the
browser.
Or,
2 Reinstall the Browser
DHCP Server Not Functioning
Issue
DHCP server is not functioning.
During login to nrcmd with user name 'cnradmin' and proper password, it shows
groups and roles as 'superuser'; but if any command related to DHCP is entered,
the following message is displayed.
"You do not have permission to perform this action."
Cause
The property, "prop:Cnr_License_IPNode" is set incorrectly in the descriptor file.
Cisco RAN Management System Installation Guide, Release 5.1
July 6, 2015
9
Troubleshooting
DHCP Server Not Functioning
Solution
1 Edit the following product.license file with proper license key for PNR by
logging into central node.
/rms/app/nwreg2/local/conf/product.licenses
Sample license file for reference:
INCREMENT count-dhcp cisco 8.1 permanent uncounted
VENDOR_STRING=<Count>10000</Count>
HOSTID=ANY
NOTICE="<LicFileID>20130715144658047</LicFileID><LicLineID>1</LicLineID>
<PAK></PAK><CompanyName></CompanyName>" SIGN=176CCF90B694
INCREMENT base-dhcp cisco 8.1 permanent uncounted
VENDOR_STRING=<Count>1000</Count>
HOSTID=ANY
NOTICE="<LicFileID>20130715144658047</LicFileID><LicLineID>2</LicLineID>
<PAK></PAK><CompanyName></CompanyName>" SIGN=0F10E6FC871E
INCREMENT base-system cisco 8.1 permanent uncounted
VENDOR_STRING=<Count>1</Count>
HOSTID=ANY
NOTICE="<LicFileID>20130715144658047</LicFileID><LicLineID>3</LicLineID>
<PAK></PAK><CompanyName></CompanyName>" SIGN=9242CBD0FED0
2 Log in to PNR GUI.
http://<central nb ip>:8090
User Name: cnradmin
Password: <prop:Cnradmin_Password> (Property value from the descriptor
file)
3 Click Administration > Licenses from Home page.
The following three types of license keys should be present. If not present, add
them using browser.
1 Base-dhcp
2 Count-dhcp
3 Base-system
4 Click Administration > Clusters.
5 Click Resynchronize.
Go to Serving Node Console and restart PNR server using the following
command:
/etc/init.d/nwreglocal stop
/etc/init.d/nwreglocal start
After restarting the PNR server, check the status using the following command:
/rms/app/nwreg2/local/usrbin/cnr_status
Output:
DHCP Server running
(pid:
Server Agent running
(pid:
CCM Server running
(pid:
WEB Server running
(pid:
CNRSNMP Server running
(pid:
RIC Server Running
(pid:
TFTP Server is not running
DNS Server is not running
DNS Caching Server is not running
8056)
8050)
8055)
8057)
8060)
8058)
Cisco RAN Management System Installation Guide, Release 5.1
10
July 6, 2015
Troubleshooting
DPE Processes are Not Running
DPE Processes are Not Running
Scenario 1:
DPE Installation Fails with error log:
Issue
This DPE is not licensed. Your request cannot be serviced"
Cause
Configure the property prop:Dpe_Cnrquery_Client_Socket_Address=NB IP
address of serving node in the descriptor file. If other than NB IP address of
serving node is given then "DPE is not licensed error" will appear in OVA first
boot log.
Solution
1 Log in to DPE CLI using the command [admin1@blr-rms11-serving ~]$
2 Execute the command telnet localhost 2323.
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
blr-rms11-serving BAC Device Provisioning Engine
User Access Verification
Password:
blr-rms11-serving>
Password:
blr-rms11-serving#
blr-rms11-serving#
blr-rms11-serving#
en
dpe cnrquery giaddr x.x.x.x
dpe cnrquery server-port 61610
dhcp reload
Scenario 2:
Issue
DPE process might not run when the password of keystore and key mismatches
from the descriptor file.
Cause
The Keystore was tampered with, or password entered is incorrect resulting in a
password verification failure. This occurs when the password used to generate the
Keystore file is different than the one given for the property
"prop:RMS_App_Password" in descriptor file.
Cisco RAN Management System Installation Guide, Release 5.1
July 6, 2015
11
Troubleshooting
Connection to Remote Object Unsuccessful
Solution
1 Navigate to /rms/app/CSCObac/dpe/conf and execute the below command to
change the password of the Keystore file.
Input:
"[root@rtpfga-s1-upload1 conf]# keytool -storepasswd –keystore
dpe.keystore"
Output:
Enter keystore password:OLD PASSWORD
New keystore password: NEW PASSWORD
Re-enter new keystore password: NEW PASSWORD
Input:
[root@rtpfga-s1-upload1 conf]# keytool -keypasswd -keystore
dpe.keystore -alias dpe –key
Output:
Enter keystore password: NEW AS PER LAST COMMAND
Enter key password for <dpe-key> : OLD PASSWORD
New key password for <dpe-key>: NEW PASSWORD
Re-enter new key password for <dpe-key>: NEW PASSWORD
The new keystore password should be same as given in the descriptor
file.
2 Restart the server process.
Note
[root@rtpfga-s1-upload1 conf]# /etc/init.d/bprAgent restart dpe
[root@rtpfga-s1-upload1 conf]# /etc/init.d/bprAgent status dpe
BAC Process Watchdog is running.
Process [dpe] is running.
Broadband Access Center [BAC 3.8.1.2
(LNX_BAC3_8_1_2_20140918_1230_12)].
Connected to RDU [10.5.1.200].
Caching [3] device configs and [52] files.
188 sessions succeeded and 1 sessions failed.
6 file requests succeeded and 0 file requests failed.
68 immediate device operations succeeded, and 2 failed.
0 home PG redirections succeeded, and 0 failed.
Using signature key name [] with a validity of [3600].
Abbreviated ParamList is enabled.
Running for [4] hours [23] mins [17] secs.
Connection to Remote Object Unsuccessful
Issue
A connection to the remote object could not be made. OVF Tool does not support
this server.
Completed with errors
Cause
The errors are triggered by ovftool command during ova deployment. the errors
can be found in both Console and vCenter logs.
Solution
User must have Administrator privileges to VMware vCenter and ESXi.
Cisco RAN Management System Installation Guide, Release 5.1
12
July 6, 2015
Troubleshooting
VLAN Not Found
VLAN Not Found
Issue
VLAN not found.
Cause
The errors are triggered by ovftool command during ova deployment. the errors
can be found in both Console and vCenter logs.
Solution
Check for the appropriate "portgroup" name on virtual switch of Elastic Sky X
Integrated (ESXi) host or Distributed Virtual Switch (DVS) on VMware vCenter.
Unable to Get Live Data in DCC UI
Issue
Cause
Solution
Live Data of an AP is not coming and Connection request fails.
1 Device is offline.
2 Device is not having its radio activated/ device is registered but not activated.
1 In the Serving Node, add one more route with Destination IP as HNB-GW
SCTP IP and Gateway as Serving Node North Bound IP as in the following
example:
Serving NB Gateway IP-10.5.1.1
HNBGW SCTP IP- 10.5.1.83
Add the following route in Serving node:
route add -net 10.5.1.83 netmask 255.255.255.0 gw 10.5.1.1
2 Activate the device from DCC UI post registration.
3 Verify trouble shooting logs in bac.
4 Verify DPE logs and ZGTT logs from ACS simulator.
Installation Warnings about Removed Parameters
These properties have been completely removed from the 4.0 OVA installation. A warning is given by the
installer, if these properties are found in the OVA descriptor file. However, installation still continues.
prop:vami.gateway.Upload-Node
prop:vami.DNS.Upload-Node
prop:vami.ip0.Upload-Node
prop:vami.netmask0.Upload-Node
prop:vami.ip1.Upload-Node
prop:vami.netmask1.Upload-Node
prop:vami.gateway.Central-Node
prop:vami.DNS.Central-Node
prop:vami.ip0.Central-Node
prop:vami.netmask0.Central-Node
Cisco RAN Management System Installation Guide, Release 5.1
July 6, 2015
13
Troubleshooting
Upload Server is Not Up
prop:vami.ip1.Central-Node
prop:vami.netmask1.Central-Node
prop:vami.gateway.Serving-Node
prop:vami.DNS.Serving-Node
prop:vami.ip0.Serving-Node
prop:vami.netmask0.Serving-Node
prop:vami.ip1.Serving-Node
prop:vami.netmask1.Serving-Node
prop:Debug_Mode
prop:Server_Crl_Urls
prop:Bacadmin_Password
prop:Dccapp_Password
prop:Opstools_Password
prop:Dccadmin_Password
prop:Postgresql_Password
prop:Central_Keystore_Password
prop:Upload_Stat_Password
prop:Upload_Calldrop_Password
prop:Upload_Demand_Password
prop:Upload_Lostipsec_Password
prop:Upload_Lostgwconnection_Password
prop:Upload_Nwlscan_Password
prop:Upload_Periodic_Password
prop:Upload_Restart_Password
prop:Upload_Crash_Password
prop:Upload_Lowmem_Password
prop:Upload_Unknown_Password
prop:Serving_Keystore_Password
prop:Cnradmin_Password
prop:Caradmin_Password
prop:Dpe_Cli_Password
prop:Dpe_Enable_Password
prop:Fc_Realm
prop:Fc_Log_Periodic_Upload_Enable
prop:Fc_Log_Periodic_Upload_Interval
prop:Fc_On_Nwl_Scan_Enable
prop:Fc_On_Lost_Ipsec_Enable
prop:Fc_On_Crash_Upload_Enable
prop:Fc_On_Call_Drop_Enable
prop:Fc_On_Lost_Gw_Connection_Enable
prop:Upload_Keystore_Password
prop:Dpe_Keystore_Password
prop:Bac_Secret
prop:Admin2_Username
prop:Admin2_Password
prop:Admin2_Firstname
prop:Admin2_Lastname
prop:Admin3_Username
prop:Admin3_Password
prop:Admin3_Firstname
prop:Admin3_Lastname
prop:Upgrade_Mode
prop:Asr5k_Hnbgw_Address
Upload Server is Not Up
The upload server fails with java.lang.ExceptionInInitializerError in the following scenarios.
The errors can be seen in opt/CSCOuls/logs/uploadServer.console.log file.
Scenario 1:
Cisco RAN Management System Installation Guide, Release 5.1
14
July 6, 2015
Troubleshooting
Upload Server is Not Up
Issue
Upload Server failed with java.lang.ExceptionInInitializerError
java.lang.ExceptionInInitializerError
at com.cisco.ca.rms.upload.server.UlsSouthBoundServer.getInstance
(UlsSouthBoundServer.java:58)
at com.cisco.ca.rms.upload.server.UlsServer.<init>(UlsServer.java:123)
at com.cisco.ca.rms.upload.server.UlsServer.<init>(UlsServer.java:25)
at com.cisco.ca.rms.upload.server.UlsServer$SingleInstanceHolder.<clinit>
(UlsServer.java:70)
at com.cisco.ca.rms.upload.server.UlsServer.getInstance(UlsServer.java:82)
at com.cisco.ca.rms.upload.server.UlsServer.main(UlsServer.java:55)
Caused by: org.jboss.netty.channel.ChannelException: Failed to bind to:
/10.6.22.12:8080
at org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:298)
at com.cisco.ca.rms.upload.server.UlsSouthBoundServer.<init>
(UlsSouthBoundServer.java:109)
at com.cisco.ca.rms.upload.server.UlsSouthBoundServer.<init>
(UlsSouthBoundServer.java:22)
at
com.cisco.ca.rms.upload.server.UlsSouthBoundServer$SingleInstanceHolder.<clinit>
(UlsSouthBoundServer.java:46)
... 6 more
Caused by: java.net.BindException: Cannot assign requested address
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Unknown Source)
at sun.nio.ch.Net.bind(Unknown Source)
at sun.nio.ch.ServerSocketChannelImpl.bind(Unknown Source)
at sun.nio.ch.ServerSocketAdaptor.bind(Unknown Source)
at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.bind
(NioServerSocketPipelineSink.java:140)
at
org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleServerSocket
(NioServerSocketPipelineSink.java:90)
at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk
(NioServerSocketPipelineSink.java:64)
at org.jboss.netty.channel.Channels.bind(Channels.java:569)
at org.jboss.netty.channel.AbstractChannel.bind(AbstractChannel.java:189)
at org.jboss.netty.bootstrap.ServerBootstrap$Binder.channelOpen(
ServerBootstrap.java:343)
at org.jboss.netty.channel.Channels.fireChannelOpen(Channels.java:170)
at org.jboss.netty.channel.socket.nio.NioServerSocketChannel.<init>
(NioServerSocketChannel.java:80)
at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.newChannel
(NioServerSocketChannelFactory.java:158)
at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.newChannel
(NioServerSocketChannelFactory.java:86)
at org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:277)
... 9 more
Cause
The server failed to bind to the IP /10.6.22.12:8080 because the requested address was
unavailable.
Solution
Navigate to /opt/CSCOuls/conf and modify the UploadServer.properties file with proper
SB and NB IP address.
Scenario 2:
Cisco RAN Management System Installation Guide, Release 5.1
July 6, 2015
15
Troubleshooting
Upload Server is Not Up
Issue
Upload Server failed with java.lang.ExceptionInInitializerError
java.lang.ExceptionInInitializerError
at com.cisco.ca.rms.upload.server.security.UlsSbSslContextMgr.getInstance
(UlsSbSslContextMgr.java:65)
at com.cisco.ca.rms.upload.server.UlsSouthBoundPipelineFactory.<init>
(UlsSouthBoundPipelineFactory.java:86)
at com.cisco.ca.rms.upload.server.UlsSouthBoundServer.<init>
(UlsSouthBoundServer.java:102)
at com.cisco.ca.rms.upload.server.UlsSouthBoundServer.<init>
(UlsSouthBoundServer.java:22)
at
com.cisco.ca.rms.upload.server.UlsSouthBoundServer$SingleInstanceHolder.<clinit>
(UlsSouthBoundServer.java:46)
at com.cisco.ca.rms.upload.server.UlsSouthBoundServer.getInstance
(UlsSouthBoundServer.java:58)
at com.cisco.ca.rms.upload.server.UlsServer.<init>(UlsServer.java:123)
at com.cisco.ca.rms.upload.server.UlsServer.<init>(UlsServer.java:25)
at com.cisco.ca.rms.upload.server.UlsServer$SingleInstanceHolder.<clinit>
(UlsServer.java:70)
at com.cisco.ca.rms.upload.server.UlsServer.getInstance(UlsServer.java:82)
at com.cisco.ca.rms.upload.server.UlsServer.main(UlsServer.java:55)
Caused by: java.lang.IllegalStateException: java.io.IOException:
Keystore was tampered with, or password was incorrect
at com.cisco.ca.rms.commons.security.SslContextManager.<init>
(SslContextManager.java:79)
at com.cisco.ca.rms.upload.server.security.UlsSbSslContextMgr.<init>
(UlsSbSslContextMgr.java:72)
at com.cisco.ca.rms.upload.server.security.UlsSbSslContextMgr.<init>
(UlsSbSslContextMgr.java:28)
at
com.cisco.ca.rms.upload.server.security.UlsSbSslContextMgr$SingleInstanceHolder.<clinit>
(UlsSbSslContextMgr.java:53)
... 11 more
Caused by: java.io.IOException: Keystore was tampered with, or password was
incorrect
at sun.security.provider.JavaKeyStore.engineLoad(Unknown Source)
at sun.security.provider.JavaKeyStore$JKS.engineLoad(Unknown Source)
at java.security.KeyStore.load(Unknown Source)
at com.cisco.ca.rms.upload.server.security.UlsSbSslContextMgr.loadKeyManagers
(UlsSbSslContextMgr.java:91)
at
com.cisco.ca.rms.commons.security.SslContextManager.<init>(SslContextManager.java:48)
... 14 more
Caused by: java.security.UnrecoverableKeyException: Password verification
failed
... 19 more
Cause
The Keystore was tampered with, or password entered is incorrect resulting in a password
verification failure.
This occurs when the password used to generate the Keystore file is different than the one
given for the property “Upload_Keystore_Password” in descriptor file.
Cisco RAN Management System Installation Guide, Release 5.1
16
July 6, 2015
Troubleshooting
Upload Server is Not Up
Solution
1 Navigate to /opt/CSCOuls/conf and execute the below command to change the password
of the Keystore file.
"[root@rtpfga-s1-upload1 conf]# keytool -storepasswd -keystore
uls.keystore"
Output:
keytool -storepasswd -keystore dpe.keystore
Enter keystore password:OLD PASSWORD
New keystore password: NEW PASSWORD
Re-enter new keystore password: NEW PASSWORD
The new Keystore password should be same as given in the descriptor
file.
2 Run another command before restarting the server to change the key password.
Note
keytool -keypasswd -keystore dpe.keystore -alias dpe -key
Enter keystore password: NEW AS PER LAST COMMAND
Enter key password for <dpe-key> : OLD PASSWORD
New key password for <dpe-key>: NEW PASSWORD
Re-enter new key password for <dpe-key>: NEW PASSWORD
3 Restart the server process.
[root@rtpfga-s1-upload1 conf]# service god restart
[root@rtpfga-s1-upload1 conf]# service god status
UploadServer: up
Scenario 3:
Cisco RAN Management System Installation Guide, Release 5.1
July 6, 2015
17
Troubleshooting
Upload Server is Not Up
Issue
Upload Server failed with java.lang.ExceptionInInitializerError
java.lang.ExceptionInInitializerError
at com.cisco.ca.rms.dcc.lus.server.LusNorthBoundServer.getInstance
(LusNorthBoundServer.java:65)
at com.cisco.ca.rms.dcc.lus.server.LusServer.<init>(LusServer.java:98)
at com.cisco.ca.rms.dcc.lus.server.LusServer.<init>(LusServer.java:17)
at com.cisco.ca.rms.dcc.lus.server.LusServer$SingleInstanceHolder.<clinit>
(LusServer.java:45)
at com.cisco.ca.rms.dcc.lus.server.LusServer.getInstance(LusServer.java:57)
at com.cisco.ca.rms.dcc.lus.server.LusServer.main(LusServer.java:30)
Caused by: org.jboss.netty.channel.ChannelException: Failed to bind to:
/0.0.0.0:8082
at org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:298)
at com.cisco.ca.rms.dcc.lus.server.LusNorthBoundServer.<init>
(LusNorthBoundServer.java:120)
at com.cisco.ca.rms.dcc.lus.server.LusNorthBoundServer.<init>
(LusNorthBoundServer.java:30)
at
com.cisco.ca.rms.dcc.lus.server.LusNorthBoundServer$SingleInstanceHolder.<clinit>
(LusNorthBoundServer.java:53)
... 6 more
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:137)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:77)
at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.bind
(NioServerSocketPipelineSink.java:140)
at
org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleServerSocket
(NioServerSocketPipelineSink.java:92)
at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk
(NioServerSocketPipelineSink.java:66)
at org.jboss.netty.channel.Channels.bind(Channels.java:462)
at org.jboss.netty.channel.AbstractChannel.bind(AbstractChannel.java:186)
at org.jboss.netty.bootstrap.ServerBootstrap$Binder.channelOpen
(ServerBootstrap.java:343)
at org.jboss.netty.channel.Channels.fireChannelOpen(Channels.java:170)
at org.jboss.netty.channel.socket.nio.NioServerSocketChannel.<init>
(NioServerSocketChannel.java:77)
at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.newChannel
(NioServerSocketChannelFactory.java:137)
at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.newChannel
(NioServerSocketChannelFactory.java:85)
at org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:277)
... 9 more
Cause
The server failed to bind to the IP /0.0.0.0:8082 because the requested address is already
in use.
Solution
Execute the command:
netstat –anp |grep <port number>
For example:
[root@rtpfga-s1-upload1 conf]# netstat -anp |grep 8082
tcp
0
0 10.6.23.16:8082
0.0.0.0:*
LISTEN
26842/java
Kill the particular process.
[root@rtpfga-s1-upload1 conf]# kill -9 26842
Start the server.
“[root@rtpfga-s1-upload1 conf]# service god start
[root@rtpfga-s1-upload1 conf]# service god status
UploadServer: up”
Cisco RAN Management System Installation Guide, Release 5.1
18
July 6, 2015
Troubleshooting
OVA Installation failures
OVA Installation failures
Issue
If the OVA installer displays an error on the installation Console.
Cause
OVA Installation failures
Solution
If there are any issues during OVA installation, the ova-first-boot.log should be
referred that is present in the Central node and Serving node. Validate the appropriate
errors in the boot log files.
Update failures in group type, Site - DCC UI throws an error
Issue
SITE Creation Fails While Importing All Mandatory and Optional Parameters.
Cause
Invalid parameter value- FC-CSON-STATUS-HSCO-INNER with Optimised.
Solution
For FC-CSON-STATUS-HSCO-INNER parameter, allowed value is Optimized
not Optimised. The spelling for Optimized should be corrected.
Kernel Panic While Upgrading to RMS, Release 5.1
To recover the system from kernel panic while upgrading, follow these steps
Note
Follow this procedure only when the following error is seen:
Kernel panic-not syncing: VFS: unable to mount root fs on
unknown block(0,0)
Cisco RAN Management System Installation Guide, Release 5.1
July 6, 2015
19
Troubleshooting
Network Unreachable on Cloning RMS VM
Procedure
Step 1
Step 2
Open the VM console when you encounter the kernel panic error.
In the VM console, click on the VM option, then select guest > Send ctrl+alt+del.
Step 3
Wait for the "Booting Red Hat Enterprise Linux Server in X seconds..." countdown to begin and press any
key to enter the menu.
Select the kernel in the second line (older kernel) when the kernel list is displayed and press Enter. The
selected older kernel will boot.
In the login screen, provide the admin username/password. Then switch to root user using the root credentials.
Navigate to the /tmp directory and copy the upgraded kernel rpm file in the system (that is, if upgrading to
RHEL 6.6, the rpm file name will be kernel-2.6.32-504.el6.x86_64.rpm).
Navigate to /boot directory and rename the latest initrd-2.6.32-504.el6.x86_64.img (assuming upgrading
to RHEL 6.6) files to 2.6.32-504.el6.x86_64.img.old.
Step 4
Step 5
Step 6
Step 7
Step 8
Verify the kernel rpm already installed on the system.
rpm –qa|grep kernel
The output of the above command will list the available kernel rpms in this system. Check that the latest kernel
rpm is seen in this list (example, kernel-2.6.32-504.el6.x86_64).
Step 9
Remove the package (upgraded kernel) if the kernel-2.6.32-504.el6.x86_64.rpm is already installed (in case
of RHEL 6.6) by using the following command.
rpm -e kernel-2.6.32-504.e16.x86_64
Step 10 Verify that the upgraded kernel is removed if it was already installed using the following command:
rpm -qa|grep kernel
Step 11 Navigate to the /tmp location and reinstall the latest rpm copied in Step 6 using the following command:
rpm -ivh -force kernel-2.6.32-504.e16.x86_64.rpm
Step 12 Navigate to the /boot location after reinstallation and verify if the initrd-2.6.32-504.el6.x86_64.img is
copied.
Step 13 Verify if the /boot/grub/grub.conf points to the latest kernel ("default" should be zero if latest kernel
is placed in the first place in the grub.conf file).
Step 14 Reboot the system. The system will now boot accurately.
Network Unreachable on Cloning RMS VM
When the network is unreachable on cloning RMS VM due to MAC address change, perform the following
steps to resolve it.
Cisco RAN Management System Installation Guide, Release 5.1
20
July 6, 2015
Troubleshooting
Network Unreachable on Cloning RMS VM
Procedure
Step 1
Step 2
Step 3
Step 4
Step 5
Step 6
Step 7
Log in to vCenter.
Open console of the affected VM.
Reboot VM from "VM" > "guest " > "Send ctrl+alt+del ".
Wait for the "Booting Red Hat Enterprise Linux Server in X seconds..." countdown to begin and press any
key to enter the menu.
Select the first kernel, that is, Red Hat Enterprise Linux Server (2.6. 32-504.e16 x86_64), when the kernel
list is displayed and press the e key, once to edit the command before booting.
Use the arrow key to select the second line starting with "kernel" in the next screen, and press the e key to
edit the selected command in the boot sequence.
Next, press the spacebar once and add number "1" and press Enter.
It will return to previous screen again where "kernel " line was selected.
Step 8
Press the b key, once to boot.
The system will boot in run level 1 and come to # prompt.
Step 9
Go to vCenter UI and click VM > Edit Settings to open the Virtual Machine Properties window.
Step 10 Note down both the network interface listed in the Hardware column and the "MAC address". The network
adaptor1 is treated as eth0 by RHEL.
Step 11 Exit the Virtual Machine Properties window.
Step 12 Return to the VM console and edit the /etc/udev/rules.d/70-persistent-net.rules file.
Step 13 Comment the lines that are not matching with above-noted MAC address.
Step 14 Change the interface ID in the order noted in the VM > Edit Settings window (see, Step 10).
Step 15 Save the file and reboot the system.
After rebooting, the system will be available.
Cisco RAN Management System Installation Guide, Release 5.1
July 6, 2015
21
Troubleshooting
Network Unreachable on Cloning RMS VM
Cisco RAN Management System Installation Guide, Release 5.1
22
July 6, 2015
Was this manual useful for you? yes no
Thank you for your participation!

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Download PDF

advertising