Use Case Scenario Restoring ACI Fabric to Factory Defaults tdeleon

Use Case Scenario
Restoring ACI Fabric to Factory Defaults
The following use scenario is where a customer wants to restore their ACI lab fabric back to
factory defaults and want to start over. The customer does not need to export\import
configuration. The will be configuring the ACI Fabric setup configuration from scratch. So the
admin user ran the command "eraseconfig setup" on their APIC controllers. The APIC
controllers rebooted and they configured each APIC using the “setup script”.
After the APICs reloaded, the admin user cannot access & login to any of the APIC controllers.
The admin tried access the APICs using the GUI, SSH, & the CIMC KVM Console. All attempts
failed. When trying to access the APIC Admin GUI, they get an an error like:
"REST Endpoint user authorization datastore is not initialized - Check Fabric Membership
Status of this fabric node"
What can the customer do to resolve this issue?
ASSESSMENT & STEPS TO RECOVER:
We see this problem scenario often when admin users are trying to restore the ACI fabric to
factory defaults. When resetting an “ACI Fabric" to factory defaults, you will also need to restore
the switches to factory default also. The switches maintain information from the previous ACI
fabric and there will be a conflict between the switches and the APICs. This will prevent the ACI
Cluster fabric to form.
Steps to Restore the Switch to Factory Defaults:
• access each switch via the console
• execute the command “setup-clean-config.sh"
• execute the command "ls -al /bootflash/aci-n9000-dk9*". This will list the ACI switch images
on the current leaf or spine node.
• execute the command "setup-bootvars.sh <aci-n9000-dk9.image_name>". This will set the
boot variable for firmware for the ACI leaf or spine node.
• execute the command "cat /mnt/cfg/0/boot/grub/menu.lst.local" and "cat /mnt/cfg/1/boot/
grub/menu.lst.local". Verify that each boot partition was set to the "'aci-n9000dk9.image_name" that you configured above.
• execute the command "reload" and answer "y" to confirm the reload.
The switch will reload and will enter into fabric discovery mode. Since the APICs have also
been restored to factory default, the switches will need to added in the Fabric-> Inventory->
Fabric Membership screen. Once the Leaf that is connected to APIC1 is learned and added to
the fabric inventory, the spines and the other leafs will also need to be added via fabric
discovery.
for example:
leaf1# setup-clean-config.sh
In progress
In progress
In progress
In progress
In progress
Done
tdeleon@cisco.com
Page 1 of 4
version1
Use Case Scenario
Restoring ACI Fabric to Factory Defaults
leaf1# ls -al /bootflash/aci-n9000-dk9*
-rw-rw-rw- 1 root root 1069883063 Sep 30 12:20 /bootflash/aci-n9000-dk9.12.1.1g.bin
-rwxrwxrwx 1 admin admin 1069875494 Oct 2 01:06 /bootflash/aci-n9000-dk9.12.1.1h.bin
leaf1# setup-bootvars.sh aci-n9000-dk9.12.1.1h.bin
In progress
In progress
In progress
In progress
In progress
In progress
Done
leaf1# cat /mnt/cfg/0/boot/grub/menu.lst.local
boot aci-n9000-dk9.12.1.1h.bin
leaf1# cat /mnt/cfg/1/boot/grub/menu.lst.local
boot aci-n9000-dk9.12.1.1h.bin
leaf1# reload
This command will reload the chassis, Proceed (y/n)? [n]: y
[35718.013795] nvram_klm wrote rr=9 rr_str=PolicyElem Ch reload to nvram
[35718.088901] Collected 8 ext4 filesystems
[35718.138682] Freezing filesystems
[35718.316737] Collected 1 ubi filesystems
[35718.363438] Freezing filesystems
[35718.403932] Done freezing filesystems
[35718.449601] Putting SSD in stdby
[35718.995860] Done putting SSD in stdby 0
[35719.042661] Done offlining SSD
[35719.080083] obfl_klm writing reset reason 9, PolicyElem Ch reload
Note: If you clear the configuration on the APIC or a leaf and for some reason cannot login via
the "admin" user. You may be able to login (via the console or CIMC) using the "rescue-user"
account. There should be no password for "rescue-user" account when the node's
configuration has been cleared. If you cannot login to a node due to fabric discovery issues, the
password for the "rescue-user" account will be the last known "admin" password.
tdeleon@cisco.com
Page 2 of 4
version1
Use Case Scenario
Restoring ACI Fabric to Factory Defaults
BEST PRACTICE FOR RESETTING AN ACI FABRIC BACK TO FACTORY DEFAULTS:
The following steps are for restoring an entire fabric to factory defaults. This is a normal
procedure which excludes the exporting & importing of a configuration file.
pre-requisites:
✓ Console access to all leaf & spine nodes .
✓ Console or CIMC access to all APIC controllers.
✓ All necessary cabling & connections are connected for APICs, leaf, and spine nodes.
Note: Console & CIMC access is necessary for troubleshooting and viewing the boot up
process.
•
•
Access each APIC as admin user using the Console, CIMC, or SSH connection.
Erase the existing APIC configuration using the following command(s):
For Newer Releases of ACI firmware versions:
acidiag touch clean
acidiag touch setup
acidiag reboot
For Older Releases of ACI firmware versions:
eraseconfig setup
Note: For an APIC Cluster that has more than one APIC, "Power Off" (using the CIMC) all
APICs except APIC1.
•
After APIC1 reboots, Go thru the SETUP script and configure at the requested parameters
for this ACI Fabric.
At this time, the admin user will need to erase the configuration from all of the Leaf & Spine
nodes so they can go thru the fabric discovery process. The "Old" known fabric information will
be incorrect and can cause issues with the Leaf & Spine nodes joining the ACI fabric.
•
•
Access each SWITCH as admin user using the Console, CIMC, or SSH connection.
Erase the existing switch configuration using the following command(s):
- execute the command "setup-clean-config.sh"
- execute the command "ls -al /bootflash/aci-n9000-dk9*". This will list the ACI switch
images on the current leaf or spine node.
- execute the command "setup-bootvars.sh <aci-n9000-dk9.image_name>". This will
set the boot variable for firmware for the ACI leaf or spine node.
- execute the command "cat /mnt/cfg/0/boot/grub/menu.lst.local" and "cat /mnt/cfg/1/
boot/grub/menu.lst.local". Verify that each boot partition was set to the "aci-n9000dk9.image_name" that you configured above.
- execute the command "reload" and answer "y" to confirm the reload.
tdeleon@cisco.com
Page 3 of 4
version1
Use Case Scenario
Restoring ACI Fabric to Factory Defaults
The switch will reload and will enter into fabric discovery mode. Since the APICs have also
been restored to factory default, the switches will need to added in the Fabric-> Inventory->
Fabric Membership screen. Once the Leaf that is connected to APIC1 is learned and added to
the fabric inventory, the spines and the other leafs will also need to be added via fabric
discovery.
•
After "ALL" of the Leaf & Spine nodes have been confgured and successfully joined the ACI
fabric, you can proceed to configure the other remaining APICs. Execute the Configuration
in Sequence and make sure the APIC Cluster is fully fit before proceeding on to the next
APIC in the Cluster.
- Access the CIMC of APIC2 and power on the APIC.
- Go thru the SETUP script and configure at the requested parameters for this ACI
Fabric.
- After the APIC reloads, verify that APIC2 joins the APIC Cluster and the APICs move
to a "Fully Fit" State. Once fully fit, Repeat the Process for APIC3 or all of the
remaining APICs.
•
Once all of the APICs have been configured and are "Fully Fit", execute the following
commands to verify the successful restoration of the intended ACI Fabric configuration:
- acidiag avread
- acidiag fnvread
- show controller
- show switch detail
- show version
Note: The listed steps above should work in most cases of restoring the ACI fabric to factory
defaults. That said, issues do occur and you may need additional help from the Cisco ACI
Solutions TAC Teams. So please record your steps used and record any error messages that
you may encounter. Capturing the Terminal output for your Terminal sessions while executing
these steps will aid the Engineers in determining where in the process you may have
encountered the errors or failures.
tdeleon@cisco.com
Page 4 of 4
version1
Open as PDF
Similar pages