2.13.5 PDF - Read the Docs

xCAT3 Documentation
Release 2.12
IBM Corporation
Jun 30, 2017
Contents
1
Table of Contents
1.1 Overview . . . .
1.2 Install Guides . .
1.3 Admin Guide . .
1.4 Advanced Topics
1.5 Troubleshooting
1.6 Developers . . .
1.7 Need Help . . .
1.8 Security Notices
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
3
3
18
37
758
957
961
978
979
i
ii
xCAT3 Documentation, Release 2.12
xCAT stands for Extreme Cloud/Cluster Administration Toolkit.
xCAT offers complete management of clouds, clusters, HPC, grids, datacenters, renderfarms, online gaming infrastructure, and whatever tomorrows next buzzword may be.
xCAT enables the administrator to:
1. Discover the hardware servers
2. Execute remote system management
3. Provision operating systems on physical or virtual machines
4. Provision machines in Diskful (stateful) and Diskless (stateless)
5. Install and configure user applications
6. Parallel system management
7. Integrate xCAT in Cloud
You’ve reached xCAT documentation site, The main page product page is http://xcat.org
xCAT is an open source project hosted on GitHub. Go to GitHub to view the source, open issues, ask questions, and
participate in the project.
Enjoy!
Contents
1
xCAT3 Documentation, Release 2.12
2
Contents
CHAPTER
1
Table of Contents
Overview
xCAT enables you to easily manage large number of servers for any type of technical computing workload. xCAT is
known for exceptional scaling, wide variety of supported hardware and operating systems, virtualization platforms,
and complete “day0” setup capabilities.
Differentiators
• xCAT Scales
Beyond all IT budgets, up to 100,000s of nodes with distributed architecture.
• Open Source
Eclipse Public License. Support contracts are also available, contact IBM.
• Supports Multiple Operating Systems
RHEL, SLES, Ubuntu, Debian, CentOS, Fedora, Scientific Linux, Oracle Linux, Windows, Esxi, RHEV, and
more!
• Support Multiple Hardware
IBM Power, IBM Power LE, x86_64
• Support Multiple Virtualization Infrastructures
IBM PowerKVM, KVM, IBM zVM, ESXI, XEN
• Support Multiple Installation Options
Diskful (Install to Hard Disk), Diskless (Runs in memory), Cloning
• Built in Automatic discovery
No need to power on one machine at a time for discovery. Nodes that fail can be replaced and back in action
simply by powering the new one on.
3
xCAT3 Documentation, Release 2.12
• RestFUL API
Provides a Rest API interface for the third-party software to integrate with
Features
1. Discover the hardware servers
• Manually define
• MTMS-based discovery
• Switch-based discovery
• Sequential-based discovery
2. Execute remote system management against the discovered server
• Remote power control
• Remote console support
• Remote inventory/vitals information query
• Remote event log query
3. Provision Operating Systems on physical (Bare-metal) or virtual machines
• RHEL
• SLES
• Ubuntu
• Debian
• Fedora
• CentOS
• Scientific Linux
• Oracle Linux
• PowerKVM
• Esxi
• RHEV
• Windows
• AIX
4. Provision machines in
• Diskful (Scripted install, Clone)
• Stateless
5. Install and configure user applications
• During OS install
• After the OS install
• HPC products - GPFS, Parallel Environment, LSF, compilers ...
• Big Data - Hadoop, Symphony
4
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
• Cloud - Openstack, Chef
6. Parallel system management
• Parallel shell (Run shell command against nodes in parallel)
• Parallel copy
• Parallel ping
7. Integrate xCAT in Cloud
• Openstack
• SoftLayer
Operating System & Hardware Support Matrix
RHEL
SLES
Ubuntu
CentOS
AIX
Windows
Power
yes
yes
no
no
yes
no
Power LE
yes
yes
yes
no
no
no
zVM
yes
yes
no
no
no
no
Power KVM
yes
yes
yes
no
no
no
x86_64
yes
yes
yes
yes
no
yes
x86_64 KVM
yes
yes
yes
yes
no
yes
x86_64 Esxi
yes
yes
yes
yes
no
yes
Architecture
The following diagram shows the basic structure of xCAT:
1.1. Overview
5
xCAT3 Documentation, Release 2.12
xCAT Management Node (xCAT Mgmt Node): The server where xCAT software is installed and used as the single
point to perform system management over the entire cluster. On this node, a database is configured to store
the xCAT node definitions. Network services (dhcp, tftp, http, etc) are enabled to respond in Operating system
deployment.
Service Node: One or more defined “slave” servers operating under the Management Node to assist in system management to reduce the load (cpu, network bandwidth) when using a single Management Node. This concept is
necessary when managing very large clusters.
Compute Node: The compute nodes are the target servers which xCAT is managing.
Network Services (dhcp, tftp, http,etc): The various network services necessary to perform Operating System deployment over the network. xCAT will bring up and configure the network services automatically without any
intervention from the System Administrator.
Service Processor (SP): A module embedded in the hardware server used to perform the out-of-band hardware control. (e.g. Integrated Management Module (IMM), Flexible Service Processor (FSP), Baseboard Management
Controller (BMC), etc)
Management network: The network used by the Management Node (or Service Node) to install operating systems
and manage the nodes. The Management Node and in-band Network Interface Card (NIC) of the nodes are
6
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
connected to this network. If you have a large cluster utilizing Service Nodes, sometimes this network is
segregated into separate VLANs for each Service Node.
Service network: The network used by the Management Node (or Service Node) to control the nodes using out-ofband management using the Service Processor. If the Service Processor is configured in shared mode (meaning
the NIC of the Service process is used for the SP and the host), then this network can be combined with the
management network.
Application network: The network used by the applications on the Compute nodes to communicate among each
other.
Site (Public) network: The network used by users to access the Management Nodes or access the Compute Nodes
directly.
RestAPIs: The RestAPI interface can be used by the third-party application to integrate with xCAT.
Quick Start Guide
If xCAT looks suitable for your requirement, following steps are recommended procedure to set up an xCAT cluster.
1. Find a server as your xCAT management node
The server can be a bare-metal server or a virtual machine. The major factor for selecting a server is the number
of machines in your cluster. The bigger the cluster is, the performance of server need to be better.
NOTE: The architecture of xCAT management node is recommended to be same as the target compute node in
the cluster.
2. Install xCAT on your selected server
The server which installed xCAT will be the xCAT Management Node.
Refer to the doc: xCAT Install Guide to learn how to install xCAT on a server.
3. Start to use xCAT management node
Refer to the doc: xCAT Admin Guide.
4. Discover target nodes in the cluster
You have to define the target nodes in the xCAT database before managing them.
For a small cluster (less than 5), you can collect the information of target nodes one by one and then define them
manually through mkdef command.
For a bigger cluster, you can use the automatic method to discover the target nodes. The discovered nodes will
be defined to xCAT database. You can use lsdef to display them.
Refer to the doc: xCAT discovery Guide to learn how to discover and define compute nodes.
5. Try to perform the hardware control against the target nodes
Now you have the node definition. Verify the hardware control for defined nodes is working. e.g. rpower
<node> stat.
Refer to the doc: Hardware Management to learn how to perform the remote hardware control.
6. Deploy OS on the target nodes
• Prepare the OS images
• Customize the OS images (Optional)
• Perform the OS deployment
Refer to the doc: Diskful Install, Diskless Install to learn how to deploy OS for a target node.
1.1. Overview
7
xCAT3 Documentation, Release 2.12
7. Update the OS after the deployment
You may require to update the OS of certain target nodes after the OS deployment, try the updatenode
command. updatenode command can execute the following tasks for target nodes:
• Install additional software/application for the target nodes
• Sync some files to the target nodes
• Run some postscript for the target nodes
Refer to the doc: Updatenode to learn how to use updatenode command.
8. Run parallel commands
When managing a cluster with hundreds or thousands of nodes, operating on many nodes in parallel might be necessary. xCAT has some parallel commands for that.
• Parallel shell
• Parallel copy
• Parallel ping
Refer to the Parallel Commands to learn how to use parallel commands.
1. Contribute to xCAT (Optional)
While using xCAT, if you find something (code, documentation, ...) that can be improved and you want to contribute
that to xCAT, do that for your and other xCAT users benefit. And welcome to xCAT community!
Refer to the Developers to learn how to contribute to xCAT community.
xCAT2 Release Information
The following table is a summary of the new operating system (OS), hardware, and features that are added to each
xCAT release. The OS and hardware listed in the table have been fully tested with xCAT. For a more detailed list of
new function, bug fixes, restrictions and known problems, refer to the individual release notes for a specific release.
• RHEL - Red Hat Enterprise Linux
• SLES - Suse Linux Enterprise Server
• UBT - Ubuntu
8
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
xCAT 2.13.x
xCAT Version
New OS
• OpenBMC
support(experimental):
rinv rinstall bmcdiscover
• RHEL 6.9
• OpenBMC
support(experimental):
rpower rcons
• Add -C for rmdef to
run nodeset offline
xCAT 2.13.3
2017/4/14
2.13.3 Release Notes
• Refine ONIE switch
support doc
• Add -p for xcatprobe
osdeploy
to support performance calculate
• To support PDU
xCAT 2.13.2
2017/2/24
2.13.2 Release Notes
• ONIE switch support
• refine
xcatprobe
subcommand:
xcatmn, osdeploy
• add ntp-wait for
genesis-base
xCAT 2.13.1
2017/1/13
2.13.1 Release Notes
• SLES 12.2
1.1. Overview
New Feature
• RHV 4.1
xCAT 2.13.4
2017/5/19
2.13.4 Release Notes
xCAT 2.13
2016/12/09
2.13 Release Notes
New Hardware
• update drivers for
genesis-base mlx4en 3.2-1.0.1.1 i40e
1.5.16
• rflash saving flashing progress
• Update configureRAID document
• statelite image create by copyds
9
xCAT3 Documentation, Release 2.12
xCAT 2.12.x
xCAT Version
xCAT 2.12.4
2016/11/11
2.12.4 Release Notes
New OS
• RHEL 7.3 LE
• RHEV 4.0
• UBT 16.04.1
xCAT 2.12.2
2016/08/19
2.12.2 Release Notes
• noboot added to
dhcpinterface
• new xcatprobe subcommand: xcatmn,
deploy and discover
• nodeset
<noderange>
offline
• Enhance: node status update
• Support Bond for
install nics
• xcatprobe osdeploy
-r (BETA)
• New opt: packimage -m -c
• New xCAT install
tool: go-xcat
• New
opt:
mkdef/lsdef
–
template
• Support rinstall for
all OS/ARCH
• site.xcatdebugmode
for diskless
• Refine
discovery
framework
• rscan <kvmhost>
• New:
xcatprobe
(experimental)
xCAT 2.12.1
2016/07/08
2.12.1 Release Notes
10
New Feature
• GitHub Issues resolved
• rinv options for
OpenPOWER
• switch based switch
discovery
• additional options
added to xcatprobe
command
• mkdef takes file
redirection
xCAT 2.12.3
2016/09/30
2.12.3 Release Notes
xCAT 2.12
2016/5/20
2.12 Release Notes
New Hardware
• RHEL 6.8
• UBT 14.4.4 LE
• UBT 16.04
• Docker: xCAT in
Docker
• Docker: container
life cycle mgt
• Docker:
Set up
Chapter 1. Docker
Table of
Contents
Registry
• New
command:
getadapter
• Add
xCAT3 Documentation, Release 2.12
xCAT 2.11.x
xCAT Version
New OS
New Hardware
New Feature
• Bug fix
xCAT 2.11.1
2016/04/22
2.11.1 Release Notes
xCAT 2.11
2015/12/11
2.11 Release Notes
1.1. Overview
•
•
•
•
RHEL 7.2 LE
UBT 14.4.3 LE
UBT 15.10 LE
PowerKVM 3.1
•
•
•
•
•
S822LC(GCA)
S822LC(GTA)
S812LC
NeuCloud OP
ZoomNet RP
• NVIDIA GPU for
OpenPOWER
• Infiniband
for
OpenPOWER
• SW KIT support for
OpenPOWER
• renergy command
for OpenPOWER
• rflash command for
OpenPOWER
• Add xCAT Troubleshooting Log
• xCAT Log Classification
• RAID Configuration
• Accelerate genimage process
• Add bmcdiscover
Command
• Enhance
xcatdebugmode
• new xCAT doc in
ReadTheDocs
11
xCAT3 Documentation, Release 2.12
xCAT 2.10.x
xCAT Version
xCAT 2.10
2015/07/31
2.10 Release Notes
12
New OS
•
•
•
•
•
•
RHEL 7.1 LE
UBT 15.4 LE
SLES 12 LE
RHEL 6.7
CentOS 7.1
SLES 11 SP4
New Hardware
• Power 8 LE
New Feature
• Ubuntu LE -> RH
7.1 Mix
• Cuda install for
Ubuntu 14.4.2
• additional
kernel
parameters
• customized
disk
part (Ubuntu)
• RAID
configure
base iprconfig
• New
command:
switchdiscover
• New
command:
makentp
• New
command:
bmcdiscovery
• Support getmacs –
noping
• site.xcatdebugmode
• validate netboot attribute
• buildcore on local
server
• copycds generates
fewer osimage
• nodeset
only
accepts osimage=
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
xCAT 2.9.x
xCAT Version
xCAT 2.9.3 for AIX
2016/03/11
2.9.3 Release Notes
xCAT 2.9.2 for AIX
2015/11/11
2.9.2 Release Notes
1
xCAT 2.9.1
2015/03/20
2.9.1 Release Notes
xCAT 2.9
2014/12/12
2.9 Release Notes
New OS
New Hardware
• AIX 7.2.0
• AIX 7.1.4.1
• AIX 6.1.8.6
• AIX 6.1.9.5
• AIX 7.1.3.5
• new format in synclist (node)
• Power 8 for AIX
• RHEL 7.1
• UBT 14.04.2
• SLES 11 SP3 and
later ONLY
•
•
•
•
•
•
•
UBT 14.4 LE
UBT 14.4.1 LE
UBT 14.10
SLES 12
RHEL 6.6
AIX 7.1.3.15
PowerKVM
New Feature
• ssl version control
in xcatd
• Nvidia GPU
• Ubuntu Local Mirror
• SLES12 diskless
• Energy
management for Power
8
• RHEL 7.1 LE ->
BE mix cluster
• nics.nicextraparams
• xCAT in Docker
Image
• confluent replaces
conserver
• TLSv1 in xcatd
• New GPG key for
xCAT packages
• fast restart xcatd
(systemd)
• netboot
method:
grub2-tftp
• netboot
method:
grub2-http
• Power 8 LE
• sysclone enhancements
• site.auditnosyslog
• site.nmapoptions
• customize
postscripts
• Power 8 LE hw discover
• IB support for P8
LE
1 xCAT 2.9.1 onwards provides support for Kernel-based Virtual Machines (KVM) and requires an operating system that ships the
perl-Sys-Virt package.
1.1. Overview
13
xCAT3 Documentation, Release 2.12
14
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
xCAT 2.8.x
xCAT Version
xCAT 2.8.4
2014/03/23
2.8.4 Release Notes
xCAT 2.8.3
2013/11/15
2.8.3 Release Notes
New OS
• RHEL 6.5
• RHEL 5.10
xCAT 2.8
2013/02/28
2.8 Release Notes
1.1. Overview
New Feature
• RHEL 7 experimental,
• support xCAT clusterzones
• commands
enhancements
• AIX 7.3.1.1
• AIX 7.3.1.0
• AIX 7.1.2
• Xeon Phi (P2)
• NS nx360M4
• xcatd flow control
• sysclone
x86_64
image
• enhance
genitird
and nodeset
• enhance confignics,
KIT
• enhance sequential
discovery
• deploy OpenStack
on Ubuntu
• SLES 11 SP3
• Xeon Phi (P1)
• HPC KIT for ppc64
• sysclone
x86_64
image (P1)
• enhance xdsh, updatenode
• localdisk for diskless
• enhance sequential
discovery
• deploy OpenStack
on Ubuntu
xCAT 2.8.2
2013/06/26
2.8.2 Release Notes
xCAT 2.8.1
2013/06/26
2.8.1 Release Notes
New Hardware
• RHEL 6.4
• RHEL 5.9
• energy
management for flex
• sequential discovery
• KIT enhancements
• osimage enhancements
• IPv6 enhancements
• def/xdsh/xdcp
enhancements
• updatenode
enhancements
• UBT 12.04
• WIN S 2012
• WIN 8 Hv
•
•
•
•
•
•
•
•
Flex IMM setup
Multiple Hostname
KIT support
KVM/zVM
enhancements
RHEV Support
Localdisk
for15
statelite
Manage MN itslef
site auditskipcmds
xCAT3 Documentation, Release 2.12
16
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
xCAT 2.7.x
xCAT Version
xCAT 2.7.8
2014/01/24
2.7.8 Release Notes
New OS
New Hardware
• AIX 7.1.3.1
• AIX 7.1.3.0
• AIX 6.1.9.1
• RHEL 6.4
• sinv for devices
• Flex energy mgt
and rbeacon
• SLES 10 SP4
• AIX 6.1.8
• AIX 7.1.2
• HPC
Integration
updates
• RHEL 6.3
• virtualization with
RHEV
• hardware discovery
for x Flex
• enhanced
AIX
HASN
xCAT 2.7.7
2013/03/17
2.7.7 Release Notes
xCAT 2.7.6
2012/11/30
2.7.6 Release Notes
xCAT 2.7.5
2012/10/29
2.7.5 Release Notes
• SLES11 SP2
• Flex
• improved IPMI for
large systems
• SLES11 SP2
• RHEL 6.2
• Flex
• HPC
Integration
updates
• AIX 7.1.1.3
• Power 775
• Flex for P
• SLES 11 kdump
• HPC
Integration
updates
xCAT 2.7.4
2012/08/27
2.7.4 Release Notes
xCAT 2.7.3
2012/06/22
2.7.3 Release Notes
xCAT 2.7.2
2012/05/25
2.7.2 Release Notes
• RHEL 6.3
• minor
enhancements
• bug fixes
• RHEL 6.2
• xcatd memory usage reduced
• xcatdebug for xcatd
and plugins
17
• lstree command
• x86_64
genesis
boot image
xCAT 2.7.1
2012/04/20
2.7.1 Release Notes
xCAT 2.7
2012/03/19
1.1.
Overview
2.7 Release
Notes
New Feature
xCAT3 Documentation, Release 2.12
Install Guides
Installation Guide for Red Hat Enterprise Linux
For the current list of operating systems supported and verified by the development team for the different releases of
xCAT, see the xCAT2 Release Notes.
Disclaimer These instructions are intended to only be guidelines and specific details may differ slightly based on the
operating system version. Always refer to the operating system documentation for the latest recommended procedures.
Prepare the Management Node
These steps prepare the Management Node for xCAT Installation
Install an OS on the Management Node
Install one of the supported operating systems on your target management node.
The system requirements for your xCAT management node largely depend on the size of the cluster you plan to
manage and the type of provisioning used (diskful, diskless, system clones, etc). The majority of system load comes
during cluster provisioning time.
Memory Requirements:
Cluster Size
Small (< 16)
Medium
Large
Memory (GB)
4-6
6-8
> 16
Configure the Base OS Repository
xCAT uses the yum package manager on RHEL Linux distributions to install and resolve dependency packages provided by the base operating system. Follow this section to create the repository for the base operating system on the
Management Node
1. Copy the DVD iso file to /tmp on the Management Node. This example will use file RHEL-LE-7.
1-20150219.1-Server-ppc64le-dvd1.iso
2. Mount the iso to /mnt/iso/rhels7.1 on the Management Node.
mkdir -p /mnt/iso/rhels7.1
mount -o loop /tmp/RHEL-LE-7.1-20150219.1-Server-ppc64le-dvd1.iso /mnt/iso/rhels7.
˓→1
3. Create a yum repository file /etc/yum.repos.d/rhels71-dvd.repo that points to the locally mounted
iso image from the above step. The file contents should appear as the following:
[rhel-7.1-dvd-server]
name=RHEL 7 SERVER packages
baseurl=file:///mnt/iso/rhels7.1/Server
enabled=1
gpgcheck=1
18
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
Configure the Management Node
By setting properties on the Management Node before installing the xCAT software will allow xCAT to automatically
configure key attributes in the xCAT site table during the install.
1. Ensure a hostname is configured on the management node by issuing the hostname command. [It’s recommended to use a fully qualified domain name (FQDN) when setting the hostname]
(a) To set the hostname of xcatmn.cluster.com:
hostname xcatmn.cluster.com
(b) Add the hostname to the /etc/sysconfig/network in order to persist the hostname on reboot.
(c) Reboot the server and verify the hostname by running the following commands:
• hostname
• hostname -d - should display the domain
2. Reduce the risk of the Management Node IP address being lost by setting the IP to STATIC in the /etc/
sysconfig/network-scripts/ifcfg-<dev> configuration files.
3. Configure any domain search strings and nameservers to the /etc/resolv.conf file.
Installing xCAT
The following sections describe the different methods for installing xCAT.
Automatic Install Using go-xcat
go-xcat is a tool that can be used to fully install or update xCAT. go-xcat will automatically download the correct
package manager repository file from xcat.org and use the public repository to install xCAT. If the xCAT management
node does not have internet connectivity, use process described in the Manual Installation section of the guide.
1. Download the go-xcat tool using wget:
wget https://raw.githubusercontent.com/xcat2/xcat-core/master/xCAT-server/share/
˓→xcat/tools/go-xcat -O - >/tmp/go-xcat
chmod +x /tmp/go-xcat
2. Run the go-xcat tool:
/tmp/go-xcat install
/tmp/go-xcat -x devel install
# installs the latest stable version of xCAT
# installs the latest development version of xCAT
Manual Install Using Software Repositories
xCAT consists of two software packages: xcat-core and xcat-dep
1. xcat-core xCAT’s main software package and is provided in one of the following options:
• Latest Release (Stable) Builds
This is the latest GA (Generally Availability) build that has been tested thoroughly
• Latest Snapshot Builds
1.2. Install Guides
19
xCAT3 Documentation, Release 2.12
This is the latest snapshot of the GA version build that may contain bug fixes but has not yet been
tested thoroughly
• Development Builds
This is the snapshot builds of the new version of xCAT in development. This version has not been
released yet, use as your own risk
2. xcat-dep xCAT’s dependency package. This package is provided as a convenience for the user and contains
dependency packages required by xCAT that are not provided by the operating system.
xCAT is installed by configuring software repositories for xcat-core and xcat-dep and using yum package
manager. The repositories can be publicly hosted or locally hosted.
Configure xCAT Software Repository
xCAT software and repo files can be obtained from: http://xcat.org/download.html
Internet Repository
[xcat-core]
For the xCAT build you want to install, download the xCAT-core.repo file and copy to /etc/yum.repos.d
[xcat-dep]
From the xCAT-dep Online Repository, navigate to the correct subdirectory for the target machine and download the
xCAT-dep.repo file and copy to /etc/yum.repos.d.
If using internet repositories, continue to the next step to install xCAT.
Local Repository
[xcat-core]
1. Download xcat-core:
# downloading the latest development build, core-rpms-snap.tar.bz2
mkdir -p ~/xcat
cd ~/xcat/
wget http://xcat.org/files/xcat/xcat-core/devel/Linux/core-snap/core-rpms-snap.
˓→tar.bz2
2. Extract xcat-core:
tar jxvf core-rpms-snap.tar.bz2
3. Configure the local repository for xcat-core by running mklocalrepo.sh script in the xcat-core directory:
cd ~/xcat/xcat-core
./mklocalrepo.sh
[xcat-dep]
Unless you are downloading xcat-dep to match a specific package tested with a GA release, it’s recommended to
download the latest version of xcat-dep.
20
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
1. Download xcat-dep:
# if downloading xcat-dep from June 11, 2015, xcat-dep-201506110324.tar.bz2
mkdir -p ~/xcat/
cd ~/xcat
wget http://xcat.org/files/xcat/xcat-dep/2.x_Linux/xcat-dep-201506110324.tar.bz2
2. Extract xcat-dep:
tar jxvf xcat-dep-201506110324.tar.bz2
3. Configure the local repository for xcat-dep by switching to the architecture and os subdirectory of the node you
are installing on, then run the mklocalrepo.sh script:
cd ~/xcat/xcat-dep/
# Example, on redhat 7.1 ppc64le: cd rh7/ppc64le
cd <os>/<arch>
./mklocalrepo.sh
Install xCAT
Install xCAT with the following command:
yum clean all (optional)
yum install xCAT
Note: During the install, you must accept the xCAT Security Key to continue:
Retrieving key from file:///root/xcat/xcat-dep/rh6/ppc64/repodata/repomd.xml.key
Importing GPG key 0xC6565BC9:
Userid: "xCAT Security Key <xcat@cn.ibm.com>"
From : /root/xcat/xcat-dep/rh6/ppc64/repodata/repomd.xml.key
Is this ok [y/N]:
Verify xCAT Installation
Quick verification of the xCAT Install can be done running the following steps:
1. Source the profile to add xCAT Commands to your path:
source /etc/profile.d/xcat.sh
2. Check the xCAT version:
lsxcatd -a
3. Check to verify that the xCAT database is initialized by dumping out the site table:
tabdump site
The output should be similar to the following:
#key,value,comments,disable
"blademaxp","64",,
"domain","pok.stglabs.ibm.com",,
1.2. Install Guides
21
xCAT3 Documentation, Release 2.12
"fsptimeout","0",,
"installdir","/install",,
"ipmimaxp","64",,
"ipmiretries","3",,
...
Starting and Stopping
xCAT is started automatically after the installation, but the following commands can be used to start, stop, restart, and
check xCAT status.
• start xCAT:
service xcatd start
[systemd] systemctl start xcatd.service
• stop xCAT:
service xcatd stop
[systemd] systemctl stop xcatd.service
• restart xCAT:
service xcatd restart
[systemd] systemctl restart xcatd.service
• check xCAT status:
service xcatd status
[systemd] systemctl status xcatd.service
Updating xCAT
If at a later date you want to update xCAT, first, update the software repositories and then run:
yum clean metadata # or, yum clean all
yum update '*xCAT*'
# To check and update the packages provided by xcat-dep:
yum update '*xcat*'
Installation Guide for SUSE Linux Enterprise Server
For the current list of operating systems supported and verified by the development team for the different releases of
xCAT, see the xCAT2 Release Notes.
Disclaimer These instructions are intended to only be guidelines and specific details may differ slightly based on the
operating system version. Always refer to the operating system documentation for the latest recommended procedures.
Prepare the Management Node
These steps prepare the Management Node for xCAT Installation
22
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
Install an OS on the Management Node
Install one of the supported operating systems on your target management node.
The system requirements for your xCAT management node largely depend on the size of the cluster you plan to
manage and the type of provisioning used (diskful, diskless, system clones, etc). The majority of system load comes
during cluster provisioning time.
Memory Requirements:
Cluster Size
Small (< 16)
Medium
Large
Memory (GB)
4-6
6-8
> 16
Configure the Base OS Repository
xCAT uses the zypper package manager on SLES Linux distributions to install and resolve dependency packages
provided by the base operating system. Follow this section to create the repository for the base operating system on
the Management Node
1. Copy the DVD iso file to /tmp on the Management Node:
# This example will use SLE-12-Server-DVD-ppc64le-GM-DVD1.iso
2. Mount the iso to /mnt/iso/sles12 on the Management Node.
mkdir -p /mnt/iso/sles12
mount -o loop /tmp/SLE-12-Server-DVD-ppc64le-GM-DVD1.iso /mnt/iso/sles12
3. Create a zypper repository file /etc/zypp/repos.d/sles12le-base.repo that points to the locally
mounted iso image from the above step. The file contents should appear as the following:
[sles-12-le-server]
name=SLES 12 ppc64le Server Packages
baseurl=file:///mnt/iso/sles12/suse
enabled=1
gpgcheck=1
Configure the Management Node
By setting properties on the Management Node before installing the xCAT software will allow xCAT to automatically
configure key attributes in the xCAT site table during the install.
1. Ensure a hostname is configured on the management node by issuing the hostname command. [It’s recommended to use a fully qualified domain name (FQDN) when setting the hostname]
(a) To set the hostname of xcatmn.cluster.com:
hostname xcatmn.cluster.com
(b) Add the hostname to the /etc/hostname in order to persist the hostname on reboot.
(c) Reboot the server and verify the hostname by running the following commands:
• hostname
1.2. Install Guides
23
xCAT3 Documentation, Release 2.12
• hostname -d - should display the domain
2. Reduce the risk of the Management Node IP address being lost by setting the IP to STATIC in the /etc/
sysconfig/network/ifcfg-<dev> configuration files.
3. Configure any domain search strings and nameservers to the /etc/resolv.conf file.
Installing xCAT
The following sections describe the different methods for installing xCAT.
Automatic Install Using go-xcat
go-xcat is a tool that can be used to fully install or update xCAT. go-xcat will automatically download the correct
package manager repository file from xcat.org and use the public repository to install xCAT. If the xCAT management
node does not have internet connectivity, use process described in the Manual Installation section of the guide.
1. Download the go-xcat tool using wget:
wget https://raw.githubusercontent.com/xcat2/xcat-core/master/xCAT-server/share/
˓→xcat/tools/go-xcat -O - >/tmp/go-xcat
chmod +x /tmp/go-xcat
2. Run the go-xcat tool:
/tmp/go-xcat install
/tmp/go-xcat -x devel install
# installs the latest stable version of xCAT
# installs the latest development version of xCAT
Manual Install Using Software Repositories
xCAT consists of two software packages: xcat-core and xcat-dep
1. xcat-core xCAT’s main software package and is provided in one of the following options:
• Latest Release (Stable) Builds
This is the latest GA (Generally Availability) build that has been tested thoroughly
• Latest Snapshot Builds
This is the latest snapshot of the GA version build that may contain bug fixes but has not yet been
tested thoroughly
• Development Builds
This is the snapshot builds of the new version of xCAT in development. This version has not been
released yet, use as your own risk
2. xcat-dep xCAT’s dependency package. This package is provided as a convenience for the user and contains
dependency packages required by xCAT that are not provided by the operating system.
xCAT is installed by configuring software repositories for xcat-core and xcat-dep and using yum package
manager. The repositories can be publicly hosted or locally hosted.
Configure xCAT Software Repository
xCAT software and repo files can be obtained from: http://xcat.org/download.html
24
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
Internet Repository
[xcat-core]
For the xCAT build you want to install, download the xCAT-core.repo file and copy to /etc/zypp/repos.d
[xcat-dep]
From the xCAT-dep Online Repository, navigate to the correct subdirectory for the target machine and download the
xCAT-dep.repo file and copy to /etc/zypp/repos.d.
If using internet repositories, continue to the next step to install xCAT.
Local Repository
[xcat-core]
1. Download xcat-core:
# downloading the latest development build, core-rpms-snap.tar.bz2
mkdir -p ~/xcat
cd ~/xcat/
wget http://xcat.org/files/xcat/xcat-core/devel/Linux/core-snap/core-rpms-snap.
˓→tar.bz2
2. Extract xcat-core:
tar jxvf core-rpms-snap.tar.bz2
3. Configure the local repository for xcat-core by running mklocalrepo.sh script in the xcat-core directory:
cd ~/xcat/xcat-core
./mklocalrepo.sh
[xcat-dep]
Unless you are downloading xcat-dep to match a specific package tested with a GA release, it’s recommended to
download the latest version of xcat-dep.
1. Download xcat-dep:
# if downloading xcat-dep from June 11, 2015, xcat-dep-201506110324.tar.bz2
mkdir -p ~/xcat/
cd ~/xcat
wget http://xcat.org/files/xcat/xcat-dep/2.x_Linux/xcat-dep-201506110324.tar.bz2
2. Extract xcat-dep:
tar jxvf xcat-dep-201506110324.tar.bz2
3. Configure the local repository for xcat-dep by switching to the architecture and os subdirectory of the node you
are installing on, then run the mklocalrepo.sh script:
cd ~/xcat/xcat-dep/
# Example, on redhat 7.1 ppc64le: cd rh7/ppc64le
cd <os>/<arch>
./mklocalrepo.sh
1.2. Install Guides
25
xCAT3 Documentation, Release 2.12
Install xCAT
Install xCAT with the following command:
zypper clean all (optional)
zypper install xCAT
Note: During the install, you must accept the xCAT Security Key to continue
Verify xCAT Installation
Quick verification of the xCAT Install can be done running the following steps:
1. Source the profile to add xCAT Commands to your path:
source /etc/profile.d/xcat.sh
2. Check the xCAT version:
lsxcatd -a
3. Check to verify that the xCAT database is initialized by dumping out the site table:
tabdump site
The output should be similar to the following:
#key,value,comments,disable
"blademaxp","64",,
"domain","pok.stglabs.ibm.com",,
"fsptimeout","0",,
"installdir","/install",,
"ipmimaxp","64",,
"ipmiretries","3",,
...
Starting and Stopping
xCAT is started automatically after the installation, but the following commands can be used to start, stop, restart, and
check xCAT status.
• start xCAT:
service xcatd start
[systemd] systemctl start xcatd.service
• stop xCAT:
service xcatd stop
[systemd] systemctl stop xcatd.service
• restart xCAT:
service xcatd restart
[systemd] systemctl restart xcatd.service
26
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
• check xCAT status:
service xcatd status
[systemd] systemctl status xcatd.service
Updating xCAT
If at a later date you want to update xCAT, first, update the software repositories and then run:
zypper refresh
zypper update "*xCAT*"
# To check and update the packages provided by xcat-dep:
zypper update "*xcat*"
Installation Guide for Ubuntu Server LTS
For the current list of operating systems supported and verified by the development team for the different releases of
xCAT, see the xCAT2 Release Notes.
Disclaimer These instructions are intended to only be guidelines and specific details may differ slightly based on the
operating system version. Always refer to the operating system documentation for the latest recommended procedures.
Prepare the Management Node
These steps prepare the Management Node or xCAT Installation
Install an OS on the Management Node
Install one of the supported operating systems on your target management node.
The system requirements for your xCAT management node largely depend on the size of the cluster you plan to
manage and the type of provisioning used (diskful, diskless, system clones, etc). The majority of system load comes
during cluster provisioning time.
Memory Requirements:
Cluster Size
Small (< 16)
Medium
Large
Memory (GB)
4-6
6-8
> 16
Configure the Base OS Repository
xCAT uses the apt package manager on Ubuntu Linux distributions to install and resolve dependency packages provided by the base operating system. Follow this section to create the repository for the base operating system on the
Management Node
1. Copy the DVD iso file to /tmp on the Management Node:
# This example will use ubuntu-14.04.3-server-ppc64el.iso
1.2. Install Guides
27
xCAT3 Documentation, Release 2.12
2. Mount the iso to /mnt/iso/ubuntu14 on the Management Node.
mkdir -p /mnt/iso/ubuntu14
mount -o loop /tmp/ubuntu-14.04.3-server-ppc64el.iso /mnt/iso/ubuntu14
3. Create an apt repository file /etc/apt.repos.d/ubuntu14-dvd.repo that points to the locally
mounted iso image from the above step. The file contents should appear as the following:
[ubuntu-14-dvd-server]
name=UBUNTU 14 SERVER packages
baseurl=file:///mnt/iso/ubuntu14/Server
enabled=1
gpgcheck=1
Configure the Management Node
By setting properties on the Management Node before installing the xCAT software will allow xCAT to automatically
configure key attributes in the xCAT site table during the install.
1. Ensure a hostname is configured on the management node by issuing the hostname command. [It’s recommended to use a fully qualified domain name (FQDN) when setting the hostname]
(a) To set the hostname of xcatmn.cluster.com:
hostname xcatmn.cluster.com
(b) Add the hostname to the /etc/hostname and /etc/hosts to persist the hostname on reboot.
(c) Reboot or run service hostname restart to allow the hostname to take effect and verify the
hostname command returns correctly:
• hostname
• hostname -d - should display the domain
2. Reduce the risk of the Management Node IP address being lost by setting the interface IP to STATIC in the
/etc/network/interfaces configuration file.
3. Configure any domain search strings and nameservers using the resolvconf command.
Installing xCAT
The following sections describe the different methods for installing xCAT.
Automatic Install Using go-xcat
go-xcat is a tool that can be used to fully install or update xCAT. go-xcat will automatically download the correct
package manager repository file from xcat.org and use the public repository to install xCAT. If the xCAT management
node does not have internet connectivity, use process described in the Manual Installation section of the guide.
1. Download the go-xcat tool using wget:
wget https://raw.githubusercontent.com/xcat2/xcat-core/master/xCAT-server/share/
˓→xcat/tools/go-xcat -O - >/tmp/go-xcat
chmod +x /tmp/go-xcat
28
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
2. Run the go-xcat tool:
/tmp/go-xcat install
/tmp/go-xcat -x devel install
# installs the latest stable version of xCAT
# installs the latest development version of xCAT
Manual Install Using Software Repositories
xCAT consists of two software packages: xcat-core and xcat-dep
1. xcat-core xCAT’s main software package and is provided in one of the following options:
• Latest Release (Stable) Builds
This is the latest GA (Generally Availability) build that has been tested thoroughly
• Latest Snapshot Builds
This is the latest snapshot of the GA version build that may contain bug fixes but has not yet been
tested thoroughly
• Development Builds
This is the snapshot builds of the new version of xCAT in development. This version has not been
released yet, use as your own risk
2. xcat-dep xCAT’s dependency package. This package is provided as a convenience for the user and contains
dependency packages required by xCAT that are not provided by the operating system.
xCAT is installed by configuring software repositories for xcat-core and xcat-dep and using yum package
manager. The repositories can be publicly hosted or locally hosted.
Configure xCAT Software Repository
xCAT software and repo files can be obtained from: http://xcat.org/download.html
Internet Repository
[xcat-core]
From the xCAT download page, find the build you want to install and add to /etc/apt/sources.list.
To configure the xCAT development build, add the following line to /etc/apt/sources.list:
[For x86_64 servers]
deb [arch=amd64] http://xcat.org/files/xcat/repos/apt/devel/core-snap trusty main
[For ppc64el servers]
deb [arch=ppc64el] http://xcat.org/files/xcat/repos/apt/devel/core-snap trusty main
[xcat-dep]
To configure the xCAT deps online repository, add the following line to /etc/apt/sources.list:
[For x86_64 servers]
deb [arch=amd64] http://xcat.org/files/xcat/repos/apt/xcat-dep trusty main
[For ppc64el servers]
deb [arch=ppc64el] http://xcat.org/files/xcat/repos/apt/xcat-dep trusty main
If using internet repositories, continue to the next step to install xCAT.
1.2. Install Guides
29
xCAT3 Documentation, Release 2.12
Local Repository
[xcat-core]
1. Download xcat-core:
# downloading the latest development build, core-rpms-snap.tar.bz2
mkdir -p ~/xcat
cd ~/xcat/
wget http://xcat.org/files/xcat/xcat-core/devel/Ubuntu/core-snap/core-debs-snap.
˓→tar.bz2
2. Extract xcat-core:
tar jxvf core-debs-snap.tar.bz2
3. Configure the local repository for xcat-core by running mklocalrepo.sh script in the xcat-core directory:
cd ~/xcat/xcat-core
./mklocalrepo.sh
[xcat-dep]
Unless you are downloading xcat-dep to match a specific package tested with a GA release, it’s recommended to
download the latest version of xcat-dep.
1. Download xcat-dep:
# if downloading xcat-dep from June 11, 2015, xcat-dep-ubuntu-snap20150611.tar.bz
mkdir -p ~/xcat/
cd ~/xcat
wget http://xcat.org/files/xcat/xcat-dep/2.x_Ubuntu/xcat-dep-ubuntu-snap20150611.
˓→tar.bz
2. Extract xcat-dep:
tar jxvf xcat-dep-ubuntu-snap20150611.tar.bz
3. Configure the local repository for xcat-dep by running the mklocalrepo.sh script:
cd ~/xcat/xcat-dep/
./mklocalrepo.sh
Install xCAT
The xCAT GPG Public Key must be added for apt to verify the xCAT packages
wget -O - "http://xcat.org/files/xcat/repos/apt/apt.key" | apt-key add -
Add the necessary apt-repositories to the management node
# Install the add-apt-repository command
apt-get install software-properties-common
# For x86_64:
add-apt-repository "deb http://archive.ubuntu.com/ubuntu $(lsb_release -sc) main"
30
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
add-apt-repository "deb http://archive.ubuntu.com/ubuntu $(lsb_release -sc)-updates
˓→main"
add-apt-repository "deb http://archive.ubuntu.com/ubuntu $(lsb_release -sc) universe"
add-apt-repository "deb http://archive.ubuntu.com/ubuntu $(lsb_release -sc)-updates
˓→universe"
# For ppc64el:
add-apt-repository "deb
add-apt-repository "deb
˓→updates main"
add-apt-repository "deb
˓→universe"
add-apt-repository "deb
˓→updates universe"
http://ports.ubuntu.com/ubuntu-ports $(lsb_release -sc) main"
http://ports.ubuntu.com/ubuntu-ports $(lsb_release -sc)http://ports.ubuntu.com/ubuntu-ports $(lsb_release -sc)
http://ports.ubuntu.com/ubuntu-ports $(lsb_release -sc)-
Install xCAT1 with the following command:
apt-get clean all
apt-get update
apt-get install xcat
Verify xCAT Installation
Quick verification of the xCAT Install can be done running the following steps:
1. Source the profile to add xCAT Commands to your path:
source /etc/profile.d/xcat.sh
2. Check the xCAT version:
lsxcatd -a
3. Check to verify that the xCAT database is initialized by dumping out the site table:
tabdump site
The output should be similar to the following:
#key,value,comments,disable
"blademaxp","64",,
"domain","pok.stglabs.ibm.com",,
"fsptimeout","0",,
"installdir","/install",,
"ipmimaxp","64",,
"ipmiretries","3",,
...
Starting and Stopping
xCAT is started automatically after the installation, but the following commands can be used to start, stop, restart, and
check xCAT status.
1
Starting with Ubuntu 16.04, the package name ‘xCAT’ is required to be all lowercase
1.2. Install Guides
31
xCAT3 Documentation, Release 2.12
• start xCAT:
service xcatd start
[systemd] systemctl start xcatd.service
• stop xCAT:
service xcatd stop
[systemd] systemctl stop xcatd.service
• restart xCAT:
service xcatd restart
[systemd] systemctl restart xcatd.service
• check xCAT status:
service xcatd status
[systemd] systemctl status xcatd.service
Updating xCAT
If at a later date you want to update xCAT, first, update the software repositories and then run:
apt-get update
apt-get -y --only-upgrade install .*xcat.*
Maintenance
Backup and Restore xCAT
It’s useful to backup xcat data sometime. For example, you need to upgrading to another version of xCAT, or you need
to change management server and move xcat form one to another, or you need to make backups regularly and restore
production environment for any accident. Below section will help you backup and restore xcat data.
Backup User Data
If need to backup xcat database, you can use dumpxCATdb command like below.
dumpxCATdb -p <path_to_save_the_database>
[Note] Maybe you need to dump some environment data for problem report when you hit defect, you can use xcatsnap
command like below.
xcatsnap -B -d <path_to_save_the_data>
Restore User Data
If need to restore xCAT environment, after xCAT software installation, you can restore xCAT DB using the restorexCATdb command pointing to the data files dumped in the past.
32
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
restorexCATdb -p
<path_to_save_the_database>
Remove xCAT
We’re sorry to see you go! Here are some steps for removing the xCAT product.
Clean Up xCAT Related Configuration
1. To clean up the node information from dhcp
makedhcp -d -a
2. To clean up the node information in tftpboot
nodeset all offline
3. To clean up the node information from /etc/hosts (optional)
Keeping xCAT nodes information in /etc/hosts is harmless, but if you really want to remove them
from /etc/hosts run:
makehosts -d all
4. To clean up the node information from DNS (optional)
After removing all the nodes from /etc/hosts, run below command to clean up the node information
from DNS.
makedns -n
Stop xCAT Service
1. Stop xCAT service
service xcatd stop
2. Stop xCAT related services(Optional)
XCAT uses various network services on the management node and service nodes, the network services
setup by xCAT may need to be cleaned up on the management node and service nodes before uninstalling
xCAT.
• NFS : Stop nfs service, unexport all the file systems exported by xCAT, and remove the xCAT file systems from
/etc/exports.
• HTTP: Stop http service, remove the xcat.conf in the http configuration directory.
• TFTP: Stop tftp service, remove the tftp files created by xCAT in tftp directory.
• DHCP: Stop dhcp service, remove the configuration made by xCAT in dhcp configuration files.
• DNS : Stop the named service, remove the named entries created by xCAT from the named database.
1.2. Install Guides
33
xCAT3 Documentation, Release 2.12
Remove xCAT Files
1. Remove the xCAT RPMs
There is no easy way to distinct all the packages depending by xCAT. For packages shipped by xCAT, you
can remove them by the commands below.
[RHEL and SLES]
rpm -qa |grep -i xcat
[Ubuntu]
dpkg -l | awk '/xcat/ { print $2 }'
If you want to remove more cleanly, the list below maybe helpful. Listed are the packages of xcat installation tarball. Some RPMs may not to be installed in a specific environment.
• XCAT Core Packages list (xcat-core):
[RHEL and SLES]
perl-xCAT
xCAT
xCAT-buildkit
xCAT-client
xCAT-confluent
xCAT-genesis-scripts-ppc64
xCAT-genesis-scripts-x86_64
xCAT-server
xCATsn
xCAT-SoftLayer
xCAT-test
xCAT-vlan
[Ubuntu]
perl-xcat
xcat
xcat-buildkit
xcat-client
xcat-confluent
xcat-genesis-scripts
xcat-server
xcatsn
xcat-test
xcat-vlan
• XCAT Dependency Packages (xcat-dep):
[RHEL and SLES]
conserver-xcat
cpio
cpio-lang
elilo-xcat
esxboot-xcat
fping
ganglia-devel
ganglia-gmetad
34
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
ganglia-gmond
ganglia-gmond-modules-python
ganglia-web
grub2-xcat
ipmitool-xcat
libconfuse
libconfuse-devel
libganglia
lldpd
net-snmp-perl
perl-AppConfig
perl-Compress-Raw-Zlib
perl-Crypt-Blowfish
perl-Crypt-CBC
perl-Crypt-Rijndael
perl-Crypt-SSLeay
perl-DBD-DB2
perl-DBD-DB2Lite
perl-DBD-Pg
perl-DBD-SQLite
perl-Expect
perl-HTML-Form
perl-IO-Compress-Base
perl-IO-Compress-Zlib
perl-IO-Socket-SSL
perl-IO-Stty
perl-IO-Tty
perl-JSON
perl-Net-DNS
perl-Net-Telnet
perl-SOAP-Lite
perl-Test-Manifest
perl-version
perl-XML-Simple
pyodbc
rrdtool
scsi-target-utils
stunnel
syslinux-xcat
systemconfigurator
systemimager-client
systemimager-common
systemimager-server
xCAT-genesis-base-ppc64
xCAT-genesis-base-x86_64
xCAT-genesis-x86_64
xCAT-UI-deps
xnba-kvm
xnba-undi
yaboot-xcat
zhcp
[Ubuntu]
conserver-xcat
elilo-xcat
grub2-xcat
ipmitool-xcat
1.2. Install Guides
35
xCAT3 Documentation, Release 2.12
syslinux
syslinux-extlinux
syslinux-xcat
xcat-genesis-base-amd64
xcat-genesis-base-ppc64
xnba-undi
Along with xCAT development, above lists maybe change, you can get the latest list through below links:
• XCAT Core Packages List (xcat-core)
[RHEL and SLES]
http://xcat.org/files/xcat/repos/yum/<version>/xcat-core/
[Ubuntu]
http://xcat.org/files/xcat/repos/apt/<version>/xcat-core/
• XCAT Dependency Packages (xcat-dep)
RPM Packages List (RHEL and SLES)
Debian Packages List (Ubuntu)
Generally, we use yum install xCAT to install xCAT, so these are some RPMs shipped by operating
system are installed during xCAT installation. We don’t have an easy way to find out all of them, but keep
these RPMs are harmless.
2. Remove xCAT certificate file
rm -rf /root/.xcat
3. Remove xCAT data file
By default, xCAT use SQLite, remove SQLite data file under /etc/xcat/.
rm -rf /etc/xcat
4. Remove xCAT related file(Optional)
XCAT has ever operated below directory when it was running. Do judgment by yourself before removing
these directory, to avoid removing some directories used for other purpose in your environment.
/install
/tftpboot
/etc/yum.repos.d/xCAT-*
/etc/sysconfig/xcat
/etc/apache2/conf.d/xCAT-*
/etc/logrotate.d/xCAT-*
/etc/rsyslogd.d/xCAT-*
/var/log/xcat
/opt/xcat/
/mnt/xcat
Remove Databases
• Removing xCAT DB from PostgreSQL.
36
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
• Removing xCAT DB from MySQL/MariaDB.
Admin Guide
The admin guide is intended to help with learning how to manage a cluster using xCAT with the following major
sections:
• Basic Concepts Introduces some of the basic concepts in xCAT.
• Manage Cluster Describes managing clusters under xCAT. The management procedures are organized based
on the hardware type since management may vary depending on the hardware architecture.
• Reference xCAT reference sections.
Basic Concepts
xCAT is not hard to use but you still need to learn some basic concepts of xCAT before starting to manage a real
cluster.
• xCAT Objects
The unit which can be managed in the xCAT is defined as an object. xCAT abstracts several types of objects
from the cluster information to represent the physical or logical entities in the cluster. Each xCAT object has a
set of attributes, each attribute is mapped from a specified field of a xCAT database table. The xCAT users can
get cluster information and perform cluster management work through operations against the objects.
• xCAT Database
All the data for the xCAT Objects (node, group, network, osimage, policy ... and global configuration) are
stored in xCAT Database. Tens of tables are created as the back-end of xCAT Objects. Generally the data in
the database is used by user through xCAT Objects. But xCAT also offers a bunch of commands to handle the
database directly.
• Global Configuration
xCAT has a bunch of Global Configuration for xCAT user to control the behaviors of xCAT. Some of the
configuration items are mandatory for an xCAT cluster that you must set them correctly before starting to use
xCAT.
• xCAT Network
xCAT’s goal is to manage and configure a significant number of servers remotely and automatically through a
central management server. All the hardware discovery/management, OS deployment/configuration and application install/configuration are performed through network. You need to have a deep understand of how xCAT
will use network before setting up a cluster.
Get Into the Detail of the Cencepts:
xCAT Objects
Basically, xCAT has 20 types of objects. They are:
auditlog
kit
node
policy
boottarget
kitcomponent
notification
rack
1.3. Admin Guide
eventlog
kitrepo
osdistro
route
firmware
monitoring
osdistroupdate
site
group
network
osimage
zone
37
xCAT3 Documentation, Release 2.12
This section will introduce you to several important types of objects and give you an overview of how to view and
manipulate them.
You can get the detail description of each object by man <object type> e.g. man node.
• node Object
The node is the most important object in xCAT. Any physical server, virtual machine or SP (Service Processor
for Hardware Control) can be defined as a node object.
For example, I have a physical server which has the following attributes:
groups: all,x86_64
The groups that this node belongs to.
arch: x86_64
The architecture of the server is x86_64.
bmc: 10.4.14.254
The IP of BMC which will be used for hardware control.
bmcusername: ADMIN
The username of bmc.
bmcpassword: admin
The password of bmc.
mac: 6C:AE:8B:1B:E8:52
The mac address of the ethernet adapter that will be used to
deploy OS for the node.
mgt: ipmi
The management method which will be used to manage the node.
This node will use ipmi protocol.
netboot: xnba
The network bootloader that will be used to deploy OS for the node.
provmethod: rhels7.1-x86_64-install-compute
The osimage that will be deployed to the node.
I want to name the node to be cn1 (Compute Node #1) in xCAT. Then I define this node in xCAT with following
command:
$mkdef -t node cn1 groups=all,x86_64 arch=x86_64 bmc=10.4.14.254
bmcusername=ADMIN bmcpassword=admin mac=6C:AE:8B:1B:E8:52
mgt=ipmi netboot=xnba provmethod=rhels7.1-x86_64-install˓→compute
After the define, I can use lsdef command to display the defined node:
$lsdef cn1
Object name: cn1
arch=x86_64
bmc=10.4.14.254
bmcpassword=admin
bmcusername=ADMIN
groups=all,x86_64
mac=6C:AE:8B:1B:E8:52
mgt=ipmi
netboot=xnba
postbootscripts=otherpkgs
postscripts=syslog,remoteshell,syncfiles
provmethod=rhels7.1-x86_64-install-compute
Then I can try to remotely power on the node cn1:
38
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
$rpower cn1 on
• group Object
group is an object which includes multiple node object. When you set group attribute for a node object to a
group name like x86_64, the group x86_64 is automatically generated and the node is assigned to the group.
The benefits of using group object:
– Handle multiple nodes through group
I defined another server cn2 which is similar with cn1, then my group x86_64 has two nodes: cn1 and
cn2.
$ lsdef -t group x86_64
Object name: x86_64
cons=ipmi
members=cn1,cn2
Then I can power on all the nodes in the group x86_64.
$ rpower x86_64 on
– Inherit attributes from group
If the group object of node object has certain attribute that node object does not have, the node will
inherit this attribute from its group.
I set the cons attribute for the group object x86_64.
$ chdef -t group x86_64 cons=ipmi
1 object definitions have been created or modified.
$ lsdef -t group x86_64
Object name: x86_64
cons=ipmi
members=cn1,cn2
The I can see the cn1 inherits the attribute cons from the group x86_64:
$ lsdef cn1
Object name: cn1
arch=x86_64
bmc=10.4.14.254
bmcpassword=admin
bmcusername=ADMIN
cons=ipmi
groups=all,x86_64
mac=6C:AE:8B:1B:E8:52
mgt=ipmi
netboot=xnba
postbootscripts=otherpkgs
postscripts=syslog,remoteshell,syncfiles
provmethod=rhels7.1-x86_64-install-compute
It is useful to define common attributes in group object so that newly added node will inherit them automatically. Since the attributes are defined in the group object, you don’t need to touch the individual
nodes attributes.
– Use Regular Expression to generate value for node attributes
1.3. Admin Guide
39
xCAT3 Documentation, Release 2.12
This is powerful feature in xCAT that you can generate individual attribute value from node name instead
of assigning them one by one. Refer to Use Regular Expression in xCAT Database Table.
• osimage Object
An osimage object represents an Operating System which can be deployed in xCAT. xCAT always generates
several default osimage objects for certain Operating System when executing copycds command to generate
the package repository for the OS.
You can display all the defined osimage object:
$ lsdef -t osimage
Display the detail attributes of one osimage named rhels7.1-x86_64-install-compute:
$ lsdef -t osimage rhels7.1-x86_64-install-compute
Object name: rhels7.1-x86_64-install-compute
imagetype=linux
osarch=x86_64
osdistroname=rhels7.1-x86_64
osname=Linux
osvers=rhels7.1
otherpkgdir=/install/post/otherpkgs/rhels7.1/x86_64
pkgdir=/install/rhels7.1/x86_64
pkglist=/opt/xcat/share/xcat/install/rh/compute.rhels7.pkglist
profile=compute
provmethod=install
synclists=/root/syncfiles.list
template=/opt/xcat/share/xcat/install/rh/compute.rhels7.tmpl
This osimage represents a Linux rhels7.1 Operating System.
The package repository is
in /install/rhels7.1/x86_64 and the packages which will be installed is listed in the file
/opt/xcat/share/xcat/install/rh/compute.rhels7.pkglist ...
I can bind the osimage to node when I want to deploy osimage rhels7.1-x86_64-install-compute on my node
cn1:
$ nodeset cn1 osimage=rhels7.1-x86_64-install-compute
Then in the next network boot, the node cn1 will start to deploy rhles7.1.
• Manipulating Objects
You already saw that I used the commands mkdef, lsdef, chdef to manipulate the objects. xCAT has 4
objects management commands to manage all the xCAT objects.
– mkdef : create object definitions
– chdef : modify object definitions
– lsdef : list object definitions
– rmdef : remove object definitions
To get the detail usage of the commands, refer to the man page. e.g. man mkdef
Get Into the Detail of the xCAT Objects:
node
40
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
Description
The definition of physical units in the cluster, such as lpar, virtual machine, frame, cec, hmc, switch.
Key Attributes
• os: The operating system deployed on this node. Valid values: AIX, rhels*, rhelc*, rhas*, centos*, SL*,
fedora*, sles* (where * is the version #)
• arch: The hardware architecture of this node. Valid values: x86_64, ppc64, x86, ia64.
• groups: Usually, there are a set of nodes with some attributes in common, xCAT admin can define a node group
containing these nodes, so that the management task can be issued against the group instead of individual
nodes. A node can be a member of different groups, so the value of this attributes is a comma-delimited
list of groups. At least one group is required to create a node. The new created group names should not be
prefixed with “__” as this token has been preserved as the internal group name.
• mgt: The method to do general hardware management of the node. This attribute can be determined by the
machine type of the node. Valid values: ipmi, blade, hmc, ivm, fsp, bpa, kvm, esx, rhevm.
• mac: The mac address of the network card on the node, which is connected with the installation server and can
be used as the network installation device.
• ip: The IP address of the node.
• netboot: The type of network boot method for this node, determined by the OS to provision, the architecture
and machine type of the node. Valid values:
Arch and Machine Type OS
x86, x86_64
ALL
ppc64
<=rhel6, <=sles11.3
ppc64
>=rhels7, >=sles11.4
ppc64le NonVirtualize
ALL
ppc64le PowerKVM Guest | ALL
valid netboot options
pxe, xnba
yaboot
grub2,grub2-http,grub2-tftp
petitboot
grub2,grub2-http,grub2-tftp
• postscripts: Comma separated list of scripts, that should be run on this node after diskful installation or diskless
boot, finish some system configuration and maintenance work. For installation of RedHat, CentOS, Fedora,
the scripts will be run before the reboot. For installation of SLES, the scripts will be run after the reboot
but before the init.d process.
• postbootscripts: Comma separated list of scripts, that should be run on this node as a SysV init job on the 1st
reboot after installation or diskless boot, finish some system configuration and maintenance work.
• provmethod: The provisioning method for node deployment. Usually, this attribute is an osimage object
name.
• status: The current status of the node, which is updated by xCAT. This value can be used to monitor the
provision process. Valid values: powering-off, installing, booting/netbooting, booted.
Use Cases
• Case 1: There is a ppc64le node named “cn1”, the mac of installation NIC is “ca:68:d3:ae:db:03”, the ip assigned
is “10.0.0.100”, the network boot method is “grub2”, place it into the group “all”. Use the following command
mkdef -t node -o cn1 arch=ppc64 mac="ca:68:d3:ae:db:03" ip="10.0.0.100" netboot=
˓→"grub2" groups="all"
1.3. Admin Guide
41
xCAT3 Documentation, Release 2.12
• Case 2:
List all the node objects
nodels
This can also be done with
lsdef -t node
• Case 3: List the mac of object “cn1”
lsdef -t node -o cn1 -i mac
• Case 4: There is a node definition “cn1”, modify its network boot method to “yaboot”
chdef -t node -o cn1 netboot=yaboot
• Case 5: There is a node definition “cn1”, create a node definition “cn2” with the same attributes with “cn1”,
except the mac addr(ca:68:d3:ae:db:04) and ip address(10.0.0.101)
step 1: write the definition of “cn1” to a stanza file named “cn.stanza”
lsdef -z cn1 > /tmp/cn.stanza
The content of “/tmp/cn.stanza” will look like
# <xCAT data object stanza file>
cn1:
objtype=node
groups=all
ip=10.0.0.100
mac=ca:68:d3:ae:db:03
netboot=grub2
step 2: modify the “/tmp/cn.stanza” according to the “cn2” attributes
# <xCAT data object stanza file>
cn2:
objtype=node
groups=all
ip=10.0.0.101
mac=ca:68:d3:ae:db:04
netboot=grub2
step 3: create “cn2” definition with “cn.stanza”
cat /tmp/cn.stanza |mkdef -z
group
XCAT supports both static and dynamic groups. A static group is defined to contain a specific set of cluster nodes. A
dynamic node group is one that has its members determined by specifying a selection criteria for node attributes. If a
nodes attribute values match the selection criteria then it is dynamically included as a member of the group. The actual
group membership will change over time as nodes have attributes set or unset. This provides flexible control over
group membership by defining the attributes that define the group, rather than the specific node names that belong to
42
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
the group. The selection criteria is a list of attr<operator>val pairs that can be used to determine the members
of a group, (see below).
Note : Dynamic node group support is available in xCAT version 2.3 and later.
In xCAT, the definition of a static group has been extended to include additional attributes that would normally be
assigned to individual nodes. When a node is part of a static group definition, it can inherit the attributes assigned to
the group. This feature can make it easier to define and manage cluster nodes in that you can generally assign nodes
to the appropriate group and then just manage the group definition instead of multiple node definitions. This feature is
not supported for dynamic groups.
To list all the attributes that may be set for a group definition you can run
lsdef -t group -h
When a node is included in one or more static groups, a particular node attribute could actually be stored in several
different object definitions. It could be in the node definition itself or it could be in one or more static group definitions.
The precedence for determining which value to use is to choose the attribute value specified in the node definition if
it is provided. If not, then each static group that the node belongs to will be checked to see if the attribute is set. The
first value that is found is the value that is used. The static groups are checked in the order that they are specified in
the groups attribute of the node definition.
NOTE : In a large cluster environment it is recommended to focus on group definitions as much as possible and
avoid setting the attribute values in the individual node definition. (Of course some attribute values, such as a MAC
addresses etc., are only appropriate for individual nodes.) Care must be taken to avoid confusion over which values
will be inherited by the nodes.
Group definitions can be created using the mkdef command, changed using the chdef command, listed using the
lsdef command and removed using the rmdef command.
Creating a static node group
There are two basic ways to create xCAT static node groups. You can either set the groups attribute of the node
definition or you can create a group definition directly.
You can set the groups attribute of the node definition when you are defining the node with the mkdef or nodeadd
command or you can modify the attribute later using the chdef or nodech command. For example, if you want a
set of nodes to be added to the group “aixnodes”,you could run chdef or nodech as follows
chdef -t node -p -o node01,node02,node03 groups=aixnodes
or
nodech node01,node02,node03 groups=aixnodes
The -p (plus) option specifies that “aixnodes” be added to any existing value for the groups attribute. The -p (plus)
option is not supported by nodech command.
The second option would be to create a new group definition directly using the mkdef command as follows
mkdef -t group -o aixnodes members="node01,node02,node03"
These two options will result in exactly the same definitions and attribute values being created in the xCAT database.
1.3. Admin Guide
43
xCAT3 Documentation, Release 2.12
Creating a dynamic node group
The selection criteria for a dynamic node group is specified by providing a list of attr<operator>val pairs that
can be used to determine the members of a group. The valid operators include: ==, !=, =~ and !~. The attr
field can be any node definition attribute returned by the lsdef command. The val field in selection criteria can
be a simple sting or a regular expression. A regular expression can only be specified when using the =~ or !~
operators. See http://www.perl.com/doc/manual/html/pod/perlre.html for information on the format and syntax of
regular expressions.
Operator descriptions
==
!=
=~
!~
Select
Select
Select
Select
nodes
nodes
nodes
nodes
where
where
where
where
the
the
the
the
attribute
attribute
attribute
attribute
value
value
value
value
is exactly this value.
is not this specific value.
matches this regular expression.
does not match this regular expression.
The selection criteria can be specified using one or more -w attr<operator>val options on the command line.
If the val field includes spaces or any other characters that will be parsed by shell then the attr<operator>val
needs to be quoted.
For example, to create a dynamic node group called “mygroup”, where the hardware control point is “hmc01” and the
partition profile is not set to service
mkdef -t group -o mygroup -d -w hcp==hmc01 -w pprofile!=service
To create a dynamic node group called “pslesnodes”, where the operating system name includes “sles” and the architecture includes “ppc”
mkdef -t group -o pslesnodes -d -w os=~sles[0-9]+ -w arch=~ppc
To create a dynamic node group called nonpbladenodes where the node hardware management method is not set to
blade and the architecture does not include ppc
mkdef -t group -o nonpbladenodes -d -w mgt!=blade -w 'arch!~ppc'
osimage
Description
A logical definition of image which can be used to provision the node.
Key Attributes
• imagetype: The type of operating system this definition represents (linux, AIX).
• osarch: The hardware architecture of the nodes this image supports. Valid values: x86_64, ppc64, ppc64le.
• osvers: The Linux distribution name and release number of the image. Valid values: rhels*, rhelc*, rhas*,
centos*, SL*, fedora*, sles* (where * is the version #).
• pkgdir: The name of the directory where the copied OS distro content are stored.
• pkglist: The fully qualified name of a file, which contains the list of packages shipped in Linux distribution
ISO which will be installed on the node.
44
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
• otherpkgdir When xCAT user needs to install some additional packages not shipped in Linux distribution
ISO, those packages can be placed in the directory specified in this attribute. xCAT user should take
care of dependency problems themselves, by putting all the dependency packages not shipped in Linux
distribution ISO in this directory and creating repository in this directory.
• otherpkglist: The fully qualified name of a file, which contains the list of user specified additional packages
not shipped in Linux distribution ISO which will be installed on the node.
• template: The fully qualified name of the template file that will be used to create the OS installer configuration
file for stateful installation (e.g. kickstart for RedHat, autoyast for SLES and preseed for Ubuntu).
Use Cases
• Case 1:
List all the osimage objects
lsdef -t osimage
• Case 2:
Create a osimage definition “customized-rhels7-ppc64-install-compute” based on an existing osimage “rhels7-ppc64install-compute”, the osimage “customized-rhels7-ppc64-install-compute” will inherit all the attributes of “rhels7ppc64-install-compute” except installing the additional packages specified in the file “/tmp/otherpkg.list”:
step 1 : write the osimage definition “rhels7-ppc64-install-compute” to a stanza file “osimage.stanza”
lsdef -z -t osimage -o rhels7-ppc64-install-compute > /tmp/osimage.stanza
The content will look like
# <xCAT data object stanza file>
rhels7-ppc64-install-compute:
objtype=osimage
imagetype=linux
osarch=ppc64
osdistroname=rhels7-ppc64
osname=Linux
osvers=rhels7
otherpkgdir=/install/post/otherpkgs/rhels7/ppc64
pkgdir=/install/rhels7/ppc64
pkglist=/opt/xcat/share/xcat/install/rh/compute.rhels7.pkglist
profile=compute
provmethod=install
template=/opt/xcat/share/xcat/install/rh/compute.rhels7.tmpl
step 2 : modify the stanza file according to the attributes of “customized-rhels7-ppc64-install-compute”
# <xCAT data object stanza file>
customized-rhels7-ppc64-install-compute:
objtype=osimage
imagetype=linux
osarch=ppc64
osdistroname=rhels7-ppc64
osname=Linux
osvers=rhels7
1.3. Admin Guide
45
xCAT3 Documentation, Release 2.12
otherpkglist=/tmp/otherpkg.list
otherpkgdir=/install/post/otherpkgs/rhels7/ppc64
pkgdir=/install/rhels7/ppc64
pkglist=/opt/xcat/share/xcat/install/rh/compute.rhels7.pkglist
profile=compute
provmethod=install
template=/opt/xcat/share/xcat/install/rh/compute.rhels7.tmpl
step 3 : create the osimage “customized-rhels7-ppc64-install-compute” from the stanza file
cat /tmp/osimage.stanza |mkdef -z
xCAT Database
All of the xCAT Objects and Configuration data are stored in xCAT database. By default, xCAT uses SQLite - an OS
contained simple database engine. The powerful open source database engines like MySQL, MariaDB, PostgreSQL
are also supported for a large cluster.
xCAT defines about 70 tables to store different data. You can get the xCAT database definition from file /opt/xcat/
lib/perl/xCAT/Schema.pm.
You can run tabdump command to get all the xCAT database tables. Or run tabdump -d <tablename> or man
<tablename> to get the detail information on columns and table definitions.
$
$
$
$
tabdump
tabdump site
tabdump -d site
man site
For a complete reference, see the man page for xcatdb: man xcatdb.
The tables in xCAT:
• site table
Global settings for the whole cluster. This table is different from the other tables. Each entry in site table is a
key=>value pair. Refer to the Global Configuration page for the major global attributes or run man site to
get all global attributes.
• policy table
Controls who has authority to run specific xCAT operations. It is the Access Control List (ACL) in xCAT.
• passwd table
Contains default userids and passwords for xCAT to access cluster components. In most cases, xCAT will also
set the userid/password in the relevant component (Generally for SP like bmc, fsp.) when it is being configured
or installed. The default userids/passwords in passwd table for specific cluster components can be overridden
by the columns in other tables, e.g. mpa , ipmi , ppchcp , etc.
• networks table
Contains the network definitions in the cluster.
You can manipulate the networks through *def command against the network object.
$ lsdef -t network
• ...
46
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
Manipulate xCAT Database Tables
xCAT offers 5 commands to manipulate the database tables:
• tabdump
Displays the header and all the rows of the specified table in CSV (comma separated values) format.
• tabedit
Opens the specified table in the user’s editor, allows them to edit any text, and then writes changes back to the
database table. The table is flattened into a CSV (comma separated values) format file before giving it to the
editor. After the editor is exited, the CSV file will be translated back into the database format.
• tabgrep
List table names in which an entry for the given node appears.
• dumpxCATdb
Dumps all the xCAT db tables to CSV files under the specified directory, often used to backup the xCAT database
for xCAT reinstallation or management node migration.
• restorexCATdb
Restore the xCAT db tables from the CSV files under the specified directory.
Advanced Topic: How to use Regular Expression in xCAT tables:
Groups and Regular Expressions in Tables
Using Regular Expressions in the xCAT Tables
The xCAT database has a number of tables, some with rows that are keyed by node name (such as noderes and
nodehm ) and others that are not keyed by node name (for example, the policy table). The tables that are keyed by
node name have some extra features that enable a more template-based style to be used:
Any group name can be used in lieu of a node name in the node field, and that row will then provide “default” attribute
values for any node in that group. A row with a specific node name can then override one or more attribute values for
that specific node. For example, if the nodehm table contains
#node,power,mgt,cons,termserver,termport,conserver,serialport,serialspeed,serialflow,
˓→getmac,cmdmapping,comments,disable
"mygroup",,"ipmi",,,,,,"19200",,,,,
"node1",,,,,,,,"115200",,,,,
In the above example, the node group called “mygroup” sets mgt=ipmi and serialspeed=19200. Any nodes
that are in this group will have those attribute values, unless overridden. For example, if “node2” is a member of
“mygroup”, it will automatically inherit these attribute values (even though it is not explicitly listed in this table). In
the case of “node1” above, it inherits mgt=ipmi, but overrides the serialspeed to be 115200, instead of 19200.
A useful, typical way to use this capability is to create a node group for your nodes and for all the attribute values that
are the same for every node, set them at the group level. Then you only have to set attributes for each node that vary
from node to node.
xCAT extends the group capability so that it can also be used for attribute values that vary from node to node in a very
regular pattern. For example, if in the ipmi table you want the bmc attribute to be set to whatever the nodename is
with “-bmc” appended to the end of it, then use this in the ipmi table
#node,bmc,bmcport,taggedvlan,bmcid,username,password,comments,disable
"compute","/\z/-bmc/",,,,,,,
1.3. Admin Guide
47
xCAT3 Documentation, Release 2.12
In this example, “compute” is a node group that contains all of the compute nodes. The 2nd attribute (bmc) is a regular
expression that is similar to a substitution pattern. The 1st part \z matches the end of the node name and substitutes
-bmc, effectively appending it to the node name.
Another example is if “node1” is assigned the IP address “10.0.0.1”, node2 is assigned the IP address “10.0.0.2”, etc.,
then this could be represented in the hosts table with the single row
#node,ip,hostnames,otherinterfaces,comments,disable
"compute","|node(\d+)|10.0.0.($1+0)|",,,,
In this example, the regular expression in the ip attribute uses | to separate the 1st and 2nd part. This means that
xCAT will allow arithmetic operations in the 2nd part. In the 1st part, (\d+), will match the number part of the node
name and put that in a variable called $1. The 2nd part is what value to give the ip attribute. In this case it will set it
to the string “10.0.0.” and the number that is in $1. (Zero is added to $1 just to remove any leading zeros.)
A more involved example is with the vm table. If your kvm nodes have node names c01f01x01v01, c01f02x03v04,
etc., and the kvm host names are c01f01x01, c01f02x03, etc., then you might have an vm table like
#node,mgr,host,migrationdest,storage,storagemodel,storagecache,storageformat,
cfgstore,memory,cpus,nics,nicmodel,bootorder,clockoffset,virtflags,master,vncport,
˓→textconsole,powerstate,beacon,datacenter,cluster,guestostype,othersettings,physlots,
˓→vidmodel,vidproto,vidpassword,comments,disable
"kvms",,"|\D+(\d+)\D+(\d+)\D+(\d+)\D+(\d+)|c($1)f($2)x($3)|",,
˓→"|\D+(\d+)\D+(\d+)\D+(\d+)\D+(\d+)|dir:///install/vms/vm($4+0)|",,,,,"3072","2",
˓→"virbr2","virtio",,,,,,,,,,,,,,,,,,
˓→
Before you panic, let me explain each column:
kvms
This is a group name. In this example, we are assuming that all of your kvm nodes belong to this
group. Each time the xCAT software accesses the vm table to get the kvm host host and storage file
vmstorage of a specific kvm node (e.g. c01f02x03v04), this row will match (because c01f02x03v04
is in the kvms group). Once this row is matched for c01f02x03v04, then the processing described in the
following items will take place.
|\D+(\d+)\D+(\d+)\D+(\d+)\D+(\d+)|c($1)f($2)x($3)|
This is a perl substitution pattern that will produce the value for the 3rd column of the table (the kvm host).
The text \D+(\d+)\D+(\d+)\D+(\d+)\D+(\d+) between the 1st two vertical bars is a regular
expression that matches the node name that was searched for in this table (in this example c01f02x03v04).
The text that matches within the 1st set of parentheses is set to $1, 2nd set of parentheses is set to $2
,3rd set of parentheses is set to $3,and so on. In our case, the \D+ matches the non-numeric part of the
name (“c”,”f”,”x”,”v”) and the \d+ matches the numeric part (“01”,”02”,”03”,”04”). So $1 is set to “01”,
$2 is set to “02”, $3 is set to “03”, and $4 is set to “04”. The text c($1)f($2)x($3) between the
2nd and 3rd vertical bars produces the string that should be used as the value for the host attribute for
c01f02x03v04, i.e,”c01f02x03”.
|\D+(\d+)\D+(\d+)\D+(\d+)\D+(\d+)|dir:///install/vms/vm($4+0)|
This item is similar to the one above. This substitution pattern will produce the value for the 5th column
(a list of storage files or devices to be used). Because this row was the match for “c01f02x03v04”, the
produced value is “dir:///install/vms/vm4”.
Just as the explained above, when the node definition “c01f02x03v04” is created with
# mkdef -t node -o c01f02x03v04 groups=kvms
1 object definitions have been created or modified.
The generated node definition is
48
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
# lsdef c01f02x03v04
Object name: c01f02x03v04
groups=kvms
postbootscripts=otherpkgs
postscripts=syslog,remoteshell,syncfiles
vmcpus=2
vmhost=c01f02x03
vmmemory=3072
vmnicnicmodel=virtio
vmnics=virbr2
vmstorage=dir:///install/vms/vm4
See perlre for more information on perl regular expressions.
Easy Regular expressions
As of xCAT 2.8.1, you can use a modified version of the regular expression support described in the previous section.
You do not need to enter the node information (1st part of the expression), it will be derived from the input nodename.
You only need to supply the 2nd part of the expression to determine the value to give the attribute.
For example:
If node1 is assigned the IP address 10.0.0.1, node2 is assigned the IP address 10.0.0.2, etc., then this could be represented in the hosts table with the single row:
Using full regular expression support you would put this in the hosts table.
chdef -t group compute ip="|node(\d+)|10.0.0.($1+0)|"
tabdump hosts
#node,ip,hostnames,otherinterfaces,comments,disable
"compute","|node(\d+)|10.0.0.($1+0)|",,,,
Using easy regular expression support you would put this in the hosts table.
chdef -t group compute ip="|10.0.0.($1+0)|"
tabdump hosts
#node,ip,hostnames,otherinterfaces,comments,disable
"compute","|10.0.0.($1+0)|",,,,
In the easy regx example, the expression only has the 2nd part of the expression from the previous example. xCAT
will evaluate the node name, matching the number part of the node name, and create the 1st part of the expression .
The 2nd part supplied is what value to give the ip attribute. The resulting output is the same.
Verify your regular expression
After you create your table with regular expression, make sure they are evaluating as you expect.
lsdef node1 | grep ip
ip=10.0.0.1
Global Configuration
All the xCAT global configurations are stored in site table, xCAT Admin can adjust the configuration by modifying
the site attribute with tabedit.
1.3. Admin Guide
49
xCAT3 Documentation, Release 2.12
This section only presents some key global configurations, for the complete reference on the xCAT global configurations, refer to the tabdump -d site.
Database Attributes
• excludenodes: A set of comma separated nodes and/or groups that would automatically be subtracted from any
noderange, it can be used for excluding some failed nodes from any xCAT command. See noderange for details
on supported formats.
• nodestatus: If set to n, the nodelist.status column will not be updated during the node deployment, node
discovery and power operations. The default is to update.
DHCP Attributes
• dhcpinterfaces: The network interfaces DHCP should listen on. If it is the same for all nodes, use a simple
comma-separated list of NICs. To specify different NICs for different nodes
xcatmn|eth1,eth2;service|bond0.
In this example xcatmn is the name of the xCAT MN, and DHCP there should listen on eth1 and eth2. On all of
the nodes in group service DHCP should listen on the bond0 nic.
• dhcplease: The lease time for the dhcp client. The default value is 43200.
• managedaddressmode: The mode of networking configuration during node provision. If set to static, the
network configuration will be configured in static mode based on the node and network definition on MN. If set
to dhcp, the network will be configured with dhcp protocol. The default is dhcp.
DNS Attributes
• domain: The DNS domain name used for the cluster.
• forwarders: The DNS servers at your site that can provide names outside of the cluster. The makedns command
will configure the DNS on the management node to forward requests it does not know to these servers. Note
that the DNS servers on the service nodes will ignore this value and always be configured to forward requests to
the management node.
• master: The hostname of the xCAT management node, as known by the nodes.
• nameservers: A comma delimited list of DNS servers that each node in the cluster should use. This value
will end up in the nameserver settings of the /etc/resolv.conf on each node. It is common (but not
required) to set this attribute value to the IP addr of the xCAT management node, if you have set up the DNS
on the management node by running makedns. In a hierarchical cluster, you can also set this attribute to
<xcatmaster> to mean the DNS server for each node should be the node that is managing it (either its
service node or the management node).
• dnsinterfaces: The network interfaces DNS server should listen on. If it is the same for all nodes, use a simple
comma-separated list of NICs. To specify different NICs for different nodes
xcatmn|eth1,eth2;service|bond0.
In this example xcatmn is the name of the xCAT MN, and DNS there should listen on eth1 and eth2. On all of
the nodes in group service DNS should listen on the bond0 nic.
NOTE: if using this attribute to block certain interfaces, make sure the ip that maps to your hostname of xCAT
MN is not blocked since xCAT needs to use this ip to communicate with the local DNS server on MN.
50
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
Install/Deployment Attributes
• installdir: The local directory name used to hold the node deployment packages.
• runbootscripts: If set to yes the scripts listed in the postbootscripts attribute in the osimage and postscripts
tables will be run during each reboot of stateful (diskful) nodes. This attribute has no effect on stateless nodes.
Run the following command after you change the value of this attribute
updatenode <nodes> -P setuppostbootscripts
• precreatemypostscripts: (yes/1 or no/0). Default is no. If yes, it will instruct xCAT at nodeset and
updatenode time to query the db once for all of the nodes passed into the cmd and create the mypostscript
file for each node, and put them in a directory of tftpdir(such as: /tftpboot). If no, it will not generate the
mypostscript file in the tftpdir.
• xcatdebugmode: the xCAT debug level. xCAT provides a batch of techniques to help user debug problems while
using xCAT, especially on OS provision, such as collecting logs of the whole installation process and accessing
the installing system via ssh, etc. These techniques will be enabled according to different xCAT debug levels
specified by ‘xcatdebugmode’, currently supported values:
'0':
'1':
'2':
disable debug mode
enable basic debug mode
enable expert debug mode
For the details on ‘basic debug mode’ and ‘expert debug mode’, refer to xCAT documentation.
Remoteshell Attributes
• sshbetweennodes: Comma separated list of groups of compute nodes to enable passwordless root ssh during
install, or xdsh -K. Default is ALLGROUPS. Set to NOGROUPS if you do not wish to enable it for any group
of compute nodes. If using the zone table, this attribute in not used.
Services Attributes
• consoleondemand: When set to yes, conserver connects and creates the console output only when the user
opens the console. Default is no on Linux, yes on AIX.
• timezone: The timezone for all the nodes in the cluster(e.g. America/New_York).
• tftpdir: tftp directory path. Default is /tftpboot.
• tftpflags:
The flags used to start tftpd.
Default is -v -l -s /tftpboot -m /etc/
tftpmapfile4xcat.conf if tftplfags is not set.
Virtualization Attributes
• persistkvmguests: Keep the kvm definition on the kvm hypervisor when you power off the kvm guest node.
This is useful for you to manually change the kvm xml definition file in virsh for debugging. Set anything
means enable.
xCAT Daemon attributes
• xcatdport: The port used by xcatd daemon for client/server communication.
1.3. Admin Guide
51
xCAT3 Documentation, Release 2.12
• xcatiport: The port used by xcatd to receive installation status updates from nodes.
• xcatlport: The port used by xcatd command log writer process to collect command output.
• xcatsslversion: The ssl version by xcatd. Default is SSLv3.
• xcatsslciphers: The ssl cipher by xcatd. Default is 3DES.
Network Planning
For a cluster, several networks are necessary to enable the cluster management and production.
• Management network
This network is used by the management node to install and manage the OS of the nodes. The MN and in-band
NIC of the nodes are connected to this network. If you have a large cluster with service nodes, sometimes this
network is segregated into separate VLANs for each service node.
Following network services need be set up in this network to supply the OS deployment, application install/configuration service.
– DNS(Domain Name Service)
The dns server, usually the management node or service node, provides the domain name service for the
entire cluster.
– HTTP(HyperText Transfer Protocol)
The http server,usually the management node or service node, acts as the download server for the initrd
and kernel, the configuration file for the installer and repository for the online installation.
– DHCP(Dynamic Host Configuration Protocol)
The dhcp server, usually the management node or service node, provides the dhcp service for the entire
cluster.
– TFTP(Trivial File Transfer Protocol)
The tftp server, usually the management node or service node, acts as the download server for bootloader
binaries, bootloader configuration file, initrd and kernel.
– NFS(Network File System)
The NFS server, usually the management node or service node, provides the file system sharing between
the management node and service node, or persistent file system support for the stateless node.
– NTP(Network Time Protocol)
The NTP server, usually the management node or service node, provide the network time service for the
entire cluster.
• Service network
This network is used by the management node to control the nodes out of band via the SP like BMC, FSP. If the
BMCs are configured in shared mode1 , then this network can be combined with the management network.
• Application network
This network is used by the applications on the compute nodes. Usually an IB network for HPC cluster.
• Site (Public) network This network is used to access the management node and sometimes for the compute
nodes to provide services to the site.
1
52
shared mode: In “Shared” mode, the BMC network interface and the in-band network interface will share the same network port.
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
From the system management perspective, the Management network and Service network are necessary to perform
the hardware control and OS deployment.
xCAT Network Planning for a New Cluster:
xCAT Network Planning
Before setting up your cluster, there are a few things that are important to think through first, because it is much easier
to go in the direction you want right from the beginning, instead of changing course midway through.
Do You Need Hierarchy in Your Cluster?
Service Nodes
For very large clusters, xCAT has the ability to distribute the management operations to service nodes. This allows
the management node to delegate all management responsibilities for a set of compute or storage nodes to a service
node so that the management node doesn’t get overloaded. Although xCAT automates a lot of the aspects of deploying
and configuring the services, it still adds complexity to your cluster. So the question is: at what size cluster do you
need to start using service nodes? The exact answer depends on a lot of factors (mgmt node size, network speed, node
type, OS, frequency of node deployment, etc.), but here are some general guidelines for how many nodes a single
management node (or single service node) can handle:
• [Linux]:
– Stateful or Stateless: 500 nodes
– Statelite: 250 nodes
• [AIX]: 150 nodes
These numbers can be higher (approximately double) if you are willing to “stage” the more intensive operations, like
node deployment.
Of course, there are some reasons to use service nodes that are not related to scale, for example, if some of your nodes
are far away (network-wise) from the mgmt node.
Network Hierarchy
For large clusters, you may want to divide the management network into separate subnets to limit the broadcast
domains. (Service nodes and subnets don’t have to coincide, although they often do.) xCAT clusters as large as 3500
nodes have used a single broadcast domain.
Some cluster administrators also choose to sub-divide the application interconnect to limit the network contention
between separate parallel jobs.
Design an xCAT Cluster for High Availability
Everyone wants their cluster to be as reliable and available as possible, but there are multiple ways to achieve that end
goal. Availability and complexity are inversely proportional. You should choose an approach that balances these 2 in
a way that fits your environment the best. Here’s a few choices in order of least complex to more complex.
1.3. Admin Guide
53
xCAT3 Documentation, Release 2.12
Service Node Pools With No HA Software
Service node pools is an xCAT approach in which more than one service node (SN) is in the broadcast domain for a
set of nodes. When each node netboots, it chooses an available SN by which one responds to its DHCP request 1st.
When services are set up on the node (e.g. DNS), xCAT configures the services to use at that SN and one other SN in
the pool. That way, if one SN goes down, the node can keep running, and the next time it netboots it will automatically
choose another SN.
This approach is most often used with stateless nodes because that environment is more dynamic. It can possibly be
used with stateful nodes (with a little more effort), but that type of node doesn’t netboot nearly as often so a more
manual operation (snmove) is needed in that case move a node to different SNs.
It is best to have the SNs be as robust as possible, for example, if they are diskful, configure them with at least 2 disks
that are RAID’ed together.
In smaller clusters, the management node (MN) can be part of the SN pool with one other SN.
In larger clusters, if the network topology dictates that the MN is only for managing the SNs (not the compute nodes),
then you need a plan for what to do if the MN fails. Since the cluster can continue to run if the MN is down temporarily,
the plan could be as simple as have a backup MN w/o any disks. If the primary MN fails, move its RAID’ed disks to
the backup MN and power it on.
HA Management Node
If you want to use HA software on your management node to synchronize data and fail over services to a backup MN,
see [TODO Highly_Available_Management_Node], which discusses the different options and the pros and cons.
It is important to note that some HA-related software like DRDB, Pacemaker, and Corosync is not officially supported
by IBM, meaning that if you have a problem specifically with that software, you will have to go to the open source
community or another vendor to get a fix.
HA Service Nodes
When you have NFS-based diskless (statelite) nodes, there is sometimes the motivation make the NFS serving highly
available among all of the service nodes. This is not recommended because it is a very complex configuration. In our
opinion, the complexity of this setup can nullify much of the availability you hope to gain. If you need your compute
nodes to be highly available, you should strongly consider stateful or stateless nodes.
If you still have reasons to pursue HA service nodes:
• For [AIX] , see [TODO XCAT_HASN_with_GPFS]
• For [Linux] , a couple prototype clusters have been set up in which the NFS service on the SNs is provided by
GPFS CNFS (Clustered NFS). A howto is being written to describe the setup as an example. Stay tuned.
xCAT Cluster OS Running Type
Whether a node is a physical server or a virtual machine, it needs to run an Operating System to support user applications. Generally, the OS is installed in the hard disk of the compute node. But xCAT also support the type that running
OS in the RAM.
This section gives the pros and cons of each OS running type, and describes the cluster characteristics that will impact
from each.
54
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
Stateful (diskful)
Traditional cluster with OS on each node’s local disk.
• Main advantage
This approach is familiar to most admins, and they typically have many years of experience with it.
• Main disadvantage
Admin has to manage all of the individual OS copies, has to face the failure of hard disk. For certain application
which requires all the compute nodes have exactly same state, this is also changeable for admin.
Stateless (diskless)
Nodes boot from a RAMdisk OS image downloaded from the xCAT mgmt node or service node at boot time.
• Main advantage
Central management of OS image, but nodes are not tethered to the mgmt node or service node it booted from.
Whenever you need a new OS for the node, just reboot the node.
• Main disadvantage
You can’t use a large image with many different applications in the image for varied users, because it uses
too much of the node’s memory to store the ramdisk. (To mitigate this disadvantage, you can put your large
application binaries and libraries in shared storage to reduce the ramdisk size. This requires some manual
configuration of the image).
Each node can also have a local “scratch” disk for swap, /tmp, /var, log files, dumps, etc. The
purpose of the scratch disk is to provide a location for files that are written to by the node that can
become quite large or for files that you don’t want to disappear when the node reboots. There should
be nothing put on the scratch disk that represents the node’s “state”, so that if the disk fails you can
simply replace it and reboot the node. A scratch disk would typically be used for situations like:
job scheduling preemption is required (which needs a lot of swap space), the applications write large
temp files, or you want to keep gpfs log or trace files persistently. (As a partial alternative to using the
scratch disk, customers can choose to put /tmp /var/tmp, and log files (except GPFS logs files)
in GPFS, but must be willing to accept the dependency on GPFS). This can be done by enabling the
‘localdisk’ support. For the details, refer to the section [TODO Enabling the localdisk Option].
OSimage Definition
The attribute provmethod is used to identify that the osimage is diskful or diskless:
$ lsdef -t osimage rhels7.1-x86_64-install-compute -i provmethod
Object name: rhels7.1-x86_64-install-compute
provmethod=install
install: Diskful
netboot: Diskless
Manage Clusters
The following provides detailed information to help start managing your cluster using xCAT.
The sections are organized based on hardware architecture.
1.3. Admin Guide
55
xCAT3 Documentation, Release 2.12
IBM POWER LE / OpenPOWER
Most of the content is general information for xCAT, the focus and examples are for management of IBM OpenPOWER servers.
IBM OpenPOWER Servers
• based on POWER8 Processor Technology is IPMI managed
• based on POWER9 Processor Technology is OpenBMC managed [Alpha]
Configure xCAT
After installing xCAT onto the management node, configure some basic attributes for your cluster into xCAT.
Set attributes in the site table
1. Verify the following attributes have been correctly set in the xCAT site table.
• domain
• forwarders
• master1
• nameservers
For more information on the keywords, see the DHCP ATTRIBUTES in the site table.
If the fields are not set or need to be changed, use the xCAT chdef command:
chdef
chdef
chdef
chdef
-t
-t
-t
-t
site
site
site
site
domain="domain_string"
fowarders="forwarders"
master="xcat_master_ip"
nameservers="nameserver1,nameserver2,etc"
Initialize DNS services
1. Initialize the DNS2 services on the xCAT Management Node:
makedns -n
Verify DNS is working by running nslookup against your Management Node:
nslookup <management_node_hostname>
For more information on DNS, refer to Cluster Name Resolution
Set attributes in the networks table
1. Display the network settings defined in the xCAT networks table using: tabdump networks
1
2
56
The value of the master attribute in the site table should be set as the IP address of the management node responsible for the compute node.
Setting up name resolution and the ability to have hostname resolved to IP addresses is required for xCAT.
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
#netname,net,mask,mgtifname,gateway,dhcpserver,tftpserver,nameservers,ntpservers,
˓→logservers,
dynamicrange,staticrange,staticrangeincrement,nodehostname,ddnsdomain,vlanid,
˓→domain,mtu,
comments,disable
"10_0_0_0-255_0_0_0","10.0.0.0","255.0.0.0","eth0","10.0.0.101",,"10.4.27.5",,,,,,
˓→,,,,,,,
A default network is created for the detected primary network using the same netmask and gateway. There may
be additional network entries in the table for each network present on the management node where xCAT is
installed.
2. To define additional networks, use one of the following options:
• [Recommended] Use mkdef to create/update an entry into networks table.
To create a network entry for 192.168.X.X/16 with a gateway of 192.168.1.254:
mkdef -t network -o net1 net=192.168.0.0 mask=255.255.0.0 gateway=192.168.1.
˓→254
• Use the tabedit command to modify the networks table directly in an editor:
tabedit networks
• Use the makenetworks command to automatically generate a entry in the networks table:
makenetworks
3. Verify the network statements
Domain and nameserver attributes must be configured in the networks table or in the site table for xCAT
to function properly.
Initialize DHCP services
Configure DHCP to listen on different network interfaces [Optional]
The default behavior of xCAT is to configure DHCP to listen on all interfaces defined in the networks
table.
The dhcpinterfaces keyword in the site table allows administrators to limit the interfaces that
DHCP will listen over. If the management node has 4 interfaces, (eth0, eth1, eth2, and eth3), and you
want DHCP to listen only on “eth1” and “eth3”, set dhcpinterfaces using:
chdef -t site dhcpinterfaces="eth1,eth3"
To set “eth1” and “eth3” on the management node and “bond0” on all nodes in the nodegroup=”service”,
set dhcpinterfaces using:
chdef -t site dhcpinterfaces="eth1,eth3;service|bond0"
or, to explicitly identify the management node with hostname xcatmn:
chdef -t site dhcpinterfaces="xcatmn|eth1,eth3;service|bond0"
1.3. Admin Guide
57
xCAT3 Documentation, Release 2.12
noboot
For the IBM OpenPOWER S822LC for HPC (“Minsky”) nodes, the BMC and compute “eth0” share the
left-side integrated ethernet port and compute “eth1” is the right-side integrated ethernet port. For these
servers, it is recommended to use two physical cables allowing the BMC port to be dedicated and “eth1”
used by the OS. When an open range is configured on the two networks, the xCAT Genesis kernel will be
sent to the BMC interface and causes problems during hardware discovery. To support this scenario, on
the xCAT management node, if “eth1” is connected to the BMC network and “eth3” is connected to the
compute network, disable genesis boot for the BMC network by setting :noboot in dhcpinterfaces
using:
chdef -t site dhcpinterfaces="eth1:noboot,eth3"
# run the mknb command to remove the genesis
# configuration file for the specified network
mknb ppc64
For more information, see dhcpinterfaces keyword in the site table.
After making any DHCP changes, create a new DHCP configuration file with the networks defined using the
makedhcp command.
makedhcp -n
Configure passwords
1. Configure the system password for the root user on the compute nodes.
• Set using the chtab command:
chtab key=system passwd.username=root passwd.password=abc123
To encrypt the password using openssl, use the following command:
chtab key=system passwd.username=root passwd.password=`openssl passwd -1
˓→abc123`
2. Configure the passwords for Management modules of the compute nodes.
• For OpenBMC managed systems:
chtab key=openbmc passwd.username=root passwd.password=0penBmc
• For IPMI/BMC managed systems:
chtab key=ipmi passwd.username=ADMIN passwd.password=admin
• For HMC managed systems:
chtab key=hmc passwd.username=hscroot passwd.password=abc123
If the username/password is different for multiple HMCs, set the username and password attribute for
each HMC node object in xCAT
• For Blade managed systems:
58
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
chtab key=blade passwd.username=USERID passwd.password=PASSW0RD
• For FSP/BPA (Flexible Service Processor/Bulk Power Assembly) the factory default passwords must be
changed before running commands against them.
rspconfig frame general_passwd=general,<newpassword>
rspconfig frame admin_passwd=admin,<newpassword>
rspconfig frame HMC_passwd=,<newpassword>
3. If using the xCAT REST API
(a) Create a non-root user that will be used to make the REST API calls.
useradd xcatws
passwd xcatws # set the password
(b) Create an entry for the user into the xCAT passwd table.
chtab key=xcat passwd.username=xcatws passwd.password=<xcatws_
˓→password>
(c) Set a policy in the xCAT policy table to allow the user to make calls against xCAT.
mkdef -t policy 6 name=xcatws rule=allow
When making calls to the xCAT REST API, pass in the credentials using the following attributes:
userName and userPW
Hardware Discovery & Define Node
In order to manage machines using xCAT, the machines need to be defined as xCAT node objects in the database.
The xCAT Objects documentation describes the process for manually creating node objects one by one using the
xCAT mkdef command. This is valid when managing a small sizes cluster but can be error prone and cumbersome
when managing large sized clusters.
xCAT provides several automatic hardware discovery methods to assist with hardware discovery by helping to simplify
the process of detecting service processors (SP) and collecting various server information. The following are methods
that xCAT supports:
MTMS-based Discovery
MTMS stands for Machine Type/Model and Serial. This is one way to uniquely identify each physical server.
MTMS-based hardware discovery assumes the administrator has the model type and serial number information for the
physical servers and a plan for mapping the servers to intended hostname/IP addresses.
Overview
1. Automatically search and collect MTMS information from the servers
2. Write discovered-bmc-nodes to xCAT (recommended to set different BMC IP address)
3. Create predefined-compute-nodes to xCAT providing additional properties
4. Power on the nodes which triggers xCAT hardware discovery engine
Pros
1.3. Admin Guide
59
xCAT3 Documentation, Release 2.12
• Limited effort to get servers defined using xCAT hardware discovery engine
Cons
• When compared to switch-based discovery, the administrator needs to create the predefined-compute-nodes
for each of the discovered-bmc-nodes. This could become difficult for a large number of servers.
Verification
Before starting hardware discovery, ensure the following is configured to make the discovery process as smooth as
possible.
Password Table
In order to communicate with IPMI-based hardware (with BMCs), verify that the xCAT passwd table contains an
entry for ipmi which defines the default username and password to communicate with the IPMI-based servers.
tabdump passwd | grep ipmi
If not configured, use the following command to set usernam=ADMIN and password=admin.
chtab key=ipmi passwd.username=ADMIN passwd.password=admin
Genesis Package
The xCAT-genesis packages provides the utility to create the genesis network boot rootimage used by xCAT when
doing hardware discovery. It should be installed during the xCAT install and would cause problems if missing.
Verify that the genesis-scripts and genesis-base packages are installed:
• [RHEL/SLES]:
rpm -qa | grep -i genesis
• [Ubuntu]:
dpkg -l | grep -i genesis
If missing:
1. Install them from the xcat-dep repository using the Operating Specific package manager (yum, zypper,
apt-get, etc)
• [RHEL]:
yum install xCAT-genesis
• [SLES]:
zypper install xCAT-genesis
• [Ubuntu]:
apt-get install xCAT-genesis
60
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
2. Create the network boot rootimage with the following command: mknb ppc64.
The genesis kernel should be copied to /tftpboot/xcat.
Discovery
When the IPMI-based servers are connected to power, the BMCs will boot up and attempt to obtain an IP address from
an open range dhcp server on your network. In the case for xCAT managed networks, xCAT should be configured
serve an open range dhcp IP addresses with the dynamicrange attribute in the networks table.
When the BMCs have an IP address and is pingable from the xCAT management node, administrators can discover the
BMCs using the xCAT’s bmcdiscover command and obtain basic information to start the hardware discovery process.
xCAT Hardware discover uses the xCAT genesis kernel (diskless) to discover additional attributes of the compute node
and automatically populate the node definitions in xCAT.
Set static BMC IP using different IP address (recommended)
The following example outlines the MTMS based hardware discovery for a single IPMI-based compute node.
Compute Node Information
Model Type
Serial Number
Hostname
IP address
Value
8247-22l
10112CA
cn01
10.1.2.1
The BMC IP address is obtained by the open range dhcp server and the plan in this scenario is to change the IP address
for the BMC to a static IP address in a different subnet than the open range addresses. The static IP address in this
example is in the same subnet as the open range to simplify the networking configuration on the xCAT management
node.
BMC Information
IP address - dhcp
IP address - static
Value
172.30.0.1
172.20.2.1
1. Detect the BMCs and add the node definitions into xCAT.
Use the bmcdiscover command to discover the BMCs responding over an IP range and write the output into the
xCAT database. This discovered BMC node is used to control the physical server during hardware discovery
and will be deleted after the correct server node object is matched to a pre-defined node. You must use the -w
option to write the output into the xCAT database.
To discover the BMC with an IP address range of 50.0.100.1-100:
bmcdiscover --range 50.0.100.1-100 -z -w
The discovered nodes will be written to xCAT database. The discovered BMC nodes are in the form nodemodel_type-serial. To view the discovered nodes:
lsdef /node-.*
Note: The bmcdiscover command will use the username/password from the passwd table corresponding
to key=ipmi. To overwrite with a different username/password use the -u and -p option to bmcdiscover.
2. Pre-define the compute nodes:
Use the bmcdiscover command to help discover the nodes over an IP range and easily create a starting file
to define the compute nodes into xCAT.
1.3. Admin Guide
61
xCAT3 Documentation, Release 2.12
To discover the compute nodes for the BMCs with an IP address of 172.30.0.1, use the command:
bmcdiscover --range 172.30.0.1 -z > predefined.stanzas
The discovered nodes have the naming convention: node-<model-type>-<serial-number>
# cat predefined.stanzas
node-8247-22l-10112ca:
objtype=node
groups=all
bmc=172.30.0.1
cons=ipmi
mgt=ipmi
mtm=8247-22L
serial=10112CA
3. Edit the predefined.stanzas file and change the discovered nodes to the intended hostname and IP
address.
(a) Edit the predefined.stanzas file:
vi predefined.stanzas
(b) Rename the discovered object names to their intended compute node hostnames based on the
MTMS mapping:
node-8247-22l-10112ca ==> cn01
(c) Add a ip attribute and give it the compute node IP address:
ip=10.1.2.1
(d) Remove nodetype and hwtype if defined in the predefined.stanza.
(e) Repeat for additional nodes in the predefined.stanza file based on the MTMS mapping.
In this example, the predefined.stanzas file now looks like the following:
# cat predefined.stanzas
cn01:
objtype=node
groups=all
bmc=172.30.0.1
cons=ipmi
mgt=ipmi
mtm=8247-22L
serial=10112CA
ip=10.1.2.1
4. Set the chain table to run the bmcsetup script, this will set the BMC IP to static.
chdef cn01 chain="runcmd=bmcsetup"
5. Set the target osimage into the chain table to automatically provision the operating system after the node discovery is complete.
chdef cn01 -p chain="osimage=<osimage_name>"
6. Change the BMC IP address
62
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
Set the BMC IP address to a different value for the predefined compute node definitions.
To change the dhcp obtained IP address of 172.30.0.1 to a static IP address of 172.20.2.1, run the following
command:
chdef cn01 bmc=172.20.2.1
7. Define the compute nodes into xCAT:
cat predefined.stanzas | mkdef -z
8. Add the compute node IP information to /etc/hosts:
makehosts cn01
9. Refresh the DNS configuration for the new hosts:
makedns -n
10. [Optional] Monitor the node discovery process using rcons
Configure the conserver for the discovered node to watch the discovery process using rcons:
makeconservercf node-8247-22l-10112ca
In another terminal window, open the remote console:
rcons node-8247-22l-10112ca
11. Start the discovery process by booting the discovered node definition:
rsetboot node-8247-22l-10112ca net
rpower node-8247-22l-10112ca on
12. The discovery process will network boot the machine into the diskless xCAT genesis kernel and perform the
discovery process. When the discovery process is complete, doing lsdef on the compute nodes should show
discovered attributes for the machine. The important mac information should be discovered, which is necessary
for xCAT to perform OS provisioning.
Set static BMC IP using dhcp provided IP address
The following example outlines the MTMS based hardware discovery for a single IPMI-based compute node.
Compute Node Information
Model Type
Serial Number
Hostname
IP address
Value
8247-22l
10112CA
cn01
10.1.2.1
The BMC IP address is obtained by the open range dhcp server and the plan is to leave the IP address the same, except
we want to change the IP address to be static in the BMC.
BMC Information
IP address - dhcp
IP address - static
Value
172.30.0.1
172.30.0.1
1. Pre-define the compute nodes:
1.3. Admin Guide
63
xCAT3 Documentation, Release 2.12
Use the bmcdiscover command to help discover the nodes over an IP range and easily create a starting file
to define the compute nodes into xCAT.
To discover the compute nodes for the BMCs with an IP address of 172.30.0.1, use the command:
bmcdiscover --range 172.30.0.1 -z > predefined.stanzas
The discovered nodes have the naming convention: node-<model-type>-<serial-number>
# cat predefined.stanzas
node-8247-22l-10112ca:
objtype=node
groups=all
bmc=172.30.0.1
cons=ipmi
mgt=ipmi
mtm=8247-22L
serial=10112CA
2. Edit the predefined.stanzas file and change the discovered nodes to the intended hostname and IP
address.
(a) Edit the predefined.stanzas file:
vi predefined.stanzas
(b) Rename the discovered object names to their intended compute node hostnames based on the
MTMS mapping:
node-8247-22l-10112ca ==> cn01
(c) Add a ip attribute and give it the compute node IP address:
ip=10.1.2.1
(d) Repeat for additional nodes in the predefined.stanza file based on the MTMS mapping.
In this example, the predefined.stanzas file now looks like the following:
# cat predefined.stanzas
cn01:
objtype=node
groups=all
bmc=172.30.0.1
cons=ipmi
mgt=ipmi
mtm=8247-22L
serial=10112CA
ip=10.1.2.1
3. Set the chain table to run the bmcsetup script, this will set the BMC IP to static.
chdef cn01 chain="runcmd=bmcsetup"
4. Set the target osimage into the chain table to automatically provision the operating system after the node discovery is complete.
chdef cn01 -p chain="osimage=<osimage_name>"
64
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
5. Define the compute nodes into xCAT:
cat predefined.stanzas | mkdef -z
6. Add the compute node IP information to /etc/hosts:
makehosts cn01
7. Refresh the DNS configuration for the new hosts:
makedns -n
8. [Optional] Monitor the node discovery process using rcons
Configure the conserver for the predefined node to watch the discovery process using rcons:
makeconservercf cn01
In another terminal window, open the remote console:
rcons cn01
9. Start the discovery process by booting the predefined node definition:
rsetboot cn01 net
rpower cn01 on
10. The discovery process will network boot the machine into the diskless xCAT genesis kernel and perform the
discovery process. When the discovery process is complete, doing lsdef on the compute nodes should show
discovered attributes for the machine. The important mac information should be discovered, which is necessary
for xCAT to perform OS provisioning.
Switch-based Discovery
For switch based hardware discovery, the servers are identified through the switches and switchposts they are directly
connected to.
In this document, the following configuration is used in the example
Management Node info:
MN Hostname: xcat1
MN NIC info for Management Network(Host network): eth1, 10.0.1.1/16
MN NIC info for Service Network(FSP/BMC nework): eth2, 50.0.1.1/16
Dynamic IP range for Hosts: 10.0.100.1-10.0.100.100
Dynamic IP range for FSP/BMC: 50.0.100.1-50.0.100.100
Compute Node info:
CN Hostname: cn1
Machine type/model: 8247-22L
Serial: 10112CA
IP Address: 10.0.101.1
Root Password: cluster
Desired FSP/BMC IP Address: 50.0.101.1
DHCP assigned FSP/BMC IP Address: 50.0.100.1
FSP/BMC username: ADMIN
FSP/BMC Password: admin
1.3. Admin Guide
65
xCAT3 Documentation, Release 2.12
Switch info:
Switch
Switch
Switch
Switch
Switch
name: switch1
username: xcat
password: passw0rd
IP Address: 10.0.201.1
port for Compute Node: port0
Configure xCAT
Configure network table
Normally, there will be at least two entries for the two subnet on MN in networks table after xCAT is installed:
#tabdump networks
#netname,net,mask,mgtifname,gateway,dhcpserver,tftpserver,nameservers,ntpservers,
˓→logservers,dynamicrange,staticrange,staticrangeincrement,nodehostname,ddnsdomain,
˓→vlanid,domain,mtu,comments,disable
"10_0_0_0-255_255_0_0","10.0.0.0","255.255.0.0","eth1","<xcatmaster>",,"10.0.1.1",,,,,
˓→,,,,,,,,
"50_0_0_0-255_255_0_0","50.0.0.0","255.255.0.0","eth2","<xcatmaster>",,"50.0.1.1",,,,,
˓→,,,,,,,,
Run the following command to add networks in networks table if there are no entries in it:
makenetworks
Setup DHCP
Set the correct NIC from which DHCP server provide service:
chdef -t site dhcpinterfaces=eth1,eth2
Add dynamic range in purpose of assigning temporary IP address for FSP/BMCs and hosts:
chdef -t network 10_0_0_0-255_255_0_0 dynamicrange="10.0.100.1-10.0.100.100"
chdef -t network 50_0_0_0-255_255_0_0 dynamicrange="50.0.100.1-50.0.100.100"
Update DHCP configuration file:
makedhcp -n
makedhcp -a
Config passwd table
Set required passwords for xCAT to do hardware management and/or OS provisioning by adding entries to the xCAT
passwd table:
# tabedit passwd
# key,username,password,cryptmethod,authdomain,comments,disable
For hardware management with ipmi, add the following line:
66
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
"ipmi","ADMIN","admin",,,,
Verify the genesis packages
The xcat-genesis packages should have been installed when xCAT was installed, but would cause problems if missing.
xcat-genesis packages are required to create the genesis root image to do hardware discovery and the genesis kernel
sits in /tftpboot/xcat/. Verify that the genesis-scripts and genesis-base packages are installed:
• [RHEL/SLES]: rpm -qa | grep -i genesis
• [Ubuntu]: dpkg -l | grep -i genesis
If missing, install them from the xcat-deps package and run mknb ppc64 to create the genesis network boot root
image.
Predefined Nodes
In order to differentiate one node from another, the admin needs to predefine node in xCAT database based on the
switches information. This consists of two parts:
1. Predefine Switches
2. Predefine Server Node
Predefine Switches
The predefined switches will represent devices that the physical servers are connected to. xCAT need to access those
switches to get server related information through SNMP v3.
So the admin need to make sure those switches are configured correctly with SNMP v3 enabled. <TODO: The
document that Configure Ethernet Switches>
Then, define switch info into xCAT:
nodeadd switch1 groups=switch,all
chdef switch1 ip=10.0.201.1
tabch switch=switch1 switches.snmpversion=3 switches.username=xcat switches.
˓→password=passw0rd switches.auth=sha
Add switch into DNS using the following commands:
makehosts switch1
makedns -n
Predefine Server node
After switches are defined, the server node can be predefined with the following commands:
nodeadd cn1 groups=powerLE,all
chdef cn1 mgt=ipmi cons=ipmi ip=10.0.101.1 bmc=50.0.101.1 netboot=petitboot
˓→installnic=mac primarynic=mac
chdef cn1 switch=switch1 switchport=0
In order to do BMC configuration during the discovery process, set runcmd=bmcsetup.
chdef cn1 chain="runcmd=bmcsetup"
1.3. Admin Guide
67
xCAT3 Documentation, Release 2.12
Set the target osimage into the chain table to automatically provision the operating system after the node discovery is
complete.
chdef cn1 -p chain="osimage=<osimage_name>"
For more information about chain, refer to Chain
Add cn1 into DNS:
makehosts cn1
maekdns -n
Discover server and define
After environment is ready, and the server is powered, we can start server discovery process. The first thing to do is
discovering the FSP/BMC of the server. It is automatically powered on when the physical server is powered.
Use the bmcdiscover command to discover the BMCs responding over an IP range and write the output into the xCAT
database. This discovered BMC node is used to control the physical server during hardware discovery and will be
deleted after the correct server node object is matched to a pre-defined node. You must use the -w option to write the
output into the xCAT database.
To discover the BMC with an IP address range of 50.0.100.1-100:
bmcdiscover --range 50.0.100.1-100 -z -w
The discovered nodes will be written to xCAT database. The discovered BMC nodes are in the form node-model_typeserial. To view the discovered nodes:
lsdef /node-.*
Note: The bmcdiscover command will use the username/password from the passwd table corresponding to
key=ipmi. To overwrite with a different username/password use the -u and -p option to bmcdiscover.
Start discovery process
To start discovery process, just need to power on the PBMC node remotely with the following command, and the
discovery process will start automatically after the host is powered on:
rpower node-8247-42l-10112ca on
[Optional] If you’d like to monitor the discovery process, you can use:
makeconservercf node-8247-42l-10112ca
rcons node-8247-42l-10112ca
Verify node definition
The following is an example of the server node definition after hardware discovery:
#lsdef cn1
Object name: cn1
arch=ppc64
bmc=50.0.101.1
68
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
cons=ipmi
cpucount=192
cputype=POWER8E (raw), altivec supported
groups=powerLE,all
installnic=mac
ip=10.0.101.1
mac=6c:ae:8b:02:12:50
memory=65118MB
mgt=ipmi
mtm=8247-22L
netboot=petitboot
postbootscripts=otherpkgs
postscripts=syslog,remoteshell,syncfiles
primarynic=mac
serial=10112CA
supportedarchs=ppc64
switch=switch1
switchport=0
Sequential-based Discovery
When the physical location of the server is not so important, sequential-based hardware discovery can be used to
simplify the discovery work. The idea is: provided a node pool, each node in the pool will be assigned an IP address
for host and an IP address for FSP/BMC, then the first physical server discovery request will be matched to the first
free node in the node pool, and IP addresses for host and FSP/BMC will be assigned to that physical server.
In this document, the following configuration is used in the example
Management Node info:
MN Hostname: xcat1
MN NIC info for Management Network(Host network): eth1, 10.0.1.1/16
MN NIC info for Service Network(FSP/BMC nework): eth2, 50.0.1.1/16
Dynamic IP range for Hosts: 10.0.100.1-10.0.100.100
Dynamic IP range for FSP/BMC: 50.0.100.1-50.0.100.100
Compute Node info:
CN Hostname: cn1
Machine type/model: 8247-22L
Serial: 10112CA
IP Address: 10.0.101.1
Root Password: cluster
Desired FSP/BMC IP Address: 50.0.101.1
DHCP assigned FSP/BMC IP Address: 50.0.100.1
FSP/BMC username: ADMIN
FSP/BMC Password: admin
Configure xCAT
Configure network table
Normally, there will be at least two entries for the two subnet on MN in networks table after xCAT is installed:
1.3. Admin Guide
69
xCAT3 Documentation, Release 2.12
#tabdump networks
#netname,net,mask,mgtifname,gateway,dhcpserver,tftpserver,nameservers,ntpservers,
˓→logservers,dynamicrange,staticrange,staticrangeincrement,nodehostname,ddnsdomain,
˓→vlanid,domain,mtu,comments,disable
"10_0_0_0-255_255_0_0","10.0.0.0","255.255.0.0","eth1","<xcatmaster>",,"10.0.1.1",,,,,
˓→,,,,,,,,
"50_0_0_0-255_255_0_0","50.0.0.0","255.255.0.0","eth2","<xcatmaster>",,"50.0.1.1",,,,,
˓→,,,,,,,,
Run the following command to add networks in networks table if there are no entries in it:
makenetworks
Setup DHCP
Set the correct NIC from which DHCP server provide service:
chdef -t site dhcpinterfaces=eth1,eth2
Add dynamic range in purpose of assigning temporary IP address for FSP/BMCs and hosts:
chdef -t network 10_0_0_0-255_255_0_0 dynamicrange="10.0.100.1-10.0.100.100"
chdef -t network 50_0_0_0-255_255_0_0 dynamicrange="50.0.100.1-50.0.100.100"
Update DHCP configuration file:
makedhcp -n
makedhcp -a
Config passwd table
Set required passwords for xCAT to do hardware management and/or OS provisioning by adding entries to the xCAT
passwd table:
# tabedit passwd
# key,username,password,cryptmethod,authdomain,comments,disable
For hardware management with ipmi, add the following line:
"ipmi","ADMIN","admin",,,,
Verify the genesis packages
The xcat-genesis packages should have been installed when xCAT was installed, but would cause problems if missing.
xcat-genesis packages are required to create the genesis root image to do hardware discovery and the genesis kernel
sits in /tftpboot/xcat/. Verify that the genesis-scripts and genesis-base packages are installed:
• [RHEL/SLES]: rpm -qa | grep -i genesis
• [Ubuntu]: dpkg -l | grep -i genesis
If missing, install them from the xcat-deps package and run mknb ppc64 to create the genesis network boot root
image.
70
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
Prepare node pool
To prepare the node pool, shall predefine nodes first, then initialize the discovery process with the predefined nodes.
Predefine nodes
Predefine a group of nodes with desired IP address for host and IP address for FSP/BMC:
nodeadd cn1 groups=powerLE,all
chdef cn1 mgt=ipmi cons=ipmi ip=10.0.101.1 bmc=50.0.101.1 netboot=petitboot
˓→installnic=mac primarynic=mac
In order to do BMC configuration during the discovery process, set runcmd=bmcsetup.
chdef cn1 chain="runcmd=bmcsetup"
Set the target osimage into the chain table to automatically provision the operating system after the node discovery is
complete.
chdef cn1 -p chain="osimage=<osimage_name>"
For more information about chain, refer to Chain
Initialize the discovery process
Specify the predefined nodes to the nodediscoverstart command to initialize the discovery process:
nodediscoverstart noderange=cn1
See “nodediscoverstart man page<TBD>” for more details.
Display information about the discovery process
There are additional nodediscover commands you can run during the discovery process. See their man pages for more
details.
Verify the status of discovery:
nodediscoverstatus
Show the nodes that have been discovered so far:
nodediscoverls -t seq -l
Stop the current sequential discovery process:
nodediscoverstop
Note: The sequential discovery process will be stopped automatically when all of the node names in the node pool are
used up.
1.3. Admin Guide
71
xCAT3 Documentation, Release 2.12
Start discovery process
To start the discovery process, the system administrator needs to power on the servers one by one manually. Then the
hardware discovery process will start automatically.
Verify node definition
The following is an example of the server node definition after hardware discovery:
# lsdef cn1
Object name: cn1
arch=ppc64
bmc=50.0.100.1
cons=ipmi
cpucount=192
cputype=POWER8E (raw), altivec supported
groups=powerLE,all
installnic=mac
ip=10.0.101.1
mac=6c:ae:8b:02:12:50
memory=65118MB
mgt=ipmi
mtm=8247-22L
netboot=petitboot
postbootscripts=otherpkgs
postscripts=syslog,remoteshell,syncfiles
primarynic=mac
serial=10112CA
supportedarchs=ppc64
Manually Define Nodes
Manually Define Node means the admin knows the detailed information of the physical server and manually defines
it into xCAT database with mkdef commands.
In this document, the following configuration is used in the example
Management Node info:
MN Hostname: xcat1
MN NIC info for Management Network(Host network): eth1, 10.0.1.1/16
MN NIC info for Service Network(FSP/BMC nework): eth2, 50.0.1.1/16
Dynamic IP range for Hosts: 10.0.100.1-10.0.100.100
Dynamic IP range for FSP/BMC: 50.0.100.1-50.0.100.100
Compute Node info:
CN Hostname: cn1
Machine type/model: 8247-22L
Serial: 10112CA
IP Address: 10.0.101.1
Root Password: cluster
Desired FSP/BMC IP Address: 50.0.101.1
DHCP assigned FSP/BMC IP Address: 50.0.100.1
72
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
FSP/BMC username: ADMIN
FSP/BMC Password: admin
Manually Define Node
Execute mkdef command to define the node:
mkdef -t node cn1 groups=powerLE,all mgt=ipmi cons=ipmi ip=10.0.101.1
˓→netboot=petitboot bmc=50.0.101.1 bmcusername=ADMIN bmcpassword=admin installnic=mac
˓→primarynic=mac mac=6c:ae:8b:6a:d4:e4
The manually defined node will be like this:
# lsdef cn1
Object name: cn1
bmc=50.0.101.1
bmcpassword=admin
bmcusername=ADMIN
cons=ipmi
groups=powerLE,all
installnic=mac
ip=10.0.101.1
mac=6c:ae:8b:6a:d4:e4
mgt=ipmi
netboot=petitboot
postbootscripts=otherpkgs
postscripts=syslog,remoteshell,syncfiles
primarynic=mac
Manually Discover Nodes
If you have a few nodes which were not discovered by automated hardware discovery process, you can find them
in discoverydata table using the nodediscoverls command. The undiscovered nodes are those that have a
discovery method value of ‘undef’ in the discoverydata table.
Display the undefined nodes with the nodediscoverls command:
#nodediscoverls -t undef
UUID
˓→SERIAL
fa2cec8a-b724-4840-82c7-3313811788cd
˓→10112CA
NODE
METHOD
MTM
undef
undef
8247-22L
If you want to manually define an ‘undefined’ node to a specific free node name, use the nodediscoverdef(TODO)
command.
Before doing that, a node with desired IP address for host and FSP/BMC must be defined first:
nodeadd cn1 groups=powerLE,all
chdef cn1 mgt=ipmi cons=ipmi ip=10.0.101.1 bmc=50.0.101.1 netboot=petitboot
˓→installnic=mac primarynic=mac
For
example,
if
you
want
to
assign
the
fa2cec8a-b724-4840-82c7-3313811788cd to cn1, run:
1.3. Admin Guide
undefined
node
whose
uuid
is
73
xCAT3 Documentation, Release 2.12
nodediscoverdef -u fa2cec8a-b724-4840-82c7-3313811788cd -n cn1
After manually defining it, the ‘node name’ and ‘discovery method’ attributes of the node will be changed. You can
display the changed attributes using the nodediscoverls command:
#nodediscoverls
UUID
˓→SERIAL
fa2cec8a-b724-4840-82c7-3313811788cd
˓→10112CA
NODE
cn1
METHOD
manual
MTM
8247-22L
Following are the brief characteristics and adaptability of each method, you can select a proper one according to your
cluster size and other consideration.
• Manually Define Nodes
Manually collect information for target servers and manually define them to xCAT Node Object through mkdef
command.
This method is recommended for small cluster which has less than 10 nodes.
– pros
No specific configuration and procedure required and very easy to use.
– cons
It will take additional time to configure the SP (Management Modules like: BMC, FSP) and collect the
server information like MTMS (Machine Type and Machine Serial) and Host MAC address for OS deployment ...
This method is inefficient and error-prone for a large number of servers.
• MTMS-based Discovery
Step1: Automatically search all the servers and collect server MTMS information.
Step2: Define the searched server to a Node Object automatically. In this case, the node name will be generated
based on the MTMS string. The admin can rename the Node Object to a reasonable name like r1u1 (It means
the physical location is in Rack1 and Unit1).
Step3: Power on the nodes, xCAT discovery engine will update additional information like the MAC for deployment for the nodes.
This method is recommended for the medium scale of cluster which has less than 100 nodes.
– pros
With limited effort to get the automatic discovery benefit.
– cons
Compared to Switch-based Discovery, the admin needs to be involved to rename the automatically discovered node to a reasonable name (optional). It’s hard to rename the node to a location-based name for a
large number of server.
• Switch-based Discovery
Step1: Pre-define the Node Object for all the nodes in the cluster. The Pre-defined node must have the
attributes switch and switchport defined to specify which Switch and Port this server connected to. xCAT will
use this Switch and Port information to map a discovered node to certain Pre-defined node.
Step2: Power on the nodes, xCAT discovery engine will discover node attributes and update them to certain
Pre-defined node.
74
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
– pros
The whole discovery process is totally automatic.
Since the node is physically identified by the Switch and Port that the server connected, if a node fail and
replaced with a new one, xCAT will automatically discover the new one and assign it to the original node
name since the Switch and Port does not change.
– cons
You need to plan the cluster with planned Switch and Port mapping for each server and switch. All the
Switches need be configured with snmpv3 accessible for xCAT management node.
• Sequential-based Discovery
Step1: Pre-define the Node Object for all the nodes in the cluster.
Step2: Manually power on the node one by one. The booted node will be discovered, each new discovered node
will be assigned to one of the Pre-defined node in Sequential.
– pros
No special configuration required like Switch-based Discovery. No manual rename node step required
like MTMS-based Discovery.
– cons
You have to strictly boot on the node in order if you want the node has the expected name. Generally you
have to waiting for the discovery process finished before power on the next one.
Hardware Management
Basic Operations
rbeacon - Beacon Light
See rbeacon manpage for more information.
Most enterprise level servers have LEDs on their front and/or rear panels, one of which is a beacon light. If turned on,
this light can assist the system administrator in locating one physical machine in the cluster.
Using xCAT, administrators can turn on and off the beacon light using: rbeacon <node> on|off
rpower - Remote Power Control
See rpower manpage for more information.
Use the rpower command to remotely power on and off a single server or a range of servers.
rpower <noderange> on
rpower <noderange> off
Other actions include:
• To get the current power state of a server: rpower <noderange> state
• To boot/reboot a server: rpower <noderange> boot
• To hardware reset a server: rpower <noderange> reset
1.3. Admin Guide
75
xCAT3 Documentation, Release 2.12
rcons - Remote Console
See rcons manpage for more information.
Most enterprise servers do not have video adapters installed with the machine and often do not provide a method for
attaching a physical monitor/keyboard/mouse to get the display output. For this purpose xCAT can assist the system
administrator to view the console over a “Serial-over-LAN” (SOL) connection through the BMC.
Configure the correct console management by modifying the node definition:
• For OpenPOWER, IPMI managed server:
chdef -t node -o <noderange> cons=ipmi
• For OpenPOWER, OpenBMC managed servers:
chdef -t node -o <noderange> cons=openbmc
Open a console to compute1:
rcons compute1
Note: The keystroke ctrl+e c . will disconnect you from the console.
Troubleshooting
General
The xCAT rcons command relies on conserver (http://www.conserver.com/). The conserver package should have
been installed with xCAT as it’s part of the xCAT dependency package. If you are having problems seeing the console,
try the following.
1. Make sure conserver is configured by running makeconservercf.
2. Check if conserver is up and running
[sysvinit] service conserver status
[systemd] systemctl status conserver.service
3. If conserver is not running, start the service using:
[sysvinit] service conserver start
[systemd] systemctl start conserver.service
4. After this, try invoking the console again: rcons <node>
OpenBMC Specific
1. For OpenBMC managed servers, the root user must be able to ssh passwordless to the BMC for the rcons
function to work.
Copy the /root/.ssh/id_rsa.pub public key to the BMC’s ~/.ssh/authorized_keys file.
76
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
Advanced Operations
rinv - Remote Hardware Inventory
See rinv manpage for more information.
Use rinv command to remotely obtain inventory information of a physical machine. This will help to distinguish
one machine from another and aid in mapping the model type and/or serial number of a machine with its host name.
To get all the hardware information for node cn1:
rinv cn1 all
To get just the firmware information for cn1:
rinv cn1 firm
rvitals - Remote Hardware Vitals
See rvitals manpage for more information.
Collecting runtime information from a running physical machine is an important part of system administration. Data
can be obtained from the service processor including temperature, voltage, cooling fans, etc.
Use the rvitals command to obtain this information.
rvitals <noderange> all
To only get the temperature information of machines in a particular noderange:
rvitals <noderange> temp
rflash - Remote Firmware Flashing
See rflash manpage for more information.
The rflash command is provided to assist the system administrator in updating firmware.
To check the current firmware version on the node’s BMC and the HPM file:
rflash <noderange> -c /firmware/8335_810.1543.20151021b_update.hpm
To update the firmware on the node’s BMC to version in the HPM file:
rflash <noderange> /firmware/8335_810.1543.20151021b_update.hpm
rspconfig - Remote Configuration of Service Processors
See rspconfig manpage for more information.
The rspconfig command can be used to configure the service processor, or Baseboard Management Controller
(BMC), of a physical machine.
For example, to turn on SNMP alerts for node cn5:
1.3. Admin Guide
77
xCAT3 Documentation, Release 2.12
rspconfig cn5 alert=on
Diskful Installation
Select or Create an osimage Definition
Before creating an image on xCAT, the distro media should be prepared ahead. That can be ISOs or DVDs.
XCAT use ‘copycds’ command to create an image which will be available to install nodes. “copycds” will copy all
contents of Distribution DVDs/ISOs or Service Pack DVDs/ISOs to a destination directory, and create several relevant
osimage definitions by default.
If using an ISO, copy it to (or NFS mount it on) the management node, and then run:
copycds <path>/<specific-distro>.iso
If using a DVD, put it in the DVD drive of the management node and run:
copycds /dev/<dvd-drive-name>
To see the list of osimages:
lsdef -t osimage
To see the attributes of a particular osimage:
lsdef -t osimage <osimage-name>
Initially, some attributes of osimage are assigned default values by xCAT - they all can work correctly because the files
or templates invoked by those attributes are shipped with xCAT by default. If you need to customize those attributes,
refer to the next section Customize osimage
Below is an example of osimage definitions created by copycds:
# lsdef -t osimage
rhels7.2-ppc64le-install-compute (osimage)
rhels7.2-ppc64le-install-service (osimage)
rhels7.2-ppc64le-netboot-compute (osimage)
rhels7.2-ppc64le-stateful-mgmtnode (osimage)
In these osimage definitions shown above
• <os>-<arch>-install-compute is the default osimage definition used for diskful installation
• <os>-<arch>-netboot-compute is the default osimage definition used for diskless installation
• <os>-<arch>-install-service is the default osimage definition used for service node deployment which shall be
used in hierarchical environment
Note: There are more things needed for ubuntu ppc64le osimages:
For ubuntu ppc64le, the initrd.gz shipped with the ISO does not support network booting. In order to install ubuntu
with xCAT, you need to follow the steps below to complete the osimage definition.
• Download mini.iso from
[ubuntu 14.04.1]: http://xcat.org/files/netboot/ubuntu14.04.1/ppc64el/mini.iso
[ubuntu 14.04.2]: http://xcat.org/files/netboot/ubuntu14.04.2/ppc64el/mini.iso
78
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
[ubuntu 14.04.3]: http://xcat.org/files/netboot/ubuntu14.04.3/ppc64el/mini.iso
[ubuntu 14.04.4]: http://xcat.org/files/netboot/ubuntu14.04.4/ppc64el/mini.iso
[ubuntu 16.04]: http://xcat.org/files/netboot/ubuntu16.04/ppc64el/mini.iso
[ubuntu 16.04.1]: http://xcat.org/files/netboot/ubuntu16.04.1/ppc64el/mini.iso
• Mount mini.iso
mkdir /tmp/iso
mount -o loop mini.iso /tmp/iso
• Copy the netboot initrd.gz to osimage
mkdir -p /install/<ubuntu-version>/ppc64el/install/netboot
cp /tmp/iso/install/initrd.gz /install/<ubuntu-version>/ppc64el/install/netboot
[Below tips maybe helpful for you]
[Tips 1]
If this is the same distro version as what your management node uses, create a .repo file in /etc/yum.repos.d with
contents similar to:
[local-<os>-<arch>]
name=xCAT local <os> <version>
baseurl=file:/install/<os>/<arch>
enabled=1
gpgcheck=0
In this way, if you need to install some additional RPMs into your MN later, you can simply install them with yum.
Or if you are installing a software on your MN that depends some RPMs from this disto, those RPMs will be found
and installed automatically.
[Tips 2]
You can create/modify an osimage definition easily based on the default osimage definition. The general steps are:
• lsdef -t osimage -z <os>-<arch>-install-compute > <filename>.stanza
• modify <filename>.stanza according to your requirements
• cat <filename>.stanza| mkdef -z
For example, if you need to change the osimage name to your favorite name, this command may be helpful:
chdef -t osimage rhels6.2-x86_64-install-compute -n rhels6.2_myimage
Customize osimage (Optional)
Optional means all the subitems in this page are not necessary to finish an OS deployment. If you are new to xCAT,
you can just jump to Initialize the Compute for Deployment.
Configure RAID before deploying the OS
1.3. Admin Guide
79
xCAT3 Documentation, Release 2.12
Overview
xCAT provides an user interface linuximage.partitionfile to specify the customized partition script for diskful provision,
and provides some default partition scripts.
Deploy Diskful Nodes with RAID1 Setup on RedHat
xCAT provides a partition script raid1_rh.sh which configures RAID1 across 2 disks on RHEL 7.x operating systems.
In most scenarios, the sample partitioning script is sufficient to create a basic RAID1 across two disks and is provided
as a sample to build upon.
1. Obtain the partition script:
mkdir -p /install/custom/partition/
wget https://raw.githubusercontent.com/xcat2/xcat-extensions/master/partition/
˓→raid1_rh.sh \
-O /install/custom/partition/raid1_rh.sh
2. Associate the partition script to the osimage:
chdef -t osimage -o rhels7.3-ppc64le-install-compute \
partitionfile="s:/install/custom/partition/raid1_rh.sh"
3. Provision the node:
rinstall cn1 osimage=rhels7.3-ppc64le-install-compute
After the diskful nodes are up and running, you can check the RAID1 settings with the following process:
mount command shows the /dev/mdx devices are mounted to various file systems, the /dev/mdx indicates that
the RAID is being used on this node.
# mount
...
/dev/md1 on / type xfs (rw,relatime,attr2,inode64,noquota)
/dev/md0 on /boot type xfs (rw,relatime,attr2,inode64,noquota)
/dev/md2 on /var type xfs (rw,relatime,attr2,inode64,noquota)
The file /proc/mdstat includes the RAID devices status on the system, here is an example of /proc/mdstat
in the non-multipath environment:
# cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 sdk2[0] sdj2[1]
1047552 blocks super 1.2 [2/2] [UU]
resync=DELAYED
bitmap: 1/1 pages [64KB], 65536KB chunk
md3 : active raid1 sdk3[0] sdj3[1]
1047552 blocks super 1.2 [2/2] [UU]
resync=DELAYED
md0 : active raid1 sdk5[0] sdj5[1]
524224 blocks super 1.0 [2/2] [UU]
bitmap: 0/1 pages [0KB], 65536KB chunk
80
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
md1 : active raid1 sdk6[0] sdj6[1]
973998080 blocks super 1.2 [2/2] [UU]
[==>..................] resync = 12.8% (125356224/973998080) finish=138.1min
˓→speed=102389K/sec
bitmap: 1/1 pages [64KB], 65536KB chunk
unused devices: <none>
On the system with multipath configuration, the /proc/mdstat looks like:
# cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 dm-11[0] dm-6[1]
291703676 blocks super 1.1 [2/2] [UU]
bitmap: 1/1 pages [64KB], 65536KB chunk
md1 : active raid1 dm-8[0] dm-3[1]
1048568 blocks super 1.1 [2/2] [UU]
md0 : active raid1 dm-9[0] dm-4[1]
204788 blocks super 1.0 [2/2] [UU]
unused devices: <none>
The command mdadm can query the detailed configuration for the RAID partitions:
mdadm --detail /dev/md2
Deploy Diskful Nodes with RAID1 Setup on SLES
xCAT provides one sample autoyast template files with the RAID1 settings /opt/xcat/share/xcat/
install/sles/service.raid1.sles11.tmpl. You can customize the template file and put it under /
install/custom/install/<platform>/ if the default one does not match your requirements.
Here is the RAID1 partitioning section in service.raid1.sles11.tmpl:
<partitioning config:type="list">
<drive>
<device>/dev/sda</device>
<partitions config:type="list">
<partition>
<format config:type="boolean">false</format>
<partition_id config:type="integer">65</partition_id>
<partition_nr config:type="integer">1</partition_nr>
<partition_type>primary</partition_type>
<size>24M</size>
</partition>
<partition>
<format config:type="boolean">false</format>
<partition_id config:type="integer">253</partition_id>
<partition_nr config:type="integer">2</partition_nr>
<raid_name>/dev/md0</raid_name>
<raid_type>raid</raid_type>
<size>2G</size>
</partition>
<partition>
1.3. Admin Guide
81
xCAT3 Documentation, Release 2.12
<format config:type="boolean">false</format>
<partition_id config:type="integer">253</partition_id>
<partition_nr config:type="integer">3</partition_nr>
<raid_name>/dev/md1</raid_name>
<raid_type>raid</raid_type>
<size>max</size>
</partition>
</partitions>
<use>all</use>
</drive>
<drive>
<device>/dev/sdb</device>
<partitions config:type="list">
<partition>
<format config:type="boolean">false</format>
<partition_id config:type="integer">131</partition_id>
<partition_nr config:type="integer">1</partition_nr>
<partition_type>primary</partition_type>
<size>24M</size>
</partition>
<partition>
<format config:type="boolean">false</format>
<partition_id config:type="integer">253</partition_id>
<partition_nr config:type="integer">2</partition_nr>
<raid_name>/dev/md0</raid_name>
<raid_type>raid</raid_type>
<size>2G</size>
</partition>
<partition>
<format config:type="boolean">false</format>
<partition_id config:type="integer">253</partition_id>
<partition_nr config:type="integer">3</partition_nr>
<raid_name>/dev/md1</raid_name>
<raid_type>raid</raid_type>
<size>max</size>
</partition>
</partitions>
<use>all</use>
</drive>
<drive>
<device>/dev/md</device>
<partitions config:type="list">
<partition>
<filesystem config:type="symbol">reiser</filesystem>
<format config:type="boolean">true</format>
<mount>swap</mount>
<partition_id config:type="integer">131</partition_id>
<partition_nr config:type="integer">0</partition_nr>
<raid_options>
<chunk_size>4</chunk_size>
<parity_algorithm>left-asymmetric</parity_algorithm>
<raid_type>raid1</raid_type>
</raid_options>
</partition>
<partition>
<filesystem config:type="symbol">reiser</filesystem>
<format config:type="boolean">true</format>
<mount>/</mount>
82
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
<partition_id config:type="integer">131</partition_id>
<partition_nr config:type="integer">1</partition_nr>
<raid_options>
<chunk_size>4</chunk_size>
<parity_algorithm>left-asymmetric</parity_algorithm>
<raid_type>raid1</raid_type>
</raid_options>
</partition>
</partitions>
<use>all</use>
</drive>
</partitioning>
The samples above created one 24MB PReP partition on each disk, one 2GB mirrored swap partition and one mirrored
/ partition uses all the disk space. If you want to use different partitioning scheme in your cluster, modify this RAID1
section in the autoyast template file accordingly.
Since the PReP partition can not be mirrored between the two disks, some additional postinstall commands should be
run to make the second disk bootable, here the commands needed to make the second disk bootable:
# Set the second disk to be bootable for RAID1 setup
parted -s /dev/sdb mkfs 1 fat16
parted /dev/sdb set 1 type 6
parted /dev/sdb set 1 boot on
dd if=/dev/sda1 of=/dev/sdb1
bootlist -m normal sda sdb
The procedure listed above has been added to the file /opt/xcat/share/xcat/install/scripts/post.
sles11.raid1 to make it be automated. The autoyast template file service.raid1.sles11.tmpl will include the content of post.sles11.raid1, so no manual steps are needed here.
After the diskful nodes are up and running, you can check the RAID1 settings with the following commands:
Mount command shows the /dev/mdx devices are mounted to various file systems, the /dev/mdx indicates that
the RAID is being used on this node.
server:~ # mount
/dev/md1 on / type reiserfs (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
debugfs on /sys/kernel/debug type debugfs (rw)
devtmpfs on /dev type devtmpfs (rw,mode=0755)
tmpfs on /dev/shm type tmpfs (rw,mode=1777)
devpts on /dev/pts type devpts (rw,mode=0620,gid=5)
The file /proc/mdstat includes the RAID devices status on the system, here is an example of /proc/mdstat:
server:~ # cat /proc/mdstat
Personalities : [raid1] [raid0] [raid10] [raid6] [raid5] [raid4]
md0 : active (auto-read-only) raid1 sda2[0] sdb2[1]
2104500 blocks super 1.0 [2/2] [UU]
bitmap: 0/1 pages [0KB], 128KB chunk
md1 : active raid1 sda3[0] sdb3[1]
18828108 blocks super 1.0 [2/2] [UU]
bitmap: 0/9 pages [0KB], 64KB chunk
unused devices: <none>
1.3. Admin Guide
83
xCAT3 Documentation, Release 2.12
The command mdadm can query the detailed configuration for the RAID partitions:
mdadm --detail /dev/md1
Disk Replacement Procedure
If any one disk fails in the RAID1 array, do not panic. Follow the procedure listed below to replace the failed disk and
you will be fine.
Faulty disks should appear marked with an (F) if you look at /proc/mdstat:
# cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 dm-11[0](F) dm-6[1]
291703676 blocks super 1.1 [2/1] [_U]
bitmap: 1/1 pages [64KB], 65536KB chunk
md1 : active raid1 dm-8[0](F) dm-3[1]
1048568 blocks super 1.1 [2/1] [_U]
md0 : active raid1 dm-9[0](F) dm-4[1]
204788 blocks super 1.0 [2/1] [_U]
unused devices: <none>
We can see that the first disk is broken because all the RAID partitions on this disk are marked as (F).
Remove the failed disk from RAID array
mdadm is the command that can be used to query and manage the RAID arrays on Linux. To remove the failed disk
from RAID array, use the command:
mdadm --manage /dev/mdx --remove /dev/xxx
Where the /dev/mdx are the RAID partitions listed in /proc/mdstat file, such as md0, md1 and md2; the /
dev/xxx are the backend devices like dm-11, dm-8 and dm-9 in the multipath configuration and sda5, sda3 and sda2
in the non-multipath configuration.
Here is the example of removing failed disk from the RAID1 array in the non-multipath configuration:
mdadm --manage /dev/md0 --remove /dev/sda3
mdadm --manage /dev/md1 --remove /dev/sda2
mdadm --manage /dev/md2 --remove /dev/sda5
Here is the example of removing failed disk from the RAID1 array in the multipath configuration:
mdadm --manage /dev/md0 --remove /dev/dm-9
mdadm --manage /dev/md1 --remove /dev/dm-8
mdadm --manage /dev/md2 --remove /dev/dm-11
After the failed disk is removed from the RAID1 array, the partitions on the failed disk will be removed from /proc/
mdstat and the “mdadm –detail” output also.
# cat /proc/mdstat
Personalities : [raid1]
84
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
md2 : active raid1 dm-6[1]
291703676 blocks super 1.1 [2/1] [_U]
bitmap: 1/1 pages [64KB], 65536KB chunk
md1 : active raid1 dm-3[1]
1048568 blocks super 1.1 [2/1] [_U]
md0 : active raid1 dm-4[1]
204788 blocks super 1.0 [2/1] [_U]
unused devices: <none>
# mdadm --detail /dev/md0
/dev/md0:
Version : 1.0
Creation Time : Tue Jul 19 02:39:03 2011
Raid Level : raid1
Array Size : 204788 (200.02 MiB 209.70 MB)
Used Dev Size : 204788 (200.02 MiB 209.70 MB)
Raid Devices : 2
Total Devices : 1
Persistence : Superblock is persistent
Update Time
State
Active Devices
Working Devices
Failed Devices
Spare Devices
:
:
:
:
:
:
Wed Jul 20 02:00:04 2011
clean, degraded
1
1
0
0
Name : c250f17c01ap01:0 (local to host c250f17c01ap01)
UUID : eba4d8ad:8f08f231:3c60e20f:1f929144
Events : 26
Number
0
1
Major
0
253
Minor
0
4
RaidDevice State
0
removed
1
active sync
/dev/dm-4
Replace the disk
Depends on the hot swap capability, you may simply unplug the disk and replace with a new one if the hot swap is
supported; otherwise, you will need to power off the machine and replace the disk and the power on the machine.
Create partitions on the new disk
The first thing we must do now is to create the exact same partitioning as on the new disk. We can do this with one
simple command:
sfdisk -d /dev/<good_disk> | sfdisk /dev/<new_disk>
For the non-mulipath configuration, here is an example:
sfdisk -d /dev/sdb | sfdisk /dev/sda
For the multipath configuration, here is an example:
1.3. Admin Guide
85
xCAT3 Documentation, Release 2.12
sfdisk -d /dev/dm-1 | sfdisk /dev/dm-0
If you got error message “sfdisk: I don’t like these partitions - nothing changed.”, you can add “–force” option to the
sfdisk command:
sfdisk -d /dev/sdb | sfdisk /dev/sda --force
You can run:
fdisk -l
To check if both hard drives have the same partitioning now.
Add the new disk into the RAID1 array
After the partitions are created on the new disk, you can use command:
mdadm --manage /dev/mdx --add /dev/xxx
To add the new disk to the RAID1 array. Where the /dev/mdx are the RAID partitions like md0, md1 and md2; the
/dev/xxx are the backend devices like dm-11, dm-8 and dm-9 in the multipath configuration and sda5, sda3 and
sda2 in the non-multipath configuration.
Here is an example for the non-multipath configuration:
mdadm --manage /dev/md0 --add /dev/sda3
mdadm --manage /dev/md1 --add /dev/sda2
mdadm --manage /dev/md2 --add /dev/sda5
Here is an example for the multipath configuration:
mdadm --manage /dev/md0 --add /dev/dm-9
mdadm --manage /dev/md1 --add /dev/dm-8
mdadm --manage /dev/md2 --add /dev/dm-11
All done! You can have a cup of coffee to watch the fully automatic reconstruction running...
While the RAID1 array is reconstructing, you will see some progress information in /proc/mdstat:
# cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 dm-11[0] dm-6[1]
291703676 blocks super 1.1 [2/1] [_U]
[>....................] recovery = 0.7% (2103744/291703676) finish=86.2min
˓→speed=55960K/sec
bitmap: 1/1 pages [64KB], 65536KB chunk
md1 : active raid1 dm-8[0] dm-3[1]
1048568 blocks super 1.1 [2/1] [_U]
[=============>.......] recovery = 65.1% (683904/1048568) finish=0.1min
˓→speed=48850K/sec
md0 : active raid1 dm-9[0] dm-4[1]
204788 blocks super 1.0 [2/1] [_U]
[===================>.] recovery = 96.5% (198016/204788) finish=0.0min
˓→speed=14144K/sec
86
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
unused devices: <none>
After the reconstruction is done, the /proc/mdstat becomes like:
# cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 dm-11[0] dm-6[1]
291703676 blocks super 1.1 [2/2] [UU]
bitmap: 1/1 pages [64KB], 65536KB chunk
md1 : active raid1 dm-8[0] dm-3[1]
1048568 blocks super 1.1 [2/2] [UU]
md0 : active raid1 dm-9[0] dm-4[1]
204788 blocks super 1.0 [2/2] [UU]
unused devices: <none>
Make the new disk bootable
If the new disk does not have a PReP partition or the PReP partition has some problem, it will not be bootable, here is
an example on how to make the new disk bootable, you may need to substitute the device name with your own values.
• [RHEL]:
mkofboot .b /dev/sda
bootlist -m normal sda sdb
• [SLES]:
parted -s /dev/sda mkfs 1 fat16
parted /dev/sda set 1 type 6
parted /dev/sda set 1 boot on
dd if=/dev/sdb1 of=/dev/sda1
bootlist -m normal sda sdb
Load Additional Drivers
Overview
During the installing or netbooting of a node, the drivers in the initrd will be used to drive the devices like network
cards and IO devices to perform the installation/netbooting tasks. But sometimes the drivers for the new devices were
not included in the default initrd shipped by Red Hat or Suse. A solution is to inject the new drivers into the initrd to
drive the new device during the installation/netbooting process.
Generally there are two approaches to inject the new drivers: Driver Update Disk and Drive RPM package.
A “Driver Update Disk” is media which contains the drivers, firmware and related configuration files for certain
devices. The driver update disk is always supplied by the vendor of the device. One driver update disk can contain
multiple drivers for different OS releases and different hardware architectures. Red Hat and Suse have different driver
update disk formats.
The ‘Driver RPM Package‘ is the rpm package which includes the drivers and firmware for the specific devices. The
Driver RPM is the rpm package which is shipped by the Vendor of the device for a new device or a new kernel version.
1.3. Admin Guide
87
xCAT3 Documentation, Release 2.12
xCAT supports both. But for ‘Driver RPM Package‘ is only supported in xCAT 2.8 and later.
No matter which approach chosen, there are two steps to make new drivers work. one is locate the new driver’s path,
another is inject the new drivers into the initrd.
Locate the New Drivers
For Driver Update Disk
There are two approaches for xCAT to find the driver disk (pick one):
1. Specify the location of the driver disk in the osimage object (This is ONLY supported in 2.8 and later)
The value for the ‘driverupdatesrc’ attribute is a comma separated driver disk list. The tag ‘dud’ must be
specified before the full path of ‘driver update disk’ to specify the type of the file:
chdef -t osimage <osimagename> driverupdatesrc=dud:<full path of driver disk>
1. Put the driver update disk in the directory <installroot>/driverdisk/<os>/<arch> (example: /
install/driverdisk/sles11.1/x86_64).
During the running of the genimage, geninitrd, or nodeset commands, xCAT will look for driver update
disks in the directory <installroot>/driverdisk/<os>/<arch>.
For Driver RPM Packages
The Driver RPM packages must be specified in the osimage object.
Three attributes of osimage object can be used to specify the Driver RPM location and Driver names. If you want to
load new drivers in the initrd, the ‘netdrivers‘ attribute must be set. And one or both of the ‘driverupdatesrc‘ and
‘osupdatename‘ attributes must be set. If both of ‘driverupdatesrc’ and ‘osupdatename’ are set, the drivers in the
‘driverupdatesrc’ have higher priority.
• netdrivers - comma separated driver names that need to be injected into the initrd. The postfix ‘.ko’ can be
ignored.
The ‘netdrivers’ attribute must be set to specify the new driver list. If you want to load all the drivers from the driver
rpms, use the keyword allupdate. Another keyword for the netdrivers attribute is updateonly, which means only the
drivers located in the original initrd will be added to the newly built initrd from the driver rpms. This is useful to
reduce the size of the new built initrd when the distro is updated, since there are many more drivers in the new kernel
rpm than in the original initrd. Examples:
chdef -t osimage <osimagename> netdrivers=megaraid_sas.ko,igb.ko
chdef -t osimage <osimagename> netdrivers=allupdate
chdef -t osimage <osimagename> netdrivers=updateonly,igb.ko,new.ko
• driverupdatesrc - comma separated driver rpm packages (full path should be specified)
A tag named ‘rpm’ can be specified before the full path of the rpm to specify the file type. The tag is optional since
the default format is ‘rpm’ if no tag is specified. Example:
chdef -t osimage <osimagename> driverupdatesrc=rpm:<full path of driver disk1>,rpm:
˓→<full path of driver disk2>
• osupdatename - comma separated ‘osdistroupdate’ objects. Each ‘osdistroupdate’ object specifies a Linux distro
update.
88
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
When geninitrd is run, kernel-*.rpm will be searched in the osdistroupdate.dirpath to get all the rpm packages
and then those rpms will be searched for drivers. Example:
mkdef -t osdistroupdate update1 dirpath=/install/<os>/<arch>
chdef -t osimage <osimagename> osupdatename=update1
If ‘osupdatename’ is specified, the kernel shipped with the ‘osupdatename’ will be used to load the newly built initrd,
then only the drivers matching the new kernel will be kept in the newly built initrd. If trying to use the ‘osupdatename’,
the ‘allupdate’ or ‘updateonly’ should be added in the ‘netdrivers’ attribute, or all the necessary driver names for the
new kernel need to be added in the ‘netdrivers’ attribute. Otherwise the new drivers for the new kernel will be missed
in newly built initrd. ..
Inject the Drivers into the initrd
For Driver Update Disk
• If specifying the driver disk location in the osimage, there are two ways to inject drivers:
1. :: nodeset <noderange> osimage=<osimagename>
2. :: geninitrd <osimagename> nodeset <noderange> osimage=<osimagename> –noupdateinitrd
Note: ‘geninitrd’ + ‘nodeset –noupdateinitrd’ is useful when you need to run nodeset frequently for a diskful node.
‘geninitrd’ only needs be run once to rebuild the initrd and ‘nodeset –noupdateinitrd’ will not touch the initrd and
kernel in /tftpboot/xcat/osimage/<osimage name>/.
• If putting the driver disk in <installroot>/driverdisk/<os>/<arch>:
Running ‘nodeset <nodenrage>’ in anyway will load the driver disk
For Driver RPM Packages
There are two ways to inject drivers:
1. Using nodeset command only:
nodeset <noderange> osimage=<osimagename> [--ignorekernelchk]
2. Using geninitrd with nodeset command:
geninitrd <osimagename> [--ignorekernelchk]
nodeset <noderange> osimage=<osimagename> --noupdateinitrd
Note: ‘geninitrd’ + ‘nodeset –noupdateinitrd’ is useful when you need to run nodeset frequently for diskful nodes.
‘geninitrd’ only needs to be run once to rebuild the initrd and ‘nodeset –noupdateinitrd’ will not touch the initrd and
kernel in /tftpboot/xcat/osimage/<osimage name>/.
The option ‘–ignorekernelchk’ is used to skip the kernel version checking when injecting drivers from osimage.driverupdatesrc. To use this flag, you should make sure the drivers in the driver rpms are usable for the target
kernel. ..
Notes
• If the drivers from the driver disk or driver rpm are not already part of the installed or booted system, it’s
necessary to add the rpm packages for the drivers to the .pkglist or .otherpkglist of the osimage object to install
1.3. Admin Guide
89
xCAT3 Documentation, Release 2.12
them in the system.
• If a driver rpm needs to be loaded, the osimage object must be used for the ‘nodeset’ and ‘genimage’ command,
instead of the older style profile approach.
• Both a Driver disk and a Driver rpm can be loaded in one ‘nodeset’ or ‘genimage’ invocation.
Configure Disk Partition
By default, xCAT will install the operating system on the first disk and with default partitions layout in the node.
However, you may choose to customize the disk partitioning during the install process and define a specific disk
layout. You can do this in one of two ways: ‘partition definition file‘ or ‘partition definition script‘.
Notes
• ‘Partition definition file’ way can be used for RedHat, SLES and Ubuntu.
• ‘partition definition script’ way was tested only for RedHat and Ubuntu, use this feature on SLES at your own
risk.
• Because disk configuration for ubuntu is different from RedHat, there maybe some section special for ubuntu.
Partition Definition File
You could create a customized osimage partition file, say /install/custom/my-partitions, that contains the disk partitioning definition, then associate the partition file with osimage, the nodeset command will insert the contents of this
file directly into the generated autoinst configuration file that will be used by the OS installer.
Create Partition File
The partition file must follow the partitioning syntax of the installer(e.g. kickstart for RedHat, AutoYaST for SLES,
Preseed for Ubuntu). you could refer to the Kickstart documentation or Autoyast documentation or Preseed documentation write your own partitions layout. Meanwhile, RedHat and SuSE provides some tools that could help generate
kickstart/autoyast templates, in which you could refer to the partition section for the partitions layout information:
• [RedHat]
– The file /root/anaconda-ks.cfg is a sample kickstart file created by RedHat installer during the installation
process based on the options that you selected.
– system-config-kickstart is a tool with graphical interface for creating kickstart files
• [SLES]
– Use yast2 autoyast in GUI or CLI mode to customize the installation options and create autoyast file
– Use yast2 clone_system to create autoyast configuration file /root/autoinst.xml to clone an existing system
• [Ubuntu]
– For detailed information see the files partman-auto-recipe.txt and partman-auto-raid-recipe.txt included in
the debian-installer package. Both files are also available from the debian-installer source repository. Note
that the supported functionality may change between releases.
Here is partition definition file example for Ubuntu standard partition in ppc64le machines
90
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
ubuntu-boot ::
8 1 1 prep
$primary{ } $bootable{ } method{ prep }
.
500 10000 1000000000 ext4
method{ format } format{ } use_filesystem{ } filesystem{ ext4 } mountpoint{ /
˓→}
.
2048 512 300% linux-swap
method{ swap } format{ }
.
Associate Partition File with Osimage
Run the following commands to associate the partition with the osimage:
chdef -t osimage <osimagename> partitionfile=/install/custom/my-partitions
nodeset <nodename> osimage=<osimage>
• For RedHat, when nodeset runs and generates the /install/autoinst file for a node, it will replace the
#XCAT_PARTITION_START#...#XCAT_PARTITION_END# directives from your osimage template with the
contents of your custom partitionfile.
• For Ubuntu, when nodeset runs and generates the /install/autoinst file for a node, it will generate a script to write the partition configuration to /tmp/partitionfile, this script will replace the
#XCA_PARTMAN_RECIPE_SCRIPT# directive in /install/autoinst/<node>.pre.
Partitioning disk file(For Ubuntu only)
The disk file contains the name of the disks to partition in traditional, non-devfs format and delimited with space ” ”,
for example :
/dev/sda /dev/sdb
If not specified, the default value will be used.
Associate partition disk file with osimage
chdef -t osimage <osimagename> -p partitionfile='d:/install/custom/partitiondisk'
nodeset <nodename> osimage=<osimage>
• the ‘d:’ preceding the filename tells nodeset that this is a partition disk file.
• For Ubuntu, when nodeset runs and generates the /install/autoinst file for a node, it will generate a script to
write the content of the partition disk file to /tmp/install_disk, this context to run the script will replace the
#XCA_PARTMAN_DISK_SCRIPT# directive in /install/autoinst/<node>.pre.
Additional preseed configuration file(For Ubuntu only)
To support other specific partition methods such as RAID or LVM in Ubuntu, some additional preseed configuration
entries should be specified.
1.3. Admin Guide
91
xCAT3 Documentation, Release 2.12
If using file way, ‘c:<the absolute path of the additional preseed config file>’, the additional preseed config file contains
the additional preseed entries in “d-i ...” syntax. When “nodeset”, the #XCA_PARTMAN_ADDITIONAL_CFG#
directive in /install/autoinst/<node> will be replaced with content of the config file. For example:
d-i partman-auto/method string raid
d-i partman-md/confirm boolean true
If not specified, the default value will be used. ..
Partition Definition Script
Create a shell script that will be run on the node during the install process to dynamically create the disk partitioning
definition. This script will be run during the OS installer %pre script on RedHat or preseed/early_command on
Unbuntu execution and must write the correct partitioning definition into the file /tmp/partitionfile on the node
Create Partition Script
The purpose of the partition script is to create the /tmp/partionfile that will be inserted into the kickstart/autoyast/preseed template, the script could include complex logic like select which disk to install and even
configure RAID, etc
Note: the partition script feature is not thoroughly tested on SLES, there might be problems, use this feature on SLES
at your own risk.
Here is an example of the partition script on RedHat and SLES, the partitioning script is /install/custom/
my-partitions.sh:
instdisk="/dev/sda"
modprobe ext4 >& /dev/null
modprobe ext4dev >& /dev/null
if grep ext4dev /proc/filesystems > /dev/null; then
FSTYPE=ext3
elif grep ext4 /proc/filesystems > /dev/null; then
FSTYPE=ext4
else
FSTYPE=ext3
fi
BOOTFSTYPE=ext3
EFIFSTYPE=vfat
if uname -r|grep ^3.*el7 > /dev/null; then
FSTYPE=xfs
BOOTFSTYPE=xfs
EFIFSTYPE=efi
fi
if [ `uname -m` = "ppc64" ]; then
echo 'part None --fstype "PPC PReP Boot" --ondisk '$instdisk' --size 8' >> /tmp/
˓→partitionfile
fi
if [ -d /sys/firmware/efi ]; then
echo 'bootloader --driveorder='$instdisk >> /tmp/partitionfile
echo 'part /boot/efi --size 50 --ondisk '$instdisk' --fstype $EFIFSTYPE' >> /tmp/
˓→partitionfile
else
echo 'bootloader' >> /tmp/partitionfile
92
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
fi
echo "part /boot --size 512 --fstype $BOOTFSTYPE --ondisk $instdisk" >> /tmp/
˓→partitionfile
echo "part swap --recommended --ondisk $instdisk" >> /tmp/partitionfile
echo "part / --size 1 --grow --ondisk $instdisk --fstype $FSTYPE" >> /tmp/
˓→partitionfile
The following is an example of the partition script on Ubuntu, the partitioning script is /install/custom/my-partitions.sh:
if [ -d /sys/firmware/efi ]; then
echo "ubuntu-efi ::" > /tmp/partitionfile
echo "
512 512 1024 fat16" >> /tmp/partitionfile
echo '
$iflabel{ gpt } $reusemethod{ } method{ efi } format{ }' >> /tmp/
˓→partitionfile
echo "
." >> /tmp/partitionfile
else
echo "ubuntu-boot ::" > /tmp/partitionfile
echo "100 50 100 ext3" >> /tmp/partitionfile
echo '
$primary{ } $bootable{ } method{ format } format{ } use_filesystem{
˓→} filesystem{ ext3 } mountpoint{ /boot }' >> /tmp/partitionfile
echo "
." >> /tmp/partitionfile
fi
echo "500 10000 1000000000 ext3" >> /tmp/partitionfile
echo "
method{ format } format{ } use_filesystem{ } filesystem{ ext3 } mountpoint{
˓→/ }" >> /tmp/partitionfile
echo "
." >> /tmp/partitionfile
echo "2048 512 300% linux-swap" >> /tmp/partitionfile
echo "
method{ swap } format{ }" >> /tmp/partitionfile
echo "
." >> /tmp/partitionfile
Associate partition script with osimage
Run below commands to associate partition script with osimage:
chdef -t osimage <osimagename> partitionfile='s:/install/custom/my-partitions.sh'
nodeset <nodename> osimage=<osimage>
- The "s:" preceding the filename tells nodeset that this is a script.
- For RedHat, when nodeset runs and generates the /install/autoinst file for a node,
˓→it will add the execution of the contents of this script to the %pre section of
˓→that file. The nodeset command will then replace the #XCAT_PARTITION_START#...#XCAT_
˓→PARTITION_END# directives from the osimage template file with "%include /tmp/
˓→partitionfile" to dynamically include the tmp definition file your script created.
- For Ubuntu, when nodeset runs and generates the /install/autoinst file for a node,
˓→it will replace the "#XCA_PARTMAN_RECIPE_SCRIPT#" directive and add the execution
˓→of the contents of this script to the /install/autoinst/<node>.pre, the /install/
˓→autoinst/<node>.pre script will be run in the preseed/early_command.
Partitioning disk script(For Ubuntu only)
The disk script contains a script to generate a partitioning disk file named “/tmp/install_disk”. for example:
1.3. Admin Guide
93
xCAT3 Documentation, Release 2.12
rm /tmp/devs-with-boot 2>/dev/null || true;
for d in $(list-devices partition); do
mkdir -p /tmp/mymount;
rc=0;
mount $d /tmp/mymount || rc=$?;
if [[ $rc -eq 0 ]]; then
[[ -d /tmp/mymount/boot ]] && echo $d >>/tmp/devs-with-boot;
umount /tmp/mymount;
fi
done;
if [[ -e /tmp/devs-with-boot ]]; then
head -n1 /tmp/devs-with-boot | egrep -o '\S+[^0-9]' > /tmp/install_disk;
rm /tmp/devs-with-boot 2>/dev/null || true;
else
DEV=`ls /dev/disk/by-path/* -l | egrep -o '/dev.*[s|h|v]d[^0-9]$' | sort -t : -k
˓→1 -k 2 -k 3 -k 4 -k 5 -k 6 -k 7 -k 8 -g | head -n1 | egrep -o '[s|h|v]d.*$'`;
if [[ "$DEV" == "" ]]; then DEV="sda"; fi;
echo "/dev/$DEV" > /tmp/install_disk;
fi;
If not specified, the default value will be used.
Associate partition disk script with osimage
chdef -t osimage <osimagename> -p partitionfile='s:d:/install/custom/
˓→partitiondiskscript'
nodeset <nodename> osimage=<osimage>
• the ‘s:’ prefix tells nodeset that is a script, the ‘s:d:’ preceding the filename tells nodeset that this is a script to
generate the partition disk file.
• For Ubuntu, when nodeset runs and generates the /install/autoinst file for a node, this context to run the script
will replace the #XCA_PARTMAN_DISK_SCRIPT# directive in /install/autoinst/<node>.pre.
Additional preseed configuration script(For Ubuntu only)
To support other specific partition methods such as RAID or LVM in Ubuntu, some additional preseed configuration
entries should be specified.
If using script way, ‘s:c:<the absolute path of the additional preseed config script>’, the additional preseed config script is a script to set the preseed values with “debconf-set”.
When “nodeset”, the
#XCA_PARTMAN_ADDITIONAL_CONFIG_SCRIPT# directive in /install/autoinst/<node>.pre will be replaced
with the content of the script. For example:
debconf-set partman-auto/method string raid
debconf-set partman-md/confirm boolean true
If not specified, the default value will be used. ..
Prescripts and Postscripts
Using Prescript
The prescript table will allow you to run scripts before the install process. This can be helpful for performing advanced actions such as manipulating system services or configurations before beginning to install a node, or to prepare
94
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
application servers for the addition of new nodes. Check the man page for more information.
man prescripts
The scripts will be run as root on the MASTER for the node. If there is a service node for the node, then the scripts
will be run on the service node.
Identify the scripts to be run for each node by adding entries to the prescripts table:
tabedit prescripts
Or:
chdef -t node -o <noderange> prescripts-begin=<beginscripts> prescripts-end=
˓→<endscripts>
Or:
chdef -t group -o <nodegroup> prescripts-begin=<beginscripts> prescripts-end=
˓→<endscripts>
tabdump prescripts
#node,begin,end,comments,disable
begin or prescripts-begin - This attribute lists the scripts to be run at the
˓→beginning of the nodeset.
end or prescripts-end - This attribute lists the scripts to be run at the end of the
˓→nodeset.
Format for naming prescripts
The general format for the prescripts-begin or prescripts-end attribute is:
[action1:]s1,s2...[|action2:s3,s4,s5...]
where:
- action1 and action2 are the nodeset actions ( 'install', 'netboot',etc) specified
˓→in the command .
- s1 and s2 are the scripts to run for _action1_ in order.
- s3, s4, and s5 are the scripts to run for action2.
If actions are omitted, the scripts apply to all actions.
Examples:
• myscript1,myscript2 - run scripts for all supported commands
• install:myscript1,myscript2|netboot:myscript3
Run scripts 1,2 for nodeset(install), runs script3 for nodeset(netboot).
All the scripts should be copied to /install/prescripts directory and made executable for root and world readable for
mounting. If you have service nodes in your cluster with a local /install directory (i.e. /install is not mounted from the
xCAT management node to the service nodes), you will need to synchronize your /install/prescripts directory to your
service node anytime you create new scripts or make changes to existing scripts.
The following two environment variables will be passed to each script:
• NODES - a comma separated list of node names on which to run the script
• ACTION - current nodeset action.
1.3. Admin Guide
95
xCAT3 Documentation, Release 2.12
By default, the script will be invoked once for all nodes. However, if ‘#xCAT setting:MAX_INSTANCE=number’
is specified in the script, the script will be invoked for each node in parallel, but no more than number of instances
specified in number will be invoked at at a time.
Exit values for prescripts
If there is no error, a prescript should return with 0. If an error occurs, it should put the error message on the stdout
and exit with 1 or any non zero values. The command (nodeset for example) that runs prescripts can be divided into 3
sections.
1. run begin prescripts
2. run other code
3. run end prescripts
If one of the prescripts returns 1, the command will finish the rest of the prescripts in that section and then exit out
with value 1. For example, a node has three begin prescripts s1,s2 and s3, three end prescripts s4,s5,s6. If s2 returns
1, the prescript s3 will be executed, but other code and the end prescripts will not be executed by the command.
If one of the prescripts returns 2 or greater, then the command will exit out immediately. This only applies to the
scripts that do not have ‘#xCAT setting:MAX_INSTANCE=number’.
Using Postscript
xCAT automatically runs a few postscripts and postbootscripts that are delivered with xCAT to set up the nodes. You
can also add your own scripts to further customize the nodes. This explains the xCAT support to do this.
Types of scripts
There are two types of scripts in the postscripts table ( postscripts and postbootscripts). The types are based on when
in the install process they will be executed. Run the following for more information:
man postscripts
• postscripts attribute - List of scripts that should be run on this node after diskful installation or diskless boot.
– [RHEL]
Postscripts will be run before the reboot.
– [SLES]
Postscripts will be run after the reboot but before the init.d process. For Linux diskless deployment,
the postscripts will be run at the init.d time, and xCAT will automatically add the list of postscripts
from the postbootscripts attribute to run after postscripts list.
• postbootscripts attribute - list of postbootscripts that should be run on this Linux node at the init.d time after
diskful installation reboot or diskless boot
• xCAT, by default, for diskful installs only runs the postbootscripts on the install and not on reboot. In xCAT
a site table attribute runbootscripts is available to change this default behavior. If set to yes, then the postbootscripts will be run on install and on reboot.
xCAT automatically adds the postscripts from the xcatdefaults.postscripts attribute of the table to run first on
the nodes after install or diskless boot.
96
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
Adding your own postscripts
To add your own script, place it in /install/postscripts on the management node. Make sure it is executable and world
readable. Then add it to the postscripts table for the group of nodes you want it to be run on (or the “all” group if you
want it run on all nodes in the appropriate attribute, according to when you want it to run).
To check what scripts will be run on your node during installation:
lsdef node1 | grep scripts
postbootscripts=otherpkgs
postscripts=syslog,remoteshell,syncfiles
You can pass parameters to the postscripts. For example:
script1 p1 p2,script2,....
p1 p2 are the parameters to script1.
Postscripts could be placed in the subdirectories in /install/postscripts on management node, and specify “subdir/postscriptname” in the postscripts table to run the postscripts in the subdirectories. This feature could be used
to categorize the postscripts for different purposes. Here is an example:
mkdir -p /install/postscripts/subdir1
mkdir -p /install/postscripts/subdir2
cp postscript1 /install/postscripts/subdir1/
cp postscript2 /install/postscripts/subdir2/
chdef node1 -p postscripts=subdir1/postscript1,subdir2/postscript2
updatenode node1 -P
If some of your postscripts will affect the network communication between the management node and compute node,
like restarting network or configuring bond, the postscripts execution might not be able to be finished successfully
because of the network connection problems, even if we put this postscript be the last postscript in the list, xCAT still
may not be able to update the node status to be “booted”. The recommendation is to use the Linux “at” mechanism to
schedule this network-killing postscript to be run at a later time. Here is an example:
The user needs to add a postscript to customize the nics bonding setup, the nics bonding setup will break the network
between the management node and compute node, then we could use “at” to run this nic bonding postscripts after all
the postscripts processes have been finished.
We could write a script, say, /install/postscripts/nicbondscript, the nicbondscript simply calls the confignicsbond using
“at”:
[root@xcatmn ~]#cat /install/postscripts/nicbondscript
#!/bin/bash
at -f ./confignicsbond now + 1 minute
[root@xcatmn ~]#
Then
chdef <nodename> -p postbootscripts=nicbondscript
1.3. Admin Guide
97
xCAT3 Documentation, Release 2.12
Recommended Postscript design
• Postscripts that you want to run anywhere, Linux, should be written in shell. This should be available on all
OS’s. If only on the service nodes, you can use Perl .
• Postscripts should log errors using the following command local4 is the default xCAT syslog class. logger -t
xCAT -p local4.info “your info message”.
• Postscripts should have good and error exit codes (i.e 0 and 1).
• Postscripts should be well documented. At the top of the script, the first few lines should describe the function
and inputs and output. You should have comments throughout the script. This is especially important if using
regx.
PostScript/PostbootScript execution
When your script is executed on the node, all the attributes in the site table are exported as variables for your scripts
to use. You can add extra attributes for yourself. See the sample mypostscript file below.
To run the postscripts, a script is built, so the above exported variables can be input. You can usually find that script in
/xcatpost on the node and for example in the Linux case it is call mypostscript. A good way to debug problems is to
go to the node and just run mypostscript and see errors. You can also check the syslog on the Management Node for
errors.
When writing you postscripts, it is good to follow the example of the current postscripts and write errors to syslog and
in shell. See Suggestions for writing scripts.
All attributes in the site table are exported and available to the postscript/postbootscript during execution. See the
mypostscript file, which is generated and executed on the nodes to run the postscripts.
Example of mypostscript
#subroutine used to run postscripts
run_ps () {
logdir="/var/log/xcat"
mkdir -p $logdir
logfile="/var/log/xcat/xcat.log"
if [_-f_$1_]; then
echo "Running postscript: $@" | tee -a $logfile
./$@ 2>&1 | tee -a $logfile
else
echo "Postscript $1 does NOT exist." | tee -a $logfile
fi
}
# subroutine end
AUDITSKIPCMDS='tabdump,nodels'
export AUDITSKIPCMDS
TEST='test'
export TEST
NAMESERVERS='7.114.8.1'
export NAMESERVERS
NTPSERVERS='7.113.47.250'
export NTPSERVERS
INSTALLLOC='/install'
export INSTALLLOC
DEFSERIALPORT='0'
export DEFSERIALPORT
DEFSERIALSPEED='19200'
98
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
export DEFSERIALSPEED
DHCPINTERFACES="'xcat20RRmn|eth0;rra000-m|eth1'"
export DHCPINTERFACES
FORWARDERS='7.113.8.1,7.114.8.2'
export FORWARDERS
NAMESERVER='7.113.8.1,7.114.47.250'
export NAMESERVER
DB='postg'
export DB
BLADEMAXP='64'
export BLADEMAXP
FSPTIMEOUT='0'
export FSPTIMEOUT
INSTALLDIR='/install'
export INSTALLDIR
IPMIMAXP='64'
export IPMIMAXP
IPMIRETRIES='3'
export IPMIRETRIES
IPMITIMEOUT='2'
export IPMITIMEOUT
CONSOLEONDEMAND='no'
export CONSOLEONDEMAND
SITEMASTER=7.113.47.250
export SITEMASTER
MASTER=7.113.47.250
export MASTER
MAXSSH='8'
export MAXSSH
PPCMAXP='64'
export PPCMAXP
PPCRETRY='3'
export PPCRETRY
PPCTIMEOUT='0'
export PPCTIMEOUT
SHAREDTFTP='1'
export SHAREDTFTP
SNSYNCFILEDIR='/var/xcat/syncfiles'
export SNSYNCFILEDIR
TFTPDIR='/tftpboot'
export TFTPDIR
XCATDPORT='3001'
export XCATDPORT
XCATIPORT='3002'
export XCATIPORT
XCATCONFDIR='/etc/xcat'
export XCATCONFDIR
TIMEZONE='America/New_York'
export TIMEZONE
USENMAPFROMMN='no'
export USENMAPFROMMN
DOMAIN='cluster.net'
export DOMAIN
USESSHONAIX='no'
export USESSHONAIX
NODE=rra000-m
export NODE
NFSSERVER=7.113.47.250
1.3. Admin Guide
99
xCAT3 Documentation, Release 2.12
export NFSSERVER
INSTALLNIC=eth0
export INSTALLNIC
PRIMARYNIC=eth1
OSVER=fedora9
export OSVER
ARCH=x86_64
export ARCH
PROFILE=service
export PROFILE
PATH=`dirname $0`:$PATH
export PATH
NODESETSTATE='netboot'
export NODESETSTATE
UPDATENODE=1
export UPDATENODE
NTYPE=service
export NTYPE
MACADDRESS='00:14:5E:5B:51:FA'
export MACADDRESS
MONSERVER=7.113.47.250
export MONSERVER
MONMASTER=7.113.47.250
export MONMASTER
OSPKGS=bash,openssl,dhclient,kernel,openssh-server,openssh-clients,busybox-anaconda,
˓→vimminimal,rpm,bind,bind-utils,ksh,nfs-utils,dhcp,bzip2,rootfiles,vixie-cron,wget,vsftpd,
˓→ntp,rsync
OTHERPKGS1=xCATsn,xCAT-rmc,rsct/rsct.core,rsct/rsct.core.utils,rsct/src,yaboot-xcat
export OTHERPKGS1
OTHERPKGS_INDEX=1
export OTHERPKGS_INDEX
export NOSYNCFILES
# postscripts-start-here\n
run_ps ospkgs
run_ps script1 p1 p2
run_ps script2
# postscripts-end-here\n
The mypostscript file is generated according to the mypostscript.tmpl file.
Using the mypostscript template
Using the mypostscript template
xCAT provides a way for the admin to customize the information that will be provided to the postscripts/postbootscripts
when they run on the node. This is done by editing the mypostscript.tmpl file. The attributes that are provided in the
shipped mypostscript.tmpl file should not be removed. They are needed by the default xCAT postscripts.
The mypostscript.tmpl, is shipped in the /opt/xcat/share/xcat/mypostscript directory.
If the admin customizes the mypostscript.tmpl, they should copy the mypostscript.tmpl to /install/postscripts/mypostscript.tmpl, and then edit it. The mypostscript for each node will be named mypostscript.<nodename>. The generated mypostscript.<nodename>. will be put in the /tftpboot/mypostscripts
directory.
100
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
site table precreatemypostscripts attribute
If the site table precreatemypostscripts attribute is set to 1 or yes, it will instruct xCAT at nodeset and updatenode
time to query the db once for all of the nodes passed into the command and create the mypostscript file for each node
and put them in a directory in $TFTPDIR(for example /tftpboot). The created mypostscript.<nodename>. file in the
/tftpboot/mypostscripts directory will not be regenerated unless another nodeset or updatenode command is run to that
node. This should be used when the system definition has stabilized. It saves time on the updatenode or reboot by not
regenerating the mypostscript file.
If the precreatemyposcripts attribute is yes, and a database change is made or xCAT code is upgraded, then you should
run a new nodeset or updatenode to regenerate the /tftpboot/mypostscript/mypostscript.<nodename>. file to pick up
the latest database setting. The default for precreatemypostscripts is no/0.
When you run nodeset or updatenode, it will search the /install/postscripts/mypostscript.tmpl first. If the /install/postscripts/mypostscript.tmpl exists, it will use that template to generate the mypostscript for each node. Otherwise, it will use /opt/xcat/share/xcat/mypostscript/mypostscript.tmpl.
Content of the template for mypostscript
The attributes that are defined in the shipped mypostscript.tmpl file should not be removed. The xCAT default postscripts rely on that information to run successfully. The following will explain the entries in the mypostscript.tmpl file.
The SITE_TABLE_ALL_ATTRIBS_EXPORT line in the file directs the code to export all attributes defined in the site
table. Note: the attributes are not always defined exactly as in the site table to avoid conflict with other table attributes
of the same name. For example, the site table master attribute is named SITEMASTER in the generated mypostscript
file.
#SITE_TABLE_ALL_ATTRIBS_EXPORT#
The following line exports ENABLESSHBETWEENNODES by running the internal xCAT routine (enablesshbetweennodes).
ENABLESSHBETWEENNODES=#Subroutine:xCAT::Template::enablesshbetweennodes:$NODE#
export ENABLESSHBETWEENNODES
tabdump(<TABLENAME>) is used to get all the information in the <TABLENAME> table
tabdump(networks)
These line export the node name based on its definition in the database.
NODE=$NODE
export NODE
These lines get a comma separated list of the groups to which the node belongs.
GROUP=#TABLE:nodelist:$NODE:groups#
export GROUP
These lines reads the nodesres table, the given attributes (nfsserver,installnic,primarynic,xcatmaster,routenames) for
the node ($NODE), and exports it.
NFSSERVER=#TABLE:noderes:$NODE:nfsserver#
export NFSSERVER
INSTALLNIC=#TABLE:noderes:$NODE:installnic#
1.3. Admin Guide
101
xCAT3 Documentation, Release 2.12
export INSTALLNIC
PRIMARYNIC=#TABLE:noderes:$NODE:primarynic#
export PRIMARYNIC
MASTER=#TABLE:noderes:$NODE:xcatmaster#
export MASTER
NODEROUTENAMES=#TABLE:noderes:$NODE:routenames#
export NODEROUTENAMES
The following entry exports multiple variables from the routes table. Not always set.
#ROUTES_VARS_EXPORT#
The following lines export nodetype table attributes.
OSVER=#TABLE:nodetype:$NODE:os#
export OSVER
ARCH=#TABLE:nodetype:$NODE:arch#
export ARCH
PROFILE=#TABLE:nodetype:$NODE:profile#
export PROFILE
PROVMETHOD=#TABLE:nodetype:$NODE:provmethod#
export PROVMETHOD
The following adds the current directory to the path for the postscripts.
PATH=`dirname $0`:$PATH
export PATH
The following sets the NODESETSTATE by running the internal xCAT getnodesetstate script.
NODESETSTATE=#Subroutine:xCAT::Postage::getnodesetstate:$NODE#
export NODESETSTATE
The following says the postscripts are not being run as a result of updatenode.(This is changed =1, when updatenode
runs).
UPDATENODE=0
export UPDATENODE
The following sets the NTYPE to compute,service or MN.
NTYPE=$NTYPE
export NTYPE
The following sets the mac address.
MACADDRESS=#TABLE:mac:$NODE:mac#
export MACADDRESS
If vlan is setup, then the #VLAN_VARS_EXPORT# line will provide the following exports:
VMNODE='YES'
export VMNODE
VLANID=vlan1...
export VLANID
VLANHOSTNAME=..
..
#VLAN_VARS_EXPORT#
102
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
If monitoring is setup, then the #MONITORING_VARS_EXPORT# line will provide:
MONSERVER=11.10.34.108
export MONSERVER
MONMASTER=11.10.34.108
export MONMASTER
#MONITORING_VARS_EXPORT#
The OSIMAGE_VARS_EXPORT# line will provide, for example:
OSPKGDIR=/install/<os>/<arch>
export OSPKGDIR
OSPKGS='bash,nfs-utils,openssl,dhclient,kernel,openssh-server,openssh-clients,busybox,
˓→wget,rsyslog,dash,vim-minimal,ntp,rsyslog,rpm,rsync,
ppc64-utils,iputils,dracut,dracut-network,e2fsprogs,bc,lsvpd,irqbalance,procps,yum'
export OSPKGS
#OSIMAGE_VARS_EXPORT#
THE NETWORK_FOR_DISKLESS_EXPORT# line will provide diskless networks information, if defined.
NETMASK=255.255.255.0
export NETMASK
GATEWAY=8.112.34.108
export GATEWAY
..
#NETWORK_FOR_DISKLESS_EXPORT#
Note: the #INCLUDE_POSTSCRIPTS_LIST# and the #INCLUDE_POSTBOOTSCRIPTS_LIST# sections in
/tftpboot/mypostscript(mypostbootscripts) on the Management Node will contain all the postscripts and postbootscripts defined for the node. When running an updatenode command for only some of the scripts , you will
see in the /xcatpost/mypostscript file on the node, the list has been redefined during the execution of updatenode to
only run the requested scripts. For example, if you run updatenode <nodename> -P syslog.
The #INCLUDE_POSTSCRIPTS_LIST# flag provides a list of postscripts defined for this $NODE.
#INCLUDE_POSTSCRIPTS_LIST#
For example, you will see in the generated file the following stanzas:
# postscripts-start-here
# defaults-postscripts-start-here
syslog
remoteshell
# defaults-postscripts-end-here
# node-postscripts-start-here
syncfiles
# node-postscripts-end-here
The #INCLUDE_POSTBOOTSCRIPTS_LIST# provides a list of postbootscripts defined for this $NODE.
#INCLUDE_POSTBOOTSCRIPTS_LIST#
For example, you will see in the generated file the following stanzas:
# postbootscripts-start-here
# defaults-postbootscripts-start-here
otherpkgs
1.3. Admin Guide
103
xCAT3 Documentation, Release 2.12
# defaults-postbootscripts-end-here
# node-postbootscripts-end-here
# postbootscripts-end-here
Kinds of variables in the template
Type 1: For the simple variable, the syntax is as follows. The mypostscript.tmpl has several examples of this. $NODE
is filled in by the code. UPDATENODE is changed to 1, when the postscripts are run by updatenode. $NTYPE is
filled in as either compute,service or MN.
NODE=$NODE
export NODE
UPDATENODE=0
export UPDATENODE
NTYPE=$NTYPE
export NTYPE
Type 2: This is the syntax to get the value of one attribute from the <tablename> and its key is $NODE. It does not
support tables with two keys. Some of the tables with two keys are (litefile,prodkey,deps,monsetting,mpa,networks).
VARNAME=#TABLE:tablename:$NODE:attribute#
For example, to get the new updatestatus attribute from the nodelist table:
UPDATESTATUS=#TABLE:nodelist:$NODE:updatestatus#
export UPDATESTATUS
Type 3: The syntax is as follows:
VARNAME=#Subroutine:modulename::subroutinename:$NODE#
or
VARNAME=#Subroutine:modulename::subroutinename#
Examples in the mypostscript.tmpl are the following:
NODESETSTATE=#Subroutine:xCAT::Postage::getnodesetstate:$NODE#
export NODESETSTATE
ENABLESSHBETWEENNODES=#Subroutine:xCAT::Template::enablesshbetweennodes:$NODE#
export ENABLESSHBETWEENNODES
Note: Type 3 is not an open interface to add extensions to the template.
Type 4: The syntax is #FLAG#. When parsing the template, the code generates all entries defined by #FLAG#, if
they are defined in the database. For example: To export all values of all attributes from the site table. The tag is
#SITE_TABLE_ALL_ATTRIBS_EXPORT#
For the #SITE_TABLE_ALL_ATTRIBS_EXPORT# flag, the related subroutine will get the attributes’ values
and deal with the special case. such as : the site.master should be exported as “SITEMASTER”. And if the
noderes.xcatmaster exists, the noderes.xcatmaster should be exported as “MASTER”, otherwise, we also should
export site.master as the “MASTER”.
Other examples are:
#VLAN_VARS_EXPORT# - gets all vlan related items
#MONITORING_VARS_EXPORT# - gets all monitoring configuration and setup da ta
104
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
#OSIMAGE_VARS_EXPORT# - get osimage related variables, such as ospkgdir, ospkgs ...
#NETWORK_FOR_DISKLESS_EXPORT# - gets diskless network information
#INCLUDE_POSTSCRIPTS_LIST# - includes the list of all postscripts for the node
#INCLUDE_POSTBOOTSCRIPTS_LIST# - includes the list of all postbootscripts for the node
Note: Type4 is not an open interface to add extensions to the templatel.
Type 5: Get all the data from the specified table. The <TABLENAME> should not be a node table, like nodelist. This
should be handles with TYPE 2 syntax to get specific attributes for the $NODE. tabdump would result in too much
data for a nodetype table. Also the auditlog, eventlog should not be in tabdump for the same reason. site table should
not be specified, it is already provided with the #SITE_TABLE_ALL_ATTRIBS_EXPORT# flag. It can be used to
get the data from the two key tables (like switch).
The syntax is:
tabdump(<TABLENAME>)
Edit mypostscript.tmpl
Add new attributes into mypostscript.tmpl
When you add new attributes into the template, you should edit the /install/postscripts/mypostscript.tmpl which
you created by copying /opt/xcat/share/xcat/mypostscript/mypostscript.tmpl. Make all additions before the #
postscripts-start-here section. xCAT will first look in /install/postscripts/mypostscript.tmpl for a file and then
if not found will use the one in /opt/xcat/share/xcat/mypostcript/mypostscript.tmpl.
For example:
UPDATESTATUS=#TABLE:nodelist:$NODE:updatestatus#
export UPDATESTATUS
...
# postscripts-start-here
#INCLUDE_POSTSCRIPTS_LIST#
## The following flag postscripts-end-here must not be deleted.
# postscripts-end-here
Note:
If you have a hierarchical cluster, you must copy your new mypostscript.tmpl to /install/postscripts/mypostscript.tmpl on the service nodes, unless /install/postscripts directory is mounted from the
MN to the service node.
Remove attribute from mypostscript.tmpl
If you want to remove an attribute that you have added, you should remove all the related lines or comment them out
with ##. For example, comment out the added lines.
##UPDATESTATUS=#TABLE:nodelist:$NODE:updatestatus#
##export UPDATESTATUS
Test the new template
There are two quick ways to test the template.
#. If the node is up:
1.3. Admin Guide
105
xCAT3 Documentation, Release 2.12
updatenode <nodename> -P syslog
Check your generated template :
Check the generated mypostscript file on compute node /xcatpost.
#. Another way, is set the precreate option
chdef -t site -o clustersite precreatemypostscripts=1
Then run
nodeset <nodename> ....
Check your generated template
vi /tftpboot/mypostscripts/mypostscript.<nodename>
Sample /xcatpost/mypostscript
This is an example of the generated postscript for a servicenode install. It is found in /xcatpost/mypostscript on the
node.
# global value to store the running status of the postbootscripts,the value
#is non-zero if one postbootscript failed
return_value=0
# subroutine used to run postscripts
run_ps () {
local ret_local=0
logdir="/var/log/xcat"
mkdir -p $logdir
logfile="/var/log/xcat/xcat.log"
if [ -f $1 ]; then
echo "`date` Running postscript: $@" | tee -a $logfile
#./$@ 2>&1 1> /tmp/tmp4xcatlog
#cat /tmp/tmp4xcatlog | tee -a $logfile
./$@ 2>&1 | tee -a $logfile
ret_local=${PIPESTATUS[0]}
if [ "$ret_local" -ne "0" ]; then
return_value=$ret_local
fi
echo "Postscript: $@ exited with code $ret_local"
else
echo "`date` Postscript $1 does NOT exist." | tee -a $logfile
return_value=-1
fi
return 0
}
# subroutine end
SHAREDTFTP='1'
export SHAREDTFTP
TFTPDIR='/tftpboot'
export TFTPDIR
CONSOLEONDEMAND='yes'
export CONSOLEONDEMAND
PPCTIMEOUT='300'
106
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
export PPCTIMEOUT
VSFTP='y'
export VSFTP
DOMAIN='cluster.com'
export DOMAIN
XCATIPORT='3002'
export XCATIPORT
DHCPINTERFACES="'xcatmn2|eth1;service|eth1'"
export DHCPINTERFACES
MAXSSH='10'
export MAXSSH
SITEMASTER=10.2.0.100
export SITEMASTER
TIMEZONE='America/New_York'
export TIMEZONE
INSTALLDIR='/install'
export INSTALLDIR
NTPSERVERS='xcatmn2'
export NTPSERVERS
EA_PRIMARY_HMC='c76v2hmc01'
export EA_PRIMARY_HMC
NAMESERVERS='10.2.0.100'
export NAMESERVERS
SNSYNCFILEDIR='/var/xcat/syncfiles'
export SNSYNCFILEDIR
DISJOINTDHCPS='0'
export DISJOINTDHCPS
FORWARDERS='8.112.8.1,8.112.8.2'
export FORWARDERS
VLANNETS='|(\d+)|10.10.($1+0).0|'
export VLANNETS
XCATDPORT='3001'
export XCATDPORT
USENMAPFROMMN='no'
export USENMAPFROMMN
DNSHANDLER='ddns'
export DNSHANDLER
ROUTENAMES='r1,r2'
export ROUTENAMES
INSTALLLOC='/install'
export INSTALLLOC
ENABLESSHBETWEENNODES=YES
export ENABLESSHBETWEENNODES
NETWORKS_LINES=4
export NETWORKS_LINES
NETWORKS_LINE1='netname=public_net||net=8.112.154.64||mask=255.255.255.
˓→192||mgtifname=eth0||gateway=8.112.154.126||dhcpserver=||tftpserver=8.112.154.
˓→69||nameservers=8.112.8.
˓→1||ntpservers=||logservers=||dynamicrange=||staticrange=||staticrangeincrement=||nodehostname=||ddn
˓→'
export NETWORKS_LINE2
NETWORKS_LINE3='netname=sn21_net||net=10.2.1.0||mask=255.255.255.
˓→0||mgtifname=eth1||gateway=<xcatmaster>||dhcpserver=||tftpserver=||nameservers=10.2.
˓→1.100,10.2.1.
˓→101||ntpservers=||logservers=||dynamicrange=||staticrange=||staticrangeincrement=||nodehostname=||d
˓→'
export NETWORKS_LINE3
NETWORKS_LINE4='netname=sn22_net||net=10.2.2.0||mask=255.255.255.
˓→0||mgtifname=eth1||gateway=10.2.2.100||dhcpserver=10.2.2.100||tftpserver=10.2.2.
˓→100||nameservers=10.2.2.100||ntpservers=||logservers=||dynamicrange=10.2.2.120-10.2.
˓→2.
1.3. Admin Guide
107
˓→250||staticrange=||staticrangeincrement=||nodehostname=||ddnsdomain=||vlanid=||domain=||mtu=||disab
˓→'
xCAT3 Documentation, Release 2.12
export NETWORKS_LINE4
NODE=xcatsn23
export NODE
NFSSERVER=10.2.0.100
export NFSSERVER
INSTALLNIC=eth0
export INSTALLNIC
PRIMARYNIC=eth0
export PRIMARYNIC
MASTER=10.2.0.100
export MASTER
OSVER=sles11
export OSVER
ARCH=ppc64
export ARCH
PROFILE=service-xcattest
export PROFILE
PROVMETHOD=netboot
export PROVMETHOD
PATH=`dirname $0`:$PATH
export PATH
NODESETSTATE=netboot
export NODESETSTATE
UPDATENODE=1
export UPDATENODE
NTYPE=service
export NTYPE
MACADDRESS=16:3d:05:fa:4a:02
export MACADDRESS
NODEID=EA163d05fa4a02EA
export NODEID
MONSERVER=8.112.154.69
export MONSERVER
MONMASTER=10.2.0.100
export MONMASTER
MS_NODEID=0360238fe61815e6
export MS_NODEID
OSPKGS='kernel-ppc64,udev,sysconfig,aaa_base,klogd,device-mapper,bash,openssl,nfs˓→utils,ksh,syslog-ng,openssh,openssh-askpass,busybox,vim,rpm,bind,bind-utils,dhcp,
˓→dhcpcd,dhcp-server,dhcp-client,dhcp-relay,bzip2,cron,wget,vsftpd,util-linux,module˓→init-tools,mkinitrd,apache2,apache2-prefork,perl-Bootloader,psmisc,procps,dbus-1,
˓→hal,timezone,rsync,powerpc-utils,bc,iputils,uuid-runtime,unixODBC,gcc,zypper,tar'
export OSPKGS
OTHERPKGS1='xcat/xcat-core/xCAT-rmc,xcat/xcat-core/xCATsn,xcat/xcat-dep/sles11/ppc64/
˓→conserver,perl-DBD-mysql,nagios/nagios-nsca-client,nagios/nagios,nagios/nagios˓→plugins-nrpe,nagios/nagios-nrpe'
export OTHERPKGS1
OTHERPKGS_INDEX=1
export OTHERPKGS_INDEX
## get the diskless networks information. There may be no information.
NETMASK=255.255.255.0
export NETMASK
GATEWAY=10.2.0.100
export GATEWAY
# NIC related attributes for the node for confignics postscript
NICIPS=""
export NICIPS
NICHOSTNAMESUFFIXES=""
108
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
export NICHOSTNAMESUFFIXES
NICTYPES=""
export NICTYPES
NICCUSTOMSCRIPTS=""
export NICCUSTOMSCRIPTS
NICNETWORKS=""
export NICNETWORKS
NICCOMMENTS=
export NICCOMMENTS
# postscripts-start-here
# defaults-postscripts-start-here
run_ps test1
run_ps syslog
run_ps remoteshell
run_ps syncfiles
run_ps confNagios
run_ps configrmcnode
# defaults-postscripts-end-here
# node-postscripts-start-here
run_ps servicenode
run_ps configeth_new
# node-postscripts-end-here
run_ps setbootfromnet
# postscripts-end-here
# postbootscripts-start-here
# defaults-postbootscripts-start-here
run_ps otherpkgs
# defaults-postbootscripts-end-here
# node-postbootscripts-start-here
run_ps test
# The following line node-postbootscripts-end-here must not be deleted.
# node-postbootscripts-end-here
# postbootscripts-end-here
exit $return_value
Suggestions
For writing scripts
• Some compute node profiles exclude perl to keep the image as small as possible. If this is your case, your
postscripts should obviously be written in another shell language, e.g. bash,ksh.
• If a postscript is specific for an os, name your postscript mypostscript.osname.
• Add logger statements to send errors back to the Management Node. By default, xCAT configures the syslog
service on compute nodes to forward all syslog messages to the Management Node. This will help debug.
Using Hierarchical Clusters
If you are running a hierarchical cluster, one with Service Nodes. If your /install/postscripts directory is not mounted
on the Service Node. You are going to need to sync or copy the postscripts that you added or changed in the /install/postscripts on the MN to the SN, before running them on the compute nodes. To do this easily, use the xdcp
command and just copy the entire /install/postscripts directory to the servicenodes ( usually in /xcatpost ).
1.3. Admin Guide
109
xCAT3 Documentation, Release 2.12
xdcp service -R /install/postscripts/* /xcatpost
or
prsync /install/postscripts service:/xcatpost
If your /install/postscripts is not mounted on the Service Node, you should also:
xdcp service -R /install/postscripts/* /install
or
prsync /install/postscripts service:/install
Synchronizing Files
Overview
Synchronizing (sync) files to the nodes is a feature of xCAT used to distribute specific files from the management node
to the new-deploying or deployed nodes.
This function is supported for diskful or RAMdisk-based diskless nodes. Generally, the specific files are usually the
system configuration files for the nodes in the /etc/directory, like /etc/hosts, /etc/resolve.conf; it also could be the
application programs configuration files for the nodes. The advantages of this function are: it can parallel sync files
to the nodes or nodegroup for the installed nodes; it can automatically sync files to the newly-installing node after the
installation. Additionally, this feature also supports the flexible format to define the synced files in a configuration file,
called ‘synclist’.
The synclist file can be a common one for a group of nodes using the same profile or osimage, or can be the special one
for a particular node. Since the location of the synclist file will be used to find the synclist file, the common synclist
should be put in a given location for Linux nodes or specified by the osimage.
xdcp command supplies the basic Syncing File function. If the ‘-F synclist’ option is specified in the xdcp command,
it syncs files configured in the synclist to the nodes. If the ‘-i PATH’ option is specified with ‘-F synclist’, it syncs files
to the root image located in the PATH directory. (Note: the ‘-i PATH’ option is only supported for Linux nodes)
xdcp supports hierarchy where service nodes are used. If a node is serviced by a service node, xdcp will sync
the files to the service node first, then sync the files from service node to the compute node. The files are place in
an intermediate directory on the service node defined by the SNsyncfiledir attribute in the site table. The default is
/var/xcat/syncfiles.
Since updatenode -F calls the xdcp to handle the Syncing File function, the updatenode -F also supports
the hierarchy.
For a new-installing nodes, the Syncing File action will be triggered when performing the postscripts for the nodes. A
special postscript named ‘syncfiles’ is used to initiate the Syncing File process.
The postscript ‘syncfiles’ is located in the /install/postscripts/. When running, it sends a message to the xcatd on the
management node or service node, then the xcatd figures out the corresponding synclist file for the node and calls the
xdcp command to sync files in the synclist to the node.
If installing nodes in a hierarchical configuration, you must sync the Service Nodes first to make sure they
are updated. The compute nodes will be sync’d from their service nodes. You can use the updatenode
<computenodes> -f command to sync all the service nodes for range of compute nodes provided.
For an installed nodes, the Syncing File action happens when performing the updatenode -F or xdcp -F
synclist command to update a nodes. If performing the updatenode -F, it figures out the location of the
synclist files for all the nodes and classify the nodes which using same synclist file and then calls the xdcp -F
synclist to sync files to the nodes.
110
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
The synclist file
The Format of synclist file
The synclist file contains the configuration entries that specify where the files should be synced to. In the synclist file,
each line is an entry which describes the location of the source files and the destination location of files on the target
node.
The basic entry format looks like following:
path_of_src_file1 -> path_of_dst_file1
path_of_src_file1 -> path_of_dst_directory
path_of_src_file1 path_of_src_file2 ... -> path_of_dst_directory
The path_of_src_file* should be the full path of the source file on the Management Node.
The path_of_dst_file* should be the full path of the destination file on target node.
The path_of_dst_directory should be the full path of the destination directory.
Since the synclist file is for common purpose, the target node need not be configured in it.
Example: the following synclist formats are supported:
sync file /etc/file2 to the file /etc/file2 on the node (with same file name)
/etc/file2 -> /etc/file2
sync file /etc/file2 to the file /etc/file3 on the node (with different file name)
/etc/file2 -> /etc/file3
sync file /etc/file4 to the file /etc/tmp/file5 on the node( different file name and directory). The directory will be
automatically created for you.
/etc/file4 -> /etc/tmp/file5
sync the multiple files /etc/file1, /etc/file2, /etc/file3, ... to the directory /tmp/etc (/tmp/etc must be a directory when
multiple files are synced at one time). If the directory does not exist,**xdcp** will create it.
/etc/file1 /etc/file2 /etc/file3 -> /tmp/etc
sync file /etc/file2 to the file /etc/file2 on the node
/etc/file2 -> /etc/
sync all files in /home/mikev to directory /home/mikev on the node
/home/mikev/* -> /home/mikev/
Note: Don’t try to sync files to the read only directory on the target node.
An example of synclist file
Assume a user wants to sync files to a node as following, the corresponding entries should be added in a synclist file.
Sync the file /etc/common_hosts to the two places on the target node: put one to the /etc/hosts, the other to the
/tmp/etc/hosts. Following configuration entries should be added
1.3. Admin Guide
111
xCAT3 Documentation, Release 2.12
/etc/common_hosts -> /etc/hosts
/etc/common_hosts -> /tmp/etc/hosts
Sync files in the directory /tmp/prog1 to the directory /prog1 on the target node, and the postfix ‘.tmpl’ needs to
be removed on the target node. (directory /tmp/prog1/ contains two files: conf1.tmpl and conf2.tmpl) Following
configuration entries should be added
/tmp/prog1/conf1.tmpl -> /prog1/conf1
/tmp/prog1/conf2.tmpl -> /prog1/conf2
Sync the files in the directory /tmp/prog2 to the directory /prog2 with same name on the target node. (directory
/tmp/prog2 contains two files: conf1 and conf2) Following configuration entries should be added:
/tmp/prog2/conf1 /tmp/prog2/conf2 -> /prog2
Sample synclist file
/etc/common_hosts -> /etc/hosts
/etc/common_hosts -> /tmp/etc/hosts
/tmp/prog1/conf1.tmpl -> /prog1/conf1
/tmp/prog1/conf2.tmpl -> /prog1/conf2
/tmp/prog2/conf1 /tmp/prog2/conf2 -> /prog2
/tmp/* -> /tmp/
/etc/testfile -> /etc/
If the above syncfile is performed by the updatenode/xdcp commands, or performed in a node installation process,
the following files will exist on the target node with the following contents.
/etc/hosts(It has the same content with /etc/common_hosts on the MN)
/tmp/etc/hosts(It has the same content with /etc/common_hosts on the MN)
/prog1/conf1(It has the same content with /tmp/prog1/conf1.tmpl on the MN)
/prog1/conf2(It has the same content with /tmp/prog1/conf2.tmpl on the MN)
/prog2/conf1(It has the same content with /tmp/prog2/conf1 on the MN)
/prog2/conf2(It has the same content with /tmp/prog2/conf2 on the MN)
Support nodes in synclist file
Note: From xCAT 2.9.2 on AIX and from xCAT 2.12 on Linux, xCAT support a new format for syncfile. The new
format is
file -> (noderange for permitted nodes) file
The noderange would have several format. Following examples show that /etc/hosts file is synced to the nodes which
is specified before the file name
/etc/hosts -> (node1,node2) /etc/hosts
˓→node1 and node2
/etc/hosts -> (node1-node4) /etc/hosts
˓→node1,node2,node3 and node4
/etc/hosts -> (node[1-4]) /etc/hosts
˓→node1, node2, node3 and node4
/etc/hosts -> (node1,node[2-3],node4) /etc/hosts
˓→node1, node2, node3 and node4
/etc/hosts -> (group1) /etc/hosts
˓→nodes in group1
/etc/hosts -> (group1,group2) /etc/hosts
˓→nodes in group1 and group2
112
# The /etc/hosts file is synced to
# The /etc/hosts file is synced to
# The /etc/hosts file is synced to
# The /etc/hosts file is synced to
# The /etc/hosts file is synced to
#
The /etc/hosts file is synced to
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
postscript support
Putting the filename.post in the rsyncfile to rsync to the node is required for hierarchical clusters. It is optional for
non-hierarchical cluster.
Advanced synclist file features
After you define the files to rsync in the syncfile, you can add an EXECUTEALWAYS clause in the syncfile. The
EXECUTEALWAYS clause will list all the postscripts you would always like to run after the files are sync’d, whether
or not any file is actually updated. The files in this list must be added to the list of files to rsync, if hierarchical.
For example, your rsyncfile may look like this. Note: the path to the file to EXECUTE, is the location of the *.post
file on the MN.
/tmp/share/file2 -> /tmp/file2
/tmp/share/file2.post -> /tmp/file2.post (required for hierarchical clusters)
/tmp/share/file3 -> /tmp/file3
/tmp/share/file3.post -> /tmp/file3.post (required for hierarchical clusters)
/tmp/myscript1 -> /tmp/myscript1
/tmp/myscript2 -> /tmp/myscript2
# the below are postscripts
EXECUTE:
/tmp/share/file2.post
/tmp/share/file3.post
EXECUTEALWAYS:
/tmp/myscript1
/tmp/myscript2
If /tmp/file2 is updated on the node in /tmp/file2, then /tmp/file2.post is automatically run on that node. If /tmp/file3
is updated on the node in /tmp/filex, then /tmp/file3.post is automatically run on that node.
You can add an APPEND clause to your syncfile.
The APPEND clause is used to append the contents of the input file to an existing file on the node. The file to be
appended must already exist on the node and not be part of the synclist that contains the APPEND clause.
For example, your synclist file may look like this:
/tmp/share/file2 -> /tmp/file2
/tmp/share/file2.post -> /tmp/file2.post
/tmp/share/file3 -> /tmp/filex
/tmp/share/file3.post -> /tmp/file3.post
/tmp/myscript -> /tmp/myscript
# the below are postscripts
EXECUTE:
/tmp/share/file2.post
/tmp/share/file3.post
EXECUTEALWAYS:
/tmp/myscript
APPEND:
/etc/myappenddir/appendfile -> /etc/mysetup/setup
/etc/myappenddir/appendfile2 -> /etc/mysetup/setup2
1.3. Admin Guide
113
xCAT3 Documentation, Release 2.12
When you use the APPEND clause, the file (left) of the arrow is appended to the file right of the arrow. In this example,
/etc/myappenddir/appendfile is appended to /etc/mysetup/setup file, which must already exist on the node. The
/opt/xcat/share/xcat/scripts/xdcpappend.sh is used to accomplish this.
The script creates a backup of the original file on the node in the directory defined by the site table nodesyncfiledir
attribute, which is /var/xcat/node/syncfiles by default. To update the original file when using the function, you need
to rsync a new original file to the node, removed the old original from the /var/xcat/node/syncfiles/org directory. If
you want to cleanup all the files for the append function on the node, you can use the xdsh -c flag. See man page
for xdsh.
Note:no order of execution may be assumed by the order that the EXECUTE,EXECUTEALWAYS and APPEND
clause fall in the synclist file.
You can add an MERGE clause to your syncfile. This is only supported on Linux.
The MERGE clause is used to append the contents of the input file to either the /etc/passwd, /etc/shadow or
/etc/group files. They are the only supported files. You must not put the /etc/passwd, /etc/shadow, /etc/group
files in an APPEND clause if using a MERGE clause. For these three file you should use a MERGE clause. The
APPEND will add the information to the end of the file. The MERGE will add or replace the information and insure
that there are no duplicate entries in these files.
For example, your synclist file may look like this
/tmp/share/file2 -> /tmp/file2
/tmp/share/file2.post -> /tmp/file2.post
/tmp/share/file3 -> /tmp/filex
/tmp/share/file3.post -> /tmp/file3.post
/tmp/myscript -> /tmp/myscript
# the below are postscripts
EXECUTE:
/tmp/share/file2.post
/tmp/share/file3.post
EXECUTEALWAYS:
/tmp/myscript
MERGE:
/etc/mydir/mergepasswd -> /etc/passwd
/etc/mydir/mergeshadow -> /etc/shadow
/etc/mydir/mergegroup -> /etc/group
When you use the MERGE clause, the file (left) of the arrow is merged into the file right of the arrow. It will replace
any common userid’s found in those files and add new userids. The /opt/xcat/share/xcat/scripts/xdcpmerge.sh is used
to accomplish this.
Note: no order of execution may be assumed by the order that the EXECUTE,EXECUTEALWAYS,APPEND and
MERGE clause fall in the synclist file.
The location of synclist file for updatenode and install process
In the installation process or updatenode process, xCAT needs to figure out the location of the synclist file automatically, so the synclist should be put into the specified place with the proper name.
If the provisioning method for the node is an osimage name, then the path to the synclist will be read from the osimage
definition synclists attribute. You can display this information by running the following command, supplying your
osimage name.
lsdef -t osimage -l <os>-<arch>-netboot-compute
Object name: <os>-<arch>-netboot-compute
exlist=/opt/xcat/share/xcat/netboot/<os>/compute.exlist
114
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
imagetype=linux
osarch=<arch>
osname=Linux
osvers=<os>
otherpkgdir=/install/post/otherpkgs/<os>/<arch>
pkgdir=/install/<os>/<arch>
pkglist=/opt/xcat/share/xcat/netboot/<os>/compute.pkglist
profile=compute
provmethod=netboot
rootimgdir=/install/netboot/<os>/<arch>/compute
**synclists=/install/custom/netboot/compute.synclist**
You can set the synclist path using the following command
chdef -t osimage -o
˓→compute.synclist
<os>-<arch>-netboot-compute synclists="/install/custom/netboot/
If the provisioning method for the node is install,or netboot then the path to the synclist should be of the following
format
/install/custom/<inst_type>/<distro>/<profile>.<os>.<arch>.synclist
<inst_type>: "install", "netboot"
<distro>:
"rh", "centos", "fedora", "sles"
<profile>,<os>and <arch> are what you set for the node
For example: The location of synclist file for the diskful installation of <os> with ‘compute’ as the profile
/install/custom/<inst_type>/<distro>/<profile>.<os>.synclist
The location of synclist file for the diskless netboot of <os> with ‘<profile>’ as the profile
/install/custom/<inst_type>/<distro>/<profile>.<os>.synclist
Run xdcp command to perform Syncing File action
xdcp command supplies three options ‘-F’ , -s, and ‘-i’ to support the Syncing File function.
• -F|–File rsync input file
Specifies the full path to the synclist file that will be used to build the rsync command
• -s
Specifies to rsync to the service nodes only for the input compute noderange.
• -i|–rootimg install image for Linux
Specifies the full path to the install image on the local node. By default, if the -F option is specified, the ‘rsync’
command is used to perform the syncing file function. For the rsync in xdcp, only the *ssh remote shell is supported
for rsync. xdcp uses the ‘-Lpotz’ as the default flags to call the rsync command. More flags for rsync command
can be specified by adding ‘-o’ flag to the call to xdcp.
For example:
1.3. Admin Guide
115
xCAT3 Documentation, Release 2.12
Using xdcp '-F' option to sync files which are listed in the /install/custom/
˓→commonsyncfiles/<profile>.synclist directory to the node group named 'compute'. If
˓→the node group compute is serviced by servicenodes, then the files will be
˓→automatically staged to the correct service nodes, and then synced to the compute
˓→nodes from those service nodes. The files will be stored in /var/xcat/syncfiles
˓→directory on the service nodes by default, or in the directory indicated in the
˓→site.SNsyncfiledir attribute. See -s option below.
xdcp compute -F /install/custom/commonsynfiles/<profile>.synclist
For Linux nodes, using xdcp ‘-i’ option with ‘-F’ to sync files created in the
stall/custom/<inst_type>/<os><profile>.synclist
to
the
osimage
in
the
directory
stall/<inst_type>/<os>/<arch>/<profile>/rootimg:
/in/in-
xdcp -i /install/<inst_type>/<os>/<arch>/<profile>/rootimg -F /install/custom/<inst_
˓→type>/<os>/<profile>.synclist
Using the xdcp ‘-s’ option to sync the files only to the service nodes for the node group named ‘compute’. The files will
be placed in the default /var/xcat/syncfiles directory or in the directory as indicated in the site.SNsyncfiledir attribute.
If you want the files synched to the same directory on the service node that they come from on the Management Node,
set site.SNsyncfiledir=/. This can be setup before a node install, to have the files available to be synced during the
install:
xdcp compute -s -F /install/custom/<inst_type>/<os>/<profile>.synclist
Synchronizing Files during the installation process
The policy table must have the entry to allow syncfiles postscript to access the Management Node. Make sure this
entry is in your table:
tabdump policy
#priority,name,host,commands,noderange,parameters,time,rule,comments,disable
.
.
"4.6",,,"syncfiles",,,,"allow",,
.
.
Hierarchy and Service Nodes
If using Service nodes to manage you nodes, you should make sure that the service nodes have been synchronized with
the latest files from the Management Node before installing. If you have a group of compute nodes (compute) that are
going to be installed that are serviced by SN1, then run the following before the install to sync the current files to SN1.
Note: the noderange is the compute node names, updatenode will figure out which service nodes need updating.
updatenode compute -f
Diskful installation
The ‘syncfiles’ postscript is in the defaults section of the postscripts table. To enable the syn files postscript to sync
files to the nodes during install the user need to do the following:
116
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
• Create the synclist file with the entries indicating which files should be synced. (refer to The Format of synclist
file )
• Put the synclist into the proper location for the node type (refer to The location of synclist file for updatenode
and install process)
Make sure your postscripts table has the syncfiles postscript listed
tabdump postscripts
#node,postscripts,postbootscripts,comments,disable
"xcatdefaults","syslog,remoteshell,syncfiles","otherpkgs",,
Diskless Installation
The diskless boot is similar with the diskful installation for the synchronizing files operation, except that the packimage
commands will sync files to the root directories of image during the creating image process.
Creating the synclist file as the steps in Diskful installation section, then the synced files will be synced to the os image
during the packimage and mkdsklsnode commands running.
Also the files will always be re-synced during the booting up of the diskless node.
Run the Sync’ing File action in the creating diskless image process
Different approaches are used to create the diskless image. The Sync’ing File action is also different.
The packimage command is used to prepare the root image files and package the root image. The Syncing File
action is performed here.
Steps to make the Sync’ing File working in the packimage command:
1. Prepare the synclist file and put it into the appropriate location as describe above in (refer The location of synclist
file for updatenode and install process)
2. Run packimage as is normally done.
Run the Syncing File action in the updatenode process
If run updatenode command with -F option, it syncs files which configured in the synclist to the nodes.
updatenode does not sync images, use xdcp -i -F option to sync images.
updatenode can be used to sync files to to diskful or diskless nodes. updatenode cannot be used to sync files to
statelite nodes.
Steps to make the Syncing File working in the updatenode -F command:
1. Create the synclist file with the entries indicating which files should be synced. (refer to The Format of synclist
file)
2. Put the synclist into the proper location (refer to The location of synclist file for updatenode and install process).
3. Run the updatenode node -F command to initiate the Syncing File action.
Note: Since Syncing File action can be initiated by the updatenode -F flag, the updatenode -P does NOT support to re-run the ‘syncfiles’ postscript, even if you specify the ‘syncfiles’ postscript in the updatenode command
line or set the ‘syncfiles’ in the postscripts.postscripts attribute.
1.3. Admin Guide
117
xCAT3 Documentation, Release 2.12
Run the Syncing File action periodically
If the admins want to run the Syncing File action automatically or periodically, the xdcp -F, xdcp -i -F and
updatenode -F commands can be used in the script, crontab or FAM directly.
For example:
Use the cron daemon to sync files in the /install/custom/<inst_type>/<distro>/<profile>.<os>.synclist to the nodegroup ‘compute’ every 10 minutes by the xdcp command by adding this to crontab. :
*/10 * * * * root /opt/xcat/bin/xdcp compute -F /install/custom/<inst_type>/<distro>/
˓→<profile>.<distro>.synclist
Use the cron daemon to sync files for the nodegroup ‘compute’ every 10 minutes by updatenode command.
*/10 * * * * root /opt/xcat/bin/updatenode compute -F
** Related To do** Add reference
Add Additional Software Packages
Overview
The name of the packages that will be installed on the node are stored in the packages list files. There are two kinds
of package list files:
• The package list file contains the names of the packages that comes from the os distro. They are stored in .pkglist
file.
• The other package list file contains the names of the packages that do NOT come from the os distro. They are
stored in .otherpkgs.pkglist file.
The path to the package lists will be read from the osimage definition. Which osimage a node is using is specified by
the provmethod attribute. To display this value for a node:
lsdef node1 -i provmethod
Object name: node
provmethod=<osimagename>
You can display this details of this osimage by running the following command, supplying your osimage name:
lsdef -t osimage <osimagename>
Object name: <osimagename>
exlist=/opt/xcat/share/xcat/<inst_type>/<os>/<profile>.exlist
imagetype=linux
osarch=<arch>
osname=Linux
osvers=<os>
otherpkgdir=/install/post/otherpkgs/<os>/<arch>
otherpkglist=/install/custom/<inst_type>/<distro>/<profile>.otherpkgs.pkglist
pkgdir=/install/<os>/<arch>
pkglist=/opt/xcat/share/xcat/<inst_type>/<os>/<profile>.pkglist
postinstall=/opt/xcat/share/xcat/<inst_type>/<distro>/<profile>.<os>.<arch>.
˓→postinstall
profile=<profile>
provmethod=<profile>
118
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
rootimgdir=/install/<inst_type>/<os>/<arch>/<profile>
synclists=/install/custom/<inst_type>/<profile>.synclist
You can set the pkglist and otherpkglist using the following command:
chdef -t osimage <osimagename> pkglist=/opt/xcat/share/xcat/<inst_type>/<distro>/
˓→<profile>.pkglist\
otherpkglist=/install/custom/<inst_type>/
˓→<distro>/my.otherpkgs.pkglist
Install Additional OS Packages for RHEL and SLES
Install Additional Packages using OS Packages steps
For rpms from the OS distro, add the new rpm names (without the version number) in the .pkglist file. For example,
file /install/custom/<inst_type>/<os>/<profile>.pkglist will look like this after adding perl-DBI:
bash
nfs-utils
openssl
dhcpcd
kernel-smp
openssh
procps
psmisc
resmgr
wget
rsync
timezone
perl-DBI
For the format of the .pkglist file, see File Format for .ospkgs.pkglist File
If you have newer updates to some of your operating system packages that you would like to apply to your OS image,
you can place them in another directory, and add that directory to your osimage pkgdir attribute. For example, with
the osimage defined above, if you have a new openssl package that you need to update for security fixes, you could
place it in a directory, create repository data, and add that directory to your pkgdir:
mkdir -p /install/osupdates/<os>/<arch>
cd /install/osupdates/<os>/<arch>
cp <your new openssl rpm> .
createrepo .
chdef -t osimage <os>-<arch>-<inst_type>-<profile> pkgdir=/install/<os>/<arch>,/
˓→install/osupdates/<os>/<arch>
Note:If the objective node is not installed by xCAT, make sure the correct osimage pkgdir attribute so that you could
get the correct repository data.
File Format for .ospkgs.pkglist File
The .pklist file is used to specify the rpm and the group/pattern names from os distro that will be installed on the nodes.
It can contain the following types of entries:
1.3. Admin Guide
119
xCAT3 Documentation, Release 2.12
* rpm name without version numbers
* group/pattern name marked with a '@' (for full install only)
* rpms to removed after the installation marked with a "-" (for full install only)
These are described in more details in the following sections.
RPM Names
A simple .pkglist file just contains the name of the rpm file without the version numbers.
For example
openssl
xntp
rsync
glibc-devel.i686
Include pkglist Files
The #INCLUDE statement is supported in the pkglist file.
You can group some rpms in a file and include that file in the pkglist file using #INCLUDE:<file># format.
openssl
xntp
rsync
glibc-devel.1686
#INCLUDE:/install/post/custom/<distro>/myotherlist#
where /install/post/custom/<distro>/myotherlist is another package list file that follows the same format.
Note: the trailing “#” character at the end of the line. It is important to specify this character for correct pkglist parsing.
Group/Pattern Names
It is only supported for stateful deployment.
In Linux, a groups of rpms can be packaged together into one package. It is called a group on RedHat, CentOS, Fedora
and Scientific Linux. To get the a list of available groups, run
• [RHEL]
yum grouplist
• [SLES]
zypper se -t pattern
You can specify in this file the group/pattern names by adding a ‘@’ and a space before the group/pattern names. For
example:
@ base
120
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
Remove RPMs After Installing
It is only supported for stateful deployment.
You can specify in this file that certain rpms to be removed after installing the new software. This is done by adding
‘-‘ before the rpm names you want to remove. For example:
-ntp
Install Additional Other Packages for RHEL and SLES
Install Additional Other Packages Steps
If you have additional rpms (rpms not in the distro) that you also want installed, make a directory to hold them, create
a list of the rpms you want installed, and add that information to the osimage definition:
• Create a directory to hold the additional rpms:
mkdir -p /install/post/otherpkgs/<distro>/<arch>
cd /install/post/otherpkgs/<distro>/<arch>
cp /myrpms/* .
createrepo .
• Create a file that lists the additional rpms that should be installed.
stall/custom/<inst_type>/<distro>/<profile>.otherpkgs.pkglist put:
For example, in /in-
myrpm1
myrpm2
myrpm3
• Add both the directory and the file to the osimage definition:
chdef -t osimage mycomputeimage otherpkgdir=/install/post/otherpkgs/<os>/<arch>
˓→otherpkglist=/install/custom/<inst_type>/<os>/<profile>.otherpkgs.pkglist
If you add more rpms at a later time, you must run createrepo again. The createrepo command is in the
createrepo rpm, which for RHEL is in the 1st DVD, but for SLES is in the SDK DVD.
If you have multiple sets of rpms that you want to keep separate to keep them organized, you can put them in
separate sub-directories in the otherpkgdir. If you do this, you need to do the following extra things, in addition to the
steps above:
• Run createrepo in each sub-directory
• In your otherpkgs.pkglist, list at least 1 file from each sub-directory. (During installation, xCAT will define a
yum or zypper repository for each directory you reference in your otherpkgs.pkglist.) For example:
xcat/xcat-core/xCATsn
xcat/xcat-dep/<os>/<arch>/conserver-xcat
There are some examples of otherpkgs.pkglist in /opt/xcat/share/xcat/<inst_type>/<distro>/
<profile>.*.otherpkgs.pkglist that show the format.
Note: the otherpkgs postbootscript should by default be associated with every node. Use lsdef to check:
lsdef node1 -i postbootscripts
1.3. Admin Guide
121
xCAT3 Documentation, Release 2.12
If it is not, you need to add it. For example, add it for all of the nodes in the “compute” group:
chdef -p -t group compute postbootscripts=otherpkgs
For the format of the .Otherpkgs file,see File Format for .otherpkgs.pkglist File
File Format for .otherpkgs.pkglist File
The otherpkgs.pklist file can contain the following types of entries:
rpm name without version numbers
otherpkgs subdirectory plus rpm name
blank lines
comment lines starting with #
#INCLUDE: <full file path># to include other pkglist files
#NEW_INSTALL_LIST# to signify that the following rpms will be installed with a new
˓→rpm install command (zypper, yum, or rpm as determined by the function using this
˓→file)
* #ENV:<variable list># to specify environment variable(s) for a sparate rpm install
˓→command
* rpms to remove before installing marked with a "-"
* rpms to remove after installing marked with a "--"
*
*
*
*
*
*
These are described in more details in the following sections.
RPM Names
A simple otherpkgs.pkglist file just contains the name of the rpm file without the version numbers.
For example, if you put the following three rpms under /install/post/otherpkgs/<os>/<arch>/ directory,
rsct.core-2.5.3.1-09120.ppc.rpm
rsct.core.utils-2.5.3.1-09118.ppc.rpm
src-1.3.0.4-09118.ppc.rpm
The otherpkgs.pkglist file will be like this:
src
rsct.core
rsct.core.utils
RPM Names with otherpkgs Subdirectories
If you create a subdirectory under /install/post/otherpkgs/<os>/<arch>/, say rsct, the otherpkgs.pkglist file will be
like this:
rsct/src
rsct/rsct.core
rsct/rsct.core.utils
122
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
Include Other pkglist Files
You can group some rpms in a file and include that file in the otherpkgs.pkglist file using #INCLUDE:<file># format.
rsct/src
rsct/rsct.core
rsct/rsct.core.utils
#INCLUDE:/install/post/otherpkgs/myotherlist#
where /install/post/otherpkgs/myotherlist is another package list file that follows the same format.
Note the trailing “#” character at the end of the line. It is important to specify this character for correct pkglist parsing.
Multiple Install Lists
You can specify that separate calls should be made to the rpm install program (zypper, yum, rpm) for groups of rpms
by specifying the entry #NEW_INSTALL_LIST# on a line by itself as a separator in your pkglist file. All rpms listed
up to this separator will be installed together. You can have as many separators as you wish in your pkglist file, and
each sublist will be installed separately in the order they appear in the file.
For example:
compilers/vacpp.rte
compilers/vac.lib
compilers/vacpp.lib
compilers/vacpp.rte.lnk
#NEW_INSTALL_LIST#
pe/IBM_pe_license
Environment Variable List
You can specify environment variable(s) for each rpm install call by entry “#ENV:<variable list>#”. The environment
variables also apply to rpm(s) remove call if there is rpm(s) needed to be removed in the sublist.
For example:
#ENV:INUCLIENTS=1 INUBOSTYPE=1#
rsct/rsct.core
rsct/rsct.core.utils
rsct/src
Be same as,
#ENV:INUCLIENTS=1#
#ENV:INUBOSTYPE=1#
rsct/rsct.core
rsct/rsct.core.utils
rsct/src
Remove RPMs Before Installing
You can also specify in this file that certain rpms to be removed before installing the new software. This is done by
adding ‘-‘ before the rpm names you want to remove. For example:
1.3. Admin Guide
123
xCAT3 Documentation, Release 2.12
rsct/src
rsct/rsct.core
rsct/rsct.core.utils
#INCLUDE:/install/post/otherpkgs/myotherlist#
-perl-doc
If you have #NEW_INSTALL_LIST# separators in your pkglist file, the rpms will be removed before the install of
the sublist that the "-<rpmname>" appears in.
Remove RPMs After Installing
You can also specify in this file that certain rpms to be removed after installing the new software. This is done by
adding -- before the rpm names you want to remove. For example:
pe/IBM_pe_license
--ibm-java2-ppc64-jre
If you have #NEW_INSTALL_LIST# separators in your pkglist file, the rpms will be removed after the install of the
sublist that the "--<rpmname>" appears in.
Install Additional Other Packages with Ubuntu official mirror
The Ubuntu ISO used to install the compute nodes only include packages to run a minimal base operating system,
it is likely that users will want to install additional Ubuntu packages from the internet Ubuntu repositories or local
repositories, this section describes how to install additional Ubuntu packages.
Compute nodes can access the internet
1. Specify the repository
Define the otherpkgdir attribute in osimage to use the internet repository directly.:
chdef -t osimage <osimage name> otherpkgdir="http://us.archive.ubuntu.com/ubuntu/
˓→\
$(lsb_release -sc) main,http://us.archive.ubuntu.com/ubuntu/ $(lsb_release -sc)˓→update main"
2. Define the otherpkglist file
create an otherpkglist file,**/install/custom/install/ubuntu/compute.otherpkgs.pkglist**. Add the packages’
name into thist file. And modify the otherpkglist attribute for osimage object.
chdef -t osimage <osimage name> otherpkglist=/install/custom/install/ubuntu/
˓→compute.otherpkgs.pkglist
3. Run updatenode <noderange> -S or updatenode <noderange> -P otherpkgs
Run updatenode -S to install/update the packages on the compute nodes
updatenode <noderange> -S
Run updatenode -P otherpkgs to install/update the packages on the compute nodes
124
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
updatenode <noderange> -P otherpkgs
Compute nodes can not access the internet
If compute nodes cannot access the internet, there are two ways to install additional packages
• use apt proxy
• use local mirror
Customize network adapter
Configure Additional Network Interfaces - confignics
The nics table and the confignics postscript can be used to automatically configure additional network interfaces
(multiple ethernets adapters, InfiniBand, etc) on the nodes as they are being deployed.
The way the confignics postscript decides what IP address to give the secondary adapter is by checking the nics table,
in which the nic configuration information is stored.
To use the nics table and confignics postscript to define a secondary adapter on one or more nodes, follow these steps:
Define configuration information for the Secondary Adapters in the nics table
There are 3 ways to complete this operation.
1. Using the mkdef and chdef commands
# mkdef cn1 groups=all nicips.eth1="11.1.89.7|12.1.89.7" nicnetworks.eth1=
˓→"net11|net12" nictypes.eth1="Ethernet"
1 object definitions have been created or modified.
# chdef cn1 nicips.eth2="13.1.89.7|14.1.89.7" nicnetworks.eth2="net13|net14"
˓→nictypes.eth2="Ethernet"
1 object definitions have been created or modified.
2. Using an xCAT stanza file
• Prepare a stanza file <filename>.stanza with content similiar to the following:
# <xCAT data object stanza file>
cn1:
objtype=node
arch=x86_64
groups=kvm,vm,all
nichostnamesuffixes.eth1=-eth1-1|-eth1-2
nichostnamesuffixes.eth2=-eth2-1|-eth2-2
nicips.eth1=11.1.89.7|12.1.89.7
nicips.eth2=13.1.89.7|14.1.89.7
nicnetworks.eth1=net11|net12
nicnetworks.eth2=net13|net14
nictypes.eth1=Ethernet
nictypes.eth2=Ethernet
1.3. Admin Guide
125
xCAT3 Documentation, Release 2.12
• Using the mkdef -z option, define the stanza file to xCAT:
# cat <filename>.stanza | mkdef -z
3. Using tabedit to edit the nics database table directly
The tabedit command opens the specified xCAT database table in a vi like editor and allows the user to edit
any text and write the changes back to the database table.
WARNING: Using the tabedit command is not the recommended method because it is tedious and error
prone.
After changing the content of the nics table, here is the result from tabdump nics
# tabdump nics
#node,nicips,nichostnamesuffixes,nictypes,niccustomscripts,nicnetworks,nicaliases,
˓→comments,disable
"cn1","eth1!11.1.89.7|12.1.89.7,eth2!13.1.89.7|14.1.89.7","eth1!-eth1-1|-eth1-2,
˓→eth2!-eth2-1|-eth2-2,"eth1!Ethernet,eth2!Ethernet",,"eth1!net11|net12,eth2!
˓→net13|net14",,,
After you have defined the configuration information in any of the ways above, run the makehosts command to add
the new configuration to the /etc/hosts file.
# makehosts cn1
# cat /etc/hosts
11.1.89.7 cn1-eth1-1
12.1.89.7 cn1-eth1-2
13.1.89.7 cn1-eth2-1
14.1.89.7 cn1-eth2-2
cn1-eth1-1.ppd.pok.ibm.com
cn1-eth1-2.ppd.pok.ibm.com
cn1-eth2-1.ppd.pok.ibm.com
cn1-eth2-2.ppd.pok.ibm.com
Add confignics into the node’s postscripts list
Use command below to add confignics into the node’s postscripts list
chdef cn1 -p postscripts=confignics
By default, confignics does not configure the install nic. if need, using flag “-s” to allow the install nic to be configured.
chdef cn1 -p prostscripts="confignics -s"
Option “-s” writes the install nic’s information into configuration file for persistence. All install nic’s data defined in
nics table will be written also.
Add network object into the networks table
The nicnetworks attribute only defines the nic that uses the IP address. Other information about the network
should be defined in the networks table.
Use the tabedit command to add/modify the networks in the‘‘networks‘‘ table
tabdump networks
#netname,net,mask,mgtifname,gateway,dhcpserver,tftpserver,nameservers,ntpservers,
˓→logservers,dynamicrange,staticrange,staticrangeincrement,nodehostname,ddnsdomain,
˓→vlanid,domain,mtu,comments,disable
126
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
...
"net11",
"net12",
"net13",
"net14",
"11.1.89.0",
"12.1.89.0",
"13.1.89.0",
"14.1.89.0",
"255.255.255.0",
"255.255.255.0",
"255.255.255.0",
"255.255.255.0",
"eth1",,,,,,,,,,,,,,,,
"eth1",,,,,,,,,,,,,,,,
"eth2",,,,,,,,,,,,,,,,
"eth2",,,,,,,,,,,,,,,,
Option -r to remove the undefined NICS
If the compute node’s nics were configured by confignics and the nics configuration changed in the nics table,
user the confignics -r to remove the undefined nic.
For example, if on a compute node the eth0, eth1, and eth2 nics were configured:
# ifconfig
eth0
Link encap:Ethernet
...
eth1
Link encap:Ethernet
...
eth2
Link encap:Ethernet
...
HWaddr 00:14:5e:d9:6c:e6
HWaddr 00:14:5e:d9:6c:e7
HWaddr 00:14:5e:d9:6c:e8
Delete the eth2 definition in nics table using the chdef command. Then run the following to remove the undefined
eth2 nic on the compute node:
updatenode <noderange> -P "confignics -r"
The result should have eth2 disabled:
# ifconfig
eth0
Link encap:Ethernet
...
eth1
Link encap:Ethernet
...
HWaddr 00:14:5e:d9:6c:e6
HWaddr 00:14:5e:d9:6c:e7
Deleting the installnic will result in strange problems, so confignics -r will not delete the nic set as the
installnic.
Advanced Networking Configuration - confignetwork
Note: confignetwork postscript is only supported on RHEL releases.
Configure BOND, VLAN and BRIDGES
The confignetwork postscript can be used to configure the network interfaces on the compute nodes to support
VLAN, BONDs, and BRIDGES. In order for the confignetwork postscript to run successfully, the following
attributes must be configured for the node in the nics table:
• nicips
• nictypes
• nicnetworks
• nicdevices - resolves the relationship among the physical network interface devices
1.3. Admin Guide
127
xCAT3 Documentation, Release 2.12
The following example set the xCAT properties for compute node cn1 to achieve the following network configuration
using the confignetwork postscript:
• Compute node cn1 has two physical NICs: eth2 and eth3
• Bond eth2 and eth3 as bond0
• From bond0, create 2 VLANs: bond0.1 and bond0.2
• Make bridge br1 using bond0.1 with IP (10.0.0.1)
• Make bridge br2 using bond0.2 with IP (20.0.0.1)
Define attributes in the nics table
Chose one of two methods described below:
1. Using the mkdef or chdef commands
(a) Compute node cn1 has two physical NICs: eth2 and eth3
chdef cn1 nictypes.eth2=ethernet nictypes.eth3=ethernet
(b) Define bond0 and bond eth2 and eth3 as bond0
chdef cn1 nictypes.bond0=bond \
nicdevices.bond0="eth2|eth3"
(c) From bond0, create 2 VLANs: bond0.1 and bond0.2
chdef cn1 nictypes.bond0.1=vlan \
nictypes.bond0.2=vlan \
nicdevices.bond0.1=bond0 \
nicdevices.bond0.2=bond0
(d) Create bridge br1 using bond0.1 with IP (10.0.0.1)
chdef cn1 nictypes.br1=bridge \
nicdevices.br1=bond0.1 \
nicips.br1=10.0.0.1 \
nicnetworks.br1="net10"
(e) Create bridge br2 using bond0.2 with IP (20.0.0.1)
chdef cn1 nictypes.br2=bridge \
nicdevices.br2=bond0.2 \
nicips.br2=20.0.0.1 \
nicnetworks.br2="net20"
2. Using an xCAT stanza file
• Prepare a stanza file <filename>.stanza with content similiar to the following:
# <xCAT data object stanza file>
cn1:
objtype=node
arch=x86_64
groups=kvm,vm,all
nicdevices.br1=bond0.1
nicdevices.br2=bond0.2
128
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
nicdevices.bond0.1=bond0
nicdevices.bond0.2=bond0
nicdevices.bond0=eth2|eth3
nictypes.eth2=ethernet
nictypes.eth3=ethernet
nictypes.bond0=bond
nictypes.bond0.1=vlan
nictypes.bond0.2=vlan
nictypes.br1=bridge
nictypes.br2=bridge
nicips.br1=10.0.0.1
nicips.br2=20.0.0.1
nicnetworks.br1=net10
nicnetworks.br2=net20
• Using the mkdef -z option, define the stanza file to xCAT:
cat <filename>.stanza | mkdef -z
Define the additional networks to xCAT
If this is a new network being created on the compute nodes, an entry needs to be created into the xCAT database.
The nicnetworks attribute only defines the nic that uses the IP address. Other information about the network
should be defined in the networks table.
Use the chdef command to add/modify the networks in the networks table
chdef -t network net10 net=10.0.0.0 mask=255.0.0.0
chdef -t network net20 net=20.0.0.0 mask=255.0.0.0
Add confignetwork into the node’s postscripts list
Use the following command to add confignetwork into postscript list to execute on reboot:
chdef cn1 -p postscripts=confignetwork
If the compute node is already running, use updatenode command to run confignetwork postscript without
rebooting the node:
updatenode cn1 -P confignetwork
Configure Two Bonded Adapters
The following example set the xCAT properties for compute node cn1 to create:
• Compute node cn1 has two physical NICs: eth2 and eth3
• Bond eth2 and eth3 as bond0
• Assign ip 40.0.0.1 to the bonded interface bond0
1.3. Admin Guide
129
xCAT3 Documentation, Release 2.12
Define attributes in the nics table
1. Using the mkdef or chdef commands
(a) Compute node cn1 has two physical NICs: eth2 and eth3
chdef cn1 nictypes.eth2=ethernet nictypes.eth3=ethernet
(b) Define bond0 and bond eth2 and eth3 as bond0
chdef cn1 nictypes.bond0=bond nicdevices.bond0="eth2|eth3"
chdef cn1 nicips.bond0=40.0.0.1
Add network object into the networks table
Use the chdef command to add/modify the networks in the networks table
chdef -t network net40 net=40.0.0.0 mask=255.0.0.0
chdef cn1 nicnetworks.bond0=net40
Add confignetwork into the node’s postscripts list
Use command below to add confignetwork into the node’s postscripts list
chdef cn1 -p postscripts=confignetwork
During OS deployment on compute node, confignetwork postscript will be executed. If the compute node is
already running, use updatenode command to run confignetwork postscript without rebooting the node:
updatenode cn1 -P confignetwork
Verify bonding mode
Login to compute node cn1 and check bonding options in /etc/sysconfig/network-scripts/
ifcfg-bond0 file
BONDING_OPTS="mode=802.3ad xmit_hash_policy=layer2+3"
• mode=802.3ad requires additional configuration on the switch.
• mode=2 can be used for bonding without additional switch configuration.
If changes are made to /etc/sysconfig/network-scripts/ifcfg-bond0 file, restart network service.
Initialize the Compute for Deployment
XCAT use ‘nodeset‘ command to associate a specific image to a node which will be installed with this image.
nodeset <nodename> osimage=<osimage>
There are more attributes of nodeset used for some specific purpose or specific machines, for example:
130
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
• runimage: If you would like to run a task after deployment, you can define that task with this attribute.
• runcmd: This instructs the node to boot to the xCAT nbfs environment and proceed to configure BMC for basic
remote access. This causes the IP, netmask, gateway, username, and password to be programmed according to
the configuration table.
• shell: This instructs the node to boot to the xCAT genesis environment, and present a shell prompt on console.
The node will also be able to be sshed into and have utilities such as wget, tftp, scp, nfs, and cifs. It will have
storage drivers available for many common systems.
Choose such additional attribute of nodeset according to your requirement, if want to get more information about
nodeset, refer to nodeset’s man page.
Start the OS Deployment
Start the deployment involves two key operations. First specify the boot device of the next boot to be network, then
reboot the node:
For Power servers, those two operations can be completed by one command rnetboot:
rnetboot <node>
For x86_64 servers, those two operations need two independent commands.
1. set the next boot device to be from the “network”
rsetboot <node> net
2. Reboot the xSeries server: ::
rpower <node> reset
Diskless Installation
Select or Create an osimage Definition
Before creating an image on xCAT, the distro media should be prepared ahead. That can be ISOs or DVDs.
XCAT use ‘copycds’ command to create an image which will be available to install nodes. “copycds” will copy all
contents of Distribution DVDs/ISOs or Service Pack DVDs/ISOs to a destination directory, and create several relevant
osimage definitions by default.
If using an ISO, copy it to (or NFS mount it on) the management node, and then run:
copycds <path>/<specific-distro>.iso
If using a DVD, put it in the DVD drive of the management node and run:
copycds /dev/<dvd-drive-name>
To see the list of osimages:
lsdef -t osimage
To see the attributes of a particular osimage:
1.3. Admin Guide
131
xCAT3 Documentation, Release 2.12
lsdef -t osimage <osimage-name>
Initially, some attributes of osimage are assigned default values by xCAT - they all can work correctly because the files
or templates invoked by those attributes are shipped with xCAT by default. If you need to customize those attributes,
refer to the next section Customize osimage
Below is an example of osimage definitions created by copycds:
# lsdef -t osimage
rhels7.2-ppc64le-install-compute (osimage)
rhels7.2-ppc64le-install-service (osimage)
rhels7.2-ppc64le-netboot-compute (osimage)
rhels7.2-ppc64le-stateful-mgmtnode (osimage)
In these osimage definitions shown above
• <os>-<arch>-install-compute is the default osimage definition used for diskful installation
• <os>-<arch>-netboot-compute is the default osimage definition used for diskless installation
• <os>-<arch>-install-service is the default osimage definition used for service node deployment which shall be
used in hierarchical environment
Note: There are more things needed for ubuntu ppc64le osimages:
For ubuntu ppc64le, the initrd.gz shipped with the ISO does not support network booting. In order to install ubuntu
with xCAT, you need to follow the steps below to complete the osimage definition.
• Download mini.iso from
[ubuntu 14.04.1]: http://xcat.org/files/netboot/ubuntu14.04.1/ppc64el/mini.iso
[ubuntu 14.04.2]: http://xcat.org/files/netboot/ubuntu14.04.2/ppc64el/mini.iso
[ubuntu 14.04.3]: http://xcat.org/files/netboot/ubuntu14.04.3/ppc64el/mini.iso
[ubuntu 14.04.4]: http://xcat.org/files/netboot/ubuntu14.04.4/ppc64el/mini.iso
[ubuntu 16.04]: http://xcat.org/files/netboot/ubuntu16.04/ppc64el/mini.iso
[ubuntu 16.04.1]: http://xcat.org/files/netboot/ubuntu16.04.1/ppc64el/mini.iso
• Mount mini.iso
mkdir /tmp/iso
mount -o loop mini.iso /tmp/iso
• Copy the netboot initrd.gz to osimage
mkdir -p /install/<ubuntu-version>/ppc64el/install/netboot
cp /tmp/iso/install/initrd.gz /install/<ubuntu-version>/ppc64el/install/netboot
[Below tips maybe helpful for you]
[Tips 1]
If this is the same distro version as what your management node uses, create a .repo file in /etc/yum.repos.d with
contents similar to:
[local-<os>-<arch>]
name=xCAT local <os> <version>
baseurl=file:/install/<os>/<arch>
132
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
enabled=1
gpgcheck=0
In this way, if you need to install some additional RPMs into your MN later, you can simply install them with yum.
Or if you are installing a software on your MN that depends some RPMs from this disto, those RPMs will be found
and installed automatically.
[Tips 2]
You can create/modify an osimage definition easily based on the default osimage definition. The general steps are:
• lsdef -t osimage -z <os>-<arch>-install-compute > <filename>.stanza
• modify <filename>.stanza according to your requirements
• cat <filename>.stanza| mkdef -z
For example, if you need to change the osimage name to your favorite name, this command may be helpful:
chdef -t osimage rhels6.2-x86_64-install-compute -n rhels6.2_myimage
Customize osimage (Optional)
Optional means all the subitems in this page are not necessary to finish an OS deployment. If you are new to xCAT,
you can just jump to Initialize the Compute for Deployment.
Load Additional Drivers
Overview
During the installing or netbooting of a node, the drivers in the initrd will be used to drive the devices like network
cards and IO devices to perform the installation/netbooting tasks. But sometimes the drivers for the new devices were
not included in the default initrd shipped by Red Hat or Suse. A solution is to inject the new drivers into the initrd to
drive the new device during the installation/netbooting process.
Generally there are two approaches to inject the new drivers: Driver Update Disk and Drive RPM package.
A “Driver Update Disk” is media which contains the drivers, firmware and related configuration files for certain
devices. The driver update disk is always supplied by the vendor of the device. One driver update disk can contain
multiple drivers for different OS releases and different hardware architectures. Red Hat and Suse have different driver
update disk formats.
The ‘Driver RPM Package‘ is the rpm package which includes the drivers and firmware for the specific devices. The
Driver RPM is the rpm package which is shipped by the Vendor of the device for a new device or a new kernel version.
xCAT supports both. But for ‘Driver RPM Package‘ is only supported in xCAT 2.8 and later.
No matter which approach chosen, there are two steps to make new drivers work. one is locate the new driver’s path,
another is inject the new drivers into the initrd.
Locate the New Drivers
For Driver Update Disk
There are two approaches for xCAT to find the driver disk (pick one):
1.3. Admin Guide
133
xCAT3 Documentation, Release 2.12
1. Specify the location of the driver disk in the osimage object (This is ONLY supported in 2.8 and later)
The value for the ‘driverupdatesrc’ attribute is a comma separated driver disk list. The tag ‘dud’ must be
specified before the full path of ‘driver update disk’ to specify the type of the file:
chdef -t osimage <osimagename> driverupdatesrc=dud:<full path of driver disk>
1. Put the driver update disk in the directory <installroot>/driverdisk/<os>/<arch> (example: /
install/driverdisk/sles11.1/x86_64).
During the running of the genimage, geninitrd, or nodeset commands, xCAT will look for driver update
disks in the directory <installroot>/driverdisk/<os>/<arch>.
For Driver RPM Packages
The Driver RPM packages must be specified in the osimage object.
Three attributes of osimage object can be used to specify the Driver RPM location and Driver names. If you want to
load new drivers in the initrd, the ‘netdrivers‘ attribute must be set. And one or both of the ‘driverupdatesrc‘ and
‘osupdatename‘ attributes must be set. If both of ‘driverupdatesrc’ and ‘osupdatename’ are set, the drivers in the
‘driverupdatesrc’ have higher priority.
• netdrivers - comma separated driver names that need to be injected into the initrd. The postfix ‘.ko’ can be
ignored.
The ‘netdrivers’ attribute must be set to specify the new driver list. If you want to load all the drivers from the driver
rpms, use the keyword allupdate. Another keyword for the netdrivers attribute is updateonly, which means only the
drivers located in the original initrd will be added to the newly built initrd from the driver rpms. This is useful to
reduce the size of the new built initrd when the distro is updated, since there are many more drivers in the new kernel
rpm than in the original initrd. Examples:
chdef -t osimage <osimagename> netdrivers=megaraid_sas.ko,igb.ko
chdef -t osimage <osimagename> netdrivers=allupdate
chdef -t osimage <osimagename> netdrivers=updateonly,igb.ko,new.ko
• driverupdatesrc - comma separated driver rpm packages (full path should be specified)
A tag named ‘rpm’ can be specified before the full path of the rpm to specify the file type. The tag is optional since
the default format is ‘rpm’ if no tag is specified. Example:
chdef -t osimage <osimagename> driverupdatesrc=rpm:<full path of driver disk1>,rpm:
˓→<full path of driver disk2>
• osupdatename - comma separated ‘osdistroupdate’ objects. Each ‘osdistroupdate’ object specifies a Linux distro
update.
When geninitrd is run, kernel-*.rpm will be searched in the osdistroupdate.dirpath to get all the rpm packages
and then those rpms will be searched for drivers. Example:
mkdef -t osdistroupdate update1 dirpath=/install/<os>/<arch>
chdef -t osimage <osimagename> osupdatename=update1
If ‘osupdatename’ is specified, the kernel shipped with the ‘osupdatename’ will be used to load the newly built initrd,
then only the drivers matching the new kernel will be kept in the newly built initrd. If trying to use the ‘osupdatename’,
the ‘allupdate’ or ‘updateonly’ should be added in the ‘netdrivers’ attribute, or all the necessary driver names for the
new kernel need to be added in the ‘netdrivers’ attribute. Otherwise the new drivers for the new kernel will be missed
in newly built initrd. ..
134
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
Inject the Drivers into the initrd
For Driver Update Disk
• If specifying the driver disk location in the osimage
Run the following command:
genimage <osimagename>
• If putting the driver disk in <installroot>/driverdisk/<os>/<arch>:
Running ‘genimage’ in anyway will load the driver disk ..
For Driver RPM Packages
Run the following command:
genimage <osimagename> [--ignorekernelchk]
The option ‘–ignorekernelchk’ is used to skip the kernel version checking when injecting drivers from osimage.driverupdatesrc. To use this flag, you should make sure the drivers in the driver rpms are usable for the target
kernel. ..
Notes
• If the drivers from the driver disk or driver rpm are not already part of the installed or booted system, it’s
necessary to add the rpm packages for the drivers to the .pkglist or .otherpkglist of the osimage object to install
them in the system.
• If a driver rpm needs to be loaded, the osimage object must be used for the ‘nodeset’ and ‘genimage’ command,
instead of the older style profile approach.
• Both a Driver disk and a Driver rpm can be loaded in one ‘nodeset’ or ‘genimage’ invocation.
Prescripts and Postscripts
Using postinstall scripts
While running genimage to generate diskless or statelite osimage, you may want to customize the root image after
the package installation step. The postinstall attribute of the osimage definition provides a hook to run user
specified script(s), in non-chroot mode, against the directory specified by rootimgdir attribute.
xCAT ships a default postinstall script for the diskless/statelite osimages that must be executed to ensure a
successful provisioning of the OS:
lsdef -t osimage -o rhels7.3-ppc64le-netboot-compute -i postinstall
Object name: rhels7.3-ppc64le-netboot-compute
postinstall=/opt/xcat/share/xcat/netboot/rh/compute.rhels7.ppc64le.postinstall
Customizing the postinstall script, can be done by either one of the methods below:
• Append your own postinstall scripts
1.3. Admin Guide
135
xCAT3 Documentation, Release 2.12
chdef -t osimage -o <osimage> -p postinstall=/install/custom/postinstall/rh7/
˓→mypostscript
• Create your own postinstall script based on the default postinstall script
cp /opt/xcat/share/xcat/netboot/rh/compute.rhels7.ppc64le.postinstall /install/
˓→custom/postinstall/rh7/mypostscript
# edit /install/custom/postinstall/rh7/mypostscript
chdef -t osimage -o <osimage> postinstall=/install/custom/postinstall/rh7/
˓→mypostscript
Common questions about the usage of postinstall scripts:
When do postinstall scripts run?
High level flow of genimage process:
1. install the packages specified by pkglist into rootimgdir directory
2. cumstomize the rootimgdir directory
3. generate the initrd based on the rootimgdir directory
The postinstall scripts are executed in step b).
Do postinstall scripts execute in chroot mode under rootimgdir directory?
No. Unlike postscripts and postbootscripts, the postinstall scripts are run in non-chroot environment, directly on
the management node. In the postinstall scripts, all the paths of the directories and files are based on / of the management node. To reference inside the rootimgdir, use the $IMG_ROOTIMGDIR environment variable, exported by
genimage.
What are some of the environment variables available to my customized postinstall scripts?
Environment variables, available to be used in the postinstall scripts are listed in postinstall attribute
section of linuximage
Synchronizing Files
Add Additional Software Packages
Customize network adapter
Configure Additional Network Interfaces - confignics
The nics table and the confignics postscript can be used to automatically configure additional network interfaces
(multiple ethernets adapters, InfiniBand, etc) on the nodes as they are being deployed.
The way the confignics postscript decides what IP address to give the secondary adapter is by checking the nics table,
in which the nic configuration information is stored.
To use the nics table and confignics postscript to define a secondary adapter on one or more nodes, follow these steps:
136
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
Define configuration information for the Secondary Adapters in the nics table
There are 3 ways to complete this operation.
1. Using the mkdef and chdef commands
# mkdef cn1 groups=all nicips.eth1="11.1.89.7|12.1.89.7" nicnetworks.eth1=
˓→"net11|net12" nictypes.eth1="Ethernet"
1 object definitions have been created or modified.
# chdef cn1 nicips.eth2="13.1.89.7|14.1.89.7" nicnetworks.eth2="net13|net14"
˓→nictypes.eth2="Ethernet"
1 object definitions have been created or modified.
2. Using an xCAT stanza file
• Prepare a stanza file <filename>.stanza with content similiar to the following:
# <xCAT data object stanza file>
cn1:
objtype=node
arch=x86_64
groups=kvm,vm,all
nichostnamesuffixes.eth1=-eth1-1|-eth1-2
nichostnamesuffixes.eth2=-eth2-1|-eth2-2
nicips.eth1=11.1.89.7|12.1.89.7
nicips.eth2=13.1.89.7|14.1.89.7
nicnetworks.eth1=net11|net12
nicnetworks.eth2=net13|net14
nictypes.eth1=Ethernet
nictypes.eth2=Ethernet
• Using the mkdef -z option, define the stanza file to xCAT:
# cat <filename>.stanza | mkdef -z
3. Using tabedit to edit the nics database table directly
The tabedit command opens the specified xCAT database table in a vi like editor and allows the user to edit
any text and write the changes back to the database table.
WARNING: Using the tabedit command is not the recommended method because it is tedious and error
prone.
After changing the content of the nics table, here is the result from tabdump nics
# tabdump nics
#node,nicips,nichostnamesuffixes,nictypes,niccustomscripts,nicnetworks,nicaliases,
˓→comments,disable
"cn1","eth1!11.1.89.7|12.1.89.7,eth2!13.1.89.7|14.1.89.7","eth1!-eth1-1|-eth1-2,
˓→eth2!-eth2-1|-eth2-2,"eth1!Ethernet,eth2!Ethernet",,"eth1!net11|net12,eth2!
˓→net13|net14",,,
After you have defined the configuration information in any of the ways above, run the makehosts command to add
the new configuration to the /etc/hosts file.
# makehosts cn1
# cat /etc/hosts
1.3. Admin Guide
137
xCAT3 Documentation, Release 2.12
11.1.89.7
12.1.89.7
13.1.89.7
14.1.89.7
cn1-eth1-1
cn1-eth1-2
cn1-eth2-1
cn1-eth2-2
cn1-eth1-1.ppd.pok.ibm.com
cn1-eth1-2.ppd.pok.ibm.com
cn1-eth2-1.ppd.pok.ibm.com
cn1-eth2-2.ppd.pok.ibm.com
Add confignics into the node’s postscripts list
Use command below to add confignics into the node’s postscripts list
chdef cn1 -p postscripts=confignics
By default, confignics does not configure the install nic. if need, using flag “-s” to allow the install nic to be configured.
chdef cn1 -p prostscripts="confignics -s"
Option “-s” writes the install nic’s information into configuration file for persistence. All install nic’s data defined in
nics table will be written also.
Add network object into the networks table
The nicnetworks attribute only defines the nic that uses the IP address. Other information about the network
should be defined in the networks table.
Use the tabedit command to add/modify the networks in the‘‘networks‘‘ table
tabdump networks
#netname,net,mask,mgtifname,gateway,dhcpserver,tftpserver,nameservers,ntpservers,
˓→logservers,dynamicrange,staticrange,staticrangeincrement,nodehostname,ddnsdomain,
˓→vlanid,domain,mtu,comments,disable
...
"net11", "11.1.89.0", "255.255.255.0", "eth1",,,,,,,,,,,,,,,,
"net12", "12.1.89.0", "255.255.255.0", "eth1",,,,,,,,,,,,,,,,
"net13", "13.1.89.0", "255.255.255.0", "eth2",,,,,,,,,,,,,,,,
"net14", "14.1.89.0", "255.255.255.0", "eth2",,,,,,,,,,,,,,,,
Option -r to remove the undefined NICS
If the compute node’s nics were configured by confignics and the nics configuration changed in the nics table,
user the confignics -r to remove the undefined nic.
For example, if on a compute node the eth0, eth1, and eth2 nics were configured:
# ifconfig
eth0
Link encap:Ethernet
...
eth1
Link encap:Ethernet
...
eth2
Link encap:Ethernet
...
HWaddr 00:14:5e:d9:6c:e6
HWaddr 00:14:5e:d9:6c:e7
HWaddr 00:14:5e:d9:6c:e8
Delete the eth2 definition in nics table using the chdef command. Then run the following to remove the undefined
eth2 nic on the compute node:
138
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
updatenode <noderange> -P "confignics -r"
The result should have eth2 disabled:
# ifconfig
eth0
Link encap:Ethernet
...
eth1
Link encap:Ethernet
...
HWaddr 00:14:5e:d9:6c:e6
HWaddr 00:14:5e:d9:6c:e7
Deleting the installnic will result in strange problems, so confignics -r will not delete the nic set as the
installnic.
Advanced Networking Configuration - confignetwork
Note: confignetwork postscript is only supported on RHEL releases.
Configure BOND, VLAN and BRIDGES
The confignetwork postscript can be used to configure the network interfaces on the compute nodes to support
VLAN, BONDs, and BRIDGES. In order for the confignetwork postscript to run successfully, the following
attributes must be configured for the node in the nics table:
• nicips
• nictypes
• nicnetworks
• nicdevices - resolves the relationship among the physical network interface devices
The following example set the xCAT properties for compute node cn1 to achieve the following network configuration
using the confignetwork postscript:
• Compute node cn1 has two physical NICs: eth2 and eth3
• Bond eth2 and eth3 as bond0
• From bond0, create 2 VLANs: bond0.1 and bond0.2
• Make bridge br1 using bond0.1 with IP (10.0.0.1)
• Make bridge br2 using bond0.2 with IP (20.0.0.1)
Define attributes in the nics table
Chose one of two methods described below:
1. Using the mkdef or chdef commands
(a) Compute node cn1 has two physical NICs: eth2 and eth3
chdef cn1 nictypes.eth2=ethernet nictypes.eth3=ethernet
(b) Define bond0 and bond eth2 and eth3 as bond0
chdef cn1 nictypes.bond0=bond \
nicdevices.bond0="eth2|eth3"
1.3. Admin Guide
139
xCAT3 Documentation, Release 2.12
(c) From bond0, create 2 VLANs: bond0.1 and bond0.2
chdef cn1 nictypes.bond0.1=vlan \
nictypes.bond0.2=vlan \
nicdevices.bond0.1=bond0 \
nicdevices.bond0.2=bond0
(d) Create bridge br1 using bond0.1 with IP (10.0.0.1)
chdef cn1 nictypes.br1=bridge \
nicdevices.br1=bond0.1 \
nicips.br1=10.0.0.1 \
nicnetworks.br1="net10"
(e) Create bridge br2 using bond0.2 with IP (20.0.0.1)
chdef cn1 nictypes.br2=bridge \
nicdevices.br2=bond0.2 \
nicips.br2=20.0.0.1 \
nicnetworks.br2="net20"
2. Using an xCAT stanza file
• Prepare a stanza file <filename>.stanza with content similiar to the following:
# <xCAT data object stanza file>
cn1:
objtype=node
arch=x86_64
groups=kvm,vm,all
nicdevices.br1=bond0.1
nicdevices.br2=bond0.2
nicdevices.bond0.1=bond0
nicdevices.bond0.2=bond0
nicdevices.bond0=eth2|eth3
nictypes.eth2=ethernet
nictypes.eth3=ethernet
nictypes.bond0=bond
nictypes.bond0.1=vlan
nictypes.bond0.2=vlan
nictypes.br1=bridge
nictypes.br2=bridge
nicips.br1=10.0.0.1
nicips.br2=20.0.0.1
nicnetworks.br1=net10
nicnetworks.br2=net20
• Using the mkdef -z option, define the stanza file to xCAT:
cat <filename>.stanza | mkdef -z
Define the additional networks to xCAT
If this is a new network being created on the compute nodes, an entry needs to be created into the xCAT database.
140
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
The nicnetworks attribute only defines the nic that uses the IP address. Other information about the network
should be defined in the networks table.
Use the chdef command to add/modify the networks in the networks table
chdef -t network net10 net=10.0.0.0 mask=255.0.0.0
chdef -t network net20 net=20.0.0.0 mask=255.0.0.0
Add confignetwork into the node’s postscripts list
Use the following command to add confignetwork into postscript list to execute on reboot:
chdef cn1 -p postscripts=confignetwork
If the compute node is already running, use updatenode command to run confignetwork postscript without
rebooting the node:
updatenode cn1 -P confignetwork
Configure Two Bonded Adapters
The following example set the xCAT properties for compute node cn1 to create:
• Compute node cn1 has two physical NICs: eth2 and eth3
• Bond eth2 and eth3 as bond0
• Assign ip 40.0.0.1 to the bonded interface bond0
Define attributes in the nics table
1. Using the mkdef or chdef commands
(a) Compute node cn1 has two physical NICs: eth2 and eth3
chdef cn1 nictypes.eth2=ethernet nictypes.eth3=ethernet
(b) Define bond0 and bond eth2 and eth3 as bond0
chdef cn1 nictypes.bond0=bond nicdevices.bond0="eth2|eth3"
chdef cn1 nicips.bond0=40.0.0.1
Add network object into the networks table
Use the chdef command to add/modify the networks in the networks table
chdef -t network net40 net=40.0.0.0 mask=255.0.0.0
chdef cn1 nicnetworks.bond0=net40
1.3. Admin Guide
141
xCAT3 Documentation, Release 2.12
Add confignetwork into the node’s postscripts list
Use command below to add confignetwork into the node’s postscripts list
chdef cn1 -p postscripts=confignetwork
During OS deployment on compute node, confignetwork postscript will be executed. If the compute node is
already running, use updatenode command to run confignetwork postscript without rebooting the node:
updatenode cn1 -P confignetwork
Verify bonding mode
Login to compute node cn1 and check bonding options in /etc/sysconfig/network-scripts/
ifcfg-bond0 file
BONDING_OPTS="mode=802.3ad xmit_hash_policy=layer2+3"
• mode=802.3ad requires additional configuration on the switch.
• mode=2 can be used for bonding without additional switch configuration.
If changes are made to /etc/sysconfig/network-scripts/ifcfg-bond0 file, restart network service.
Enable Kdump Over Ethernet
Overview
kdump is an advanced crash dumping mechanism. When enabled, the system is booted from the context of another
kernel. This second kernel reserves a small amount of memory, and its only purpose is to capture the core dump image
in case the system crashes. Being able to analyze the core dump helps significantly to determine the exact cause of the
system failure.
xCAT Interface
The pkglist, exclude and postinstall files location and name can be obtained by running the following command:
lsdef -t osimage <osimage name>
Here is an example:
lsdef -t osimage rhels7.1-ppc64le-netboot-compute
Object name: rhels7.1-ppc64le-netboot-compute
exlist=/opt/xcat/share/xcat/netboot/rh/compute.rhels7.ppc64le.exlist
imagetype=linux
osarch=ppc64le
osdistroname=rhels7.1-ppc64le
osname=Linux
osvers=rhels7.1
otherpkgdir=/install/post/otherpkgs/rhels7.1/ppc64le
permission=755
pkgdir=/install/rhels7.1/ppc64le
pkglist=/opt/xcat/share/xcat/netboot/rh/compute.rhels7.ppc64le.pkglist
142
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
postinstall=/opt/xcat/share/xcat/netboot/rh/compute.rhels7.ppc64le.postinstall
profile=compute
provmethod=netboot
rootimgdir=/install/netboot/rhels7.1/ppc64le/compute
In above example, pkglist file is /opt/xcat/share/xcat/netboot/rh/compute.rhels7.ppc64le.pkglist, exclude files is in /opt/xcat/share/xcat/netboot/rh/compute.rhels7.ppc64le.exlist, and postinstall file is
/opt/xcat/share/xcat/netboot/rh/compute.rhels7.ppc64le.postinstall.
Setup pkglist
Before setting up kdump, the appropriate rpms should be added to the pkglist file.Here is the rpm packages list which
needs to be added to pkglist file for kdump for different OS.
• [RHEL]
kexec-tools
crash
• [SLES11]
kdump
kexec-tools
makedumpfile
• [SLES10]
kernel-kdump
kexec-tools
kdump
makedumpfile
• [Ubuntu]
<TODO>
The exclude file
The base diskless image excludes the /boot directory, but it is required for kdump. Update the exlist file and remove
the entry for /boot. Then run the packimage or liteimg command to update your image with the changes.
<TODO exclude files list>
The postinstall file
The kdump will create a new initrd which used in the dumping stage. The /tmp or /var/tmp directory will be used as
the temporary directory. These 2 directory only are allocated 10M space by default. You need to enlarge it to 200M.
Modify the postinstall file to increase /tmp space.
• [RHELS]
tmpfs
/var/tmp
1.3. Admin Guide
tmpfs
defaults,size=200m
0 2
143
xCAT3 Documentation, Release 2.12
• [SLES10]
tmpfs
/var/tmp
tmpfs
defaults,size=200m
0 2
• [SLES11]
tmpfs
/tmp
tmpfs
defaults,size=200m
0 2
• [Ubuntu]
<TODO>
The dump attribute
In order to support kdump, the dump attribute was added into linuximage table, which is used to define the remote
path where the crash information should be dumped to. Use the chdef command to change the image’s dump attribute
using the URI format.
chdef -t osimage <image name> dump=nfs://<nfs_server_ip>/<kdump_path>
The <nfs_server_ip> can be excluded if the destination NFS server is the service or management node.
chdef -t osimage <image name> dump=nfs:///<kdump_path>
The crashkernelsize attribute
For system x machine, on sles10 set the crashkernelsize attribute like this:
chdef -t osimage <image name> crashkernelsize=<size>M@16M
On sles11 and rhels6 set the crashkernelsize attribute like this:
chdef -t osimage <image name> crashkernelsize=<size>M
Where <size> recommended value is 256. For more information about the size can refer to the following information:
https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Enterprise_Linux/5/html/Deployment_Guide/
ch-kdump.html#s2-kdump-configuration-cli.
http://www.novell.com/support/kb/doc.php?id=3374462.
https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Deployment_Guide/
s2-kdump-configuration-cli.html.
For system p machine, set the crashkernelsize attribute to this:
chdef -t osimage <image name> crashkernelsize=<size>@32M
Where <size> recommended value is 256, more information can refer the kdump document for the system x.
When your node starts, and you get a kdump start error like this:
Your running kernel is using more than 70% of the amount of space you reserved for
˓→kdump, you should consider increasing your crashkernel
You should modify this attribute using this chdef command:
144
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
chdef -t osimage <image name> crashkernelsize=512M@32M
If 512M@32M is not large enough, you should change the crashkernelsize larger like 1024M until the error message
disappear.
The enablekdump postscript
This postscript enablekdump is used to start the kdump service when the node is booting up. Add it to your nodes list
of postscripts by running this command:
chdef -t node <node range> -p postscripts=enablekdump
Notes
Currently, only NFS is supported for the setup of kdump.
If the dump attribute is not set, the kdump service will not be enabled.
Make sure the NFS remote path(nfs://<nfs_server_ip>/<kdump_path>) is exported and it is read-writeable to the node
where kdump service is enabled.
How to trigger kernel panic on Linux
Normally, kernel panic() will trigger booting into capture kernel. Once the kernel panic is triggered, the node will
reboot into the capture kernel, and a kernel dump (vmcore) will be automatically saved to the directory on the specified
NFS server (<nfs_server_ip>).
1. For RHESL6 the directory is <kdump_path>/var/crash/<node_ip>-<time>/
2. For SLES11 the directory is <kdump_path>/<node hostname>/<date>
3. For SLES10 the directory is <kdump_path>/<node hostname>
For Redhat and SLES11.1 testing, you can use the following commands:
echo 1 > /proc/sys/kernel/sysrq
echo c > /proc/sysrq-trigger
This will force the Linux kernel to crash, and the address-YYYY-MM-DD-HH:MM:SS/vmcore file will be copied to
the location you have selected on the specified NFS server directory.
Dump Analysis
Once the system has returned from recovering the crash, you may wish to analyze the kernel dump file using the crash
tool.
1.Locate the recent vmcore dump file.
2.Locate the kernel file for the crash server(the kernel is under /tftpboot/xcat/netboot/<OS
name=”“>/<ARCH>/<profile>/kernel on management node).
3.Once you have located a vmcore dump file and kernel file, call crash:
1.3. Admin Guide
145
xCAT3 Documentation, Release 2.12
crash <vmcore_dump_file> <kernel_file>
If crash cannot find any files under /usr/lib/debug? Make sure you have the kernel-debuginfo package installed.
For more information about the dump analysis you can refer the following documents:
http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html/Deployment_Guide/s1-kdump-crash.
htmlRHELdocument
http://www.novell.com/support/kb/doc.php?id=3374462SLESdocument
Installing a New Kernel in the Diskless Image
[TODO : Verify on ppc64le]
Note: This procedure assumes you are using xCAT 2.6.1 or later.
The kerneldir attribute in linuximage table can be used to assign a directory containing kernel RPMs that can be
installed into diskless images. The default for kernerdir is /install/kernels. To add a new kernel, create a directory
named <kernelver> under the kerneldir, and genimage will pick them up from there.
The following examples assume you have the kernel RPM in /tmp and is using the default value for kerneldir (/install/kernels).
The RPM names below are only examples, substitute your specific level and architecture.
• [RHEL]
The RPM kernel package is usually named:
kernel-<kernelver>.rpm.
229.ael7b.ppc64le.rpm means kernelver=3.10.0-229.ael7b.ppc64le.
For example,
kernel-3.10.0-
mkdir -p /install/kernels/3.10.0-229.ael7b.ppc64le
cp /tmp/kernel-3.10.0-229.ael7b.ppc64le.rpm /install/kernels/3.10.0-229.ael7b.ppc64le
createrepo /install/kernels/3.10.0-229.ael7b.ppc64le/
Run genimage/packimage to update the image with the new kernel. Note: If downgrading the kernel, you may need
to first remove the rootimg directory.
genimage <imagename> -k 3.10.0-229.ael7b.ppc64le
packimage <imagename>
• [SLES]
The RPM kernel package is usually separated into two parts: kernel-<arch>-base and kernel<arch>. For example,
/tmp contains the following two RPMs:
kernel-default-3.12.28-4.6.ppc64le.rpm
kernel-default-base-3.12.28-4.6.ppc64le.rpm
kernel-default-devel-3.12.28-4.6.ppc64le.rpm
3.12.28-4.6.ppc64le is NOT the kernel version,3.12.28-4-ppc64le is the kernel version. The “4.6.ppc64le” is replaced
with “4-ppc64le”:
mkdir -p /install/kernels/3.12.28-4-ppc64le/
cp /tmp/kernel-default-3.12.28-4.6.ppc64le.rpm /install/kernels/3.12.28-4-ppc64le/
cp /tmp/kernel-default-base-3.12.28-4.6.ppc64le.rpm /install/kernels/3.12.28-4˓→ppc64le/
cp /tmp/kernel-default-devel-3.12.28-4.6.ppc64le.rpm /install/kernels/3.12.28-4˓→ppc64le/
146
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
Run genimage/packimage to update the image with the new kernel. Note: If downgrading the kernel, you may need
to first remove the rootimg directory.
Since the kernel version name is different from the kernel rpm package name, the -k flag MUST to be specified on the
genimage command.
genimage <imagename> -k 3.12.28-4-ppc64le 3.12.28-4.6
packimage <imagename>
Installing New Kernel Drivers to Diskless Initrd
The kernel drivers in the diskless initrd are used for the devices during the netboot. If you are missing one or more
kernel drivers for specific devices (especially for the network device), the netboot process will fail. xCAT offers two
approaches to add additional drivers to the diskless initrd during the running of genimage.
Use the ‘-n’ flag to add new drivers to the diskless initrd:
genimage <imagename> -n <new driver list>
Generally, the genimage command has a default driver list which will be added to the initrd. But if you specify the ‘-n’
flag, the default driver list will be replaced with your <new driver list>. That means you need to include any drivers
that you need from the default driver list into your <new driver list>.
The default driver list:
rh-x86:
tg3 bnx2 bnx2x e1000 e1000e igb mlx_en virtio_net be2net
rh-ppc:
e1000 e1000e igb ibmveth ehea
rh-ppcle: ext3 ext4
sles-x86: tg3 bnx2 bnx2x e1000 e1000e igb mlx_en be2net
sels-ppc: tg3 e1000 e1000e igb ibmveth ehea be2net
sles-ppcle: scsi_mod libata scsi_tgt jbd2 mbcache crc16 virtio virtio_ring libahci
˓→crc-t10dif scsi_transport_srp af_packet ext3 ext4 virtio_pci virtio_blk scsi_dh
˓→ahci megaraid_sas sd_mod ibmvscsi
Note: With this approach, xCAT will search for the drivers in the rootimage. You need to make sure the drivers
have been included in the rootimage before generating the initrd. You can install the drivers manually in an existing
rootimage (using chroot) and run genimage again, or you can use a postinstall script to install drivers to the rootimage
during your initial genimage run.
Use the driver rpm package to add new drivers from rpm packages to the diskless initrd. Refer to the Configure
Additional Network Interfaces - confignics for details.
Accelerating the diskless initrd and rootimg generating
Generating diskless initrd with genimage and compressed rootimg with packimage and liteimg is a timeconsuming process, it can be accelerated by enabling parallel compression tool pigz on the management node with
multiple processors and cores. See Appendix for an example on packimage performance optimized with pigz
enabled.
Enabling the pigz for diskless initrd and rootimg generating
The parallel compression tool pigz can be enabled by installing pigz package on the management server or diskless
rootimg. Depending on the method of generating the initrd and compressed rootimg, the steps differ in different Linux
distributions.
1.3. Admin Guide
147
xCAT3 Documentation, Release 2.12
• [RHEL]
The package pigz is shipped in Extra Packages for Enterprise Linux (or EPEL) instead of RedHat iso, this
involves some complexity.
Extra Packages for Enterprise Linux (or EPEL) is a Fedora Special Interest Group that creates, maintains, and
manages a high quality set of additional packages for Enterprise Linux, including, but not limited to, Red Hat
Enterprise Linux (RHEL), CentOS and Scientific Linux (SL), Oracle Linux (OL).
EPEL has an epel-release package that includes gpg keys for package signing and repository information.
Installing this package for your Enterprise Linux version should allow you to use normal tools such as yum to
install packages and their dependencies.
Refer to the http://fedoraproject.org/wiki/EPEL for more details on EPEL
1. Enabling the pigz in genimage (only supported in RHELS6 or above)
pigz should be installed in the diskless rootimg. Download pigz package from https://dl.fedoraproject.
org/pub/epel/ , then customize the diskless osimage to install pigz as the additional packages, see Install
Additional Other Packages for more details.
2. Enabling the pigz in packimage
pigz should be installed on the management server. Download pigz package from https://dl.
fedoraproject.org/pub/epel/ , then install the pigz with yum or rpm.
• [UBUNTU]
Make sure the pigz is installed on the management node with the following command:
dpkg -l|grep pigz
If not, pigz can be installed with the following command:
apt-get install pigz
• [SLES]
1. Enabling the pigz in genimage (only supported in SLES12 or above)
pigz should be installed in the diskless rootimg, since‘‘pigz‘‘ is shipped in the SLES iso, this can be done
by adding pigz into the pkglist of diskless osimage.
2. Enabling the pigz in packimage
Make sure the pigz is installed on the management node with the following command:
rpm -qa|grep pigz
If not, pigz can be installed with the following command:
zypper install pigz
Appendix: An example on packimage performance optimization with “pigz” enabled
This is an example on performance optimization with pigz enabled.
In this example, a xCAT command packimage rhels7-ppc64-netboot-compute is run on a Power 7 machine with 4 cores.
The system info:
148
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
# uname -a
Linux c910f03c01p03 3.10.0-123.el7.ppc64 #1 SMP Mon May 5 11:18:37 EDT 2014 ppc64
˓→ppc64 ppc64 GNU/Linux
# cat /etc/os-release
NAME="Red Hat Enterprise Linux Server"
VERSION="7.0 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="7.0"
PRETTY_NAME="Red Hat Enterprise Linux Server 7.0 (Maipo)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.0:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.0
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION=7.0
The CPU info:
# cat /proc/cpuinfo
processor
: 0
cpu
: POWER7 (architected), altivec supported
clock
: 3550.000000MHz
revision
: 2.0 (pvr 003f 0200)
processor
cpu
clock
revision
:
:
:
:
1
POWER7 (architected), altivec supported
3550.000000MHz
2.0 (pvr 003f 0200)
processor
cpu
clock
revision
:
:
:
:
2
POWER7 (architected), altivec supported
3550.000000MHz
2.0 (pvr 003f 0200)
processor
cpu
clock
revision
:
:
:
:
3
POWER7 (architected), altivec supported
3550.000000MHz
2.0 (pvr 003f 0200)
timebase
platform
model
machine
:
:
:
:
512000000
pSeries
IBM,8233-E8B
CHRP IBM,8233-E8B
The time spent on packimage with gzip:
# time packimage rhels7-ppc64-netboot-compute
Packing contents of /install/netboot/rhels7/ppc64/compute/rootimg
compress method:gzip
real
user
1m14.896s
0m0.159s
1.3. Admin Guide
149
xCAT3 Documentation, Release 2.12
sys
0m0.019s
The time spent on packimage with pigz:
# time packimage rhels7-ppc64-netboot-compute
Packing contents of /install/netboot/rhels7/ppc64/compute/rootimg
compress method:pigz
real
user
sys
0m23.177s
0m0.176s
0m0.016s
Trim diskless rootimg
To reduce the memory and boot-up time for the diskless node, the initrd and rootimg.gz should be kept as compact as
possible under the premise of meeting the user’s requirements.
Exclude list
xCAT provides an attribute exlist in the osimage object definition, that allows the user to select files to exclude
when building the rootimg.gz file for the diskless node.
Take the osimage sles12.1-ppc64le-netboot-compute for example:
# lsdef -t osimage -o sles12.1-ppc64le-netboot-compute -i exlist
Object name: sles12.1-ppc64le-netboot-compute
exlist=/opt/xcat/share/xcat/netboot/sles/compute.sles12.ppc64le.exlist
Content of the Exclude List file
The file specified in linuximage.exlist includes relative path of the directories and files that will be excluded
from the rootimg.gz generated by packimage. The relative path assumes the rootimg directory, /install/
netboot/sles12.1/ppc64le/compute/rootimg here, to be the base directory.1
The following is a sample of exlist file
...
./usr/share/X11/locale/*
./usr/lib/perl[0-9]/[0-9.]*/ppc64le-linux-thread-multi/Encode/JP*
+./usr/share/X11/locale/C*
...
The content above presents some syntax supported in exlist file:
• Exclude files:
./usr/share/X11/locale/*
All the files and subdirectories under rootimg/usr/share/X11/locale/ will be excluded.
1 The exlist file entry should not end with a slash /, For example, this entry will never match anything: ./usr/lib/perl[0-9]/[0-9.
]*/ppc64le-linux-thread-multi/Encode/.
150
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
• Exclude Files using Patterns2 :
./usr/lib/perl[0-9]/[0-9.]*/ppc64le-linux-thread-multi/Encode/JP*
Use regular expression to easily exclude files. The above example will exclude any Perl library installed under
/usr/lib/ matching ppc64le-linux-thread-multi/Encode/JP*
• Include files:
+./usr/share/locale/C*
It is useful to include files following an exclude entry to quickly remove a larger set of files using a wildcard and
then adding back the few necessary files using the + sign. In the above example, all the files and sub-directories
matching the pattern /usr/share/locale/C* will be included in the rootimg.gz file.
Customize the exlist file and the osimage definition
Check the default exlist file and make sure:
• all files and directories you do not want in the image will be excluded from the rootimg.
• no file or directory you need will be excluded from the rootimg.
If you want to customize the osimage sles12.1-ppc64le-netboot-compute with your own exlist file, follow
the following steps:
#create a customized exlist file based on the default one
cp /opt/xcat/share/xcat/netboot/sles/compute.sles12.ppc64le.exlist /install/custom/
˓→netboot/sles/compute.sles12.ppc64le.exlist
#edit the newly created exlist file according to your need
vi /install/custom/netboot/sles/compute.sles12.ppc64le.exlist
#specify the newly created exlist file in the osimage definition
chdef -t osimage -o sles12.1-ppc64le-netboot-compute exlist=/install/custom/netboot/
˓→sles/compute.sles12.ppc64le.exlist
Generate Diskless Image
The copycds command copies the contents of the Linux media to /install/<os>/<arch> so that it will be
available for installing nodes or creating diskless images. After executing copycds, there are several osimage
definitions created by default. Run tabdump osimage to view these images:
tabdump osimage
The output should be similar to the following:
"rhels7.1-ppc64le-install-compute",,"compute","linux",,"install",,"rhels7.1-ppc64le",,
˓→,"Linux","rhels7.1","ppc64le",,,,,,,,
"rhels7.1-ppc64le-install-service",,"service","linux",,"install",,"rhels7.1-ppc64le",,
˓→,"Linux","rhels7.1","ppc64le",,,,,,,,
"rhels7.1-ppc64le-stateful-mgmtnode",,"compute","linux",,"install",,"rhels7.1-ppc64le
˓→",,,"Linux","rhels7.1","ppc64le",,,,,,,,
"rhels7.1-ppc64le-netboot-compute",,"compute","linux",,"netboot",,"rhels7.1-ppc64le",,
˓→,"Linux","rhels7.1","ppc64le",,,,,,,,
2
Pattern match test applies to the whole file name, starting from one of the start points specified in the exlist file entry. The regex syntax
should comply with the regex syntax of system command find -path, refer to its doc for details.
1.3. Admin Guide
151
xCAT3 Documentation, Release 2.12
The netboot-compute is the default diskless osimage created rhels7.1 ppc64le. Run genimage to generate a
diskless image based on the “rhels7.1-ppc64le-netboot-compute” definition:
genimage rhels7.1-ppc64le-netboot-compute
Before packing the diskless image, you have the opportunity to change any files in the image by changing to
the rootimgdir and making modifications. (e.g. /install/netboot/rhels7.1/ppc64le/compute/
rootimg).
However it’s recommended that all changes to the image are made via post install scripts so that it’s easily repeatable.
Although, instead, we recommend that you make all changes to the image via your postinstall script, so that it is
repeatable. Refer to Prescripts and Postscripts for more details.
Pack Diskless Image
After you run genimage to create the image, you can go ahead to pack the image to create the ramdisk:
packimage rhels7.1-ppc64le-netboot-compute
Export and Import Image
Overview
Note: There is a current restriction that exported 2.7 xCAT images cannot be imported on 2.8 xCAT https://sourceforge.
net/p/xcat/bugs/3813/. This is no longer a restrictions, if you are running xCAT 2.8.3 or later.
We want to create a system of making xCAT images more portable so that they can be shared and prevent people from
reinventing the wheel. While every install is unique there are some things that can be shared among different sites to
make images more portable. In addition, creating a method like this allows us to create snap shots of images we may
find useful to revert to in different situations.
Image exporting and importing are supported for stateful (diskful) and stateless (diskless) clusters. The following
documentation will show how to use imgexport to export images and imgimport to import images.
Exporting an image
1, The user has a working image and the image is defined in the osimage table and linuximage table. example:
lsdef -t osimage myimage
Object name: myimage
exlist=/install/custom/netboot/sles/compute1.exlist
imagetype=linux
netdrivers=e1000
osarch=ppc64le
osname=Linux
osvers=sles12
otherpkgdir=/install/post/otherpkgs/sles12/ppc64
otherpkglist=/install/custom/netboot/sles/compute1.otherpkgs.pkglist
pkgdir=/install/sles11/ppc64le
pkglist=/install/custom/netboot/sles/compute1.pkglist
postinstall=/install/custom/netboot/sles/compute1.postinstall
profile=compute1
152
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
provmethod=netboot
rootimgdir=/install/netboot/sles12/ppc64le/compute1
synclists=/install/custom/netboot/sles/compute1.list
2, The user runs the imgexport command. example:
imgexport myimage -p node1 -e /install/postscripts/myscript1 -e /install/
˓→postscripts/myscript2
(-p and -e are optional)
A bundle file called myimage.tgz will be created under the current directory. The bundle file contains the ramdisk,
boot kernel, the root image and all the configuration files for generating the image for a diskless cluster. For diskful,
it contains the kickstart/autoyast configuration file. (see appendix). The -p flag puts the names of the postscripts for
node1 into the image bundle. The -e flags put additional files into the bundle. In this case two postscripts myscript1
and myscript2 are included. This image can now be used on other systems.
Importing an image
1. User downloads a image bundle file from somewhere. (Sumavi.com will be hosting many of these).
2. User runs the imgimport command.
example:
imgimport myimage.tgz -p group1
(-p is optional)
This command fills out the osimage and linuximage tables, and populates file directories with appropriate files from
the image bundle file such as ramdisk, boot kernel, root image, configuration files for diskless. Any additional files
that come with the bundle file will also be put into the appropriate directories. If -p flag is specified, the postscript
names that come with the image will be put the into the postscripts table for the given node or group.
Copy an image to a new image name on the MN
Very often, the user wants to make a copy of an existing image on the same xCAT mn as a start point to make
modifications. In this case, you can run imgexport first as described on chapter 2, then run imgimport with -f flag to
change the profile name of the image. That way the image will be copied into a different directory on the same xCAT
mn.
example:
imgimport myimage.tgz -p group1 -f compute2
Modify an image (optional)
Skip this section if you want to use the image as is.
1, The use can modify the image to fit his/her own need. The following can be modified.
• Modify .pkglist file to add or remove packages that are from the os distro
• Modify .otherpkgs.pkglist to add or remove packages from other sources. Refer to Using_Updatenode for
details
• For diskful, modify the .tmpl file to change the kickstart/autoyast configuration
1.3. Admin Guide
153
xCAT3 Documentation, Release 2.12
• Modify .synclist file to change the files that are going to be synchronized to the nodes
• Modify the postscripts table for the nodes to be deployed
• Modify the osimage and/or linuximage tables for the location of the source rpms and the rootimage location
2, Run genimage:
genimage image_name
3, Run packimage:
packimage image_name
Deploying nodes
You can change the provmethod of the node to the new image_name if different:
chdef <noderange> provmethod=<image_name>
nodeset <noderange> osimage=<image_name>
and the node is ready to deploy.
Appendix
You can only export/import one image at a time. Each tarball will have the following simple structure:
manifest.xml
<files>
extra/ (optional)
manifest.xml
The manifest.xml will be analogous to an autoyast or windows unattend.xml file where it tells xCAT how to store the
items. The following is an example for a diskless cluster:
manifest.xml:
<?xml version="1.0"?>
<xcatimage>
<exlist>/install/custom/netboot/sles/compute1.exlist</exlist>
<extra>
<dest>/install/postscripts</dest>
<src>/install/postscripts/myscript1</src>
</extra>
<imagename>myimage</imagename>
<imagetype>linux</imagetype>
<kernel>/install/netboot/sles12/ppc64le/compute1/kernel</kernel>
<netdrivers>e1000</netdrivers>
<osarch>ppc64le</osarch>
<osname>Linux</osname>
<osvers>sles12</osvers>
<otherpkgdir>/install/post/otherpkgs/sles12/ppc64</otherpkgdir>
<otherpkglist>/install/custom/netboot/sles/compute1.otherpkgs.pkglist</otherpkglist>
154
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
<pkgdir>/install/sles12/ppc64le</pkgdir>
<pkglist>/install/custom/netboot/sles/compute1.pkglist</pkglist>
<postbootscripts>my4,otherpkgs,my3,my4</postbootscripts>
<postinstall>/install/custom/netboot/sles/compute1.postinstall</postinstall>
<postscripts>syslog,remoteshell,my1,configrmcnode,syncfiles,my1,my2</postscripts>
<profile>compute1</profile>
<provmethod>netboot</provmethod>
<ramdisk>/install/netboot/sles12/ppc64le/compute1/initrd-diskless.gz</ramdisk>
<rootimg>/install/netboot/sles12/ppc64le/compute1/rootimg.gz</rootimg>
<rootimgdir>/install/netboot/sles12/ppc64le/compute1</rootimgdir>
<synclists>/install/custom/netboot/sles/compute1.list</synclists>
</xcatimage>
In the above example, we have a directive of where the files came from and what needs to be processed.
Note that even though source destination information is included, all files that are standard will be copied to the
appropriate place that xCAT thinks they should go.
Exported files
The following files will be exported, assuming x is the profile name:
For diskful:
x.pkglist
x.otherpkgs.pkglist
x.tmpl
x.synclist
For diskless:
kernel
initrd.gz
rootimg.gz
x.pkglist
x.otherpkgs.pkglist
x.synclist
x.postinstall
x.exlist
Note: Although the postscripts names can be exported by using the -p flag. The postscripts themselves are not included
in the bundle file by default. The use has to use -e flag to get them included one by one if needed.
Initialize the Compute for Deployment
XCAT use ‘nodeset‘ command to associate a specific image to a node which will be installed with this image.
nodeset <nodename> osimage=<osimage>
There are more attributes of nodeset used for some specific purpose or specific machines, for example:
• runimage: If you would like to run a task after deployment, you can define that task with this attribute.
• runcmd: This instructs the node to boot to the xCAT nbfs environment and proceed to configure BMC for basic
remote access. This causes the IP, netmask, gateway, username, and password to be programmed according to
the configuration table.
1.3. Admin Guide
155
xCAT3 Documentation, Release 2.12
• shell: This instructs the node to boot to the xCAT genesis environment, and present a shell prompt on console.
The node will also be able to be sshed into and have utilities such as wget, tftp, scp, nfs, and cifs. It will have
storage drivers available for many common systems.
Choose such additional attribute of nodeset according to your requirement, if want to get more information about
nodeset, refer to nodeset’s man page.
Start the OS Deployment
Start the deployment involves two key operations. First specify the boot device of the next boot to be network, then
reboot the node:
For Power servers, those two operations can be completed by one command rnetboot:
rnetboot <node>
For x86_64 servers, those two operations need two independent commands.
1. set the next boot device to be from the “network”
rsetboot <node> net
2. Reboot the xSeries server: ::
rpower <node> reset
Statelite Installation
Overview
This document details the design and setup for the statelite solution of xCAT. Statelite is an intermediate mode between
diskful and diskless.
Statelite provides two kinds of efficient and flexible solutions, most of the OS image can be NFS mounted read-only,
or the OS image can be in the ramdisk with tmpfs type. Different from the stateless solution, statelite provides a
configurable list of directories and files that can be read-write. These read-write directories and files can be configured
to either persist or not persist across reboots.
Solutions
There are two solutions: NFSROOT-based and RAMdisk-based.
1. NFSROOT-based(default):
(a) rootfstype in the osimage xCAT data objects is left as blank, or set to nfs, the NFSROOT-base
statelite solution will be enabled.
(b) the ROOTFS is NFS mounted read-only.
2. RAMdisk-based:
(a) rootfstype in the osimage xCAT data objects is set to ramdisk.
(b) one image file will be downloaded when the node is booting up, and the file will be extracted to the
ramdisk, and used as the ROOTFS.
Advantages
Statelite offers the following advantages over xCAT’s stateless (RAMdisk) implementation:
156
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
1. Some files can be made persistent over reboot. This is useful for license files or database servers where some
state is needed. However, you still get the advantage of only having to manage a single image.
2. Changes to hundreds of machines can take place instantly, and automatically, by updating one main image. In most cases, machines do not need to reboot for these changes to take affect. This is only for the
NFSROOT-based solution.
3. Ease of administration by being able to lock down an image. Many parts of the image can be read-only, so no
modifications can transpire without updating the central image.
4. Files can be managed in a hierarchical manner. For example: Suppose you have a machine that is in one lab
in Tokyo and another in London. You could set table values for those machines in the xCAT database to allow
machines to sync from different places based on their attributes. This allows you to have one base image with
multiple sources of file overlay.
5. Ideal for virtualization. In a virtual environment, you may not want a disk image (neither stateless nor stateful)
on every virtual node as it consumes memory and disk. Virtualizing with the statelite approach allows for images
to be smaller, easier to manage, use less disk, less memory, and more flexible.
Disadvantages
However, there’re still several disadvantages, especially for the NFSROOT-based solution.
1. NFS Root requires more network traffic to run as the majority of the disk image runs over NFS. This may depend
on your workload, but can be minimized. Since the bulk of the image is read-only, NFS caching on the server
helps minimize the disk access on the server, and NFS caching on the client helps reduce the network traffic.
2. NFS Root can be complex to set up. As more files are created in different places, there are greater chances
for failures. This flexibility is also one of the great virtues of Statelite. The image can work in nearly any
environment.
Configuration
Statelite configuration is done using the following tables in xCAT:
• litefile
• litetree
• statelite
• policy
• noderes
litefile table
The litefile table specifies the directories and files on the statelite nodes that should be read/write, persistent, or readonly overlay. All other files in the statelite nodes come from the read-only statelite image.
1. The first column in the litefile table is the image name this row applies to. It can be an exact osimage definition
name, an osimage group (set in the groups attribute of osimages), or the keyword ALL.
2. The second column in the litefile table is the full path of the directory or file on the node that you are setting
options for.
3. The third column in the litefile table specifies options for the directory or file:
(a) tmpfs - It provides a file or directory for the node to use when booting, its permission will be the same as
the original version on the server. In most cases, it is read-write; however, on the next statelite boot, the
1.3. Admin Guide
157
xCAT3 Documentation, Release 2.12
original version of the file or directory on the server will be used, it means it is non-persistent. This option
can be performed on files and directories.
(b) rw - Same as above. Its name “rw” does NOT mean it always be read-write, even in most cases it is
read-write. Do not confuse it with the “rw” permission in the file system.
(c) persistent - It provides a mounted file or directory that is copied to the xCAT persistent location and then
over-mounted on the local file or directory. Anything written to that file or directory is preserved. It
means, if the file/directory does not exist at first, it will be copied to the persistent location. Next time the
file/directory in the persistent location will be used. The file/directory will be persistent across reboots. Its
permission will be the same as the original one in the statelite location. It requires the statelite table to be
filled out with a spot for persistent statelite. This option can be performed on files and directories.
(d) con - The contents of the pathname are concatenated to the contents of the existing file. For this directive
the searching in the litetree hierarchy does not stop when the first match is found. All files found in the
hierarchy will be concatenated to the file when found. The permission of the file will be “-rw-r–r–”, which
means it is read-write for the root user, but readonly for the others. It is non-persistent, when the node
reboots, all changes to the file will be lost. It can only be performed on files. Do not use it for one
directory.
(e) ro - The file/directory will be overmounted read-only on the local file/directory. It will be located in the
directory hierarchy specified in the litetree table. Changes made to this file or directory on the server will
be immediately seen in this file/directory on the node. This option requires that the file/directory to be
mounted must be available in one of the entries in the litetree table. This option can be performed on files
and directories.
(f) tmpfs,rw - Only for compatibility it is used as the default option if you leave the options column blank.
It has the same semantics with the link option, so when adding new items into the _litefile table, the link
option is recommended.
(g) link - It provides one file/directory for the node to use when booting, it is copied from the server, and will
be placed in tmpfs on the booted node. In the local file system of the booted node, it is one symbolic link
to one file/directory in tmpfs. And the permission of the symbolic link is “lrwxrwxrwx”, which is not the
real permission of the file/directory on the node. So for some application sensitive to file permissions, it
will be one issue to use “link” as its option, for example, “/root/.ssh/”, which is used for SSH, should NOT
use “link” as its option. It is non-persistent, when the node is rebooted, all changes to the file/directory
will be lost. This option can be performed on files and directories.
(h) link,ro - The file is readonly, and will be placed in tmpfs on the booted node. In the local file system of
the booted node, it is one symbolic link to the tmpfs. It is non-persistent, when the node is rebooted, all
changes to the file/directory will be lost. This option requires that the file/directory to be mounted must be
available in one of the entries in the litetree table. The option can be performed on files and directories.
(i) link,con - Similar to the “con” option. All the files found in the litetree hierarchy will be concatenated to
the file when found. The final file will be put to the tmpfs on the booted node. In the local file system of
the booted node, it is one symbolic link to the file/directory in tmpfs. It is non-persistent, when the node is
rebooted, all changes to the file will be lost. The option can only be performed on files.
(j) link,persistent - It provides a mounted file or directory that is copied to the xCAT persistent location and
then over-mounted to the tmpfs on the booted node, and finally the symbolic link in the local file system
will be linked to the over-mounted tmpfs file/directory on the booted node. The file/directory will be
persistent across reboots. The permission of the file/directory where the symbolic link points to will be the
same as the original one in the statelite location. It requires the statelite table to be filled out with a spot
for persistent statelite. The option can be performed on files and directories.
(k) localdisk - The file or directory will be stored in the local disk of the statelite node. Refer to the section To
enable the localdisk option to enable the ‘localdisk’ support.
Currently, xCAT does not handle the relative links very well. The relative links are commonly used by the system
158
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
libraries, for example, under /lib/ directory, there will be one relative link matching one .so file. So, when you
add one relative link to the litefile table (Not recommend), make sure the real file also be included, or put its directory
name into the litefile table.
Note: It is recommended that you specify at least the entries listed below in the litefile table, because most of these
files need to be writeable for the node to boot up successfully. When any changes are made to their options, make sure
they won’t affect the whole system.
Sample Data for Redhat statelite setup
This is the minimal list of files needed, you can add additional files to the litefile table.
#image,file,options,comments,disable
"ALL","/etc/adjtime","tmpfs",,
"ALL","/etc/securetty","tmpfs",,
"ALL","/etc/lvm/","tmpfs",,
"ALL","/etc/ntp.conf","tmpfs",,
"ALL","/etc/rsyslog.conf","tmpfs",,
"ALL","/etc/rsyslog.conf.XCATORIG","tmpfs",,
"ALL","/etc/udev/","tmpfs",,
"ALL","/etc/ntp.conf.predhclient","tmpfs",,
"ALL","/etc/resolv.conf","tmpfs",,
"ALL","/etc/yp.conf","tmpfs",,
"ALL","/etc/resolv.conf.predhclient","tmpfs",,
"ALL","/etc/sysconfig/","tmpfs",,
"ALL","/etc/ssh/","tmpfs",,
"ALL","/etc/inittab","tmpfs",,
"ALL","/tmp/","tmpfs",,
"ALL","/var/","tmpfs",,
"ALL","/opt/xcat/","tmpfs",,
"ALL","/xcatpost/","tmpfs",,
"ALL","/etc/systemd/system/multi-user.target.wants/","tmpfs",,
"ALL","/root/.ssh/","tmpfs",,
"ALL","/etc/rc3.d/","tmpfs",,
"ALL","/etc/rc2.d/","tmpfs",,
"ALL","/etc/rc4.d/","tmpfs",,
"ALL","/etc/rc5.d/","tmpfs",,
Sample Data for SLES statelite setup
This is the minimal list of files needed, you can add additional files to the litefile table.
#image,file,options,comments,disable
"ALL","/etc/lvm/","tmpfs",,
"ALL","/etc/ntp.conf","tmpfs",,
"ALL","/etc/ntp.conf.org","tmpfs",,
"ALL","/etc/resolv.conf","tmpfs",,
"ALL","/etc/hostname","tmpfs",,
"ALL","/etc/ssh/","tmpfs",,
"ALL","/etc/sysconfig/","tmpfs",,
"ALL","/etc/syslog-ng/","tmpfs",,
"ALL","/etc/inittab","tmpfs",,
"ALL","/tmp/","tmpfs",,
"ALL","/etc/init.d/rc3.d/","tmpfs",,
"ALL","/etc/init.d/rc5.d/","tmpfs",,
1.3. Admin Guide
159
xCAT3 Documentation, Release 2.12
"ALL","/var/","tmpfs",,
"ALL","/etc/yp.conf","tmpfs",,
"ALL","/etc/fstab","tmpfs",,
"ALL","/opt/xcat/","tmpfs",,
"ALL","/xcatpost/","tmpfs",,
"ALL","/root/.ssh/","tmpfs",,
"ALL","/etc/systemd/system/","tmpfs",,
"ALL","/etc/adjtime","tmpfs",,
litetree table
The litetree table controls where the initial content of the files in the litefile table come from, and the long term
content of the ro files. When a node boots up in statelite mode, it will by default copy all of its tmpfs files from the
.default directory of the root image, for example /install/netboot/rhels7.3/x86_64/compute/
rootimg/.default, so there is not required to set up a litetree table. If you decide that you want some of the files
pulled from different locations that are different per node, you can use this table.
You can choose to use the defaults and not set up a litetree table.
statelite table
The statelite table specifies location on an NFS server where a nodes persistent files are stored. This is done by entering
the information into the statelite table.
In the statelite table, the node or nodegroups in the table must be unique; that is a node or group should appear only
once in the first column table. This makes sure that only one statelite image can be assigned to a node. An example
would be:
"compute",,"<nfssvr_ip>:/gpfs/state",,
Any nodes in the compute node group will have their state stored in the /gpfs/state directory on the machine
with <nfssvr_ip> as its IP address.
When the node boots up, then the value of the statemnt attribute will be mounted to /.statelite/
persistent.
The code will then create the following subdirectory /.statelite/persistent/
<nodename>, if there are persistent files that have been added in the litefile table. This directory will be the root of
the image for this node’s persistent files. By default, xCAT will do a hard NFS mount of the directory. You can change
the mount options by setting the mntopts attribute in the statelite table.
Also, to set the statemnt attribute, you can use variables from xCAT database. It follows the same grammar as the
litetree table. For example:
#node,image,statemnt,mntopts,comments,disable
"cn1",,"$noderes.nfsserver:/lite/state/$nodetype.profile","soft,timeo=30",,
Note: Do not name your persistent storage directory with the node name, as the node name will be added in the
directory automatically. If you do, then a directory named /state/cn1 will have its state tree inside /state/
cn1/cn1.
Policy
Ensure policies are set up correctly in the Policy Table. When a node boots up, it queries the xCAT database to get the
litefile and litetree table information. In order for this to work, the commands (of the same name) must be set in the
160
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
policy table to allow nodes to request it. This should happen automatically when xCAT is installed, but you may want
to verify that the following lines are in the policy table:
chdef -t policy -o 4.7 commands=litefile rule=allow
chdef -t policy -o 4.8 commands=litetree rule=allow
noderes
noderes.nfsserver attribute can be set for the NFSroot server. If this is not set, then the default is the Management Node.
noderes.nfsdir can be set. If this is not set, the default is /install
Provision statelite
Show current provisioning method
To determine the current provisioning method of your node, execute:
lsdef <noderange> -i provmethod
Note: syncfiles is not currently supported for statelite nodes.
Generate default statelite image from distoro media
In this example, we are going to create a new compute node osimage for rhels7.3 on ppc64le. We will set up a
test directory structure that we can use to create our image. Later we can just move that into production.
Use the copycds command to copy the appropriate iso image into the /install directory for xCAT. The copycds
commands will copy the contents to /install/rhels7.3/<arch>. For example:
copycds RHEL-7.3-20161019.0-Server-ppc64le-dvd1.iso
The contents are copied into /install/rhels7.3/ppc64le/
The configuration files pointed to by the attributes are the defaults shipped with xCAT. We will want to copy them to
the /install directory, in our example the /install/test directory and modify them as needed.
Statelite Directory Structure
Each statelite image will have the following directories:
/.statelite/tmpfs/
/.default/
/etc/init.d/statelite
All files with link options, which are symbolic links, will link to /.statelite/tmpfs.
tmpfs files that are persistent link to /.statelite/persistent/<nodename>/, /.statelite/
persistent/<nodename> is the directory where the node’s individual storage will be mounted to.
/.default is where default files will be copied to from the image to tmpfs if the files are not found in the litetree
hierarchy.
1.3. Admin Guide
161
xCAT3 Documentation, Release 2.12
Customize your statelite osimage
Create the osimage definition
Setup your osimage/linuximage tables with new test image name, osvers,osarch, and paths to all the files for building and installing the node. So using the above generated rhels7.3-ppc64le-statelite-compute as an
example, I am going to create my own image. The value for the provisioning method attribute is osimage in my
example.:
mkdef rhels7.3-custom-statelite -u profile=compute provmethod=statelite
Check your setup:
lsdef -t osimage rhels7.3-custom-statelite
Customize the paths to your pkglist, syncfile, etc to the osimage definition, that you require. Note, if you
modify the files on the /opt/xcat/share/... path then copy to the appropriate /install/custom/...
path. Remember all files must be under /install if using hierarchy (service nodes).
Copy the sample *list files and modify as needed:
mkdir -p /install/test/netboot/rh
cp -p /opt/xcat/share/xcat/netboot/rh/compute.rhels7.ppc64le.pkglist \
/install/test/netboot/rh/compute.rhels7.ppc64le.pkglist
cp -p /opt/xcat/share/xcat/netboot/rh/compute.exlist \
/install/test/netboot/rh/compute.exlist
chdef -t osimage -o rhels7.3-custom-statelite \
pkgdir=/install/rhels7.3/ppc64le \
pkglist=/install/test/netboot/rh/compute.rhels7.ppc64le.pkglist \
exlist=/install/test/netboot/rh/compute.exlist \
rootimgdir=/install/test/netboot/rh/ppc64le/compute
Setup pkglists
In the above example, you have defined your pkglist to be in /install/test/netboot/rh/compute.
rhels7.ppc64le.pkglist.
Edit compute.rhels7.ppc64le.pkglist and compute.exlist as needed.
vi /install/test/netboot/rh/compute.rhels7.ppc64le.pkglist
vi /install/test/netboot/rh/compute.exlist
Make sure nothing is excluded in compute.exlist that you need.
Install other specific packages
Make the directory to hold additional rpms to install on the compute node.
mkdir -p /install/test/post/otherpkgs/rh/ppc64le
Now copy all the additional OS rpms you want to install into /install/test/post/otherpkgs/rh/
ppc64le.
162
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
At first you need to create one text file which contains the complete list of files to include in the repository. The name
of the text file is rpms.list and must be in /install/test/post/otherpkgs/rh/ppc64le directory. Create
rpms.list:
cd /install/test/post/otherpkgs/rh/ppc64le
ls *.rpm > rpms.list
Then, run the following command to create the repodata for the newly-added packages:
createrepo -i rpms.list /install/test/post/otherpkgs/rh/ppc64le
The createrepo command with -i rpms.list option will create the repository for the rpm packages listed in the
rpms.list file. It won’t destroy or affect the rpm packages that are in the same directory, but have been included into
another repository.
Or, if you create a sub-directory to contain the rpm packages, for example, named other in /install/test/post/
otherpkgs/rh/ppc64le. Run the following command to create repodata for the directory /install/test/
post/otherpkgs/rh/ppc64le.
createrepo /install/post/otherpkgs/<os>/<arch>/**other**
Note: Replace other with your real directory name.
Define the location of of your otherpkgs in your osimage:
chdef -t osimage -o rhels7.3-custom-statelite \
otherpkgdir=/install/test/post/otherpkgs/rh/ppc64le \
otherpkglist=/install/test/netboot/rh/compute.otherpkgs.pkglist
There are examples under /opt/xcat/share/xcat/netboot/<platform> of typical *otherpkgs.
pkglist files that can used as an example of the format.
Set up Post scripts for statelite
The rules to create post install scripts for statelite image is the same as the rules for stateless/diskless install images.
There’re two kinds of postscripts for statelite (also for stateless/diskless).
The first kind of postscript is executed at genimage time, it is executed again the image itself on the MN . It was setup
in The postinstall file section before the image was generated.
The second kind of postscript is the script that runs on the node during node deployment time. During init.d timeframe,
/etc/init.d/gettyset calls /opt/xcat/xcatdsklspost that is in the image. This script uses wget to
get all the postscripts under mn:/install/postscripts and copy them to the /xcatpost directory on the
node. It uses openssl or stunnel to connect to the xcatd on the mn to get all the postscript names for the node from the
postscripts table. It then runs the postscripts for the node.
Setting up postinstall files (optional)
Using postinstall files is optional. There are some examples shipped in /opt/xcat/share/xcat/netboot/
<platform>.
If you define a postinstall file to be used by genimage, then
chdef -t osimage -o rhels7.3-custom-statelite postinstall=<your postinstall file path>
˓→.
1.3. Admin Guide
163
xCAT3 Documentation, Release 2.12
Generate the image
Run the following command to generate the image based on your osimage named rhels7.
3-custom-statelite. Adjust your genimage parameters to your architecture and network settings. See
man genimage.
genimage rhels7.3-custom-statelite
The genimage will create a default /etc/fstab in the image, if you want to change the defaults, on the management
node, edit fstab in the image:
cd /install/netboot/rhels7/ppc64le/compute/rootimg/etc
cp fstab fstab.ORIG
vi fstab
Note: adding /tmp and /var/tmp to /etc/fstab is optional, most installations can simply use /. It was
documented her to show that you can restrict the size of filesystems, if you need to. The indicated values are just and
example, and you may need much bigger filessystems, if running applications like OpenMPI.
Pack the image
Execute liteimg
liteimg rhels7.3-custom-statelite
Boot the statelite node
Execute rinstall
rinstall node1 osimage=rhels7.3-custom-statelite
Switch to the RAMdisk based solution
It is optional, if you want to use RAMdisk-based solution, follow this section.
Set rootfstype
If you want the node to boot with a RAMdisk-based image instead of the NFS-base image, set the rootfstype attribute
for the osimage to ramdisk. For example:
chdef -t osimage -o rhels7.3-custom-statelite rootfstype=ramdisk
Run liteimg command
The liteimg command will modify your statelite image (the image that genimage just created) by creating a series
of links. Once you are satisfied with your image contains what you want it to, run liteimg <osimagename>:
liteimg rhels7.3-custom-statelite
164
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
For files with link options, the liteimg command creates two levels of indirection, so that files can be modified
while in their image state as well as during runtime. For example, a file like $imageroot/etc/ntp.conf with
link option in the litefile table, will have the following operations done to it:
In our case $imageroot is /install/netboot/rhels5.3/x86_64/compute/rootimg
The liteimg script, for example, does the following to create the two levels of indirection.
mkdir -p $imageroot/.default/etc
mkdir -p $imageroot/.statelite/tmpfs/etc
mv $imgroot/etc/ntp.conf $imgroot/.default/etc
cd $imgroot/.statelite/tmpfs/etc
ln -sf ../../../.default/etc/ntp.conf .
cd $imgroot/etc
ln -sf ../.statelite/tmpfs/etc/ntp.conf .
When finished, the original file will reside in $imgroot/.default/etc/ntp.conf. $imgroot/etc/ntp.
conf will link to $imgroot/.statelite/tmpfs/etc/ntp.conf which will in turn link to $imgroot/.
default/etc/ntp.conf.
But for files without link options, the liteimg command only creates clones in $imageroot/.default/ directory, when the node is booting up, the mount command with --bind option will get the corresponding files from the
litetree places or .default directory to the sysroot directory.
Note: If you make any changes to your litefile table after running liteimg then you will need to rerun liteimg
again. This is because files and directories need to have the two levels of redirects created.
Boot the statelite node
Make sure you have set up all the attributes in your node definitions correctly following the node installation instructions corresponding to your hardware:
You can now deploy the node by running the following commmands:
rinstall <noderange>
You can then use rcons or wcons to watch the node boot up.
Adding/updating software and files for the running nodes
Make changes to the files which configured in the litefile table
During the preparation or booting of node against statelite mode, there are specific processes to handle the files which
configured in the litefile table. The following operations need to be done after made changes to the statelite files.
1. Run liteimg against the osimage and reboot the node : Added, removed or changed the entries in the litefile
table.
2. Reboot the node :
• Changed the location directory in the litetree table.
• Changed the location directory in the statelite table.
• Changed, removed the original files in the location of litetree or statelite table.
Note: Thing should not do:
• When there are node running on the nfs-based statelite osimage, do not run the packimage against this osimage.
1.3. Admin Guide
165
xCAT3 Documentation, Release 2.12
Make changes to the common files
Because most of system files for the nodes are NFS mounted on the Management Node with read-only option,
installing or updating software and files should be done to the image. The image is located under /install/
netboot/<os>/<arch>/<profile>/rootimg directory.
To install or update an rpm, do the following:
• Install the rpm package into rootimg
rpm --root /install/netboot/<os>/<arch>/<profile>/rootimg -ivh rpm_name
• Restart the software application on the nodes
xdsh <noderange> <restart_this_software_command>
It is recommended to follow the section (Adding third party softeware) to add the new rpm to the otherpkgs.pkglist
file, so that the rpm will get installed into the new image next time the image is rebuilt.
Note: The newly added rpms are not shown when running rpm -qa on the nodes although the rpm is installed. It
will shown next time the node is rebooted.
To create or update a file for the nodes, just modify the file in the image and restart any application that uses the file.
For the ramdisk-based node, you need to reboot the node to take the changes.
Hierarchy Support
In the statelite environment, the service node needs to provide NFS service for the compute node with
statelite, the service nodes must to be setup with diskfull installation.
Setup the diskfull service node
1. Setup one diskfull service node at first.
2. Since statelite is a kind of NFS-hybrid method, you should remove the installloc attribute in the site table. This
makes sure that the service node does not mount the /install directory from the management node on the
service node.
Generate the statelite image
To generate the statelite image for your own profile follow instructions in Customize your statelite osimage.
NOTE: if the NFS directories defined in the litetree table are on the service node, it is better to setup the NFS directories
in the service node following the chapter.
Sync the /install directory
The command prsync is used to sync the /install directory to the service nodes.
Run the following:
cd /
prsync install <sn>:/
166
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
<sn> is the hostname of the service node you defined.
Since the prsync command will sync all the contents in the /install directory to the service nodes, the first time
will take a long time. But after the first time, it will take very short time to sync.
NOTE: if you make any changes in the /install directory on the management node, and the changes can affect the
statelite image, you need to sync the /install directory to the service node again.
Set the boot state to statelite
You can now deploy the node:
rinstall <noderange> osimage=rhel5.3-x86_64-statelite-compute
This will create the necessary files in /tftpboot for the node to boot correctly.
Advanced features
Both directory and its child items coexist in litefile table
As described in the above chapters, we can add the files/directories to litefile table. Sometimes, it is necessary to put
one directory and also its child item(s) into the litefile table. Due to the implementation of the statelite on Linux, some
scenarios works, but some doesn’t work.
Here are some examples of both directory and its child items coexisting:
Both the parent directory and the child file coexist:
"ALL","/root/testblank/",,,
"ALL","/root/testblank/tempfschild","tempfs",,
One more complex example:
"ALL","/root/",,,
"ALL","/root/testblank/tempfschild","tempfs",,
Another more complex example, but we don’t intend to support such one scenario:
"ALL","/root/",,,
"ALL","/root/testblank/",,,
"ALL","/root/testblank/tempfschild","tempfs",,
For example, in scenario 1, the parent is /root/testblank/, and the child is /root/testblank/
tempfschild. In scenario 2, the parent is /root/, and the child is /root/testblank/tempfschild.
In order to describe the hierarchy scenarios we can use , P to denote parent, and C to denote child.
1.3. Admin Guide
167
xCAT3 Documentation, Release 2.12
Option
P:tmpfs
P:tmpfs C:persistent
P:persistent C:tmpfs
P:persistent C:persistent
P:ro C:any
P:tmpfs C:ro
P:tmpfs C:con
P:link C:link
P: link C: link, persistent
P:link, persistent
C: link
P:link, persistent
C:link, persistent
P:link C:link,ro
P:link C:link,con
P:link, persistent
168
C:link,ro
P:link, persistent
C:link,con
Example
Remarks
“ALL”,”/root/testblank/”„,
Both the parent and the child are
“ALL”,”/root/testblanktempfschild”,”tempfs”„
mounted to tmpfs on the booted
node following their respective options. Only the parent are mounted
to the local file system.
“ALL”,”/root/testblank/”„,
Both parent and child are mounted
“ALL”,”/root/testblank/testpersfile”,”persistent”„
to tmpfs on the booted node following their respective options. Only
the parent is mounted to the local
file
“ALL”,”/root/testblank/”,”persistent”„ Not permitted now. But plan to sup“ALL”,”/root/testblank/tempfschild”„, port it.
“ALL”,”/root/testblank/”,”persistent”„ Both parent and child are mounted
“ALL”,”/root/testblank/testpersfile”,”persistent”„
to tmpfs on the booted node following their respective options. Only
the parent is mounted to local file
system.
Not permitted
Both parent and child are mounted
to tmpfs on the booted node following their respective options. Only
the parent is mounted to local file
system.
Both parent and child are mounted
to tmpfs on the booted node following their respective options. Only
the parent is mounted to local file
system.
“ALL”,”/root/testlink/”,”link”„
Both parent and child are created in
“ALL”,”/root/testlink/testlinkchild”,”link”„
tmpfs on the booted node following
their respective options; there’s only
one symbolic link of the parent is
created in the local file system.
“ALL”,”/root/testlinkpers/”,”link”„
Both parent and child are created in
“ALL”,”/root/testlink/testlinkchild”„ tmpfs on the booted node following
“link,persistent”
their respective options; there’s only
one symbolic link of the parent is
created in the local file system.
“ALL”,”/root/testlinkpers/”,”link,persistent”„
NOT permitted
“ALL”,”/root/testlink/testlinkchild”,”link”
“ALL”,”/root/testlinkpers/”,”link,persistent”„
Both parent and child are created in
“ALL”,”/root/testlink
tmpfs on the booted node following
“link,persistent” way; there’s only
one symbolic link of the parent is
created in the local file system.
“ALL”,”/root/testlink/”,”link”„
Both parent and child are created in
“ALL”,”/root/testlink/testlinkro”,”link,ro”„
tmpfs on the booted node, there’s
only one symbolic link of the parent
is created in the local file system.
“ALL”,”/root/testlink/”,”link”„
Both parent and child are created in
“ALL”,”/root/testlink/testlinkconchild”,”link,con”„
tmpfs on the booted node, there’s
only one symbolic link of the parent
in the local file system.
NOT Permitted
Chapter 1. Table of Contents
NOT Permitted
xCAT3 Documentation, Release 2.12
litetree table
The litetree table controls where the initial content of the files in the litefile table come from, and the long term content
of the ro files. When a node boots up in statelite mode, it will by default copy all of its tmpfs files from the /.
default directory of the root image, so there is not requirement to setup a litetree table. If you decide that you want
some of the files pulled from different locations that are different per node, you can use this table.
See litetree man page for description of attributes.
For example, a user may have two directories with a different /etc/motd that should be used for nodes in two
locations:
10.0.0.1:/syncdirs/newyork-590Madison/rhels5.4/x86_64/compute/etc/motd
10.0.0.1:/syncdirs/shanghai-11foo/rhels5.4/x86_64/compute/etc/motd
You can specify this in one row in the litetree table:
1,,10.0.0.1:/syncdirs/$nodepos.room/$nodetype.os/$nodetype.arch/$nodetype.profile
When each statelite node boots, the variables in the litetree table will be substituted with the values for that node to
locate the correct directory to use. Assuming that /etc/motd was specified in the litefile table, it will be searched
for in all of the directories specified in the litetree table and found in this one.
You may also want to look by default into directories containing the node name first:
$noderes.nfsserver:/syncdirs/$node
The litetree prioritizes where node files are found. The first field is the priority. The second field is the image name
(ALL for all images) and the final field is the mount point.
Our example is as follows:
1,,$noderes.nfsserver:/statelite/$node
2,,cnfs:/gpfs/dallas/
The two directories /statelite/$node on the node’s $noderes.nfsserver and the /gpfs/dallas on the node
cnfs contain root tree structures that are sparsely populated with files that we want to place in those nodes. If files are
not found in the first directory, it goes to the next directory. If none of the files can be found in the litetree hierarchy,
then they are searched for in /.default on the local image.
Installing a new Kernel in the statelite image
Obtain you new kernel and kernel modules on the MN, for example here we have a new SLES kernel.
1. Copy the kernel into /boot :
cp **vmlinux-2.6.32.10-0.5-ppc64**/boot
2. Copy the kernel modules into /lib/modules/<new kernel directory>
/lib/modules # ls -al
total 16
drwxr-xr-x 4 root root 4096 Apr 19 10:39 .
drwxr-xr-x 17 root root 4096 Apr 13 08:39 ..
drwxr-xr-x 3 root root 4096 Apr 13 08:51 2.6.32.10-0.4-ppc64
**drwxr-xr-x 4 root root 4096 Apr 19 10:12 2.6.32.10-0.5-ppc64**
1.3. Admin Guide
169
xCAT3 Documentation, Release 2.12
3. Run genimage to update the statelite image with the new kernel
genimage -k 2.6.32.10-0.5-ppc64 <osimage_name>
4. Then after a nodeset command and netbooti, shows the new kernel:
uname -a
Enabling the localdisk Option
Note: You can skip this section if not using the localdisk option in your litefile table.
Several things need to be done to enable the ‘localdisk’ support:
Define how to partition the local disk
When a node is deployed, the local hard disk needs to be partitioned and formatted before it can be used. This section
explains how provide a configuration file that tells xCAT to partition a local disk and make it ready to use for the
directories listed in the litefile table with the localdisk option.
The configuration file needs to be specified in the partitionfile attribute of the osimage definition. The configuration file includes several parts:
• Global parameters to control enabling or disabling the function
• [disk] part to control the partitioning of the disk
• [localspace] part to control which partition will be used to store the localdisk directories listed in the litefile table
• [swapspace] part to control the enablement of the swap space for the node.
An example localdisk configuration file:
enable=yes
enablepart=no
[disk]
dev=/dev/sdb
clear=yes
parts=100M-200M,1G-2G
[disk]
dev=/dev/sda
clear=yes
parts=10,20,30
[disk]
dev=/dev/sdc
clear=yes
parts=10,20,30
[localspace]
dev=/dev/sda1
fstype=ext3
[swapspace]
dev=/dev/sda2
170
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
The two global parameters enable and enablepart can be used to control the enabling/disabling of the functions:
• enable: The localdisk feature only works when enable is set to yes. If it is set to no, the localdisk configuration
will not be run.
• enablepart: The partition action (refer to the [disk] section) will be run only when enablepart=yes.
The [disk] section is used to configure how to partition a hard disk:
• dev: The path of the device file.
• clear: If set to yes it will clear all the existing partitions on this disk.
• fstype: The file system type for the new created partitions. ext3 is the default value if not set.
• parts: A comma separated list of space ranges, one for each partition that will be created on the device. The
valid format for each space range is <startpoint>-<endpoint> or <percentage of the disk>.
For example, you could set it to 100M-10G or 50. If you set it to 50, that means 50% of the disk space will be
assigned to that partition.
The [localspace] section is used to specify which partition will be used as local storage for the node.
• dev: The path of the partition.
• fstype: The file system type on the partition.
the [swapspace] section is used to configure the swap space for the statelite node.
• dev: The path of the partition file which will be used as the swap space.
To enable the local disk capability, create the configuration file (for example in /install/custom) and set the
path in the partitionfile attribute for the osimage:
chdef -t osimage partitionfile=/install/custom/cfglocaldisk
Now all nodes that use this osimage (i.e. have their provmethod attribute set to this osimage definition name), will
have its local disk configured.
Configure the files in the litefile table
For the files/directories that you would like xCAT to store on the local disk, add an entry in the litefile table like this:
"ALL","/tmp/","localdisk",,
Note: you do not need to specify the swap space in the litefile table. Just putting it in the partitionfile config file is
enough.
Add an entry in policy table to permit the running of the getpartitioin command from the node
chtab priority=7.1 policy.commands=getpartition policy.rule=allow
If Using the RAMdisk-based Image
If you want to use the local disk option with a RAMdisk-based image, remember to follow the instructions in Switch
to the RAMdisk based solution.
If your reason for using a RAMdisk image is to avoid compute node runtime dependencies on the service node or
management node, then the only entries you should have in the litefile table should be files/dirs that use the localdisk
option.
1.3. Admin Guide
171
xCAT3 Documentation, Release 2.12
Debugging techniques
When a node boots up in statelite mode, there is a script that runs called statelite that is in the root directory
of $imageroot/etc/init.d/statelite. This script is not run as part of the rc scripts, but as part
of the pre-switch root environment. Thus, all the linking is done in this script. There is a set x near the
top of the file. You can uncomment it and see what the script runs. You will then see lots of mkdirs and
links on the console.
You can also set the machine to shell. Just add the word shell on the end of the pxeboot file of the node
in the append line. This will make the init script in the initramfs pause 3 times before doing a switch_root.
When all the files are linked they are logged in /.statelite/statelite.log on the node. You
can get into the node after it has booted and look in the /.statelite directory.
Using Updatenode
Introduction
After initial node deployment, you may need to make changes/updates to your nodes. The updatenode command
is for this purpose. It allows you to add or modify the followings on your nodes:
1. Add additional software
2. Re-run postscripts or run additional postscripts
3. Synchronize new/updated configuration files
4. Update ssh keys and xCAT certificates
Each of these will be explained in the document. The basic way to use updatenode is to set the definition of nodes
on the management node the way you want it and then run updatenode to push those changes out to the actual
nodes. Using options to the command, you can control which of the above categories updatenode pushes out to the
nodes.
Most of what is described in this document applies to stateful and stateless nodes. In addition to the information in
this document, check out the updatenode man page.
Add Additional Software
The packages that will be installed on the node are stored in the packages list files. There are two kinds of package
list files:
1. The package list file contains the names of the packages that come from the os distro. They are stored in .pkglist
file.
2. The other package list file contains the names of the packages that do NOT come from the os distro. They are
stored in .otherpkgs.pkglist file.
Installing Additional OS Distro Packages
For packages from the OS distro, add the new package names (without the version number) in the .pkglist file. If you
have newer updates to some of your operating system packages that you would like to apply to your OS image, you
can place them in another directory, and add that directory to your osimage pkgdir attribute. How to add additional
OS distro packages, go to Install Additional OS Packages for RHEL and SLES
172
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
Note:If the objective node is not installed by xCAT, make sure the correct osimage pkgdir attribute so that you could
get the correct repository data.
Install Additional non-OS Packages
If you have additional packages (packages not in the distro) that you also want installed, make a directory to hold
them, create a list of the packages you want installed, and add that information to the osimage definition. How to add
Additional Other Packages, go to Install Additional Other Packages for RHEL and SLES
Update Nodes
Run the updatenode command to push the new software to the nodes:
updatenode <noderange> -S
The -S flag updates the nodes with all the new or updated packages specified in both .pkglist and .otherpkgs.pkglist.
If you have a configuration script that is necessary to configure the new software, then instead run:
cp myconfigscript /install/postscripts/
chdef -p -t compute postbootscripts=myconfigscript
updatenode <noderange> ospkgs,otherpkgs,myconfigscript
The next time you re-install these nodes, the additional software will be automatically installed.
If you update stateless nodes, you must also do this next step, otherwise the next time you reboot the stateless
nodes, the new software won’t be on the nodes. Run genimage and packimage to install the extra rpms into the image:
genimage <osimage>
packimage <osimage>
Update the delta changes in Sysclone environment
Updatenode can also be used in Sysclone environment to push delta changes to target node. After capturing the delta
changes from the golden client to management node, just run below command to push delta changes to target nodes.
See Sysclone environment related section: Update Nodes Later On for more information.
updatenode <targetnoderange> -S
Rerun Postscripts or Run Additional Postcripts
You can use the updatenode command to perform the following functions after the nodes are up and running:
• Rerun postscripts defined in the postscripts table.
• Run any additional postscript one time.
Go to Using Postscript to see how to configure postscript.
Go to Using Prescript to see how to configure prepostscript.
To rerun all the postscripts for the nodes. (In general, xCAT postscripts are structured such that it is not harmful to run
them multiple times.)
1.3. Admin Guide
173
xCAT3 Documentation, Release 2.12
updatenode <noderange> -P
To rerun just the syslog postscript for the nodes:
updatenode <noderange> -P syslog
To run a list of your own postscripts, make sure the scripts are copied to /install/postscripts directory, then:
updatenode <noderange> -P "script1,script2"
If you need to, you can also pass arguments to your scripts:
updatenode <noderange> -P "script1 p1 p2,script2"
mypostscript template for updatenode
You can customize what attributes you want made available to the postscript, using the shipped mypostscript.tmpl file
Using the mypostscript template.
Synchronize new/updated configuration files
Setting up syncfile
Use instructions in doc: The synclist file.
Syncfiles to the nodes
After compute node is installed, you would like to sync files to the nodes:
updatenode <noderange> -F
With the updatenode command the syncfiles postscript cannot be used to sync files to the nodes.Therefore, if you
run updatenode <noderange> -P syncfiles, nothing will be done. A message will be logged that you
must use updatenode <noderange> -F to sync files.
Update the ssh Keys and Credentials on the Nodes
If after node deployment, the ssh keys or xCAT ssl credentials become corrupted, xCAT provides a way to quickly fix
the keys and credentials on your service and compute nodes:
updatenode <noderange> -K
Note: this option can’t be used with any of the other updatenode options.
Appendix : Debugging Tips
Internally updatenode command uses the xdsh in the following ways:
Linux: xdsh <noderange> -e /install/postscripts/xcatdsklspost -m <server> <scripts&gt>
where <scripts> is a comma separated postscript like ospkgs,otherpkgs etc.
174
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
• wget is used in xcatdsklspost/xcataixpost to get all the postscripts from the <server> to the node. You can check
/tmp/wget.log file on the node to see if wget was successful or not. You need to make sure the /xcatpost directory
has enough space to hold the postscripts.
• A file called /xcatpost/mypostscript (Linux) is created on the node which contains the environmental variables
and scripts to be run. Make sure this file exists and it contains correct info. You can also run this file on the node
manually to debug.
• For ospkgs/otherpkgs, if /install is not mounted on the <server>, it will download all the rpms from the
<server> to the node using wget. Make sure /tmp and /xcatpost have enough space to hold the rpms and check
/tmp/wget.log for errors.
• For ospkgs/otherpkgs, If zypper or yum is installed on the node, it will be used the command to install the rpms.
Make sure to run createrepo on the source directory on the <server> every time a rpm is added or removed.
Otherwise, the rpm command will be used, in this case, make sure all the necessary depended rpms are copied
in the same source directory.
• You can append -x on the first line of ospkgs/otherpkgs to get more debug info.
Parallel Commands
xCAT provides a set of commands that can run common remote commands (ssh, scp, rsh, rcp, rsync,
ping, cons) in parallel on xCAT managed nodes. The xCAT commands will format the output making the results
easier to parse and help administrators manage large clusters.
The following commands are provided:
• pcons - runs a command on the noderange using the out-of-band console
• pping - parallel ping
• ppping - parallel ping between nodes in a cluster
• prsync - parallel rsync
• pscp - parallel remote copy ( supports scp and not hierarchy)
• psh - parallel remote shell ( supports ssh and not hierarchy)
• pasu - parallel ASU utility
• xdcp - concurrently copies files to and from multiple nodes. ( scp/rcp and hierarchy)
• xdsh - concurrently runs commands on multiple nodes. ( supports ssh/rsh and hierarchy)
• xdshbak - formats the output of the xdsh command
• xcoll - Formats command output of the psh, xdsh, rinv command
Examples for xdsh
• To set up the SSH keys for root on node1, run as root:
xdsh node1 -K
• To run the ps -ef command on node targets node1 and node2, enter:
xdsh node1,node2 "ps -ef"
• To run the ps command on node targets node1 and run the remote command with the -v and -t flag, enter:
1.3. Admin Guide
175
xCAT3 Documentation, Release 2.12
xdsh node1,node2 -o"-v -t" ps =item *
• To execute the commands contained in myfile in the XCAT context on several node targets, with a fanout of 1,
enter:
xdsh node1,node2 -f 1 -e myfile
• To run the ps command on node1 and ignore all the dsh environment variable except the DSH_NODE_OPTS,
enter:
xdsh node1 -X `DSH_NODE_OPTS' ps
• To run on Linux, the xdsh command dpkg -l| grep vim on the node ubuntu diskless image, enter:
xdsh -i /install/netboot/ubuntu14.04.2/ppc64el/compute/rootimg "dpkg -l|grep vim"
• To run xdsh with the non-root userid “user1” that has been setup as an xCAT userid and with sudo on node1
and node2 to run as root, do the following, see Granting users xCAT privileges:
xdsh node1,node2 --sudo -l user1 "cat /etc/passwd"
Examples for xdcp
• To copy the /etc/hosts file from all nodes in the cluster to the /tmp/hosts.dir directory on the local host, enter:
xdcp all -P /etc/hosts /tmp/hosts.dir
A suffix specifying the name of the target is appended to each file name. The contents of the /tmp/hosts.dir
directory are similar to:
hosts._node1
hosts._node2
hosts._node3
hosts._node4
hosts._node5
hosts._node6
hosts._node7
hosts._node8
• To copy /localnode/smallfile and /tmp/bigfile to /tmp on node1 using rsync and input -t flag to rsync, enter:
xdcp node1 -r /usr/bin/rsync -o "-t" /localnode/smallfile /tmp/bigfile /tmp
• To copy the /etc/hosts file from the local host to all the nodes in the cluster, enter:
xdcp all /etc/hosts /etc/hosts
• To rsync the /etc/hosts file to your compute nodes:
Create a rsync file /tmp/myrsync, with this line:
/etc/hosts -> /etc/hosts
or
/etc/hosts -> /etc/ (last / is required)
Run:
xdcp compute -F /tmp/myrsync
176
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
• To rsync the /etc/file1 and file2 to your compute nodes and rename to filex and filey:
Create a rsync file /tmp/myrsync, with these line:
/etc/file1 -> /etc/filex
/etc/file2 -> /etc/filey
Run:
xdcp compute -F /tmp/myrsync to update the Compute Nodes
• To rsync files in the Linux image at /install/netboot/ubuntu14.04.2/ppc64el/compute/rootimg on the MN:
Create a rsync file /tmp/myrsync, with this line:
/etc/hosts /etc/passwd -> /etc
Run:
xdcp -i /install/netboot/ubuntu14.04.2/ppc64el/compute/rootimg -F /tmp/myrsync
Virtual Machines
xCAT supports the following virtualization infrastructures:
Kernel-based Virtual Machine (KVM): A full virtualization solution for Enterprise Linux distributions, known as
the de facto open source virtualization mechanism and currently used by many software companies.
IBM PowerKVM: A product that leverages the Power resilience and performance with the openness of KVM, which
provides several advantages:
• Higher workload consolidation with processors overcommitment and memory sharing
• Dynamic addition and removal of virtual devices
• Microthreading scheduling granularity
• Integration with IBM PowerVC and OpenStack
• Simplified management using open source software
• Avoids vendor lock-in
• Uses POWER8 hardware features, such as SMT8 and microthreading
The xCAT based KVM solution offers users the ability to:
• provision the hypervisor on bare metal nodes
• provision virtual machines with the any OS supported in xCAT
• migrate virtual machines to different hosts
• install copy on write instances of virtual machines
• clone virtual machines
This section introduces the steps of management node preparation, hypervisor setup and virtual machine management,
and presents some typical problems and solutions on xCAT kvm support.
1.3. Admin Guide
177
xCAT3 Documentation, Release 2.12
Set Up the Management Node for KVM
Install the kvm related packages
Additional packages need to be installed on the management node for kvm support.
Please make sure the following packages have been installed on the management node, if not, install them manually.
perl-Sys-Virt
Set Up the kvm storage directory on the management node(optional)
It is a recommended configuration to create a shared file system for virtual machines hosting. The shared file system,
usually on a SAN, NAS or GPFS, is shared among KVM hypervisors, which simplifies VM migration from one
hypervisor to another with xCAT.
The easiest shared file system is /install directory on the management node, it can be shared among hypervisors
via NFS. Please refer to the following steps :
• Create a directory to store the virtual disk files
mkdir -p /install/vms
• export the storage directory
echo "/install/vms *(rw,no_root_squash,sync,fsid=0)" >> /etc/exports
exportfs -r
Note: make sure the root permission is turned on for nfs clients (i.e. use the no_root_squash option). Otherwise,
the virtual disk file can not work.
Install and Configure Hypervisor
Provision Hypervisor
[PowerKVM]
Obtain a PowerKVM ISO and create PowerKVM osimages with it:
copycds ibm-powerkvm-3.1.0.0-39.0-ppc64le-gold-201511041419.iso
The following PowerKVM osimage will be created
# lsdef -t osimage -o pkvm3.1-ppc64le-install-compute
Object name: pkvm3.1-ppc64le-install-compute
imagetype=linux
osarch=ppc64le
osdistroname=pkvm3.1-ppc64le
osname=Linux
osvers=pkvm3.1
otherpkgdir=/install/post/otherpkgs/pkvm3.1/ppc64le
pkgdir=/install/pkvm3.1/ppc64le
profile=compute
provmethod=install
template=/opt/xcat/share/xcat/install/pkvm/compute.pkvm3.ppc64le.
˓→tmpl
178
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
[RHEV]
Red Hat Virtualization (formally known as RHEV or Red Hat Enterprise Virtualization) is a
virtualization solution provided by Red Hat.
At the time of this writing there is no RHEV-H prebuilt hypervisor image on Power LE. The
method for creating a Red Hat Hypervisor on Power LE is to first install RHEL and apply the
KVM support on top with the provided RPMs.
Obtain and download the RHEV RPM packages from the Red Hat download site.
• Management-Agent-Power-7
• Power_Tools-7
In the following example, the RPMs are downloaded to /install/post/otherpkgs/
rhels7.3/ppc64le/RHEV4/4.0-GA
• Create a yum repository for the downloaded RPMs
createrepo /install/post/otherpkgs/rhels7.3/ppc64le/RHEV4/4.0-GA
• Create new osimage definition based on an existing RHEL7 osimage definition
mkdef -t osimage -o rhels7.3-ppc64le-RHEV4-install-compute \
--template rhels7.3-ppc64le-install-compute
• Modify otherpkgdir attribute to point to the package directory with downloaded
RPMs
chdef -t osimage rhels7.3-ppc64le-RHEV4-install-compute \
otherpkgdir=/install/post/otherpkgs/rhels7.3/ppc64le/RHEV4/4.0˓→GA
• Create a new package list file /install/custom/rhels7.3/ppc64le/rhelv4.
pkglist to include necessary packages provided from the OS.
#INCLUDE:/opt/xcat/share/xcat/install/rh/compute.rhels7.pkglist#
libvirt
screen
bridge-utils
• Modify pkglist attribute to point to the package list file from the step above
chdef -t osimage rhels7.3-snap3-ppc64le-RHEV4-install-compute \
pkglist=/install/custom/rhels7.3/ppc64le/rhelv4.pkglist
• Create a new package list file /install/custom/rhels7.3/ppc64le/rhev4.
otherpkgs.pkglist to list required packages
qemu-kvm-rhev
qemu-kvm-tools-rhev
virt-manager-common
virt-install
• Modify otherpkglist attribute to point to the package list file from the step above
1.3. Admin Guide
179
xCAT3 Documentation, Release 2.12
chdef -t osimage rhels7.3-snap3-ppc64le-RHEV4-install-compute \
otherpkglist=/install/custom/rhels7.3/ppc64le/rhev4.otherpkgs.
˓→pkglist
• The RHEV osimage should look similar to:
Object name: rhels7.3-ppc64le-RHEV4-install-compute
imagetype=linux
osarch=ppc64le
osdistroname=rhels7.3-ppc64le
osname=Linux
osvers=rhels7.3
otherpkgdir=/install/post/otherpkgs/rhels7.3/ppc64le/RHEV4/4.
˓→0-GA
otherpkglist=/install/custom/rhels7.3/ppc64le/rhev4.otherpkgs.
˓→pkglist
pkgdir=/install/rhels7.3/ppc64le
pkglist=/install/custom/rhels7.3/ppc64le/rhelv4.pkglist
profile=compute
provmethod=install
template=/opt/xcat/share/xcat/install/rh/compute.rhels7.tmpl
1. Customize the hypervisor node definition to create network bridge
xCAT ships a postscript xHRM to create a network bridge on kvm host during installation/netbooting. Specify
the xHRM with appropriate parameters in postscripts attribute. For example:
• To create a bridge named ‘br0’ against the installation network device specified by installnic:
chdef kvmhost1 -p postscripts="xHRM bridgeprereq br0"
• To create a bridge with default name ‘default’ against the installation network device specified by installnic:
chdef kvmhost1 -p postscripts="xHRM bridgeprereq"
• To create a bridge named ‘br0’ against the network device ‘eth0’:
chdef kvmhost1 -p postscripts="xHRM bridgeprereq eth0:br0"
Note: The network bridge name you use should not be the virtual bridges (vbrX) created by libvirt installation1 .
2. Customize the hypervisor node definition to mount the shared kvm storage directory on management node
(optional)
If the shared kvm storage directory on the management node has been exported, it can be mounted on PowerKVM hypervisor for virtual machines hosting.
An easy way to do this is to create another postscript named “mountvms” which creates a directory /install/vms
on hypervisor and then mounts /install/vms from the management node, the content of “mountvms” can be:
logger -t xcat "Install: setting vms mount in fstab"
mkdir -p /install/vms
echo "$MASTER:/install/vms /install/vms nfs \
rsize=8192,wsize=8192,timeo=14,intr,nfsvers=2 1 2" >> /etc/fstab
1 Every standard libvirt installation provides NAT based connectivity to virtual machines out of the box using the “virtual bridge” interfaces
(virbr0, virbr1, etc) Those will be created by default.
180
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
Then set the file permission and specify the script in postscripts attribute of hypervisor node definition:
chmod 755 /install/postscripts/mountvms
chdef kvmhost1 -p postscripts=mountvms
3. Provision the hypervisor node with the osimage
nodeset kvmhost1 osimage=<osimage_name>
rpower kvmhost1 boot
Create network bridge on hypervisor
To launch VMs, a network bridge must be created on the KVM hypervisor.
If the hypervisor is provisioned successfully according to the steps described above, a network bridge will be created
and attached to a physical interface. This can be checked by running brctl show on the hypervisor to show the
network bridge information, please make sure a network bridge has been created and configured according to the
parameters passed to postscript “xHRM”
# brctl show
bridge name
br0
bridge id
8000.000000000000
STP enabled
no
interfaces
eth0
If the network bridge is not created or configured successfully, run “xHRM” with updatenode on management node
to create it manually::
updatenode kvmhost1
-P "xHRM bridgeprereq eth0:br0"
Start libvirtd service
Verify libvirtd service is running:
systemctl status libvirtd
If service is not running, it can be started with:
systemctl start libvirtd
Manage Virtual Machine (VM)
Now the MowerKVM hypervisor “kvmhost1” is ready, this section introduces the VM management in xCAT, including
examples on how to create, remove and clone VMs.
Create Virtual Machine
Create VM Node Definition
Create a virtual machine node object “vm1”, assign it to be a member of group “vm”, its ip is “192.168.0.1”, run
makehost to add an entry in /etc/hosts file:
1.3. Admin Guide
181
xCAT3 Documentation, Release 2.12
mkdef vm1 groups=vm,all
chdef vm1 ip=192.168.0.1
makehosts vm1
Update DNS configuration and database:
makedns -n
makedns -a
Specify VM attributes
After the VM object is created, several key attributes need to be specified with chdef :
1. the number of virtual cpus in the VM:
chdef vm1 vmcpus=2
2. the kvm hypervisor of the VM:
chdef vm1 vmhost=kvmhost1
3. the virtual memory size, with the unit “Megabit”. Specify 1GB memory to “vm1” here:
chdef vm1 vmmemory=1024
Note: For diskless node, the vmmemory should be at least 2048 MB, otherwise the node cannot boot up.
4. the hardware management module, “kvm” for PowerKVM:
chdef vm1 mgt=kvm
5. Define the virtual network card, it should be set to the bridge “br0” which has been created in the hypervisor. If
no bridge is specified, no network device will be created for the VM node “vm1”:
chdef vm1 vmnics=br0
6. The vmnicnicmodel attribute is used to set the type and corresponding driver for the nic. If not set, the default
value is ‘virtio’.
chdef vm1 vmnicnicmodel=virtio
7. Define the storage for the vm1, three types of storage source format are supported.
(a) Create storage on a NFS server
The format is nfs://<IP_of_NFS_server>/dir, that means the kvm disk files will be created at
nfs://<IP_of_NFS_server>/dir:
chdef vm1 vmstorage=nfs://<IP_of_NFS_server>/install/vms/
(b) Create storage on a device of hypervisor
The format is ‘phy:/dev/sdb1’:
chdef vm1 vmstorage=phy:/dev/sdb1
182
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
(c) Create storage on a directory of hypervisor
The format is ‘dir:///var/lib/libvirt/images’:
chdef vm1 vmstorage=dir:///var/lib/libvirt/images
Note: The attribute vmstorage is only valid for diskful VM node.
8. Define the console attributes for VM:
chdef vm1 serialport=0 serialspeed=115200
9. (optional)For monitoring and access the VM with vnc client, set vidpassword value:
chtab node=vm1 vm.vidpassword=abc123
10. Set netboot attribute
• [x86_64]
chdef vm1 netboot=xnba
• [PPC64LE]
chdef vm1 netboot=grub2
Make sure “grub2” had been installed on the management node:
#rpm -aq | grep grub2
grub2-xcat-1.0-1.noarch
Make virtual machine
If vmstorage is a NFS mounted directory or a device on hypervisor, run
mkvm vm1
To create the virtual machine “vm1” with 20G hard disk on a hypervisor directory, run
mkvm vm1 -s 20G
When “vm1” is created successfully, a VM hard disk file with a name like “vm1.sda.qcow2” will be found in the
location specified by vmstorage. What’s more, the mac attribute of “vm1” is set automatically, check it with:
lsdef vm1 -i mac
Now a VM “vm1” is created, it can be provisioned like any other nodes in xCAT. The VM node can be powered on
by:
rpower vm1 on
If “vm1” is powered on successfully, the VM status can be obtained by running the following command on management node
rpower vm1 status
or running the following command on the kvm hypervisor “kvmhost1”
1.3. Admin Guide
183
xCAT3 Documentation, Release 2.12
#virsh list
Id Name
State
-------------------------------6 vm1
running
Monitoring the Virtual Machine
When the VM has been created and powered on, choose one of the following methods to monitor and access it.
• Open the console on kvm hypervisor:
virsh console vm1
• Use rcons/wcons on xCAT management node to open text console:
chdef vm1 cons=kvm
makeconservercf vm1
rcons vm1
• Connect to virtual machine through vnc console
In order to connect the virtual machine’s vnc server, a new set of credentials need to be generated by running:
xcatclient getrvidparms vm1
vm1: method: kvm
vm1: textconsole: /dev/pts/0
vm1: password: JOQTUtn0dUOBv9o3
vm1: vidproto: vnc
vm1: server: kvmhost1
vm1: vidport: 5900
Note: Now just pick a favorite vnc client to connect the hypervisor, with the password generated by
getrvidparms. If the vnc client complains “the password is not valid”, the reason might be that the hypervisor and headnode clocks are out of sync! Please try to sync them by running ntpdate <ntp server>
on both the hypervisor and the headnode.
• Use wvid on management node
Make sure firewalld service is stopped, disable it if not:
chkconfig firewalld off
or
systemctl disable firewalld
Then, run wvid on MN:
wvid vm1
• For PowerKVM, kimchi on the kvm hypervisor can be used to monitor and access the VM.
Remove the virtual machine
Remove the VM “vm1” even when it is in “power-on” status:
184
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
rmvm vm1 -f
Remove the definition of “vm1” and related storage:
rmvm vm1 -p
Clone the virtual machine
Clone is an operation that creating a VM from an existed one by inheriting most of its attributes and data.
The general step of clone a VM is like this: first creating a VM master , then creating a VM with the newly created
VM master in attaching or detaching mode.
In attaching mode
In this mode, all the newly created VMs are attached to the VM master. Since the image of the newly created VM
only includes the differences from the VM master, which requires less disk space. The newly created VMs can NOT
run without the VM master.
An example is shown below:
Create the VM master “vm5” from a VM node “vm1”:
#clonevm vm1 -t vm5
vm1: Cloning vm1.sda.qcow2 (currently is 1050.6640625 MB and has a capacity of 4096MB)
vm1: Cloning of vm1.sda.qcow2 complete (clone uses 1006.74609375 for a disk size of
˓→4096MB)
vm1: Rebasing vm1.sda.qcow2 from master
vm1: Rebased vm1.sda.qcow2 from master
The newly created VM master “vm5” can be found in the vmmaster table.
#tabdump vmmaster
name,os,arch,profile,storage,storagemodel,nics,vintage,originator,comments,disable
"vm5","<os>","<arch>","compute","nfs://<storage_server_ip>/vms/kvm",,"br0","<date>",
˓→"root",,
Clone a new node vm2 from VM master vm5:
clonevm vm2 -b vm5
In detaching mode
Create a VM master “vm6” .
#clonevm vm2 -t vm6 -d
vm2: Cloning vm2.sda.qcow2 (currently is 1049.4765625 MB and has a capacity of 4096MB)
vm2: Cloning of vm2.sda.qcow2 complete (clone uses 1042.21875 for a disk size of
˓→4096MB)
Clone a VM “vm3” from the VM master “vm6” in detaching mode:
#clonevm vm3 -b vm6 -d
vm3: Cloning vm6.sda.qcow2 (currently is 1042.21875 MB and has a capacity of 4096MB)
1.3. Admin Guide
185
xCAT3 Documentation, Release 2.12
Migrate Virtual Machines
Virtual machine migration is a process that moves the virtual machines (guests) between different hypervisors (hosts).
Note: The VM storage directory should be accessible from both hypervisors (hosts).
Migrate the VM “kvm1” from hypervisor “hyp01” to hypervisor “hyp02”:
#rmigrate kvm1 hyp02
kvm1: migrated to hyp02
Trouble Shooting
VNC client complains the credentials are not valid
Issue: While connecting to the hypervisor with VNC, the vnc client complains “Authentication failed”.
Solution: Check whether the clocks on the hypervisor and headnode are synced
rpower fails with “Error: internal error Process exited while reading console log qemu: Permission
denied”
Issue:
#rpower vm1 on
vm1: Error: internal error Process exited while reading console log output:
˓→char device redirected to /dev/pts/1
qemu: could not open disk image /var/lib/xcat/pools/2e66895a-e09a-53d5-74d3˓→eccdd9746eb5/vm1.sda.qcow2: Permission denied: internal error Process
˓→exited while reading console log output: char device redirected to /dev/
˓→pts/1
qemu: could not open disk image /var/lib/xcat/pools/2e66895a-e09a-53d5-74d3˓→eccdd9746eb5/vm1.sda.qcow2: Permission denied
Solution: Usually caused by incorrect permission in NFS server/client configuration. NFSv4 is enabled
in some Linux distributions such as CentOS6 by default. The solution is simply to disable NFSv4
support on the NFS server by uncommenting the following line in “/etc/sysconfig/nfs”:
RPCNFSDARGS="-N 4"
Then restart the NFS services and try to power on the VM again...
Note: For stateless hypervisor, purge the VM by rmvm -p vm1, reboot the hypervisor and then
create the VM.
rpower fails with “Error: internal error: process exited while connecting to monitor qemu: Permission denied”
Issue:
#rpower vm1 on
vm1: Error: internal error: process exited while connecting to monitor: 2016˓→02-03T08:28:54.104601Z qemu-system-ppc64: -drive file=/var/lib/xcat/pools/
˓→c7953a80-89ca-53c7-64fb-2dcfc549bd45/vm1.sda.qcow2,if=none,id=drive-scsi0˓→0-0-0,format=qcow2,cache=none: Could not open '/var/lib/xcat/pools/
˓→c7953a80-89ca-53c7-64fb-2dcfc549bd45/vm1.sda.qcow2': Permission denied
186
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
Solution: Usually caused by SELinux policies.
The solution is simply to disable SELinux
on the vmhost/hypervisor by editing “/etc/selinux/config” and change the SELINUX line to
SELINUX=disabled:
SELINUX=disabled
Then reboot the hypervisor...
rmigrate fails with “Error: libvirt error code: 38, message: unable to connect to server at
‘c910f05c35:49152’: No route to host.”
Issue:
#rmigrate vm1 kvmhost2
vm1: Error: libvirt error code: 38, message: unable to connect to server at
˓→'kvmhost2:49152': No route to host: Failed migration of vm1 from kvmhost1
˓→to kvmhost2
Solution: Usually caused by active firewall. To disable the firewall issue:
systemctl disable firewalld
rmigrate fails with “Error: 38, message: failed to create directory ‘<dir-name>’: File exists: Unknown
issue libvirt error code.”
Issue:
#rmigrate vm1 kvmhost2
vm1: Error: 38, message: failed to create directory '<dir-name>': File
˓→exists: Unknown issue libvirt error code.
Solution: Ususally happens when nfs: is specified for vmstorage attribute but that NFS directory is no
longer mounted. Make sure the directory /var/lib/xcat/pools is empty on the destination kvmhost.
Error: Cannot communicate via libvirt to kvmhost1
Issue: The kvm related commands complain “Error: Cannot communicate via libvirt to kvmhost1”
Solution: Usually caused by incorrect ssh configuration between xCAT management node and hypervisor. Make sure it is possible to access the hypervisor from management node via ssh without
password.
Fail to ping the installed VM
Issue: The newly installed stateful VM node is not pingable, the following message can be observed in
the console during VM booting:
ADDRCONF(NETDEV_UP): eth0 link is not ready.
Solution: Usually caused by the incorrect VM NIC model. Try the following steps to specify “virtio”:
1.3. Admin Guide
187
xCAT3 Documentation, Release 2.12
rmvm vm1
chdef vm1 vmnicnicmodel=virtio
mkvm vm1
x86_64
This section is not available at this time.
Refer to xCAT Documentation on SourceForge for information on System X servers.
References
xCAT Man Pages
These man pages are auto generated from .pod files to .rst files using the create_man_pages.py script under
xcat-core
man1
addkit.1
NAME
addkit - Adds product software Kits to an xCAT cluster environmnet.
SYNOPSIS
addkit [-? | -h | --help] [-v | --version]
addkit [-i | --inspection] kitlist
addkit [-V | --verbose] [-p | --path path] kitlist
DESCRIPTION
The addkit command installs a kit on the xCAT management node from a kit tarfile or directory. It creates xCAT
database definitions for the kit, kitrepo, and kitcomponent.
Note: xCAT Kit support is ONLY available for Linux operating systems.
OPTIONS
-h|--help
Display usage message.
-V|--verbose
Verbose mode.
-v|--version
188
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
Command version.
-i|--inspection
Show the summary of the given kits
-p|--path path
The destination directory to which the contents of the kit tarfiles and/or kit deploy directories will be
copied. When this option is not specified, the default destination directory will be formed from the
installdir site attribute with ./kits subdirectory.
kitlist
A comma delimited list of kit_tarball_files or kit_deploy_directories to be added to the xCAT environment. Each entry can be an absolute or relative path. See xCAT documentation for more information on
building kits.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To add kits from tarball files:
addkit kit-test1.tar.bz2,kit-test2.tar.bz2
2. To add kits from directories:
addkit kit-test1,kit-test2
3. To add kits from tarball kit-test1.tar.bz2 to target path /install/test:
addkit -p /install/test kit-test1.tar.bz2
4. To see general information about kit kit-test1.tar.bz2 without adding the kit to xCAT:
addkit -i kit-test1.tar.bz2
SEE ALSO
lskit(1)|lskit.1,
rmkit(1)|rmkit.1,
comp(1)|chkkitcomp.1
addkitcomp(1)|addkitcomp.1,
rmkitcomp(1)|rmkitcomp.1,
chkkit-
addkitcomp.1
NAME
addkitcomp - Assign Kit components to an xCAT osimage.
1.3. Admin Guide
189
xCAT3 Documentation, Release 2.12
SYNOPSIS
addkitcomp [-? | -h | --help] [-v | --version]
addkitcomp [-V | --verbose] [-a | --adddeps] [-f | --force] [-n | --noupgrade] [--noscripts] -i osimage kitcompname_list
DESCRIPTION
The addkitcomp command will assign kit components to an xCAT osimage. The kit component meta rpm, package
rpm and deploy parameters will be added to osimage’s otherpkg.pkglist and postbootscripts will be added to osimages’s
postbootscripts attribute.
Note: xCAT Kit support is ONLY available for Linux operating systems.
OPTIONS
-a|--adddeps
Assign kitcomponent dependencies to the osimage.
-h|--help
Display usage message.
-V|--verbose
Verbose mode.
-v|--version
Command version.
-f|--force
Add kit component to osimage even if there is a mismatch in OS, version, arch, serverrole, or kitcompdeps
-i osimage
The osimage name that the kit component is assigning to.
-n|--noupgrade
1. Allow multiple versions of kitcomponent to be installed into the osimage, instead of kitcomponent upgrade.
2. Kit components added by addkitcomp -n will be installed separately behind all other ones which have been
added.
--noscripts
Do not add kitcomponent’s postbootscripts to osimage
kitcompname_list
A comma-delimited list of valid full kit component names or kit component basenames that are to be
added to the osimage.
190
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To add a single kit component to osimage “rhels6.2-ppc64-netboot-compute”:
addkitcomp -i rhels6.2-ppc64-netboot-compute comp-test1-1.0-1-rhels-6.2-ppc64
2. To add a kit component to osimage with dependencies, use the -a (addeps) option:
addkitcomp -a -i rhels6.2-ppc64-netboot-compute comp-test2-1.0-1-rhels-6.2-ppc64
3. To add a kit component to osimage with incompatable osarch, osversion or ostype, use the -f (force) option:
addkitcomp -f -i rhels6.2-ppc64-netboot-compute comp-test1-1.0-1-rhels-6.2-ppc64
4. To add a new version of kit component to osimage without upgrade, use the -n (noupgrade) option:
addkitcomp -n -i rhels6.2-ppc64-netboot-compute comp-test2-1.0-1-rhels-6.2-ppc64
SEE ALSO
lskit(1)|lskit.1, addkit(1)|addkit.1, rmkit(1)|rmkit.1, rmkitcomp(1)|rmkitcomp.1, chkkitcomp(1)|chkkitcomp.1
bmcdiscover.1
NAME
bmcdiscover - Discover Baseboard Management Controllers (BMCs) using a scan method
SYNOPSIS
bmcdiscover [-? | -h | --help]
bmcdiscover [-v | --version]
bmcdiscover [-s scan_method] [-u bmc_user] [-p bmc_passwd] [-z] [-w] --range ip_ranges
bmcdiscover -u bmc_user -p bmc_passwd -i bmc_ip --check
bmcdiscover [-u bmc_user] [-p bmc_passwd] -i bmc_ip --ipsource
DESCRIPTION
The bmcdiscover command will discover Baseboard Management Controllers (BMCs) using a scan method.
The command uses nmap to scan active nodes over a specified IP range. The IP range format should be a format that
is acceptable by nmap.
1.3. Admin Guide
191
xCAT3 Documentation, Release 2.12
The bmcdiscover command can also obtain some information about the BMC. (Check username/password, IP address
source, DHCP/static configuration)
Note: The scan method currently support is nmap.
OPTIONS
--range
Specify one or more IP ranges acceptable to nmap. IP range can be hostnames, IP addresses, networks,
etc. A single IP address (10.1.2.3) or an IP range (10.1.2.0/24) can be specified. If the range is very large,
the bmcdiscover command may take a long time to return.
-s
Scan method (The only supported scan method at this time is nmap)
-z
List the data returned in xCAT stanza format
-w
Write to the xCAT database.
-i|--bmcip
BMC IP address.
-u|--bmcuser
BMC user name.
-p|--bmcpasswd
BMC user password.
--check
Check BMC administrator User/Password.
--ipsource
Display the BMC IP configuration.
-h|--help
Display usage message
-v|--version
Display version information
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
192
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
EXAMPLES
1. To get all responding BMCs from IP range “10.4.23.100-254” and 50.3.15.1-2”:
bmcdiscover -s nmap --range "10.4.23.100-254 50.3.15.1-2"
Note: Input for IP range can be in the form: scanme.nmap.org, microsoft.com/24, 192.168.0.1; 10.0.0-255.1-254.
2. To get all BMSs in IP range “10.4.22-23.100-254”, displayed in xCAT stanza format:
bmcdiscover -s nmap --range "10.4.22-23.100-254" -z
3. Discover the BMCs and write the discovered-node definitions into the xCAT database and write out the stanza
foramt to the console:
bmcdiscover -s nmap --range "10.4.22-23.100-254" -w -z
4. To check if the username or password is correct against the BMC:
bmcdiscover -i 10.4.23.254 -u USERID -p PASSW0RD --check
5. Get BMC IP Address source, DHCP Address or static Address
bmcdiscover -i 10.4.23.254 -u USERID -p PASSW0RD --ipsource
SEE ALSO
lsslp(1)|lsslp.1
buildkit.1
NAME
buildkit - Used to build a software product Kit which may be used to install software in an xCAT cluster.
SYNOPSIS
buildkit [-? | -h | --help] [-v | --version]
To build a new Kit
buildkit [-V | --verbose] subcommand [kit_name] [repo_name | all] [-l | --kitloc kit_location]
To add packages to an existing Kit.
buildkit [-V | --verbose] addpkgs kit_tarfile [-p | --pkgdir package_directory_list] [-k | --kitversion version] [-r |
--kitrelease release] [-l | --kitloc kit_location]
DESCRIPTION
The buildkit command provides a collection of utilities that may be used to package a software product as a Kit tarfile
that can be used to install software on the nodes of an xCAT cluster. A Kit contains the product software packages,
configuration and control information, and install and customization scripts.
1.3. Admin Guide
193
xCAT3 Documentation, Release 2.12
Note: The xCAT support for Kits is only available for Linux operating systems.
You will need to run the buildkit command several times with different subcommands to step through the process of
building a kit:
By default the buildkit subcommands will operate in the current working directory, (ie. look for files, create directories
etc.). You could specify a different location by using the “-l | --kitloc kit_location” option.
The kit_location is the full path name of the directory that contains the kit files. You would use the same location value
for all the buildkit subcommands.
For example, to create a new kit named “prodkit” in the directory /home/mykits/ either run:
1. If no location is provided then the command will create a subdirectory called “prodkit” in the current directory
“/home/mykits” and the new kit files will be created there.
cd /home/mykits
buildkit create prodkit
or
2. If a location is provided then the Kit files will be created there. Note that the Kit name does not necessarily have
to be the directory name where the kit files are located.
buidkit create prodkit -l /home/mykits/prodkit
In both cases the /home/mykits/prodkit directory is created and the inital files for the kit are created in that directory.
The following example illustrates the basic process for building a new Kit. In this example we are building a Kit
named “mytstkit”.
1. Change to the directory where you wish to create the Kit.
2. Create a template directory for your kit:
buildkit create mytstkit
3. Change directory to the new “mytstkit” subdirectory that was just created.
cd mytstkit
4. Edit the buildkit configuration file for your kit:
vi buildkit.conf
(See xCAT Kit documentation for details.)
5. Create all required files, scripts, plugins, and packages for your kit.
6. Validate your kit build configuration and fix any errors that are reported:
buildkit chkconfig
7. List the repos defined in your buildkit configuration file:
buildkit listrepo
8. For each repo name listed, build the repository. Note that if you need to build repositories for OS distributions,
versions, or architectures that do not match the current system, you may need to copy your kit template directory
to an appropriate server to build that repository, and then copy the results back to your main build server. For
example, to build a repo named “rhels6.3” you would run the following command.
buildkit buildrepo rhels6.3
or, you can build all of the repos at one time if there are no OS or architecture dependencies for kitcomponent
package builds or kitpackage builds:
194
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
buildkit buildrepo all
9. Build the kit tar file:
buildkit buildtar
OPTIONS
-h |--help
Display usage message.
-k|--kitversion version
Product version.
-l|--kitloc kit_location
The directory location of the Kit files.
-p|--pkgdir package_directory_list
A comma-separated list of directory locations for product RPMs.
-r|--kitrelease release
Product release.
-V |--verbose
Verbose mode.
-v|--version
Command version.
SUB-COMMANDS
create kit_basename
Creates a new kit build directory structure for kit kit_basename using the location specified on the command line or the current directory. The sample kit files from /opt/xcat/share/xcat/kits/kit_template are
copied over, and the buildkit.conf file is modified for the specified kit_basename.
chkconfig
Reads the buildkit.conf file, verifies that the file syntax is correct and that all specified files exist.
listrepo
Reads the buildkit.conf file, lists all Kit package repositories listed in the file, and reports the build status
for each repository.
buildrepo {repo_name | all}
Reads the buildkit.conf file, and builds the specified Kit package repository. The built packages are
placed in the directory <kit_location>/build/kit_repodir/repo_name. If all is specified, all kit repositories
are built.
cleanrepo {repo_name | all}
Reads the buildkit.conf file, and deletes all the package files and package meta data files from the
<kit_location>/build/kit_repodir/repo_name directory. If all is specified, all kit repository files are
deleted.
1.3. Admin Guide
195
xCAT3 Documentation, Release 2.12
buildtar
Reads the buildkit.conf file, validates that all kit repositories have been built, and builds the Kit tar file
<kit_location>/kitname.tar.bz2.
cleantar
Reads the <kit_location>/buildkit.conf file and deletes the following:
• Kit tar files matching <kit_location>/kit_name\.tar.bz2*.
• <kit_location>/build/kit_name
• <kit_location>/rpmbuild
• <kit_location>/tmp
• <kit_location>/debbuild
Caution: Make sure you back up any tar files you would like to keep before running this subcommand.
cleanall
Equivalent to running buildkit cleanrepo all and buildkit cleantar.
addpkgs
kit_tarfile {-p | --pkgdir package_directory_list} [-k | --kitversion version] [-r | --kitrelease release]
Add product package rpms to a previously built kit tar file. This is used for partial product kits that
are built and shipped separately from the product packages, and are identified with a kit_tarfile name of
kitname.NEED_PRODUCT_PKGS.tar.bz2. Optionally, change the kit release and version values when
building the new kit tarfile. If kitcomponent version and/or release values are defaulted to the kit values,
those will also be changed and new kitcomponent rpms will be built. If kit or kitcomponent scripts,
plugins, or other files specify name, release, or version substitution strings, these will all be replaced with
the new values when built into the new complete kit tarfile kit_location/new_kitname.tar.bz2.
RETURN VALUE
<B>0
The command completed successfully.
<B>1
An error has occurred.
EXAMPLES
1. To create the sample kit shipped with the xCAT-buildkit rpm on a RHELS 6.3 server and naming it mykit, run
the following commands:
cd /home/myuserid/kits
buildkit create mykit
cd mykit
vi buildkit.conf
buildkit chkconfig
buildkit listrepo
196
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
buildkit buildrepo all
buildkit buildtar
2. To clean up a kit repository directory after build failures on a RHELS 6.3 server to prepare for a new kit
repository build, run:
buildkit cleanrepo rhels6.3
3. To clean up all kit build files, including a previously built kit tar file, run
buildkit cleanall
4. To create a kit named “tstkit” located in /home/foobar/tstkit instead of the current working directory.
buildkit create tstkit -l /home/foobar/tstkit
FILES
/opt/xcat/bin/buildkit
/opt/xcat/share/xcat/kits/kit_template
/opt/xcat/share/xcat/kits/kitcomponent.spec.template
<kit location>/buildkit.conf
<kit location>/build/kitname/kit.conf
<kit location>/kitname.tar.bz2
SEE ALSO
addkit(1), lskit(1), rmkit(1), addkitcomp(1), rmkitcomp(1), chkkitcomp(1)
cfgve.1
NAME
cfgve - Configure the elements for a virtual environment.
SYNOPSIS
cfgve -t dc -m manager -o object [-c -k nfs | localfs | -r]
cfgve -t cl -m manager -o object [-c -p cpu type | -r -f]
cfgve -t sd -m manager -o object [-c | -g | -s | -a | -b | -r -f]
cfgve -t nw -m manager -o object [-c -d data center -n vlan ID | -a -l cluster | -b | -r]
cfgve -t tpl -m manager -o object [-r]
1.3. Admin Guide
197
xCAT3 Documentation, Release 2.12
DESCRIPTION
The cfgve command can be used to configure a virtual environment for ‘Storage Domain’, ‘Network’ and ‘Template’
objects.
The mandatory parameter -m manager is used to specify the address of the manager of virtual environment. xCAT
needs it to access the RHEV manager.
The mandatory parameter -t type is used to specify the type of the target object.
Basically, cfgve command supports five types of object: dc, cl, sd, nw and tpl.
dc - The create and remove operations are supported.
cl - The create and remove operations are supported.
sd - The create, attach, detach, activate, deactivate and remove operations are supported.
nw - The create, attach, detach and remove operations are supported.
tpl - The remove operation is supported.
The mandatory parameter -o object is used to specify which object to configure.
OPTIONS
-a To attach the target object.
-b To detach the target object.
-c To create the target object.
For creating of Storage Domain, the target storage domain will be created first, then attached to data
center and activated.
The parameters that used to create the storage domain are gotten from ‘virtsd’ table. The detail parameters
in the virtsd table:
virtsd.node - The name of the storage domain.
virtsd.sdtype - The type of storage domain. Valid value: data, iso, export. Default value is ‘data’.
virtsd.stype - The storage type. “nfs” or “localfs”.
virtsd.location - The location of the storage. nfs: Format: [nfsserver:nfspath]. The NFS export directory
must be configured for read write access and must be owned by vdsm:kvm. localfs: “/data/images/rhev”
is set by default.
virtsd.host - A host must be specified for a storage domain as SPM (Storage Pool Manager) when initialize the storage domain. The role of SPM may be migrated to other host by rhev-m during the running of
the datacenter (For example, when the current SPM encountered issue or going to maintenance status.
virtsd.datacenter - The storage will be attached to. ‘Default’ data center is the default value.
-d data center
The name of data center.
Specify the ‘Data Center’ that will be used for the object to be attached to. It is used by <nw> type.
-f It can be used with -r to remove the target object by force.
For removing of Storage Domain, if -f is specified, the storage domain will be deactivated and detached
from data center before the removing.
198
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
-g To activate the target object.
-h Display usage message.
-k storage type
To specify the type of the storage type when creating the data center.
Supported type: nfs; localfs.
-l cluster
Specify the cluster for the network to attach to.
-m manager
Specify the manager of the virtual environment.
For RHEV, the FQDN (Fully Qualified Domain Name) of the rhev manager have to be specified.
-n vlan ID
To specify the vlan number when creating a network.
-o object
The name of the target object.
-p cpu type
To specify the cpu type when creating the cluster. Intel Penryn Family is default type.
Supported type: Intel Conroe Family, Intel Penryn Family, Intel Nehalem Family, Intel Westmere
Family, AMD Opteron G1, AMD Opteron G2, AMD Opteron G3
-r To remove the target object.
For removing of Storage Domain, the storage space will be formatted after removing.
-s To deactivate the target object.
-t type
Specify the type of the target object.
Supported types: dc - Data Center cl - Cluster sd - Storage Domain nw - Network tpl - Template
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To create the Storage Domain ‘sd1’, enter:
cfgve -t sd -m <FQDN of rhev manager> -o sd1 -c
2. To deactivate the Storage Domain ‘sd1’ from data center, enter:
cfgve -t sd -m <FQDN of rhev manager> -o sd1 -s
3. To remove the Storage Domain ‘sd1’, enter:
1.3. Admin Guide
199
xCAT3 Documentation, Release 2.12
cfgve -t sd -m <FQDN of rhev manager> -o sd1 -r
4. To create the network ‘nw1’, enter:
cfgve -t nw -m <FQDN of rhev manager> -o nw1 -c
5. To remove the template ‘tpl01’, enter:
cfgve -t tpl -m <FQDN of rhev manager> -o tpl01 -r
FILES
/opt/xcat/bin/cfgve
SEE ALSO
lsve(1)|lsve.1
cfm2xcat.1
NAME
cfm2xcat - Migrates the CFM setup in CSM to the xdcp rsync setup in xCAT.
SYNOPSIS
cfm2xcat [-i path of the CFM distribution files generated] [-o path of the xdcp rsync files generated from the CFM
distribution files]
cfm2xcat [-h]
DESCRIPTION
Copy the cfm2xcat command to the CSM Management Server. Run the command, indicating where you want your
files saved with the -i and -o flags. They can be in the same directory.
The cfm2xcat command will run cfmupdatenode -a, saving the generated CFM distribution files in the directory
indicates with (-i). From those distribution files, it will generate xdcp rsync input files (-F option on xdcp) in the
directory indicated by ( -o).
Check the rsync files generated. There will be a file generated (rsyncfiles) from the input -o option on the command,
and the same file with a (.nr) extension generated for each different noderange that will used to sync files based on
your CFM setup in CSM. The rsyncfiles will contain the rsync file list. The rsyncfiles.nr will contain the noderange.
If multiple noderanges then the file name (rsyncfiles) will be appended with a number.
200
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
OPTIONS
-h Display usage message.
-i Path of the CFM distribution files generated from the cfmupdatenode -a command.
-o Path of the xdcp rsync input file generated from the CFM distribution files.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To build xCAT rsync files to use with xdcp -F , enter on the CSM Management Server, make sure the path exists:
cfm2xcat -i /tmp/cfm/cfmdistfiles -o /tmp/cfm/rsyncfiles
2. To use the file on the xCAT Management Node copy to /tmp/cfm on the xCAT MN:
xdcp ^/tmp/cfm/rsyncfiles.nr -F /tmp/cfm/rsyncfiles
xdcp ^/tmp/cfm/rsyncfiles.nr1 -F /tmp/cfm/rsyncfiles1
xdcp ^/tmp/cfm/rsyncfiles.nr2 -F /tmp/cfm/rsyncfiles2
FILES
/opt/xcat/share/xcat/tools/cfm2xcat
chdef.1
NAME
chdef - Change xCAT data object definitions.
SYNOPSIS
chdef [-h | --help] [-t object-types]
chdef [-t object-types] [-o object-names] [-n new-name] [node]
chdef [-V | --verbose] [-t object-types] [-o object-names] [-d | --dynamic] [-p | --plus] [-m | --minus] [-z | --stanza] [[w attr==val] [-w attr=~val] ...] [noderange] [attr=val [attr=val...]] [-u [provmethod= {install | netboot | statelite}]
[profile=xxx] [osvers=value] [osarch=value]]
1.3. Admin Guide
201
xCAT3 Documentation, Release 2.12
DESCRIPTION
This command is used to change xCAT object definitions which are stored in the xCAT database. The default is to
replace any existing attribute value with the one specified on the command line. The command will also create a new
definition if one doesn’t exist.
This command also can be used to change the xCAT object name to a new name. Note: the site,monitoring types can
NOT be supported.
OPTIONS
attr=val [attr=val ...]
Specifies one or more “attribute equals value” pairs, separated by spaces. Attr=val pairs must be specified
last on the command line. Use the help option to get a list of valid attributes for each object type.
-d|--dynamic
Use the dynamic option to change dynamic node groups definition. This option must be used with -w
option.
-h|--help
Display usage message.
-m|--minus
If the value of the attribute is a list then this option may be used to remove one or more items from the
list.
-n new-name
Change the current object name to the new-name which is specified by the -n option. Objects of type site,
group and monitoring cannot be renamed with the -n option. Note: For the -n option, only one node can
be specified. For some special nodes such as fsp, bpa, frame, cec etc., their name is referenced in their
own hcp attribute, or the hcp attribute of other nodes. If you use -n option, you must manually change all
hcp attributes that refer to this name.
noderange
A set of comma delimited node names and/or group names. (must be the first parameter) See the
“noderange” man page for details on supported formats.
-o object-names
A set of comma delimited object names.
-p|--plus
This option will add the specified values to the existing value of the attribute. It will create a commaseparated list of values.
-t object-types
A set of comma delimited object types. Use the help option to get a list of valid object types.
-V|--verbose
Verbose mode.
-w attr==val -w attr=~val ...
Use one or multiple -w flags to specify the selection string that can be used to select objects. The operators
==, !=, =~ and !~ are available. Use the help option to get a list of valid attributes for each object type.
202
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
Operator descriptions: == Select nodes where the attribute value is exactly this value. != Select nodes
where the attribute value is not this specific value. =~ Select nodes where the attribute value matches
this regular expression. !~ Select nodes where the attribute value does not match this regular expression.
Note: the operator !~ will be parsed by shell, if you want to use !~ in the selection string, use single quote
instead. For example:-w ‘mgt!~ipmi’.
-z|--stanza
Indicates that the file being piped to the command is in stanza format. See the xcatstanzafile man page for
details on using xCAT stanza files.
-u
Fill in the attributes such as template file, pkglist file and otherpkglist file of osimage object based on the
specified parameters. It will search “/install/custom/” directory first, and then “/opt/xcat/share/”.
Note: this option only works for objtype osimage.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To change a site definition.
chdef -t site -o clustersite installdir=/xcatinstall
2. To change a basic node definition.
chdef -t node -o node01 groups="all,aix"
(The group definitions are also created if they don’t already exist.)
3. To add another group to the “groups” attribute in the previous example.
chdef -p -t node -o node01 groups="compute"
4. To remove the “all” group from the “groups” attribute in the previous example.
chdef -m -t node -o node01 groups="all"
5. To replace the current “groups” attribute value of “node01”.
chdef -t node -o node01 groups="linux"
6. To add “node01” to the “members” attribute of a group definition called “LinuxNodes”.
chdef -p -t group -o LinuxNodes members="node01"
7. To update a set of definitions based on information contained in the stanza file mystanzafile.
cat mystanzafile | chdef -z
1.3. Admin Guide
203
xCAT3 Documentation, Release 2.12
8. To update a dynamic node group definition to add the cons=hmc wherevals pair.
chdef -t group -o dyngrp -d -p -w cons==hmc
9. To change the node object name from node1 to node2.
chdef -t node -o node1 -n node2
10. To change the node hwtype, this command will change the value of ppc.nodetype.
chdef -t node -o node1 hwtype=lpar
11. To change the policy table for policy number 7.0 for admin1
chdef -t policy -o 7.0 name=admin1 rule=allow
12. To change the node nic attributes
chdef -t node -o cn1 nicips.eth0="1.1.1.1|1.2.1.1" nicnetworks.eth0="net1|net2"
˓→nictypes.eth0="Ethernet"
13. To update an osimage definition.
chdef redhat6img -u provmethod=install
FILES
$XCATROOT/bin/chdef
(The XCATROOT environment variable is set when xCAT is installed. The default value is “/opt/xcat”.)
NOTES
This command is part of the xCAT software product.
SEE ALSO
mkdef(1)|mkdef.1, lsdef(1)|lsdef.1, rmdef(1)|rmdef.1, xcatstanzafile(5)|xcatstanzafile.5
chhypervisor.1
NAME
chhypervisor - Configure the virtualization hosts.
SYNOPSIS
RHEV specific :
chhypervisor noderange [-a]
204
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
chhypervisor noderange [-n]
chhypervisor noderange [-p]
chhypervisor noderange [-e]
chhypervisor noderange [-d]
zVM specific :
chhypervisor noderange [--adddisk2pool function region volume group]
chhypervisor noderange [--addscsi device_number device_path option persist]
chhypervisor noderange [--addvlan name owner type transport]
chhypervisor noderange [--addvswitch name osa_dev_addr osa_exp_adapter controller connect (0, 1, or 2) memory_queue router transport vlan_id port_type update gvrp native_vlan]
chhypervisor noderange [--addzfcp2pool pool status wwpn lun size owner]
chhypervisor noderange [--removediskfrompool function region group]
chhypervisor noderange [--removescsi device_number persist (YES or NO)]
chhypervisor noderange [--removevlan name owner]
chhypervisor noderange [--removevswitch name]
chhypervisor noderange [--removezfcpfrompool pool lun wwpn]
chhypervisor noderange [--smcli function arguments]
DESCRIPTION
The chhypervisor command can be used to configure the RHEV-h.
The rhev-h host will register to the rhev-m automatically, but admin needs to approve the host can be added to the
‘cluster’ with -a flag .
After registering, the network interfaces of host need to be added to the ‘network’ of RHEV. And the power management for the host should be configured so that rhev-m could make proper decision when certain host encountered
error.
The chhypervisor command can also be used to configure the zVM host.
For each host, an entry should be added to the hypervisor table:
The columns of hypervisor table:
hypervisor.node - rhev-h host name or zVM host name (lower-case).
hypervisor.type - Must be set to ‘rhevh’ or ‘zvm’.
hypervisor.mgr - The rhev manager (The FQDN of rhev-m server) for the host.
hypervisor.interface - The configuration for the nics. Refer to -n.
hypervisor.cluster - The cluster that the host will be added to. The default is ‘Default’ cluster if not specified.
1.3. Admin Guide
205
xCAT3 Documentation, Release 2.12
OPTIONS
RHEV specific :
-a Approve the host that to be added to cluster.
Before approve, the status of the host must be ‘pending_approval’.
-n Configure the network interfaces for the host.
Note: This operation only can be run when host is in ‘maintenance mode’. Use -d to switch the host to
‘maintenance’ mode.
The interfaces which configured in hypervisor.interface will be added to the network of RHEV.
The format of hypervisor.interface is multiple [network:interfacename: protocol:IP:netmask:gateway]
sections separated with ‘|’. For example: [rhevm2:eth0:static:10.1.0.236:255.255.255.0:0.0.0.0].
network - The logic network which has been created by ‘cfgve -t nw’ or the default management network
‘rhevm’.
interfacename - Physical network name: ‘eth0’,’eth1’...
protocol - To identify which boot protocol to use for the interface: dhcp or static.
IP - The IP address for the interface.
netmask - The network mask for the interface.
gateway - The gateay for the interface. This field only can be set when the interface is added to ‘rhevm’
network.
-p Configure the power management for the host.
The power management must be configured for the rhev-h host to make the rhev-m to monitor the power
status of the host, so that when certain host failed to function, rhev-m will fail over certain role like SPM
to other active host.
For rack mounted server, the bmc IP and user:password need to be set for the power management (These
parameters are gotten from ipmi table). rhev-m uses the ipmi protocol to get the power status of the host.
-e To activate the host.
-d To deactivate the host to maintenance mode.
-h Display usage message.
zVM specific :
--adddisk2pool function region volume group
Add a disk to a disk pool defined in the EXTENT CONTROL. Function type can be either: (4) Define
region as full volume and add to group OR (5) Add existing region to group. If the volume already exists
in the EXTENT CONTROL, use function 5. If the volume does not exist in the EXTENT CONTROL,
but is attached to SYSTEM, use function 4.
--addscsi device_number device_path option persist
Dynamically add a SCSI disk to a running z/VM system.
--addvlan name owner type transport
Create a virtual network LAN.
206
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
--addvswitch name osa_dev_addr osa_exp_adapter controller connect (0, 1, or 2) memory_queue router transport
vlan_id port_type update gvrp native_vlan
Create a virtual switch.
--addzfcp2pool pool status wwpn lun size owner
Add a zFCP device to a device pool defined in xCAT. The device must have been carved up in the storage
controller and configured with a WWPN/LUN before it can be added to the xCAT storage pool. z/VM
does not have the ability to communicate directly with the storage controller to carve up disks dynamically.
--removediskfrompool function region group
Remove a disk from a disk pool defined in the EXTENT CONTROL. Function type can be either: (1)
Remove region, (2) Remove region from group, (3) Remove region from all groups, OR (7) Remove
entire group .
--removescsi device_number persist (YES or NO)
Delete a real SCSI disk.
--removevlan name owner
Delete a virtual network LAN.
--removevswitch name
Delete a virtual switch.
--removezfcpfrompool pool lun
Remove a zFCP device from a device pool defined in xCAT.
--smcli function arguments
Execute a SMAPI function. A list of APIs supported can be found by using the help flag, e.g. chhypervisor
pokdev61 –smcli -h. Specific arguments associated with a SMAPI function can be found by using the help
flag for the function, e.g. chhypervisor pokdev61 –smcli Image_Query_DM -h. Only z/VM 6.2 and older
SMAPI functions are supported at this time. Additional SMAPI functions will be added in subsequent
zHCP versions.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
RHEV specific :
1. To approve the host ‘host1’, enter:
chhypervisor host1 -a
2. To configure the network interface for the host ‘host1’, enter:
chhypervisor host1 -n
3. To configure the power management for the host ‘host1’, enter:
1.3. Admin Guide
207
xCAT3 Documentation, Release 2.12
chhypervisor host1 -p
4. To activate the host ‘host1’, enter:
chhypervisor host1 -e
5. To deactivate the host ‘host1’, enter:
chhypervisor host1 -d
zVM specific :
1. To add a disk to a disk pool defined in the EXTENT CONTROL, enter:
chhypervisor pokdev61 --adddisk2pool 4 DM1234 DM1234 POOL1
2. To add a zFCP device to a device pool defined in xCAT, enter:
chhypervisor pokdev61 --addzfcp2pool zfcp1 free 500501234567C890
˓→4012345600000000 8G
3. To remove a region from a group in the EXTENT CONTROL, enter:
chhypervisor pokdev61 --removediskfrompool 2 DM1234 POOL1
4. To remove a zFCP device from a device pool defined in xCAT, enter:
chhypervisor pokdev61 --removezfcpfrompool zfcp1 4012345600000000
˓→500501234567C890
5. To execute a SMAPI function (Image_Query_DM), enter:
chhypervisor pokdev61 --smcli Image_Query_DM -T LNX3
FILES
/opt/xcat/bin/chhypervisor
chkkitcomp.1
NAME
chkkitcomp - Check if Kit components are compatible with an xCAT osimage.
SYNOPSIS
chkkitcomp [-? | -h | --help] [-v | --version]
chkkitcomp [-V | --verbose] -i osimage kitcompname_list
208
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
DESCRIPTION
The chkkitcomp command will check if the kit components are compatible with the xCAT osimage.
This command will ignore the current osimage.kitcomponents setting and check if the kitcompname_list is compatible
with the osimage and kit component dependencies.
Note: xCAT Kit support is ONLY available for Linux operating systems.
OPTIONS
-h|--help
Display usage message.
-V|--verbose
Verbose mode.
-v|--version
Command version.
-i osimage
The name of the osimage to check against.
kitcompname_list
A comma-delimited list of valid full kit component names or kit component basenames that are to be
checked against the osimage.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To check if a kit component, comp-test1-1.0-1-rhels-6.2-ppc64 can be added to osimage rhels6.2-ppc64netboot-compute:
chkkitcomp -i rhels6.2-ppc64-netboot-compute comp-test1-1.0-1-rhels-6.2-ppc64
SEE ALSO
lskit(1)|lskit.1, addkit(1)|addkit.1, rmkit(1)|rmkit.1, addkitcomp(1)|addkitcomp.1, rmkitcomp(1)|rmkitcomp.1
chkosimage.1
NAME
chkosimage - Use this xCAT command to check an xCAT osimage.
1.3. Admin Guide
209
xCAT3 Documentation, Release 2.12
SYNOPSIS
chkosimage [-h | --help ]
chkosimage [-V] [-c|--clean] osimage_name
DESCRIPTION
This command is currently supported for AIX osimages only.
Use this command to verify if the NIM lpp_source directories contain the correct software. The lpp_source directory
must contain all the software that is specified in the “installp_bundle” and “otherpkgs” attributes of the osimage
definition.
The command gets the name of the lpp_source resource from the xCAT osimage definition and the location of the
lpp_source directory from the NIM resource definition.
It will check for installp, rpm and emgr type packages.
Note: Remember to use the prefixes, “I:”, “R:”, and “E:”, respectively, when specifying package names in an installp_bundle file or an otherpkgs list.
In addition to checking for missing software the chkosimage command will also check to see if there are multiple
matches. This could happen when you use wildcards in the software file names. For example, if you have perl-xCAT*
in a bundle file it could match multiple versions of the xCAT rpm package saved in your lpp_source directory.
If this happens you must remove the unwanted versions of the rpms. If the extra rpms are not removed you will get
install errors.
To help with this process you can use the “-c|–clean” option. This option will keep the rpm package with the most
recent timestamp and remove the others.
The chkosimage command should always be used to verify the lpp_source content before using the osimage to install
any AIX cluster nodes.
OPTIONS
-c |--clean
Remove any older versions of the rpms. Keep the version with the latest timestamp.
-h |--help
Display usage message.
osimage_name
The name of the xCAT for AIX osimage definition.
-V |--verbose
Verbose mode.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
210
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
EXAMPLES
1. Check the XCAT osimage called “61image” to verify that the lpp_source directories contain all the software that is
specified in the “installp_bundle” and “otherpkgs” attributes.
chkosimage -V 61image
2. Clean up the lpp_source directory for the osimage named “61img” by removing any older rpms with the same
names but different versions.
chkosimage -c 61img
FILES
/opt/xcat/bin/chkosimage
NOTES
This command is part of the xCAT software product.
SEE ALSO
mknimimage(1)|mknimimage.1
chvlan.1
NAME
chvlan - It adds or removes nodes for the vlan.
SYNOPSIS
chvlan vlanid -n | --nodes noderange [-i | --interface nic]
chvlan vlanid -n | --nodes noderange -d | --delete
chvlan [-h | --help]
chvlan [-v | --version]
DESCRIPTION
The chvlan command adds nodes to the given vlan. If -d is specified, the nodes will be removed from the vlan.
For added security, the root guard and bpdu guard will be enabled for the ports added to this vlan. However, the guards
will not be disabled if the ports are removed from the vlan using chvlan (-d) or rmvlan commands. To disable them,
you need to use the switch command line interface. Please refer to the switch command line interface manual to see
how to disable the root guard and bpdu guard for a port.
1.3. Admin Guide
211
xCAT3 Documentation, Release 2.12
Parameters
vlanid is a unique vlan number.
OPTIONS
-n|--nodes The nodes or groups to be added or removed. It can be stand alone nodes or KVM guests. It takes the
noderange format. Please check the man page for noderange for details.
-i|--interface (For adding only). The interface name where the vlan will be tagged on. If omitted, the xCAT management network will be assumed. For KVM, it is the interface name on the host.
-h|--help Display usage message.
-v|--version The Command Version.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To add node1, node2 and node3 to vlan 3.
chvlan 3 -n node1,node2,node3
2. To add node1, node2 and node3 to vlan 3 using eth1 interface.
chvlan 3 -n node1,node2,node3 -i eth1
3. TO remove node1, node2 and node3 from vlan 3.
chvlan -n node1,node2,node3 -d
4. To add KVM guests node1 and node2 to vlan 3
mkdef node1 arch=x86_64 groups=kvm,all installnic=mac primarynic=mac mgt=kvm
˓→netboot=pxe nfsserver=10.1.0.204 os=rhels6 profile=compute provmethod=install
˓→serialport=0 serialspeed=115200 vmcpus=1 vmhost=x3650n01 vmmemory=512
˓→vmnics=br0 vmstorage=nfs://10.1.0.203/vms
mkdef node2 arch=x86_64 groups=kvm,all installnic=mac primarynic=mac mgt=kvm
˓→netboot=pxe nfsserver=10.1.0.204 os=rhels6 profile=compute provmethod=install
˓→serialport=0 serialspeed=115200 vmcpus=1 vmhost=x3650n01 vmmemory=512
˓→vmnics=br0 vmstorage=nfs://10.1.0.203/vms
chvlan 3 -n node1,node2
mkvm node1,node2 -s 20G
rpower node1,node2 on
rinstall node1,node2
212
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
5. To remove KVM guests node1 and node2 from vlan 3
chvlan 3 -n node1,node2 -d
rpower node1,node2 off
rmvm node1,node2
FILES
/opt/xcat/bin/chvlan
SEE ALSO
mkvlan(1)|mkvlan.1, rmvlan(1)|rmvlan.1, lsvlan(1)|lsvlan.1
chvlanports.1
NAME
chvlanports - It adds or removes nodes’ switch interfaces for the vlan.
SYNOPSIS
chvlanports vlanid -n | --nodes noderange -i | --interface nic
chvlanports vlanid -n | --nodes noderange -i | --interface nic -d | --delete
chvlanports [-h | --help]
chvlanports [-v | --version]
DESCRIPTION
The chvlanports command adds nodes switch interfaces to the given vlan. If -d is specified, the nodes switch interfaces
will be removed from the vlan.
This command won’t create/remove vlans on switches, it just add node’s switch ports into exisitng vlan or remove
them from existing vlan on switch. Before calling chvlanports, the nodes switch interfaces should be configured in
table switch, and vlan must already existing in switches. =head1 Parameters
vlanid is a unique vlan number.
OPTIONS
-n|--nodes The nodes or groups to be added or removed. It takes the noderange format. Check the man page for
noderange for details.
-i|--interface The interface name where the vlan will be tagged on.
-h|--help Display usage message.
1.3. Admin Guide
213
xCAT3 Documentation, Release 2.12
-v|--version The Command Version.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To add node1, node2 and node3 to vlan 3 using eth1 interface.
chvlanports 3 -n node1,node2,node3 -i eth1
2. TO remove eth1 interface of node1, node2 and node3 from vlan 3.
chvlanports 3 -n node1,node2,node3 -i eth1 -d
FILES
/opt/xcat/bin/chvlanports
SEE ALSO
mkvlan(1)|mkvlan.1, rmvlan(1)|rmvlan.1, lsvlan(1)|lsvlan.1, chvlan(1)|chvlan.1
chvm.1
NAME
chvm - Changes HMC-, DFM-, IVM-, and zVM-managed partition profiles or virtual machines. For Power 775,
chvm could be used to change the octant configuration values for generating LPARs; change the I/O slots assignment
to LPARs within the same CEC.
SYNOPSIS
chvm [-h| --help]
chvm [-v| --version]
PPC (with HMC) specific:
chvm [-V| --verbose] noderange [-p profile]
chvm [-V| --verbose] noderange attr=val [attr=val...]
214
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
PPC (using Direct FSP Management) specific:
chvm noderange --p775 [-p profile]
chvm noderange --p775 -i id [-m memory_interleaving] -r partition_rule
chvm noderange [lparname={ * | name}]
chvm noderange [vmcpus= min/req/max] [vmmemory= min/req/max] [vmothersetting=hugepage:N,bsr:N]
[add_physlots= drc_index1,drc_index2...] [add_vmnics= vlan1[,vlan2..]] [add_vmstorage=<N|viosnode:slotid>]
[--vios] [del_physlots= drc_index1,drc_index2...] [del_vadapter= slotid]
KVM specific:
chvm noderange [--cpupin hostcpuset]
chvm noderange [--membind numanodeset]
chvm noderange [--devpassthru pcidevice...]
chvm noderange [--devdetach pcidevice...]
VMware/KVM specific:
chvm noderange [-a size] [-d disk] [-p disk] [--resize disk=size] [--cpus count] [--mem memory]
zVM specific:
chvm noderange [--add3390 disk_pool device_address size mode read_password write_password multi_password]
chvm noderange [--add3390active device_address mode]
chvm noderange [--add9336 disk_pool device_address size mode read_password write_password multi_password]
chvm noderange [--adddisk2pool function region volume group]
chvm noderange [--addnic device_address type device_count]
chvm noderange [--addpagespool volume_address volume_label volume_use
tem_config_type parm_disk_owner parm_disk_number parm_disk_password]
system_config_name
sys-
chvm noderange [--addprocessor device_address]
chvm noderange [--addprocessoractive device_address type]
chvm noderange [--addvdisk device_address size]
chvm noderange [--addzfcp pool device_address loaddev size tag wwpn lun]
chvm noderange [--connectnic2guestlan device_address lan owner]
chvm noderange [--connectnic2vswitch device_address vswitch]
chvm noderange [--copydisk target_address source_node source_address]
chvm noderange [--dedicatedevice virtual_device real_device mode]
chvm noderange [--deleteipl]
chvm noderange [--disconnectnic device_address]
chvm noderange [--formatdisk device_address multi_password]
1.3. Admin Guide
215
xCAT3 Documentation, Release 2.12
chvm noderange [--grantvswitch vswitch]
chvm noderange [--purgerdr]
chvm noderange [--removedisk device_address]
chvm noderange [--removenic device_address]
chvm noderange [--removeprocessor device_address]
chvm noderange [--removeloaddev wwpn lun]
chvm noderange [--removezfcp device_address wwpn lun]
chvm noderange [--replacevs directory_entry]
chvm noderange [--setipl ipl_target load_parms parms]
chvm noderange [--setpassword password]
chvm noderange [--setloaddev wwpn lun]
chvm noderange [--sharevolume volume_address share_enable]
chvm noderange [--undedicatedevice device_address]
DESCRIPTION
PPC (with HMC) specific:
The chvm command modifies the partition profile for the partitions specified in noderange. A partitions current profile
can be read using lsvm, modified, and piped into the chvm command, or changed with the -p flag.
This command also supports to change specific partition attributes by specifying one or more “attribute equals value”
pairs in command line directly, without whole partition profile.
PPC (using Direct FSP Management) specific:
For Power 755(use option –p775 to specify):
chvm could be used to change the octant configuration values for generating LPARs. chvm is designed to set the
Octant configure value to split the CPU and memory for partitions, and set Octant Memory interleaving value. The
chvm will only set the pending attributes value. After chvm, the CEC needs to be rebooted manually for the pending
values to be enabled. Before reboot the cec, the administrator can use chvm to change the partition plan. If the partition
needs I/O slots, the administrator should use chvm to assign the I/O slots.
chvm is also designed to assign the I/O slots to the new LPAR. Both the current IO owning lpar and the new IO owning
lpar must be powered off before an IO assignment. Otherwise, if the I/O slot is belonged to an Lpar and the LPAR is
power on, the command will return an error when trying to assign that slot to a different lpar.
The administrator should use lsvm to get the profile content, and then edit the content, and add the node name with
”:” manually before the I/O which will be assigned to the node. And then the profile can be piped into the chvm
command, or changed with the -p flag.
For normal power machine:
chvm could be used to modify the resources assigned to partitions. The admin shall specify the attributes with options
vmcpus, vmmemory, add_physlots, vmothersetting, add_vmnics and/or add_vmstorage. If nothing specified, nothing
will be returned.
216
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
zVM specific:
The chvm command modifies the virtual machine’s configuration specified in noderange.
OPTIONS
Common:
-h
Display usage message.
-v
Command Version.
PPC (with HMC) specific:
-p profile
Name of an existing partition profile.
attr=val
Specifies one or more “attribute equals value” pairs, separated by spaces.
-V
Verbose output.
PPC (using Direct FSP Management) specific:
--p775
Specify the operation is for Power 775 machines.
-i
Starting numeric id of the newly created partitions. For Power 775 using Direct FSP Management, the id
value only could be 1, 5, 9, 13, 17, 21, 25 and 29. Shall work with option --p775.
-m
memory interleaving. The setting value only could be 1 or 2. 2 means non-interleaved mode (also 2MC
mode), the memory cannot be shared across the processors in an octant. 1 means interleaved mode (also
8MC mode) , the memory can be shared. The default value is 1. Shall work with option --p775.
-r
partition rule. Shall work with option --p775.
If all the octants configuration value are same in one CEC, it will be ” -r 0-7:value” .
If the octants use the different configuration value in one cec, it will be “-r 0:value1,1:value2,...7:value7”,
or “-r 0:value1,1-7:value2” and so on.
The octants configuration value for one Octant could be 1, 2, 3, 4, 5. The meanings of the octants
configuration value are as following:
1.3. Admin Guide
217
xCAT3 Documentation, Release 2.12
1
2
3
4
5
------
1
2
3
4
2
partition with all cpus and memory of the octant
partitions with a 50/50 split of cpus and memory
partitions with a 25/25/50 split of cpus and memory
partitions with a 25/25/25/25 split of cpus and memory
partitions with a 25/75 split of cpus and memory
-p profile
Name of I/O slots assignment profile. Shall work with option --p775.
lparname={\* | name}
Set LPAR name for the specified lpars. If ‘*’ specified, it means to get names from xCAT database and
then set them for the specified lpars. If a string is specified, it only supports single node and the string will
be set for the specified lpar. The user can use lsvm to check the lparnames for lpars.
vmcpus=value vmmemory=value add_physlots=value vmothersetting=value
To specify the parameters that will be modified.
add_vmnics=value add_vmstorage=value [--vios]
To create new virtual adapter for the specified node.
del_physlots=drc_index1,drc_index2...
To delete physical slots which are specified by the drc_index1,drc_index2....
del_vadapter=slotid
To delete a virtual adapter specified by the slotid.
VMware/KVM specific:
-a size
Add a new Hard disk with size defaulting to GB. Multiple can be added with comma separated values.
--cpus count
Set the number of CPUs.
-d disk
Deregister the Hard disk but leave the backing files. Multiple can be done with comma separated values.
The disks are specified by SCSI id.
--mem memory
Set the memory, defaults to MB.
-p disk
Purge the Hard disk. Deregisters and deletes the files. Multiple can be done with comma separated values.
The disks are specified by SCSI id.
--resize disk=size
Change the size of the Hard disk. The disk in qcow2 format can not be set to less than its current size.
The disk in raw format can be resized smaller, use caution. Multiple disks can be resized by using comma
separated disk=size pairs. The disks are specified by SCSI id. Size defaults to GB.
218
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
KVM specific:
--cpupin hostcpuset
To pin guest domain virtual CPUs to physical host CPUs specified with hostcpuset. hostcpuset is a list of
physical CPU numbers. Its syntax is a comma separated list and a special markup using ‘-‘ and ‘^’ (ex.
‘0-4’, ‘0-3,^2’) can also be allowed. The ‘-‘ denotes the range and the ‘^’ denotes exclusive.
Note: The expression is sequentially evaluated, so “0-15,^8” is identical to “9-14,0-7,15” but not identical
to “^8,0-15”.
--membind numanodeset
It is possible to restrict a guest to allocate memory from the specified set of NUMA nodes numanodeset.
If the guest vCPUs are also pinned to a set of cores located on that same set of NUMA nodes, memory
access is local and improves memory access performance.
--devpassthru pcidevice1,pcidevice2...
The PCI passthrough gives a guest VM direct access to I/O devices pcidevice1,pcidevice2.... The PCI
devices are assigned to a virtual machine, and the virtual machine can use this I/O exclusively. The
devices list are a list of comma separated PCI device names delimited with comma, the PCI device names
can be obtained by running virsh nodedev-list on the host.
--devdetach pcidevice1,pcidevice2...
To detaching the PCI devices which are attached to VM guest via PCI passthrough from the VM guest.
The devices list are a list of comma separated PCI device names delimited with comma, the PCI device
names can be obtained by running virsh nodedev-list on the host.
zVM specific:
--add3390 disk_pool device_address size mode read_password write_password multi_password
Adds a 3390 (ECKD) disk to a virtual machine’s directory entry. The device address can be automatically
assigned by specifying ‘auto’. The size of the disk can be specified in GB, MB, or the number of cylinders.
--add3390active device_address mode
Adds a 3390 (ECKD) disk that is defined in a virtual machine’s directory entry to that virtual server’s
active configuration.
--add9336 disk_pool device_address size mode read_password write_password multi_password
Adds a 9336 (FBA) disk to a virtual machine’s directory entry. The device address can be automatically
assigned by specifying ‘auto’. The size of the disk can be specified in GB, MB, or the number of blocks.
--adddisk2pool function region volume group
Add a disk to a disk pool defined in the EXTENT CONTROL. Function type can be either: (4) Define
region as full volume and add to group OR (5) Add existing region to group. The disk has to already be
attached to SYSTEM.
--addnic device_address type device_count
Adds a network adapter to a virtual machine’s directory entry (case sensitive).
--addpagespool volume_addr volume_label volume_use system_config_name system_config_type parm_disk_owner
parm_disk_number parm_disk_password
Add a full volume page or spool disk to the virtual machine.
--addprocessor device_address
1.3. Admin Guide
219
xCAT3 Documentation, Release 2.12
Adds a virtual processor to a virtual machine’s directory entry.
--addprocessoractive device_address type
Adds a virtual processor to a virtual machine’s active configuration (case sensitive).
--addvdisk device_address size
Adds a v-disk to a virtual machine’s directory entry.
--addzfcp pool device_address loaddev size tag wwpn lun
Add a zFCP device to a device pool defined in xCAT. The device must have been carved up in the storage
controller and configured with a WWPN/LUN before it can be added to the xCAT storage pool. z/VM
does not have the ability to communicate directly with the storage controller to carve up disks dynamically.
xCAT will find the a zFCP device in the specified pool that meets the size required, if the WWPN and LUN
are not given. The device address can be automatically assigned by specifying ‘auto’. The WWPN/LUN
can be set as the LOADDEV in the directory entry if (1) is specified as the ‘loaddev’.
--connectnic2guestlan device_address lan owner
Connects a given network adapter to a GuestLAN.
--connectnic2vswitch device_address vswitch
Connects a given network adapter to a VSwitch.
--copydisk target_address source_node source_address
Copy a disk attached to a given virtual server.
--dedicatedevice virtual_device real_device mode
Adds a dedicated device to a virtual machine’s directory entry.
--deleteipl
Deletes the IPL statement from the virtual machine’s directory entry.
--disconnectnic device_address
Disconnects a given network adapter.
--formatdisk disk_address multi_password
Formats a disk attached to a given virtual server (only ECKD disks supported). The disk should not be
linked to any other virtual server. This command is best used after add3390().
--grantvswitch vswitch
Grant vSwitch access for given virtual machine.
--purgerdr
Purge the reader belonging to the virtual machine
--removedisk device_address
Removes a minidisk from a virtual machine’s directory entry.
--removenic device_address
Removes a network adapter from a virtual machine’s directory entry.
--removeprocessor device_address
Removes a processor from an active virtual machine’s configuration.
--removeloaddev wwpn lun
220
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
Removes the LOADDEV statement from a virtual machines’s directory entry.
--removezfcp device_address wwpn lun
Removes a given SCSI/FCP device belonging to the virtual machine.
--replacevs directory_entry
Replaces a virtual machine’s directory entry. The directory entry can be echoed into stdin or a text file.
--setipl ipl_target load_parms parms
Sets the IPL statement for a given virtual machine.
--setpassword password
Sets the password for a given virtual machine.
--setloaddev wwpn lun
Sets the LOADDEV statement in the virtual machine’s directory entry.
--undedicatedevice device_address
Delete a dedicated device from a virtual machine’s active configuration and directory entry.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
PPC (with HMC) specific:
1. To change the partition profile for lpar4 using the configuration data in the file /tmp/lparfile, enter:
cat /tmp/lparfile | chvm lpar4
Output is similar to:
lpar4: Success
2. To change the partition profile for lpar4 to the existing profile ‘prof1’, enter:
chvm lpar4 -p prof1
Output is similar to:
lpar4: Success
3. To change partition attributes for lpar4 by specifying attribute value pairs in command line, enter:
chvm lpar4 max_mem=4096
Output is similar to:
lpar4: Success
1.3. Admin Guide
221
xCAT3 Documentation, Release 2.12
PPC (using Direct FSP Management) specific:
1. For Power 775, to create a new partition lpar1 on the first octant of the cec cec01, lpar1 will use all the cpu and
memory of the octant 0, enter:
mkdef -t node -o lpar1 mgt=fsp groups=all parent=cec01
nodetype=lpar
hcp=cec01
then:
chvm lpar1 --p775 -i 1 -m 1 -r 0:1
Output is similar to:
lpar1: Success
cec01: Please reboot the CEC cec1 firstly, and then use chvm to assign the I/O slots
˓→to the LPARs
2. For Power 775, to create new partitions lpar1-lpar8 on the whole cec cec01, each LPAR will use all the cpu and
memory of each octant, enter:
mkdef -t node -o lpar1-lpar8 nodetype=lpar
mgt=fsp groups=all parent=cec01
hcp=cec01
then:
chvm lpar1-lpar8 --p775 -i 1 -m 1 -r 0-7:1
Output is similar to:
lpar1: Success
lpar2: Success
lpar3: Success
lpar4: Success
lpar5: Success
lpar6: Success
lpar7: Success
lpar8: Success
cec01: Please reboot the CEC cec1 firstly, and then use chvm to assign the I/O slots
˓→to the LPARs
3. For Power 775 cec1, to create new partitions lpar1-lpar9, the lpar1 will use 25% CPU and 25% memory of the
first octant, and lpar2 will use the left CPU and memory of the first octant. lpar3-lpar9 will use all the cpu and
memory of each octant, enter:
mkdef -t node -o lpar1-lpar9 mgt=fsp groups=all parent=cec1
nodetype=lpar
hcp=cec1
then:
chvm lpar1-lpar9 --p775 -i 1 -m 1
-r 0:5,1-7:1
Output is similar to:
lpar1:
lpar2:
lpar3:
lpar4:
lpar5:
lpar6:
222
Success
Success
Success
Success
Success
Success
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
lpar7: Success
lpar8: Success
lpar9: Success
cec1: Please reboot the CEC cec1 firstly, and then use chvm to assign the I/O slots
˓→to the LPARs
4.To change the I/O slot profile for lpar4 using the configuration data in the file /tmp/lparfile, the I/O slots information
is similar to:
4: 514/U78A9.001.0123456-P1-C17/0x21010202/2/1
4: 513/U78A9.001.0123456-P1-C15/0x21010201/2/1
4: 512/U78A9.001.0123456-P1-C16/0x21010200/2/1
then run the command:
cat /tmp/lparfile | chvm lpar4 --p775
5. To change the I/O slot profile for lpar1-lpar8 using the configuration data in the file /tmp/lparfile. Users can
use the output of lsvm.and remove the cec information, and modify the lpar id before each I/O, and run the
command as following:
chvm lpar1-lpar8 --p775 -p /tmp/lparfile
6. To change the LPAR name, enter:
chvm lpar1 lparname=test_lpar01
Output is similar to:
lpar1: Success
7. For Normal Power machine, to modify the resource assigned to a partition:
Before modify, the resource assigned to node ‘lpar1’ can be shown with:
lsvm lpar1
The output is similar to:
lpar1: Lpar Processor Info:
Curr Processor Min: 1.
Curr Processor Req: 4.
Curr Processor Max: 16.
lpar1: Lpar Memory Info:
Curr Memory Min: 1.00 GB(4 regions).
Curr Memory Req: 4.00 GB(16 regions).
Curr Memory Max: 32.00 GB(128 regions).
lpar1: 1,513,U78AA.001.WZSGVU7-P1-T7,0x21010201,0xc03(USB Controller)
lpar1: 1,512,U78AA.001.WZSGVU7-P1-T9,0x21010200,0x104(RAID Controller)
lpar1: 1/2/2
lpar1: 128.
To modify the resource assignment:
chvm lpar1 vmcpus=1/2/16 vmmemory=1G/8G/32G add_physlots=0x21010202
The output is similar to:
1.3. Admin Guide
223
xCAT3 Documentation, Release 2.12
lpar1: Success
The resource information after modification is similar to:
lpar1: Lpar Processor Info:
Curr Processor Min: 1.
Curr Processor Req: 2.
Curr Processor Max: 16.
lpar1: Lpar Memory Info:
Curr Memory Min: 1.00 GB(4 regions).
Curr Memory Req: 8.00 GB(32 regions).
Curr Memory Max: 32.00 GB(128 regions).
lpar1: 1,514,U78AA.001.WZSGVU7-P1-C19,0x21010202,0xffff(Empty Slot)
lpar1: 1,513,U78AA.001.WZSGVU7-P1-T7,0x21010201,0xc03(USB Controller)
lpar1: 1,512,U78AA.001.WZSGVU7-P1-T9,0x21010200,0x104(RAID Controller)
lpar1: 1/2/2
lpar1: 128.
Note: The physical I/O resources specified with add_physlots will be appended to the specified partition. The physical
I/O resources which are not specified but belonged to the partition will not be removed. For more information about
add_physlots, refer to lsvm(1)|lsvm.1.
VMware/KVM specific:
chvm vm1 -a 8,16 --mem 512 --cpus 2
Output is similar to:
vm1: node successfully changed
zVM specific:
1. To adds a 3390 (ECKD) disk to a virtual machine’s directory entry:
chvm gpok3 --add3390 POOL1 0101 2G MR
Output is similar to:
gpok3: Adding disk 0101 to LNX3... Done
2. To add a network adapter to a virtual machine’s directory entry:
chvm gpok3 --addnic 0600 QDIO 3
Output is similar to:
gpok3: Adding NIC 0900 to LNX3... Done
3. To connects a given network adapter to a GuestLAN:
chvm gpok3 --connectnic2guestlan 0600 GLAN1 LN1OWNR
Output is similar to:
224
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
gpok3: Connecting NIC 0600 to GuestLan GLAN1 on LN1OWNR... Done
4. To connects a given network adapter to a vSwitch:
chvm gpok3 --connectnic2vswitch 0600 VSW1
Output is similar to:
gpok3: Connecting NIC 0600 to vSwitch VSW1 on LNX3... Done
5. To removes a minidisk from a virtual machine’s directory entry:
chvm gpok3 --removedisk 0101
Output is similar to:
gpok3: Removing disk 0101 on LNX3... Done
6. To Removes a network adapter from a virtual machine’s directory entry:
chvm gpok3 --removenic 0700
Output is similar to:
gpok3: Removing NIC 0700 on LNX3... Done
7. To replaces a virtual machine’s directory entry:
cat /tmp/dirEntry.txt | chvm gpok3 --replacevs
Output is similar to:
gpok3: Replacing user entry of LNX3... Done
8. To resize virtual machine’s disk sdb to 10G and sdc to 15G:
chvm gpok3 --resize sdb=10G,sdc=15G
FILES
/opt/xcat/bin/chvm
SEE ALSO
mkvm(1)|mkvm.1, lsvm(1)|lsvm.1, rmvm(1)|rmvm.1
chzone.1
NAME
chzone - Changes a zone defined in the cluster.
1.3. Admin Guide
225
xCAT3 Documentation, Release 2.12
SYNOPSIS
chzone zonename [--defaultzone] [-K] [-k full path to the ssh RSA private key] [-a noderange | -r noderange] [-g] [-f]
[-s {yes|no}] [-V]
chzone [-h | -v]
DESCRIPTION
The chzone command is designed to change the definition of a zone previous defined in the cluster. The chzone
command is only supported on Linux ( No AIX support). The nodes are not updated with the new root ssh keys by
chzone. You must run updatenode -k or xdsh -K to the nodes to update the root ssh keys to the new generated zone
keys. This will also sync any service nodes with the zone keys, if you have a hierarchical cluster. Note: if any zones
in the zone table, there must be one and only one defaultzone. Otherwise, errors will occur.
OPTIONS
-h | --help
Displays usage information.
-v | --version
Displays command version and build date.
-k | --sshkeypath full path to the ssh RSA private key
This is the path to the id_rsa key that will be used to build new root’s ssh keys for the zone. If k is used, it will generate the ssh public key from the input ssh RSA private key, and store both in
/etc/xcat/sshkeys/<zonename>/.ssh directory.
-K | --genkeys
Using this flag, will generate new ssh RSA private and public keys for the zone into the
/etc/xcat/sshkeys/<zonename>/.ssh directory. The nodes are not automatically updated with the new root
ssh keys by chzone. You must run updatenode -k or xdsh -K to the nodes to update the root ssh keys to
the new generated zone keys. This will also sync any service nodes with the zone keys, if you have a
hierarchical cluster.
--defaultzone
if --defaultzone is input, then it will set the zone defaultzone attribute to yes. if --defaultzone is input and
another zone is currently the default, then the -f flag must be used to force a change to the new defaultzone.
If -f flag is not use an error will be returned and no change made. Note: if any zones in the zone table,
there must be one and only one defaultzone. Otherwise, errors will occur.
-a | --addnoderange noderange
For each node in the noderange, it will set the zonename attribute for that node to the input zonename.
If the -g flag is also on the command, then it will add the group name “zonename” to each node in the
noderange.
-r | --rmnoderange noderange
For each node in the noderange, if the node is a member of the input zone, it will remove the zonename
attribute for that node. If any of the nodes in the noderange is not a member of the zone, you will get an
error and nothing will be changed. If the -g flag is also on the command, then it will remove the group
name “zonename” from each node in the noderange.
226
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
-s| --sshbetweennodes yes|no
If -s entered, the zone sshbetweennodes attribute will be set to yes or no based on the input. When this is
set to yes, then ssh will be setup to allow passwordless root access between nodes. If no, then root will be
prompted for a password when running ssh between the nodes in the zone.
-f | --force
Used with the --defaultzone flag to override the current default zone.
-g | --assigngroup
Used with the -a or -r flag to add or remove the group zonename for all nodes in the input noderange.
-V | --Verbose
Verbose mode.
EXAMPLES
1. To chzone zone1 to the default zone, enter:
chzone> zone1 --default -f
2. To generate new root ssh keys for zone2A using the ssh id_rsa private key in /root/.ssh:
chzone zone2A -k /root/.ssh
Note: you must use xdsh -K or updatenode -k to update the nodes with the new keys
3. To generate new root ssh keys for zone2A, enter :
chzone zone2A -K
Note: you must use xdsh -K or updatenode -k to update the nodes with the new keys
4. To add a new group of nodes (compute3) to zone3 and add zone3 group to the nodes, enter:
chzone zone3 -a compute3 -g
5. To remove a group of nodes (compute4) from zone4 and remove zone4 group from the nodes, enter:
chzone> zone4 -r compute4 -g
6. To change the sshbetweennodes setting on the zone to not allow passwordless ssh between nodes, enter:
chzone zone5 -s no
Note: you must use xdsh -K or updatenode -k to update the nodes with this new setting.
FILES
/opt/xcat/bin/chzone/
Location of the chzone command.
1.3. Admin Guide
227
xCAT3 Documentation, Release 2.12
SEE ALSO
L <mkzone(1)|mkzone.1>,L <rmzone(1)|rmzone.1>,L <xdsh(1)|xdsh.1>, updatenode(1)|updatenode.1
clonevm.1
NAME
clonevm - Create masters from virtual machines and virtual machines from masters.
SYNOPSIS
clonevm noderange [ -t master_to_be_made | -b master_to_base_vms_upon ] [ -d|--detached] [-f|--force]
DESCRIPTION
Command to promote a VM’s current configuration and storage to a master as well as performing the converse operation of creating VMs based on a master.
By default, attempting to create a master from a running VM will produce an error. The --force argument will request
that a master be made of the VM anyway.
Also, by default a VM that is used to create a master will be rebased as a thin clone of that master. If the --force
argument is used to create a master of a powered on vm, this will not be done. Additionally, the --detached option can
be used to explicitly request that a clone not be tethered to a master image, allowing the clones to not be tied to the
health of a master, at the cost of additional storage.
When promoting a VM’s current state to master, all related virtual disks will be copied and merged with any prerequisite images. A master will not be tethered to other masters.
OPTIONS
-h|--help
Display usage message.
-b master_to_base_vms_upon
The master to base the clones upon
-t master_to_be_made
The target master to copy a single VM’s state to
-d|--detached
Explicitly request that the noderange be untethered from any masters.
-f|--force
Force cloning of a powered on VM. Implies -d if the VM is on.
-v|--version
Command Version.
-V|--verbose
228
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
Verbose output.
RETURN VALUE
0: The command completed successfully.
Any other value: An error has occurred.
EXAMPLES
1. Creating a master named appserver from a node called vm1:
clonevm vm1 -t appserver
2. Cleating 30 VMs from a master named appserver:
clonevm vm1-vm30 -b appserver
FILES
/opt/xcat/bin/clonevm
SEE ALSO
chvm(1)|chvm.1, lsvm(1)|lsvm.1, rmvm(1)|rmvm.1, mkvm(1)|mkvm.1, vmmaster(5)|vmmaster.5
configfpc.1
NAME
configfpc - discover the Fan Power Controllers (FPCs) and configure the FPC interface
SYNOPSIS
configfpc -i interface
configfpc -i interface --ip default ip address
configfpc [-V | --verbose]
configfpc [-h | --help | -?]
DESCRIPTION
configfpc will discover and configure all FPCs that are set to the default IP address. If not supplied the default ip is
192.168.0.100.
The -i interface is required to direct configfpc to the xCAT MN interface which is on the same VLAN as the FPCs.
There are several bits of information that must be included in the xCAT database before running this command.
1.3. Admin Guide
229
xCAT3 Documentation, Release 2.12
You must create the FPC node definitions for all FPCs being discovered including the IP address and switch port
information.
The configfpc command discovers the FPCs and collects the MAC address. The MAC address is used to relate the FPC
to a FPC node using the switch information for this MAC. Once the relationship is discovered the FPC is configured
with the FPC node IP settings.
This process is repeated until no more FPCs are discovered.
For more information on
XCAT_NeXtScale_Clusters
xCAT
support
of
NeXtScale
and
configfpc
see
the
following
doc:
OPTIONS
-i interface
Use this flag to specify which xCAT MN interface (example: eth4) that is connected to the NeXtScale
FPCs. This option is required.
--ip default ip address
Use this flag to override the default ip address of 192.168.0.100 with a new address.
-V | --verbose
Verbose mode
EXAMPLES
1. To discover and configure all NeXtScale Fan Power Controllers (FPCs) connected on eth0 interface.
configfpc -i eth0
2. To override the default ip address and run in Verbose mode.
configfpc -i eth0 --ip 196.68.0.100 -V
csm2xcat.1
NAME
csm2xcat - Allows the migration of a CSM database to an xCAT database.
SYNOPSIS
csm2xcat [--dir path]
csm2xcat [-h]
230
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
DESCRIPTION
The csm2xcat command must be run on the Management Server of the CSM system that you want to migrate to xCAT.
The command will build two xCAT stanza files that can update the xCAT database with the chdef command.
Copy the csm2xcat command to the CSM Management Server. Run the command, indicating where you want your
stanza files saved with the --dir parameter. Check the stanza files to see if the information is what you want put in the
xCAT database. Copy the two stanza files: node.stanza, device.stanza back to your xCAT Management node, and run
the chdef command to input into the xCAT database.
OPTIONS
-h Display usage message.
--dir Path to the directory containing the stanza files.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To build xCAT stanza files, enter on the CSM Management Server:
csm2xcat --dir /tmp/mydir
2. To put the data in the xCAT database on the xCAT Management Node:
cat node.stanza | chdef -z
cat device.stanza | chdef -z
FILES
/opt/xcat/share/xcat/tools/csm2xcat
$dir/conversion.log
SEE ALSO
chdef(1)|chdef.1
db2sqlsetup.1
NAME
db2sqlsetup - Sets up the IBM DB2 for xCAT to use.
1.3. Admin Guide
231
xCAT3 Documentation, Release 2.12
SYNOPSIS
db2sqlsetup [-h | --help}
db2sqlsetup [-v | --version}
db2sqlsetup [-i | --init] {-S | -C} [-o | --setupODBC] [-V | --verbose]
db2sqlsetup [-i | --init] {-S} [-N | --nostart] [-o | --setupODBC] [-V | --verbose]
db2sqlsetup [-o | --setupODBC] {-S | -C} [-V | --verbose]
db2sqlsetup [-p | --passwd] [-S | -C]
DESCRIPTION
db2sqlsetup - Sets up the IBM DB2 database for xCAT to use. The db2sqlsetup script is run on the Management
Node, after the DB2 Server code has been installed, to setup the DB2 Server (-S). The xcatd daemon will be stopped
during migration on the MN. No xCAT commands should be run during the init process, because we will be migrating
the xCAT database to DB2 and restarting the xcatd daemon.
The db2sqlsetup script must be run on each Service Node, after the DB2 Client code has been installed, to setup the
DB2 Client (-C). There are two postscripts that are provided ( db2install and odbcsetup) that will automatically setup
you Service Node as a DB2 client.
For full information on the setup of DB2, see Setting_Up_DB2_as_the_xCAT_DB.
When running of db2sqlsetup on the MN: One password must be supplied for the setup, a password for the xcatdb unix
id which will be used as the DB2 instance id and database name. The password will be prompted for interactively or
can be input with the XCATDB2PW environment variable. The script will create the xcat database instance (xcatdb)
in the /var/lib/db2 directory unless overridden by setting the site.databaseloc attribute. This attribute should not be set
to the directory that is defined in the installloc attribute and it is recommended that the databaseloc be a new filesystem
dedicated to the DB2 database, especially in very large clusters.
When running db2sqlseutp on the SN: Not only will the password for the DB2 instance Id be prompted for and must
match the one on the Management Node; but also the hostname or ip address of the Management Node as known by
the Service Node must be supplied , unless the XCATDB2SERVER environment variable is set. You can automatically
install and setup of DB2 on the SN using the db2install and odbcsetup postscripts and not need to manually run the
command. See the full documentation.
Note: On AIX , root must be running ksh and on Linux, bash shell.
OPTIONS
-h|--help
Displays the usage message.
-v|--version
Displays the release version of the code.
-V|--verbose
Displays verbose messages.
-i|--init
232
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
The init option is used to setup an installed DB2 database on AIX or Linux (p-Series) so that xCAT can
use the database. This must be combined with either the -S or -C flag to indicate whether we are setting
up the Server or the Client. With the -S flag, it involves creating the xcatdb database, the xcatdb instance
id, allowing access to the xcatdb database by the Management Node. It also backs up the current xCAT
database and restores it into the newly setup xcatdb DB2 database. It creates the /etc/xcat/cfgloc file to
point the xcatd daemon to the DB2 database and restarts the xcatd daemon using the database.
-p|--passwd
The password change option is to change the database access password for the DB2 xcatdb database. If -S
is input then it will only change the password on the DB2 Server (MN). If -C is input it will only change
on the DB2 clients (SN). If neither -S or -C are input with this flag, then it will change both the DB2
Server and Clients. When changing the password the xcatd daemon will be stopped and restarted. Any
other tools accessing the database should also be stopped before changing and restarted after changing.
-S|-C
This options says whether to setup the Server (-S) on the Management Node, or the Client (-C) on the
Service Nodes.
-N|--nostart
This option with the -S flag will create the database, but will not backup and restore xCAT tables into the
database. It will create the cfgloc file such that the next start of xcatd will try and contact the database.
This can be used to setup the xCAT DB2 database during or before install.
-o|--setupODBC
This option sets up the ODBC /etc/../odbcinst.ini, /etc/../odbc.ini and the .odbc.ini file in roots home
directory will be created and initialized to run off the xcatdb DB2 database.
ENVIRONMENT VARIABLES
* XCATDB2INSPATH overrides the default install path for DB2 which is /opt/ibm/db2/V9.7 for Linux and
/opt/IBM/db2/V9.7 for AIX.
* DATABASELOC override the where to create the xcat DB2 database, which is /var/lib/db2 by default of taken from
the site.databaseloc attribute.
* XCATDB2PW can be set to the password for the xcatdb DB2 instance id so that there will be no prompting for a
password when the script is run.
EXAMPLES
1. To setup DB2 Server for xCAT to run on the DB2 xcatdb database, on the MN:
db2sqlsetup -i -S
2. To setup DB2 Client for xCAT to run on the DB2 xcatdb database, on the SN:
db2sqlsetup -i -C
3. To setup the ODBC for DB2 xcatdb database access, on the MN :
db2sqlsetup -o -S
4. To setup the ODBC for DB2 xcatdb database access, on the SN :
1.3. Admin Guide
233
xCAT3 Documentation, Release 2.12
db2sqlsetup -o -C
5. To setup the DB2 database but not start xcat running with it:
db2sqlsetup -i -S -N
6. To change the DB2 xcatdb password on both the Management and Service Nodes:
db2sqlsetup -p
dumpxCATdb.1
NAME
dumpxCATdb - dumps the xCAT db tables .
SYNOPSIS
dumpxCATdb [-a] [-V] [{-p | --path} path]
dumpxCATdb [-b] [-V] [{-p | --path} path]
dumpxCATdb [-h | --help] [-v | --version]
DESCRIPTION
If not using the binary dump option (-b), then the dumpxCATdb command creates .csv files for xCAT database tables
and puts them in the directory given by the -p flag. These files can be used by the restorexCATdb command to restore
the database. The command will read the list of tables in the site.skiptables attribute and not backup those tables.
Supports using XCAT_SKIPTABLES env variable to provide a list of skip tables. The command will never backup
TEAL or ISNM tables, except isnm_config. To dump TEAL tables use the documented process for TEAL. For ISNM
use tabdump, after using tabprune to get to prune unnecessary records.
If using the binary dump option for the DB2 or postgreSQL database, then the routine will use the Database provide
utilities for backup of the entire database.
OPTIONS
-h Display usage message.
-v Command Version.
-V Verbose.
-a All, without this flag the eventlog and auditlog will be skipped.
-b This flag is only used for the DB2 or postgreSQL database. The routine will use the database backup utilities to
create a binary backup of the entire database. Note to use this backup on DB2, you will have first had to modify the
logging of the database and have taken an offline initial backup. Refer to the xCAT DB2 documentation for more
instructions.
-p Path to the directory to dump the database. It will be created, if it does not exist.
234
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To dump the xCAT database into the /tmp/db directory, enter:
dumpxCATdb -p /tmp/db
2. To dump the xCAT database into the /tmp/db directory, including the auditlog and eventlog enter:
dumpxCATdb -a -p /tmp/db
3. To have dumpxCATdb not backup the hosts or passwd table:
chtab key=skiptables site.value="hosts,passwd"
dumpxCATdb
-p /tmp/db
4. To have dumpxCATdb not backup the hosts or passwd table:
export XCAT_SKIPTABLES="hosts,passwd"
dumpxCATdb
-p /tmp/db
5. To have dumpxCATdb use DB2 utilities to backup the DB2 database:
dumpxCATdb -b -p /install/db2backup
FILES
/opt/xcat/sbin/dumpxCATdb
SEE ALSO
restorexCATdb(1)|restorexCATdb.1
genimage.1
NAME
genimage - Generates a stateless image to be used for a diskless install.
SYNOPSIS
genimage
1.3. Admin Guide
235
xCAT3 Documentation, Release 2.12
genimage [-o osver] [-a arch] [-p profile] [-i nodebootif ] [-n nodenetdrivers] [--onlyinitrd] [-r otherifaces] [k kernelver] [-g krpmver] [-m statelite] [-l rootlimitsize] [--permission permission] [--interactive] [--dryrun] [-ignorekernelchk] [--noupdate] imagename
genimage [-h | --help | -v | --version]
DESCRIPTION
Generates a stateless and a statelite image that can be used to boot xCAT nodes in a diskless mode.
genimage will use the osimage definition for information to generate this image. Additional options specified on the
command line will override any corresponding previous osimage settings, and will be written back to the osimage
definition.
If genimage runs on the management node, both the osimage table and linuximage table will be updated with the given
values from the options.
The genimage command will generate two initial ramdisks for stateless and statelite, one is initrd-stateless.gz, the
other one is initrd-statelite.gz.
After your image is generated, you can chroot to the image, install any additional software you would like, or make
modifications to files, and then run the following command to prepare the image for deployment.
for stateless: packimage
for statelite: liteimg
Besides prompting for some parameter values, the genimage command takes default guesses for the parameters not
specified or not defined in the osimage and linuximage tables. It also assumes default answers for questions from
the yum/zypper command when installing rpms into the image. Use --interactive flag if you want the yum/zypper
command to prompt you for the answers.
If --onlyinitrd is specified, genimage only regenerates the initrd for a stateless image to be used for a diskless install.
The genimage command must be run on a system that is the same architecture and same distro with same major release
version as the nodes it will be used on. If the management node is not the same architecture or same distro level, copy
the contents of /opt/xcat/share/xcat/netboot/<os> to a system that is the proper architecture, and mount /install from
the management node to that system. Then change directory to /opt/xcat/share/xcat/netboot/<os> and run ./genimage.
Parameters
imagename specifies the name of an os image definition to be used. The specification for the image is stored in the
osimage table and linuximage table.
OPTIONS
-a arch
The hardware architecture of this node: x86_64, ppc64, x86, ia64, etc. If omitted, the current hardware
architecture will be used.
-o osver
The operating system for the image: fedora8, rhel5, sles10, etc. The OS packages must be in /install/<osver>/<arch> (use copycds(8)|copycds.8).
-p profile
236
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
The profile (e.g. compute, service) to use to create the image. This determines what package lists are used
from /opt/xcat/share/xcat/netboot/<os> to create the image with. When deploying nodes with this image,
the nodes’ nodetype.profile attribute must be set to this same value.
-i nodebootif
This argument is now optional, and allows you to specify the network boot interface to be configured in
the image (e.g. eth0). If not specified, the interface will be determined and configured during the network
boot process.
-n nodenetdrivers
This argument is now optional, and allows you to specify the driver modules needed for the network
interface(s) on your stateless nodes. If you do not specify this option, the default is to include all recent
IBM xSeries network drivers.
If specified, nodenetdrivers should be a comma separated list of network drivers to be used by the stateless
nodes (Ie.: -n tg3,e1000). Note that the drivers will be loaded in the order that you list them, which may
prove important in some cases.
-l rootlimit
The maximum size allowed for the root file system in the image. Specify in bytes, or can append k, m, or
g.
--onlyinitrd
Regenerates the initrd for a stateless image to be used for a diskless install.
Regenerates the initrd that is part of a stateless/statelite image that is used to boot xCAT nodes in a
stateless/stateli te mode.
The genimage --onlyinitrd command will generate two initial ramdisks, one is initrd-statelite.gz for
statelite mode, the other one is initrd-stateless.gz for stateless mode.
--permission permission
The mount permission of /.statelite directory for statelite mode, which is only used for statelite mode,
and the default permission is 755.
-r otherifaces
Other network interfaces (e.g. eth1) in the image that should be configured via DHCP.
-k kernelver
Use this flag if you want to use a specific version of the kernel in the image. Defaults to the first kernel
found in the install image.
-g krpmver
Use this flag to specify the rpm version for kernel packages in the image. It must be present if -k flag is
specified in the command for SLES. Generally, the value of -g is the part after linux- and before .rpm in
a kernel rpm name.
-m statelite
This flag is for Ubuntu, Debian and Fedora12 only. Use this flag to specify if you want to generate statelite
image. The default is to generate stateless image for these three operating systems. For others, this flag is
invalid because both stateless and statelite images will be generated with this command.
--interactive
1.3. Admin Guide
237
xCAT3 Documentation, Release 2.12
This flag allows the user to answer questions from yum/zypper command when installing rpms into the
image. If it is not specified, ‘-y’ will be passed to the yum command and ‘–non-interactive –no-gpgchecks’ will be passed to the zypper command as default answers.
--dryrun
This flag shows the underlying call to the os specific genimage function. The user can copy and the paste
the output to run the command on another machine that does not have xCAT installed.
-t tmplimit
(Deprecated) This flag allows the user to setup the /tmp and the /var/tmp file system sizes. This flag is no
longer supported. You can overwrite any file system size using the .postinstall script where you can create
a new /etc/fstab file.
--ignorekernelchk
Skip the kernel version checking when injecting drivers from osimage.driverupdatesrc. That means all
drivers from osimage.driverupdatesrc will be injected to initrd for the specific target kernel.
--noupdate
This flag allows the user to bypass automatic package updating when installing other packages.
-v|--version
Display version.
-h|--help
Display usage message.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1 To prompt the user for inputs:
genimage
2 To generate an image using information from an osimage definition:
genimage myimagename
3 To run genimage in test mode without actually generating an image:
genimage --dryrun
myimagename
4 To generate an image and have yum/zypper prompt for responses:
genimage myimagename --interactive
5 To generate an image, replacing some values in the osimage definition:
genimage -i eth0 -n tg3 myimagename
238
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
FILES
/opt/xcat/bin/genimage
/opt/xcat/share/xcat/netboot/<OS>/genimage
SEE ALSO
packimage(1)|packimage.1, liteimg(1)|liteimg.1
geninitrd.1
NAME
geninitrd - Generate an initrd (initial ramfs) which to be used for stateful install or stateless netboot.
SYNOPSIS
geninitrd imagename [--ignorekernelchk]
geninitrd [-h | --help]
DESCRIPTION
Generate the initrd for the osimage: imagename which is an xCAT object of osimage type.
Diskful Osimage
If the imagename is a stateful one (The provmethod attribute for the osimage is ‘install’), this command is used to
rebuild the initrd to inject the new drivers from driver rpms or ‘update distro’ and copy the rebuilt initrd and new kernel
(If there’s new kernel in ‘update distro’) to the directory /tftpboot/xcat/<imagename>.
If the initrd has been rebuilt by geninitrd, when run nodeset, the –noupdateinitrd option should be used to skip the
rebuilding of initrd to improve the performance.
Three attributes of osimage object can be used to specify the Driver RPM location and Driver names for injecting new
drivers to initrd.
netdrivers - comma separated driver names that need to be injected to the initrd. The postfix ‘.ko’ can be ignored.
The netdrivers attribute must be set to specify the new driver list. If you want to load all the drivers from the driver
rpms, using the keyword allupdate.
driverupdatesrc - comma separated driver rpm packages (full path should be specified)
osupdatename - comma separated ‘osdistroupdate’ object. Each ‘osdistroupdate’ object specifies a Linux distro
update. When run geninitrd, ‘kernel-*.rpm’ will be searched from osdistroupdate.dirpath to get all the rpm packages
and then search the drivers from the rpm packages.
Refer to the doc: Using_Linux_Driver_Update_Disk
Stateless Osimage
If the imagename is a stateless one (The provmethod attribute for the osimage is ‘netboot’), this command is used to
generate the initrd from the rootimg which generated by ‘genimage’ command. So the ‘genimage’ must be run once
before running the geninitrd command.
1.3. Admin Guide
239
xCAT3 Documentation, Release 2.12
Two attributes of osimage object can be used to specify the Driver RPM location and Driver names for injecting new
drivers to initrd.
netdrivers - comma separated driver names that need to be injected to the initrd. The postfix ‘.ko’ can be ignored.
The netdrivers attribute must be set to specify the new driver list. If you want to load all the drivers from the driver
rpms, using the keyword allupdate.
driverupdatesrc - comma separated driver rpm packages (full path should be specified)
Parameters
imagename specifies the name of an os image definition to be used. The specification for the image is stored in the
osimage table and linuximage table.
--ignorekernelchk
Skip the kernel version checking when injecting drivers from osimage.driverupdatesrc. That means all
drivers from osimage.driverupdatesrc will be injected to initrd for the specific target kernel.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1 To generate initrd for the osimage myimagename:
geninitrd myimagename
FILES
/opt/xcat/bin/geninitrd
/opt/xcat/bin/genimage
/opt/xcat/share/xcat/netboot/<OS>/genimage
SEE ALSO
geninitrd(1)|geninitrd.1, genimage(1)|genimage.1
getadapter.1
NAME
getadapter - Obtain all network adapters’s predictable name and some other information before provision or network
configuration.
240
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
SYNOPSIS
getadapter noderange [-f]
getadapter [-h | --help | -v | --version | -V]
DESCRIPTION
Traditionally, network interfaces in Linux are enumerated as eth[0123...], but these names do not necessarily correspond to actual labels on the chassis. getadapter help customer to get predictable network device name and some
other network adapter information before provision or network configuration.
Since getadpter uses genesis to collect network adapters information, the target node will be restarted.
getadapter For each node within the <noderange>, follows below scheme:
If the target node is scanned for the first time, getadapter will trigger genesis to collect information then save the
information at the nicsadapter column of nics table. If the target node has ever been scanned, getadapter will use
the information from nics table first. If user hopes to scan the adapter information for the node but these information
already exist, -f option can be used to start rescan process.
getadapter tries to collect more information for the target network device, but doesn’t guarantee collect same much
information for every network device.
Collected information:
name: the consistent name which can be used by confignic directly in operating system which follow the same naming
scheme with rhels7
pci: the pci location
mac: the MAC address
candidatename: All the names which satisfy predictable network device naming scheme. (if xcat enhance confignic
command later, user can use these names to configure their network adapter, even customize their name)
vender: the vender of network device
model: the model of network device
linkstate: the link state of network device
OPTIONS
-h
Display usage message.
-v
Command Version.
-V
Display verbose message.
-f
Force to trigger new round scan. ignore the data collected before.
1.3. Admin Guide
241
xCAT3 Documentation, Release 2.12
EXAMPLES
1. To collect node[1-3]’s network device information, enter:
getadapter
node[1-2]
Output is similar to:
-->Starting scan for: node1,node2
The whole scan result:
-------------------------------------[node1]: Adapter information exists, no need to scan.
-------------------------------------[node2] scan successfully, below are the latest data
node2:[1]->eno1!mac=34:40:b5:be:6a:80|pci=/pci0000:00/0000:00:01.0/0000:0c:00.
˓→0|candidatename=eno1/enp12s0f0/enx3440b5be6a80
node2:[2]->enp0s29u1u1u5!mac=36:40:b5:bf:44:33|pci=/pci0000:00/0000:00:1d.0/usb2/2-1/
˓→2-1.1/2-1.1.5/2-1.1.5:1.0|candidatename=enp0s29u1u1u5/enx3640b5bf4433
Every node gets a separate section to display its all network adapters information, every network adapter owns single
line which start as node name and followed by index and other information.
2. Force to trigger new round scan
getadatper node -f
SEE ALSO
noderange(3)|noderange.3
getmacs.1
NAME
getmacs - Collects node MAC address.
SYNOPSIS
Common:
getmacs [-h| --help | -v| --version]
PPC specific:
getmacs noderange [-F filter]
getmacs noderange [-M]
getmacs noderange [-V| --verbose] [-f] [-d] [--arp] | [-D {[-S server] [-G gateway] [-C client] [-o] | [--noping]}]
242
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
blade specific:
getmacs noderange [-V| --verbose] [-d] [--arp] [-i ethN | enN]
DESCRIPTION
The getmacs command collects MAC address from a single or range of nodes. Note that on AIX systems, the returned
MAC address is not colon-separated (for example 8ee2245cf004), while on Linux systems the MAC address is colonseparated (for example 8e:e2:24:5c:f0:04). If no ping test performed, getmacs writes the first adapter MAC to the
xCAT database. If ping test performed, getmacs will write the first successfully pinged MAC to xCAT database.
For PPC (using Direct FSP Management) specific:
Note: If network adapters are physically assigned to LPARs, getmacs cannot read the MAC addresses unless perform
Discovery with option “-D”, since there is no HMC command to read them and getmacs has to login to open firmware.
And if the LPARs has never been activated before, getmacs need to be performed with the option “-D” to get theirs
MAC addresses.
For PPC (using HMC) specific:
Note: The option “-D” must be used to get MAC addresses of LPARs.
For IBM Flex Compute Node (Compute Node for short) specific:
Note: If “-d” is specified, all the MAC of the blades will be displayed. If no option specified, the first MAC address
of the blade will be written to mac table.
OPTIONS
--arp
Read MAC address with ARP protocol.
-C
Specify the IP address of the partition for ping test. The default is to read from xCAT database if no -C specified.
-d
Display MAC only. The default is to write the first valid adapter MAC to the xCAT database.
-D
Perform discovery for mac address. By default, it will run ping test to test the connection between adapter and xCAT
management node. Use ‘–noping’ can skip the ping test to save time. Be aware that in this way, the lpars will be reset.
-f
Force immediate shutdown of the partition. This flag must be used with -D flag.
-F
Specify filters to select the correct adapter. Acceptable filters are Type, MAC_Address, Phys_Port_Loc, Adapter,
Port_Group, Phys_Port, Logical_Port, VLan, VSwitch, Curr_Conn_Speed.
-G
Gateway IP address of the partition. The default is to read from xCAT database if no -G specified.
-h
Display usage message.
1.3. Admin Guide
243
xCAT3 Documentation, Release 2.12
-M
Return multiple MAC addresses for the same adapter or port, if available from the hardware. For some network
adapters (e.g. HFI) the MAC can change when there are some recoverable internal errors. In this case, the hardware
can return several MACs that the adapter can potentially have, so that xCAT can put all of them in DHCP. This allows
successful booting, even after a MAC change, but on Linux at this time, it can also cause duplicate IP addresses, so
it is currently not recommended on Linux. By default (without this flag), only a single MAC address is returned for
each adapter.
--noping
Only can be used with ‘-D’ to display all the available adapters with mac address but do NOT run ping test.
-o
Read MAC address when the lpar is in openfirmware state. This option mush be used with [-D] option to perform ping
test. Before use -o, the lpar must be in openfirmware state.
-S
The IP address of the machine to ping. The default is to read from xCAT database if no -S specified.
-v
Command Version.
-V
Verbose output.
-i
Specify the interface whose mac address will be collected and written into mac table. If 4 mac addresses are returned
by option ‘-d’, they all are the mac addresses of the blade. The N can start from 0(map to the eth0 of the blade) to 3. If
5 mac addresses are returned, the 1st mac address must be the mac address of the blade’s FSP, so the N will start from
1(map to the eth0 of the blade) to 4.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To retrieve the MAC address for the HMC-managed partition lpar4 and write the first valid adapter MAC to the
xCAT database, enter:
getmacs lpar4
Output is similar to:
lpar4:
#Type MAC_Address Phys_Port_Loc Adapter Port_Group Phys_Port Logical_Port
˓→ VSwitch
Curr_Conn_Speed
hea 7607DFB07F02 N/A N/A N/A N/A N/A 1 ETHERNET0 N/A
ent U78A1.001.99203B5-P1-T6
00145eb55788 /lhea@23c00614/ethernet@23e00514
˓→unsuccessful physical
VLan
2. To retrieve the MAC address with ARP protocol:
244
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
getmacs lpar4 --arp
Output is similar to:
lpar4:
#IP
192.168.0.10
MAC_Address
00145eb55788
3. To retrieve the MAC address for the HMC-managed partition lpar4 and display the result only, enter:
getmacs lpar4 -d
Output is similar to:
lpar4:
#Type MAC_Address Phys_Port_Loc Adapter Port_Group Phys_Port Logical_Port
˓→ VSwitch
Curr_Conn_Speed
hea 7607DFB07F02 N/A N/A N/A N/A N/A 1 ETHERNET0 N/A
ent U78A1.001.99203B5-P1-T6
00145eb55788 /lhea@23c00614/ethernet@23e00514
˓→unsuccessful physical
4. To retrieve the MAC address
Type=hea,VSwitch=ETHERNET0.
for
the
HMC-managed
partition
lpar4
with
VLan
filter
getmacs lpar4 -F Type=hea,VSwitch=ETHERNET0
Output is similar to:
lpar4:
#Type MAC_Address Phys_Port_Loc Adapter
˓→ VSwitch
Curr_Conn_Speed
hea 7607DFB07F02 N/A N/A N/A N/A N/A
Port_Group
1
Phys_Port
ETHERNET0
Logical_Port
VLan
N/A
5. To retrieve the MAC address while performing a ping test for the HMC-managed partition lpar4 and display the
result only, enter:
getmacs lpar4 -d -D -S 9.3.6.49 -G 9.3.6.1 -C 9.3.6.234
Output is similar to:
lpar4:
#Type Location Code
MAC Address
Full Path Name Ping Result
ent U9133.55A.10B7D1G-V12-C4-T1 8e:e2:24:5c:f0:04 /vdevice/l-lan@30000004 successful
˓→virtual
6. To retrieve the MAC address for Power 775 LPAR using Direct FSP Management without ping test and display
the result only, enter:
getmacs lpar4 -d
Output is similar to:
lpar4:
#Type Phys_Port_Loc MAC_Address Adapter
˓→ VSwitch
Curr_Conn_Speed
HFI N/A 02:00:02:00:00:04 N/A N/A N/A
1.3. Admin Guide
Port_Group
N/A
N/A
Phys_Port
N/A
Logical_Port
VLan
N/A
245
xCAT3 Documentation, Release 2.12
7. To retrieve multiple MAC addresses from Power 775 HFI network adapter using Direct FSP Management, enter:
getmacs lpar4 -M
Output is similar to:
lpar4:
#Type Phys_Port_Loc MAC_Address Adapter Port_Group Phys_Port Logical_Port VLan
˓→ VSwitch
Curr_Conn_Speed
HFI N/A 02:00:02:00:00:04|02:00:02:00:00:05|02:00:02:00:00:06 N/A N/A N/A N/A
˓→N/A
N/A N/A
8. To retrieve the MAC address for Power Lpar by ‘-D’ but without ping test.
getmacs lpar4 -D --noping
Output is similar to:
lpar4:
# Type Location Code
MAC Address
Full Path Name Device Type
ent U8233.E8B.103A4DP-V3-C3-T1 da:08:4c:4d:d5:03 /vdevice/l-lan@30000003
ent U8233.E8B.103A4DP-V3-C4-T1 da:08:4c:4d:d5:04 /vdevice/l-lan@30000004
ent U78A0.001.DNWHYT2-P1-C6-T1 00:21:5e:a9:50:42 /lhea@200000000000000/
˓→ethernet@200000000000003
physical
virtual
virtual
FILES
/opt/xcat/bin/getmacs
SEE ALSO
makedhcp(8)|makedhcp.8
getslnodes.1
NAME
getslnodes - queries your SoftLayer account and gets attributes for each server.
SYNOPSIS
getslnodes [-v | --verbose] [hostname-match]
getslnodes [-? | -h | --help]
DESCRIPTION
The getslnodes command queries your SoftLayer account and gets attributes for each server. The attributes can be
piped to ‘mkdef -z’ to define the nodes in the xCAT DB so that xCAT can manage them.
Before using this command, you must download and install the SoftLayer API perl module. For example:
246
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
cd /usr/local/lib
git clone https://github.com/softlayer/softlayer-api-perl-client.git
You also need to follow these directions to get your SoftLayer API key: http://knowledgelayer.softlayer.com/
procedure/retrieve-your-api-key
getslnodes requires a .slconfig file in your home directory that contains your SoftLayer userid, API key, and location
of the SoftLayer API perl module, in attr=val format. For example:
# Config
userid =
apikey =
apidir =
file used by the xcat cmd getslnodes
joe_smith
1234567890abcdef1234567890abcdef1234567890abcdef
/usr/local/lib/softlayer-api-perl-client
OPTIONS
-?|-h|--help
Display usage message.
-v|--version
Command Version.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. Display information about all of the nodes in your SoftLayer account:
getslnodes
2. Display information about all of the nodes whose hostname starts with foo:
getslnodes foo
3. Create xCAT node defintions in the xCAT DB for all of the nodes in your SoftLayer account:
getslnodes | mkdef -z
FILES
/opt/xcat/bin/getslnodes
SEE ALSO
pushinitrd(1)|pushinitrd.1
1.3. Admin Guide
247
xCAT3 Documentation, Release 2.12
gettab.1
NAME
gettab - select table rows, based on attribute criteria, and display specific attributes.
SYNOPSIS
gettab [-H | --with-fieldname] key=value,... table.attribute ...
gettab [-? | -h | --help]
DESCRIPTION
The gettab command uses the specified key values to select a row in each of the tables requested. For each selected
row, the specified attributes are displayed. The gettab command can be used instead of nodels for tables that are not
keyed by nodename (e.g. the site table), or to select rows based on an attribute value other than nodename.
OPTIONS
-H|--with-fieldname
Always display table.attribute name next to result. By default, this is done only if more than one table.attribute is requested.
-?|-h|--help
Display usage message.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To display setting for master (management node) in the site table:
gettab -H key=master site.value
The output would be similar to:
site.value: mgmtnode.cluster.com
2. To display the first node or group name that has mgt set to blade in the nodehm table:
gettab mgt=blade nodehm.node
The output would be similar to:
248
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
blades
FILES
/opt/xcat/bin/gettab
SEE ALSO
nodels(1)|nodels.1, chtab(8)|chtab.8, tabdump(8)|tabdump.8
groupfiles4dsh.1
NAME
groupfiles4dsh - Builds a directory of files for each defined nodegroup in xCAT.
SYNOPSIS
groupfiles4dsh [{-p | --path} path]
groupfiles4dsh [-h | --help] [-v | --version]
DESCRIPTION
This tool will build a directory of files, one for each defined nodegroup in xCAT. The file will be named the nodegroup
name and contain a list of nodes that belong to the nodegroup. The file can be used as input to the AIX dsh command.
The purpose of this tool is to allow backward compatibility with scripts that were created using the AIX or CSM dsh
command
Reference: man dsh.
OPTIONS
-h Display usage message.
-v Command Version.
-p Path to the directory to create the nodegroup files (must exist).
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
1.3. Admin Guide
249
xCAT3 Documentation, Release 2.12
EXAMPLES
1. To create the nodegroup files in directory /tmp/nodegroupfiles, enter:
groupfiles4dsh -p /tmp/nodegroupfiles
To use with dsh:
export DSH_CONTEXT=DSH ( default unless CSM is installed)
export DSH_NODE_RSH=/bin/ssh
(default is rsh)
export DSH_NODEGROUP_PATH= /tmp/nodegroupfiles
dsh -N all
dsh -a date
date
(where all is a group defined in xCAT)
(will look in all nodegroupfiles and build a list of all nodes)
FILES
/opt/xcat/share/xcat/tools/groupfiles4dsh
SEE ALSO
xdsh(1)|xdsh.1
imgcapture.1
NAME
imgcapture - Captures an image from a Linux diskful node and create a diskless or diskful image on the management
node.
SYNOPSIS
imgcapture node -t | --type {diskless | sysclone} -o | --osimage osimage [-i nodebootif ] [-n nodenetdrivers] [-V |
--verbose]
imgcapture [-h | --help] | [-v | --version]
DESCRIPTION
The imgcapture command will capture an image from one running diskful Linux node and create a diskless or diskful
image for later use.
The node should be one diskful Linux node, managed by the xCAT MN, and the remote shell between MN and the
node should have been configured. AIX is not supported. VMs are not supported.
The imgcapture command supports two image types: diskless and sysclone. For the diskless type, it will capture
an image from one running diskful Linux node, prepares the rootimg directory, kernel and initial ramdisks for the
liteimg/packimage command to generate the statelite/stateless rootimg. For the sysclone type, it will capture an
image from one running diskful Linux node, create an osimage which can be used to clone other diskful Linux nodes.
The diskless type:
250
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
The attributes of osimage will be used to capture and prepare the root image. The osver, arch and profile attributes
for the stateless/statelite image to be created are duplicated from the node‘s attribute. If the -p|--profile profile option
is specified, the image will be created under “/<installroot>/netboot/<osver>/<arch>/<profile>/rootimg”.
The default files/directories excluded in the image are specified by /opt/xcat/share/xcat/netboot/<os>/<profile>.<osver>.<arch>.imgcaptur
also,
you can put your customized file (<profile>.<osver>.<arch>.imgcapture.exlist) to /install/custom/netboot/<osplatform>. The directories in the default .imgcapture.exlist file are necessary to capture the
image from the diskful Linux node managed by xCAT, don’t remove it.
The image captured will be extracted into the /<installroot>/netboot/<osver>/<arch>/<profile>/rootimg directory.
After the imgcapture command returns without any errors, you can customize the rootimg and run the
liteimg/packimage command with the options you want.
The sysclone type:
xCAT leverages the Open Source Tool - Systemimager to capture the osimage from the node, and put it into /<installroot>/sysclone/images directory.
The imgcapture command will create the osimage definition after the image is captured successfully, you can use this
osimage and nodeset command to clone diskful nodes.
OPTIONS
-t | --type
Specify the osimage type you want to capture, two types are supported: diskless and sysclone.
-p|--profile profile
Assign profile as the profile of the image to be created.
-o|--osimage osimage
The osimage name.
-i nodebootif
The network interface the diskless node will boot over (e.g. eth0), which is used by the genimage command to generate initial ramdisks.
-n nodenetdrivers
The driver modules needed for the network interface, which is used by the genimage command to generate
initial ramdisks.
By default, the genimage command can provide drivers for the following network interfaces:
For x86 or x86_64 platform:
tg3 bnx2 bnx2x e1000 e1000e igb m1x_en
For ppc64 platform:
e1000 e1000e igb ibmveth ehea
For S390x:
qdio ccwgroup
If the network interface is not in the above list, you’d better specify the driver modules with this option.
-h|--help
1.3. Admin Guide
251
xCAT3 Documentation, Release 2.12
Display the usage message.
-v|--version
Display the version.
-V|--verbose
Verbose output.
RETRUN VALUE
0 The command completed sucessfully.
1 An error has occurred.
EXAMPLES
node1 is one diskful Linux node, which is managed by xCAT.
1. There’s one pre-defined osimage. In order to capture and prepare the diskless root image for osimage, run the
command:
imgcapture node1 -t diskless -o osimage
2. In order to capture the diskful image from node1 and create the osimage img1, run the command:
imgcapture node1 -t sysclone -o img1
FILES
/opt/xcat/bin/imgcapture
SEE ALSO
genimage(1)|genimage.1, imgimport(1)|imgimport.1,
liteimg(1)|liteimg.1, nodeset(8)|nodeset.8
imgexport(1)|imgexport.1,
packimage(1)|packimage.1,
imgexport.1
NAME
imgexport - Exports an xCAT image.
SYNOPSIS
imgexport [-h| --help]
imgexport image_name [destination] [[-e | --extra file:dir] ... ] [-p | --postscripts node_name] [-v | --verbose]
252
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
DESCRIPTION
The imgexport command will export an image that is being used by xCAT. To export images, you must have the
images defined in the osimage table. All the columns in the osimage and linuximage tables will be exported. If kits are
used in stateful or stateless images, kit, kitcomponent and kitrepo tables will be exported. In addition, the following
files will also be exported.
For stateful: x.pkglist x.otherpkgs.pkglist x.tmpl x.synclist kits related files
For stateless: kernel initrd.gz rootimg.cpio.xz or rootimg.cpio.gz or rootimg.tar.xz or rootimg.tar.gz or rootimg.gz(for
backward-compatibility) x.pkglist x.otherpkgs.pkglist x.synclist x.postinstall x.exlist kits related files
For statelite: kernel initrd.gz root image tree x.pkglist x.synclist x.otherpkgs.pkglist x.postinstall x.exlist
where x is the name of the profile.
Any files specified by the -e flag will also be exported. If -p flag is specified, the names of the postscripts and the
postbootscripts for the given node will be exported. The postscripts themselves need to be manualy exported using -e
flag.
For statelite, the litefile table settings for the image will also be exported. The litetree and statelite tables are not
exported.
OPTIONS
-e|--extra srcfile:destdir
Pack up extra files. If destdir is omitted, the destination directory will be the same as the source directory.
-h|--help
Display usage message.
-p|--postscripts node_name
Get the names of the postscripts and postbootscripts for the given node and pack them into the image.
-v|--verbose
Verbose output.
image_name
The name of the image. Use lsdef -t osimage to find out all the image names.
destination
The output bundle file name.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. Simplest way to export an image. If there is an image in the osimage table named ‘foo’, then run:
1.3. Admin Guide
253
xCAT3 Documentation, Release 2.12
imgexport foo
foo.tgz will be built in the current working directory. Make sure that you have enough space in the directory that you
are in to run imgexport if you have a big image to tar up.
2. To include extra files with your image:
imgexport Default_Stateless_1265981465 foo.tgz -e /install/postscripts/myscript1 -e /
˓→tmp/mydir:/usr/mydir
In addition to all the default files, this will export /install/postscripts/myscript1 and the whole directory /tmp/dir into the
file called foo.tgz. And when imgimport is called /install/postscripts/myscript1 will be copied into the same directory
and /tmp/mydir will be copied to /usr/mydir.
3. To include postscript with your image:
imgexport Default_Stateless_1265981465 foo.tgz -p node1 -e /install/postscripts/
˓→myscript1
The postscripts and the postbootscripts names specified in the postscripts table for node1 will be exported into the
image. The postscript myscript1 will also be exported.
FILES
/opt/xcat/bin/imgexport
SEE ALSO
imgimport(1)|imgimport.1
imgimport.1
NAME
imgimport - Imports an xCAT image or configuration file into the xCAT tables so that you can immediately begin
deploying with it.
SYNOPSIS
imgimport [-h|--help]
imgimport bundle_file_name [-p | --postscripts nodelist] [-f | --profile new_profile] [-v | --verbose]
DESCRIPTION
The imgimport command will import an image that has been exported by imgexport from xCAT. This is the easiest
way to transfer, backup, change or share images created by xCAT whether they be stateless or stateful. The bundle
file will be unpacked in the current working directory. The xCAT configuration such as osimage and linuximage tables
will then be updated.
254
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
For stateful, the following files will be copied to the appropriate directories. x.pkglist x.otherpkgs.pkglist x.tmpl
x.synclist kits related files
For stateless, the following files will be copied to the appropriate directories. kernel initrd.gz rootimg.cpio.xz or
rootimg.cpio.gz or rootimg.tar.xz or rootimg.tar.gz or rootimg.gz(for backward-compatibility) x.pkglist
x.otherpkgs.pkglist x.synclist x.postinstall x.exlist kits related files
For statelite, the following files will be copied to the appropriate directories. kernel initrd.gz root image tree
x.pkglist x.synclist x.otherpkgs.pkglist x.postinstall x.exlist
where x is the profile name.
Any extra files, included by --extra flag in the imgexport command, will also be copied to the appropriate directories.
For statelite, the litefile table will be updated for the image. The litetree and statelite tables are not imported.
If -p flag is specified, the postscripts table will be updated with the postscripts and the postbootscripts names from the
image for the nodes given by this flag.
If -f flag is not specified, all the files will be copied to the same directories as the source. If it is specified, the old profile
name x will be changed to the new and the files will be copied to the appropriate directores for the new profiles. For
example, /opt/xcat/share/xcat/netboot/sles/x.pkglist will be copied to /install/custom/netboot/sles/compute_new.pkglist
and /install/netboot/sles11/ppc64/x/kernel will be copied to /install/netboot/sles11/ppc64/compute_new/kernel. This
flag is commonly used when you want to copy the image on the same xCAT mn so you can make modification on the
new one.
After this command, you can run the nodeset command and then start deploying the nodes. You can also choose to
modify the files and run the following commands before the node deployment.
For stateful: nodeset
For stateless: genimage packimage nodeset
For statelite genimage liteimg nodeset
OPTIONS
-f|--profile new_profile
Import the image with a new profile name.
-h|--help
Display usage message.
-p|--postscripts nodelist
Import the postscripts. The postscripts contained in the image will be set in the postscripts table for
nodelist.
-v|--verbose
Verbose output.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
1.3. Admin Guide
255
xCAT3 Documentation, Release 2.12
EXAMPLES
1. Simplest way to import an image. If there is a bundle file named ‘foo.gz’, then run:
imgimport foo.gz
2. Import the image with postscript names.
imgimport foo.gz -p node1,node2
The postscripts table will be updated with the name of the postscripts and the postbootscripts for node1 and node2.
3. Import the image with a new profile name
imgimport foo.gz -f compute_test
FILES
/opt/xcat/bin/imgimport
SEE ALSO
imgexport(1)|imgexport.1
liteimg.1
NAME
liteimg - Modify statelite image by creating a series of links.
SYNOPSIS
liteimg [-h| --help]
liteimg [-v| --version]
liteimg imagename
DESCRIPTION
This command modifies the statelite image by creating a series of links. It creates 2 levels of indirection so that files
can be modified while in their image state as well as during runtime. For example, a file like <$imgroot>/etc/ntp.conf
will have the following operations done to it:
* mkdir -p $imgroot/.default/etc*
* mkdir -p $imgroot/.statelite/tmpfs/etc*
* mv $imgroot/etc/ntp.conf $imgroot/.default/etc*
* cd $imgroot/.statelite/tmpfs/etc*
* ln -sf ../../../.default/etc/ntp.conf .*
256
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
* cd $imgroot/etc*
* ln -sf ../.statelite/tmpfs/etc/ntp.conf .*
When finished, the original file will reside in $imgroot/.default/etc/ntp.conf. $imgroot/etc/ntp.conf will link to $imgroot/.statelite/tmpfs/etc/ntp.conf which will in turn link to $imgroot/.default/etc/ntp.conf
Note: If you make any changes to your litefile table after running liteimg then you will need to rerun liteimg again.
PARAMETERS
imagename specifies the name of a os image definition to be used. The specification for the image is stored in the
osimage table and linuximage table.
OPTIONS
-h|--help Display usage message.
-v|--version Command Version.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To lite a RHEL 6.6 statelite image for a compute node architecture x86_64 enter:
liteimg rhels6.6-x86_64-statelite-compute
FILES
/opt/xcat/bin/liteimg
NOTES
This command is part of the xCAT software product.
SEE ALSO
genimage(1)|genimage.1
1.3. Admin Guide
257
xCAT3 Documentation, Release 2.12
lsdef.1
NAME
lsdef - Use this command to list xCAT data object definitions.
SYNOPSIS
lsdef [-h | --help] [-t object-types] [-i attr-list]
lsdef [-V | --verbose] [-l | --long] [-s | --short] [-a | --all] [-S] [-t object-types] [-o object-names] [-z | --stanza] [-i
attr-list] [-c | --compress] [--osimage] [--nics] [[-w attr==val] [-w attr=~val] ...] [noderange]
lsdef [-l | --long] [-a | --all] [-t object-types] [-z | --stanza] [-i attr-list] [--template [template-object-name]]
DESCRIPTION
This command is used to display xCAT object definitions which are stored in the xCAT database and xCAT object
definition templates shipped in xCAT.
OPTIONS
-a|--all
Display all definitions. For performance consideration, the auditlog and eventlog objects will not be listed.
To list auditlog or eventlog objects, use lsdef -t auditlog or lsdef -t eventlog instead.
-c|--compress
Display information in compressed mode, each output line has format “<object name>: <data>”. The
output can be passed to command xcoll or xdshbak for formatted output. The -c flag must be used with -i
flag.
-h|--help
Display usage message.
-i attr-list
Comma separated list of attribute names to display.
-l|--long
List the complete object definition.
-s|--short
Only list the object names.
-S
List all the hidden nodes (FSP/BPA nodes) with other ones.
noderange
A set of comma delimited node names and/or group names. See the “noderange” man page for details on
supported formats.
-o object-names
258
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
A set of comma delimited object names.
--template [template-object-name]
Show the object definition templates template-object-name shipped in xCAT. If no template-object-name
is specified, all the object definition templates of the specified type -t object-types will be listed. Use
-a|--all option to list all the object definition templates.
--osimage
Show all the osimage information for the node.
--nics
Show the nics configuration information for the node.
-t object-types
A set of comma delimited object types. Use the help option to get a list of valid objects.
-V|--verbose
Verbose mode.
-w attr==val -w attr=~val ...
Use one or multiple -w flags to specify the selection string that can be used to select objects. The operators
==, !=, =~ and !~ are available. Use the help option to get a list of valid attributes for each object type.
Operator descriptions: == Select nodes where the attribute value is exactly this value. != Select nodes
where the attribute value is not this specific value. =~ Select nodes where the attribute value matches
this regular expression. !~ Select nodes where the attribute value does not match this regular expression.
Note: if the “val” fields includes spaces or any other characters that will be parsed by shell, the
“attr<operator>val” needs to be quoted. If the operator is ”!~”, the “attr<operator>val” needs to be quoted
using single quote.
-z|--stanza
Display output in stanza format. See the xcatstanzafile man page for details on using xCAT stanza files.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To display a description of all the valid attributes that could be used when defining an xCAT node.
lsdef -t node -h
2. To get a list of all the objects that have been defined.
lsdef
OR
lsdef -a
3. To get all the attributes of the node1
1.3. Admin Guide
259
xCAT3 Documentation, Release 2.12
lsdef node1
OR
lsdef -t node node1
OR
lsdef -t node -o node1
4. To get the object name of node1 instead of all the attributes
lsdef -s node1
5. To get a list of all the network definitions.
lsdef -t network
6. To get a complete listing of all network definitions.
lsdef -l -t network
7. To list the whole xCAT database and write it to a stanza file. (backup database)
lsdef -a -l -z > mydbstanzafile
8. To list the MAC and install adapter name for each node.
lsdef -t node -i mac,installnic
9. To list an osimage definition named “aix53J”.
lsdef -t osimage -l -o aix53J
10. To list all node definitions that have a status value of “booting”.
lsdef -t node -w status==booting
11. To list all the attributes of the group “service”.
lsdef -l -t group -o service
12. To list all the attributes of the nodes that are members of the group “service”.
lsdef -t node -l service
13. To get a listing of object definitions that includes information about what xCAT database tables are used to store
the data.
lsdef -V -l -t node -o node01
14. To list the hidden nodes that can’t be seen with other flags. The hidden nodes are FSP/BPAs.
lsdef -S
15. To list the nodes status and use xcoll to format the output.
lsdef -t node -i status -c | xcoll
16. To display the description for some specific attributes that could be used when defining an xCAT node.
260
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
lsdef -t node -h -i profile,pprofile
17. To display the nics configuration information for node cn1.
lsdef cn1 --nics
18. To list all the object definition templates shipped in xCAT.
lsdef --template -a
19. To display the details of “node” object definition template “ppc64le-template” shipped in xCAT.
lsdef -t node --template ppc64le-template
20. To list all the “node” object definition templates shipped in xCAT.
lsdef -t node --template
FILES
/opt/xcat/bin/lsdef
NOTES
This command is part of the xCAT software product.
SEE ALSO
mkdef(1)|mkdef.1, chdef(1)|chdef.1, rmdef(1)|rmdef.1, xcatstanzafile(5)|xcatstanzafile.5
lsdocker.1
NAME
lsdocker - List docker instance.
SYNOPSIS
lsdocker noderange [-l | --logs]
lsdocker dockerhost
lsdocker [-h | --help]
lsdocker {-v | --version}
DESCRIPTION
lsdocker To list docker instance info or all the running docker instance info if dockerhost is specified.
1.3. Admin Guide
261
xCAT3 Documentation, Release 2.12
OPTIONS
-l|--logs
To return the logs of docker instance. Only works for docker instance.
EXAMPLES
1. To get info for docker instance “host01c01”
lsdocker host01c01
Output is similar to:
host01c01: 50800dfd8b5f
˓→running /host01c01
ubuntu
/bin/bash
2016-01-13T06:32:59
2. To get info for running docker instance on dockerhost “host01”
lsdocker host01
Output is similar to:
host01: 50800dfd8b5f ubuntu
˓→minutes
/host01c01
host01: 875ce11d5987 ubuntu
˓→seconds
/host01c02
/bin/bash
2016-1-13 - 1:32:59
Up 12
/bin/bash
2016-1-21 - 1:12:37
Up 5
SEE ALSO
mkdocker(1)|mkdocker.1, rmdocker(1)|rmdocker.1
lsflexnode.1
NAME
lsflexnode - Display the information of flexible node
SYNOPSIS
lsflexnode [-h | --help]
lsflexnode [-v | --version]
lsflexnode noderange
DESCRIPTION
IBM BladeCenter HX5 offers flexibility ideal that the blades can be combined together for scalability.
There are several concepts to support the HX5 multiple blades combination:
262
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
Complex: Multiple blades which combined by a scalability card is a complex.
Partition: A logic concept which containing part of the Blade slot node in a complex. Each partition can map to
a system to install Operating System. Each partition could have 1HX5, 1HX5+1MD or 2HX5+2MD. (MD is the
Memory Drawer)
Blade slot node: The physical blade which installed in the slot of a chassis. It can be a HX5 or MD.
A Complex will be created automatically when a multiple blades combination is installed. In this Complex, every
blade belongs to it is working as a Blade slot node.
A Partition can be created base on the Complex, each Partition can have one or multiple Blade slot node.
The noderange in the SYNOPSIS can be a AMM node or a blade node.
OPTIONS
-h | --help
Display the usage message.
-v | --version
Display the version information.
ATTRIBUTES
The meaning of attributes which displayed by the lsflexnode. The word ‘node’ in this section means Blade slot node.
Complex
The unique numeric identifier for a complex installed in the chassis.
Partition number
The number of partitions currently defined for this complex.
Complex node number
The number of nodes existing in this complex, regardless of their assignment to any given partition.
Partition
The unique numeric identifier for a partition defined within a complex installed in the chassis.
Partition Mode
The currently configured mode of this partition. It can be ‘partition’ or ‘standalone’.
Partition node number
The number of nodes currently defined for this partition.
Partition status
The current power status of this partition when the partition has a valid partition configuration. It can be
‘poweredoff’, ‘poweredon’, ‘resetting’ or ‘invalid’.
Node
The unique numeric identifier for this node, unique within the partition. If this node does not belong to a
partition, the slot number will be displayed.
Node state
1.3. Admin Guide
263
xCAT3 Documentation, Release 2.12
The physical power state of this node. It can be ‘poweredoff’, ‘poweredon’ or ‘resetting’.
Node slot
The base slot number where the node exists in the chassis.
Node resource
A string providing a summary overview of the resources provided by this node. It includes the CPU
number, CPU frequency and Memory size.
Node type
The general categorization of the node. It can be ‘processor’, ‘memory’ or ‘io’.
Node role
Indicates if the node is assigned to a partition, and if so, provides an indication of whether the node is the
primary node of the partition or not.
Flexnode state
The state of a flexible node. It is the state of the partition which this node belongs to. If this node does
NOT belong to a partition, the value should be ‘invalid’.
It can be ‘poweredoff’, ‘poweredon’, ‘resetting’ or ‘invalid’.
Complex id
The identifier of the complex this node belongs to.
Partition id
The identifier of the partition this node belongs to.
EXAMPLES
1 Display all the Complex, Partition and Blade slot node which managed by a AMM.
lsflexnode amm1
The output:
amm1:
amm1:
amm1:
amm1:
amm1:
amm1:
amm1:
amm1:
amm1:
amm1:
amm1:
amm1:
amm1:
amm1:
amm1:
amm1:
amm1:
amm1:
264
Complex - 24068
..Partition number - 1
..Complex node number - 2
..Partition = 1
....Partition Mode - partition
....Partition node number - 1
....Partition status - poweredoff
....Node - 0 (logic id)
......Node state - poweredoff
......Node slot - 14
......Node type - processor
......Node resource - 2 (1866 MHz) / 8 (2 GB)
......Node role - secondary
..Partition = unassigned
....Node - 13 (logic id)
......Node state - poweredoff
......Node slot - 13
......Node type - processor
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
amm1: ......Node resource - 2 (1866 MHz) / 8 (2 GB)
amm1: ......Node role - unassigned
2 Display a flexible node.
lsflexnode blade1
The output:
blade1:
blade1:
blade1:
blade1:
blade1:
blade1:
blade1:
blade1:
Flexnode state - poweredoff
Complex id - 24068
Partition id - 1
Slot14: Node state - poweredoff
Slot14: Node slot - 14
Slot14: Node type - processor
Slot14: Node resource - 2 (1866 MHz) / 8 (2 GB)
Slot14: Node role - secondary
FILES
/opt/xcat/bin/lsflexnode
SEE ALSO
mkflexnode(1)|mkflexnode.1, rmflexnode(1)|rmflexnode.1
lshwconn.1
NAME
lshwconn - Use this command to display the connection status for CEC and Frame nodes.
SYNOPSIS
lshwconn [-h| --help]
lshwconn [-v| --version]
PPC (with HMC) specific:
lshwconn [-V| --verbose] noderange
PPC (without HMC, using FSPAPI) specific:
lshwconn noderange -T tooltype
1.3. Admin Guide
265
xCAT3 Documentation, Release 2.12
DESCRIPTION
This command is used to display the connection status for CEC and Frame node.
OPTIONS
-h|--help
Display usage message.
-V|--verbose
Verbose output.
-T
The tooltype is used to communicate to the CEC/Frame. The value could be lpar or fnm. The tooltype
value lpar is for xCAT and fnm is for CNM.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To display connection status for all CEC nodes in node group CEC:
lshwconn cec
Output is similar to:
cec1: ipaddr=192.168.200.245,alt_ipaddr=unavailable,state=Connected
cec2: Connection not found
2. To display connection status for Frame node frame1:
lshwconn frame1
Output is similar to:
frame1: side=a,ipaddr=192.168.200.247,alt_ipaddr=unavailable,state=Connected
frame1: side=b,ipaddr=192.168.200.248,alt_ipaddr=unavailable,state=Connected
3. To display connection status for all CEC nodes in node group CEC to hardware server, and using lpar tooltype:
lshwconn cec -T lpar
Output is similar to:
cec1: sp=primary,ipadd=40.3.7.1,alt_ipadd=unavailable,state=LINE UP
cec2: Connection not found
266
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
FILES
$XCATROOT/bin/lshwconn
(The XCATROOT environment variable is set when xCAT is installed. The default value is “/opt/xcat”.)
NOTES
This command is part of the xCAT software product.
SEE ALSO
rmhwconn(1)|rmhwconn.1, mkhwconn(1)|mkhwconn.1
lskit.1
NAME
lskit - Lists information for one or more Kits.
SYNOPSIS
lskit [-V | --verbose] [-F | --framework kitattr_names] [-x | --xml | --XML] [-K | --kitattr kitattr_names] [-R |
--repoattr repoattr_names] [-C | --compattr compattr_names] [kit_names]
lskit [-? | -h | --help | -v | --version]
lskit [-F | --framework kit_path_name]
DESCRIPTION
The lskit command is used to list information for one or more kits. A kit is a special kind of package that is used to
install a software product on one or more nodes in an xCAT cluster.
Note: The xCAT support for Kits is only available for Linux operating systems.
The lskit command outputs the following info for each kit: the kit’s basic info, the kit’s repositories, and the kit’s
components. The command outputs the info in two formats: human-readable format (default), and XML format. Use
the -x option to view the info in XML format.
Input to the command can specify any number or combination of the input options.
OPTIONS
-F|--framework kit_path_name
Use this option to display the framework values of the specified Kit tarfile. This information is retreived
directly from the tarfile and can be done before the Kit has been defined in the xCAT database. This option
cannot be combined with other options.
-K|--kitattr kitattr_names
1.3. Admin Guide
267
xCAT3 Documentation, Release 2.12
Where kitattr_names is a comma-delimited list of kit attribute names. The names correspond to attribute
names in the kit table. The lskit command will only display the specified kit attributes.
-R|--repoattr repoattr_names
Where repoattr_names is a comma-delimited list of kit repository attribute names. The names correspond
to attribute names in the kitrepo table. The lskit command will only display the specified kit repository
attributes.
-C|--compattr compattr_names
where compattr_names is a comma-delimited list of kit component attribute names. The names correspond to attribute names in the kitcomponent table. The lskit command will only display the specified
kit component attributes.
kit_names
is a comma-delimited list of kit names. The lskit command will only display the kits matching these
names.
-x|--xml|--XML
Need XCATXMLTRACE=1 env when using -x|–xml|–XML, for example: XCATXMLTRACE=1 lskit -x
testkit-1.0.0 Return the output with XML tags. The data is returned as:
<data>
<kitinfo>
...
</kitinfo>
</data>
...
<data>
<kitinfo>
...
</kitinfo>
</data>
Each <kitinfo> tag contains info for one kit. The info inside <kitinfo> is structured as follows:
The <kit> sub-tag contains the kit's basic info.
The <kitrepo> sub-tags store info about the kit's repositories.
The <kitcomponent> sub-tags store info about the kit's components.
The data inside <kitinfo> is returned as:
<kitinfo>
<kit>
...
</kit>
<kitrepo>
...
</kitrepo>
...
<kitcomponent>
...
</kitcomponent>
...
</kitinfo>
268
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
-V|--verbose
Display additional progress and error messages.
-v|--version
Command Version.
-?|-h|--help
Display usage message.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To list all kits, enter:
lskit
2. To list the kit “kit-test1-1.0-Linux”, enter:
lskit kit-test1-1.0-Linux
3. To list the kit “kit-test1-1.0-Linux” for selected attributes, enter:
lskit -K basename,description -R kitreponame -C kitcompname kit-test1-1.0˓→Linux
4. To list the framework value of a Kit tarfile.
lskit -F /myhome/mykits/pperte-1.3.0.2-0-x86_64.tar.bz2
Output is similar to:
Extracting the kit.conf file from /myhome/mykits/pperte-1.3.0.2-0-x86_64.tar.
˓→bz2. Please wait.
kitframework=2
compatible_kitframeworks=0,1,2
5. To list kit “testkit-1.0-1” with XML tags, enter:
XCATXMLTRACE=1 lskit -x testkit-1.0-1
FILES
/opt/xcat/bin/lskit
1.3. Admin Guide
269
xCAT3 Documentation, Release 2.12
SEE ALSO
lskitcomp(1)|lskitcomp.1, lskitdeployparam(1)|lskitdeployparam.1, addkit(1)|addkit.1, rmkit(1)|rmkit.1, addkitcomp(1)|addkitcomp.1, rmkitcomp(1)|rmkitcomp.1
lskitcomp.1
NAME
lskitcomp - Used to list information for one or more kit components.
SYNOPSIS
lskitcomp [-V | --verbose] [-x | --xml | --XML] [-C | --compattr compattr_names] [-O | --osdistro os_distro] [-S |
--serverrole server_role] [kitcomp_names]
lskitcomp [-? | -h | --help | -v | --version]
DESCRIPTION
The lskitcomp command is used to list information for one or more kit components. A kit is made up of one or more
kit components. Each kit component is a meta package used to install a software product component on one or more
nodes in an xCAT cluster.
The lskitcomp command outputs the kit component info in two formats: human-readable format (default), and XML
format. Use the -x option to view the info in XML format.
Input to the command can specify any number or combination of the input options.
Note: The xCAT support for Kits is only available for Linux operating systems.
OPTIONS
-C|--compattr compattr_names
where compattr_names is a comma-delimited list of kit component attribute names. The names correspond to attribute names in the kitcomponent table. The lskitcomp command will only display the
specified kit component attributes.
-O|--osdistro os_distro
where os_distro is the name of an osdistro in osdistro table. The lskitcomp command will only display
the kit components matching the specified osdistro.
-S|--serverrole server_role
where server_role is the name of a server role. The typical server roles are: mgtnode, servicenode,
computenode, loginnode, storagennode. The lskitcomp command will only display the kit components
matching the specified server role.
kitcomp_names
is a comma-delimited list of kit component names. The lskitcomp command will only display the kit
components matching the specified names.
270
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
-x|--xml|--XML
Need XCATXMLTRACE=1 env when using -x|–xml|–XML. Return the output with XML tags. The data
is returned as:
<data>
<kitinfo>
...
</kitinfo>
</data>
...
<data>
<kitinfo>
...
</kitinfo>
</data>
Each <kitinfo> tag contains info for a group of kit components belonging to the same kit. The info inside
<kitinfo> is structured as follows:
The <kit> sub-tag contains the kit's name.
The <kitcomponent> sub-tags store info about the kit's components.
The data inside <kitinfo> is returned as:
<kitinfo>
<kit>
...
</kit>
<kitcomponent>
...
</kitcomponent>
...
</kitinfo>
-V|--verbose
Display additional progress and error messages.
-v|--version
Command Version.
-?|-h|--help
Display usage message.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To list all kit components, enter:
1.3. Admin Guide
271
xCAT3 Documentation, Release 2.12
lskitcomp
2. To list the kit component “comp-server-1.0-1-rhels-6-x86_64”, enter:
lskitcomp comp-server-1.0-1-rhels-6-x86_64
3. To list the kit component “comp-server-1.0-1-rhels-6-x86_64” for selected kit component attributes, enter:
lskitcomp -C kitcompname,desc comp-server-1.0-1-rhels-6-x86_64
4. To list kit components compatible with “rhels-6.2-x86_64” osdistro, enter:
lskitcomp -O rhels-6.2-x86_64
5. To list kit components compatible with “rhels-6.2-x86_64” osdistro and “computenode” server role, enter:
lskitcomp -O rhels-6.2-x86_64 -S computenode
6. To list the kit component “testkit-compute-1.0-1-ubuntu-14.04-ppc64el” with XML tags, enter:
XCATXMLTRACE=1 lskitcomp -x testkit-compute-1.0-1-ubuntu-14.04-ppc64el
FILES
/opt/xcat/bin/lskitcomp
SEE ALSO
lskit(1)|lskit.1,
lskitdeployparam(1)|lskitdeployparam.1,
comp(1)|addkitcomp.1, rmkitcomp(1)|rmkitcomp.1
addkit(1)|addkit.1,
rmkit(1)|rmkit.1,
addkit-
lskitdeployparam.1
NAME
lskitdeployparam - Lists the deployment parameters for one or more Kits or Kit components
SYNOPSIS
lskitdeployparam [-V | --verbose] [-x | --xml | --XML] [-k | --kitname kit_names] [-c | --compname comp_names]
lskitdeployparam [-? | -h | --help | -v | --version]
DESCRIPTION
The lskitdeployparam command is used to list the kit deployment parameters for one or more kits, or one or more kit
components. Kit deployment parameters are used to customize the installation or upgrade of kit components.
The lskitdeployparam command outputs the kit component information in two formats: human-readable format
(default), and XML format. Use the -x option to view the information in XML format.
272
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
Input to the command can specify any combination of the input options.
Note: The xCAT support for Kits is only available for Linux operating systems.
OPTIONS
-k|--kitname kit_names
Where kit_names is a comma-delimited list of kit names. The lskitdeployparam command will only
display the deployment parameters for the kits with the matching names.
-c|--compname comp_names
Where comp_names is a comma-delimited list of kit component names. The lskitdeployparam command
will only display the deployment parameters for the kit components with the matching names.
-x|--xml|--XML
Return the output with XML tags. The data is returned as:
<data>
<kitdeployparam>
<name>KIT_KIT1_PARAM1</name>
<value>value11</value>
</kitdeployparam>
</data>
<data>
<kitdeployparam>
<name>KIT_KIT1_PARAM2</name>
<value>value12</value>
</kitdeployparam>
</data>
...
-V|--verbose
Display additional progress and error messages.
-v|--version
Command Version.
-?|-h|--help
Display usage message.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To list kit deployment parameters for kit “kit-test1-1.0-Linux”, enter:
lskitdeployparam -k kit-test1-1.0-Linux
1.3. Admin Guide
273
xCAT3 Documentation, Release 2.12
2. To list kit deployment parameters for kit component “comp-server-1.0-1-rhels-6-x86_64”, enter:
lskitdeployparam -c comp-server-1.0-1-rhels-6-x86_64
FILES
/opt/xcat/bin/lskitdeployparam
SEE ALSO
lskit(1)|lskit.1, lskitcomp(1)|lskitcomp.1, addkit(1)|addkit.1, rmkit(1)|rmkit.1, addkitcomp(1)|addkitcomp.1, rmkitcomp(1)|rmkitcomp.1
lskmodules.1
NAME
lskmodules - list kernel driver modules in rpms or driver disk image files
SYNOPSIS
lskmodules [-V | --verbose] [-i | --osimage osimage_names] [-c | --kitcomponent kitcomp_names] [-o | --osdistro
osdistro_names] [-u | --osdistropudate osdistroupdate_names] [-x | --xml | --XML]
lskmodules [-? | -h | --help | -v | --version]
DESCRIPTION
The lskmodules command finds the kernel driver module files (*.ko) in the specified input locations, runs the modinfo
command against each file, and returns the driver name and description. If -x is specified, the output is returned with
XML tags.
Input to the command can specify any number or combination of the input options.
OPTIONS
-i|--osimage osimage_names
where osimage_names is a comma-delimited list of xCAT database osimage object names. For each
osimage_name, lskmodules will use the entries in osimage.driverupdatesrc for the rpms and driver disk
image files to search.
-c|--kitcomponent kitcomponent_names
where kitcomponent_names is a comma-delimited list of xCAT database kitcomponent object names. For
each kitcomponent_name, lskmodules will use the entries in kitcomponent.driverpacks for the rpm list
and the repodir of the kitcomponent.kitreponame for the location of the rpm files to search.
-o|--osdistro osdistro_names
274
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
where osdistro_names is a comma-delimited list of xCAT database osdistro object names. For each osdistro_name, lskmodules will search each <osdistro.dirpaths>/Packages/kernel-<kernelversion>.rpm file.
-u|--osdistroupdate osdistroupdate_names
where osdistroupdate_names is a comma-delimited list of xCAT database osdistroupdate table entries. For each osdistroupdate_name, lskmodules will search the <osdistroupdate.dirpath>/kernel<kernelversion>.rpm file.
-x|--xml|--XML
Return the output with XML tags. The data is returned as:
<module>
<name> xxx.ko </name>
<description> this is module xxx </description>
</module>
This option is intended for use by other programs. The XML will not be displayed. To view the returned
XML, set the XCATSHOWXML=yes environment variable before running this command.
-V|--verbose
Display additional progress and error messages.
-v|--version
Command Version.
-?|-h|--help
Display usage message.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To list the kernel modules included in the driverpacks shipped with kitcomponent kit1_comp1-x86_64, enter:
lskmodules -c kit1_comp1-x86_64
FILES
SEE ALSO
lslite.1
NAME
lslite - Display a summary of the statelite information.
1.3. Admin Guide
275
xCAT3 Documentation, Release 2.12
SYNOPSIS
lslite [-h | --help]
lslite [-V | --verbose] [-i imagename] | [noderange]
DESCRIPTION
The lslite command displays a summary of the statelite information that has been defined for a noderange or an image.
OPTIONS
-h|--help
Display usage message.
-V|--verbose
Verbose mode.
-i imagename
The name of an existing xCAT osimage definition.
noderange
A set of comma delimited node names and/or group names. See the “noderange” man page for details on
additional supported formats.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To list the statelite information for an xCAT node named “node01”.
lslite node01
Output is similar to:
>>>Node: node01
Osimage: 61img
Persistent directory (statelite table):
xcatmn1:/statelite
Litefiles (litefile table):
tmpfs,rw
/etc/adjtime
tmpfs,rw
/etc/lvm/.cache
tmpfs,rw
/etc/mtab
........
276
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
Litetree path (litetree table):
1,MN:/etc
2,server1:/etc
2. To list the statelite information for an xCAT osimage named “osimage01”.
lslite -i osimage01
Output is similar to:
tmpfs,rw
tmpfs,rw
tmpfs,rw
........
/etc/adjtime
/etc/lvm/.cache
/etc/mtab
FILES
/opt/xcat/bin/lslite
SEE ALSO
noderange(3)|noderange.3, tabdump(8)|tabdump.8
lsslp.1
NAME
lsslp - Discovers selected networked services information within the same subnet.
SYNOPSIS
lsslp [-h| --help]
lsslp [-v| --version]
lsslp [noderange] [-V] [-i ip[,ip..]] [-w] [-r|-x|-z] [-n] [-s CEC|FRAME|MM|IVM|RSA|HMC|CMM|IMM2|FSP]
[-t tries] [-I] [-C counts] [-T timeout] [--vpdtable]
DESCRIPTION
The lsslp command discovers selected service types using the -s flag. All service types are returned if the -s flag is not
specified. If a specific IP address is not specified using the -i flag, the request is sent out all available network adapters.
The optional -r, -x, -z and –vpdtable flags format the output. If you can’t receive all the hardware, use -T to increase
the waiting time.
NOTE: SLP broadcast requests will propagate only within the subnet of the network adapter broadcast IPs specified
by the -i flag.
1.3. Admin Guide
277
xCAT3 Documentation, Release 2.12
OPTIONS
noderange The nodes which the user wants to discover. If the user specifies the noderange, lsslp will just return the
nodes in the node range. Which means it will help to add the new nodes to the xCAT database without modifying
the existed definitions. But the nodes’ name specified in noderange should be defined in database in advance. The
specified nodes’ type can be frame/cec/hmc/fsp/bpa. If the it is frame or cec, lsslp will list the bpa or fsp nodes within
the nodes(bap for frame, fsp for cec). Do not use noderange with the flag -s.
-i IP(s) the command will send out (defaults to all available adapters).
-h Display usage message.
-n Only display and write the newly discovered hardware.
-u Do unicast to a specified IP range. Must be used with -s and --range. The -u flag is not supported on AIX.
--range Specify one or more IP ranges. Must be use in unicast mode. It accepts multiple formats. For example,
192.168.1.1/24, 40-41.1-2.3-4.1-100. If the range is huge, for example, 192.168.1.1/8, lsslp may take a very long time
for node scan. So the range should be exactly specified.
-r Display Raw SLP response.
-C The number of the expected responses specified by the user. When using this flag, lsslp will not return until the it
has found all the nodes or time out. The default max time is 3 seconds. The user can use -T flag the specify the time
they want to use. A short time will limit the time costing, while a long time will help to find all the nodes.
-T The number in seconds to limit the time of lsslp.
-s Service type interested in discovering.
-t Number or service-request attempts.
--vpdtable Output the SLP response in vpdtable formatting. Easy for writing data to vpd table.
-v Command Version.
-V Verbose output.
-w Writes output to xCAT database.
-x XML format.
-z Stanza formatted output.
-I Give the warning message for the nodes in database which have no SLP responses. Note that this flag can only be
used after the database migration finished successfully.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To list all discovered HMC service types in tabular format, enter:
lsslp -s HMC
Output is similar to:
278
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
device
HMC
HMC
HMC
type-model
7310CR2
7310CR2
7310CR3
serial-number ip-addresses
103F55A
1.1.1.115
105369A
3.3.3.103
KPHHK24
3.3.3.154
hostname
hmc01
hmc02
hmc03
2. list all discovered FSP service types in raw response format on subnet 30.0.0.255, enter:
lsslp -i 3.0.0.255 -s CEC -r
Output is similar to:
(type=cec-service-processor),(serial-number=10A3AEB),(machinetype-model=9117-570),
˓→(fru-serial-number=YL11C5338102),(hostname=),(frame-number=0),(cage-number=0),(ip˓→address=3.0.0.94,1.1.1.147),(web-url=https://3.0.0.94:473 ), (slot=1),(bpc˓→machinetype-model=0),(bpc-serial-number=0),(Image=fips240/b0630a_0623.240)
(type=cec-service-processor),(serial-number=10A3E2B),(machinetype-model=9117-570),
˓→(fru-serial- number=YL11C5338250),(hostname=),(frame-number=0),(cage-number=0),(ip˓→address=3.0.0.95,1.1.1.147), (web-url=https://3.0.0.95:473 ),(slot=1),(bpc˓→machinetype-model=0),(bpc-serial-number=0),(Image=fips240/b0630a_0623.240)
3. To list all discovered MM service types in XML format and write the output to the xCAT database, enter:
lsslp -s MM -x -w
Output is similar to:
<Node>
<groups>mm,all</groups>
<id>00:14:5E:E0:CB:1E</id>
<mgt>blade</mgt>
<mtm>029310C</mtm>
<node>Server-029310C-SN100485A-A</node>
<nodetype>mm</nodetype>
<otherinterfaces>9.114.47.229</otherinterfaces>
<serial>100485A</serial>
</Node>
4. To list all discovered service types in stanza format and write the output to the xCAT database, enter:
lsslp -z -w
Output is similar to:
c76v1hmc02:
objtype=node
hcp=c76v1hmc02
nodetype=hmc
mtm=7315CR2
serial=10407DA
ip=192.168.200.125
groups=hmc,all
mgt=hmc
mac=00:1a:64:fb:7d:50
hidden=0
192.168.200.244:
objtype=node
hcp=192.168.200.244
nodetype=fsp
1.3. Admin Guide
279
xCAT3 Documentation, Release 2.12
mtm=9125-F2A
serial=0262662
side=A-0
otherinterfaces=192.168.200.244
groups=fsp,all
mgt=fsp
id=4
parent=Server-9125-F2A-SN0262662
mac=00:1a:64:fa:01:fe
hidden=1
Server-8205-E6B-SN1074CDP:
objtype=node
hcp=Server-8205-E6B-SN1074CDP
nodetype=cec
mtm=8205-E6B
serial=1074CDP
groups=cec,all
mgt=fsp
id=0
hidden=0
192.168.200.33:
objtype=node
hcp=192.168.200.33
nodetype=bpa
mtm=9458-100
serial=99201WM
side=B-0
otherinterfaces=192.168.200.33
groups=bpa,all
mgt=bpa
id=0
mac=00:09:6b:ad:19:90
hidden=1
Server-9125-F2A-SN0262652:
objtype=node
hcp=Server-9125-F2A-SN0262652
nodetype=frame
mtm=9125-F2A
serial=0262652
groups=frame,all
mgt=fsp
id=5
hidden=0
5. To list all discovered service types in stanza format and display the IP address, enter:
lsslp -w
Output is similar to:
mm01:
objtype=node
nodetype=fsp
mtm=8233-E8B
serial=1000ECP
side=A-0
groups=fsp,all
mgt=fsp
280
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
id=0
mac=00:14:5E:F0:5C:FD
otherinterfaces=50.0.0.5
bpa01:
objtype=node
nodetype=bpa
mtm=9A01-100
serial=0P1N746
side=A-1
groups=bpa,all
mgt=bpa
id=0
mac=00:1A:64:54:8C:A5
otherinterfaces=50.0.0.1
6. To list all the CECs, enter:
lsslp -s CEC
device
FSP
FSP
CEC
type-model
9117-MMB
9117-MMB
9117-MMB
serial-number
105EBEP
105EBEP
105EBEP
side
A-1
B-1
ip-addresses
20.0.0.138
20.0.0.139
hostname
20.0.0.138
20.0.0.139
Server-9117-MMB-SN105EBEP
7. To list all the nodes defined in database which have no SLP response.
lsslp -I
Output is similar to:
These nodes defined in database but can't be discovered: f17c00bpcb_b,f17c01bpcb_a,
˓→f17c01bpcb_b,f17c02bpcb_a,
device
bpa
bpa
type-model
9458-100
9458-100
serial-number
BPCF017
BPCF017
side
A-0
B-0
ip-addresses
40.17.0.1
40.17.0.2
hostname
f17c00bpca_a
f17c00bpcb_a
8. To find the nodes within the user specified. Make sure the noderange input have been defined in xCAT database.
lsslp CEC1-CEC3
or lsslp CEC1,CEC2,CEC3
device
FSP
FSP
FSP
FSP
CEC
FSP
FSP
FSP
FSP
CEC
FSP
FSP
FSP
type-model
9A01-100
9A01-100
9A01-100
9A01-100
9A01-100
8233-E8B
8233-E8B
8233-E8B
8233-E8B
8233-E8B
8205-E6B
8205-E6B
8205-E6B
1.3. Admin Guide
serial-number
0P1P336
0P1P336
0P1P336
0P1P336
0P1P336
1040C7P
1040C7P
1040C7P
1040C7P
1040C7P
1000ECP
1000ECP
1000ECP
side
A-0
B-0
A-1
B-1
ip-addresses
192.168.200.34
192.168.200.35
50.0.0.27
50.0.0.28
A-0
B-0
A-1
B-1
192.168.200.36
192.168.200.37
50.0.0.29
50.0.0.30
A-0
B-0
A-1
192.168.200.38
192.168.200.39
50.0.0.31
hostname
192.168.200.34
192.168.200.35
50.0.0.27
50.0.0.28
CEC1
192.168.200.36
192.168.200.37
50.0.0.29
50.0.0.30
CEC2
192.168.200.38
192.168.200.39
50.0.0.27
281
xCAT3 Documentation, Release 2.12
FSP
CEC
8205-E6B
8205-E6B
1000ECP
1000ECP
B-1
50.0.0.32
50.0.0.28
CEC3
9. To list all discovered CMM in stanza format, enter: lsslp -s CMM -m -z
e114ngmm1:
objtype=node
mpa=e114ngmm1
nodetype=cmm
mtm=98939AX
serial=102537A
groups=cmm,all
mgt=blade
hidden=0
otherinterfaces=70.0.0.30
hwtype=cmm
10. To use lsslp unicast, enter:
lsslp -u -s CEC --range 40-41.1-2.1-2.1-2
FILES
/opt/xcat/bin/lsslp
SEE ALSO
rscan(1)|rscan.1
lstree.1
NAME
lstree - Display the tree of service node hierarchy, hardware hierarchy, or VM hierarchy.
SYNOPSIS
lstree [-h | --help]
lstree [-s | --servicenode] [-H | --hardwaremgmt] [-v | --virtualmachine] [noderange]
DESCRIPTION
The lstree command can display the tree of service node hierarchy for the xCAT nodes which have service node
defined or which are service nodes, display the tree of hardware hierarchy only for the physical objects, display the
tree of VM hierarchy for the xCAT nodes which are virtual machines or which are the hosts of virtual machines. If
a noderange is specified, only show the part of the hierarchy that involves those nodes. For ZVM, we only support
to display VM hierarchy. By default, lstree will show both the hardware hierarchy and the VM hierarchy for all the
nodes.
282
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
OPTIONS
-h|--help
Display usage message.
-s|-- servicenode
Show the tree of service node hierarchy.
-H|--hardwaremgmt
Show the tree of hardware hierarchy.
--v|--virtualmachine
Show the tree of VM hierarchy.
noderange
A set of comma delimited node names and/or group names. See the “noderange” man page for details on
additional supported formats.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To display the tree of service node hierarchy for all the nodes.
lstree -s
Output is similar to:
Service Node: mysn01
|__mycn01
|__mycn02
|__mycn03
Service Node: mysn02
|__mycn11
|__mycn12
|__mycn13
......
2. To display the tree of service node hierarchy for service node “mysn01”.
lstree -s mysn01
Output is similar to:
Service Node: mysn01
|__mycn01
|__mycn02
|__mycn03
1.3. Admin Guide
283
xCAT3 Documentation, Release 2.12
3. To display the tree of hardware hierarchy for all the nodes.
lstree -H
Output is similar to:
HMC: myhmc01
|__Frame: myframe01
|__CEC: mycec01
|__CEC: mycec02
......
Service Focal Point: myhmc02
|__Frame: myframe01
|__CEC: mycec01
|__CEC: mycec02
|__CEC: mycec03
......
Management Module: mymm01
|__Blade 1: js22n01
|__Blade 2: js22n02
|__Blade 3: js22n03
......
BMC: 192.168.0.1
|__Server: x3650n01
4. To display the tree of hardware hierarchy for HMC “myhmc01”.
lstree -H myhmc01
Output is similar to:
HMC: myhmc01
|__Frame: myframe01
|__CEC: mycec01
|__CEC: mycec02
......
5. To display the tree of VM hierarchy for all the nodes.
lstree -v
Output is similar to:
Server: hs22n01
|__ hs22vm1
Server: x3650n01
|__ x3650n01kvm1
|__ x3650n01kvm2
6. To display the tree of VM hierarchy for the node “x3650n01”.
lstree -v x3650n01
Output is similar to:
284
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
Server: x3650n01
|__ x3650n01kvm1
|__ x3650n01kvm2
7. To display both the hardware tree and VM tree for all nodes.
lstree
Output is similar to:
HMC: myhmc01
|__Frame: myframe01
|__CEC: mycec01
|__LPAR 1: node01
|__LPAR 2: node02
|__LPAR 3: node03
......
|__CEC: mycec02
|__LPAR 1: node11
|__LPAR 2: node12
|__LPAR 3: node13
......
Service Focal Point: myhmc02
|__Frame: myframe01
|__CEC: mycec01
|__LPAR 1: node01
|__LPAR 2: node02
|__LPAR 3: node03
......
|__Frame: myframe02
|__CEC: mycec02
|__LPAR 1: node21
|__LPAR 2: node22
|__LPAR 3: node23
......
Management Module: mymm01
|__Blade 1: hs22n01
|__hs22n01vm1
|__hs22n01vm2
|__Blade 2: hs22n02
|__hs22n02vm1
|__hs22n02vm2
......
BMC: 192.168.0.1
|__Server: x3650n01
|__ x3650n01kvm1
|__ x3650n01kvm2
FILES
/opt/xcat/bin/lstree
1.3. Admin Guide
285
xCAT3 Documentation, Release 2.12
SEE ALSO
noderange(3)|noderange.3, tabdump(8)|tabdump.8
lsve.1
NAME
lsve - Lists detail attributes for a virtual environment.
SYNOPSIS
lsve [-t type] [-m manager] [-o object]
DESCRIPTION
The lsve command can be used to list a virtual environment for ‘Data Center’, ‘Cluster’, ‘Storage Domain’, ‘Network’
and ‘Template’ objects.
The mandatory parameter -m manager is used to specify the address of the manager of virtual environment. xCAT
needs it to access the RHEV manager.
The mandatory parameter -t type is used to specify the type of the target object.
Basically, lsve command supports three types of object: dc, cl, sd, nwand tpl.
The parameter -o object is used to specify which object to list. If no -o is specified, all the objects with the -t type will
be displayed.
OPTIONS
-h Display usage message.
-m Specify the manager of the virtual environment.
For RHEV, the FQDN (Fully Qualified Domain Name) of the rhev manager have to be specified.
-o The target object to display.
-t Specify the type of the target object.
Supported types:
B<dc> - Data Center (For type of 'dc', all the elements belongs to the data
˓→center will be listed.)
B<cl> - Cluster
B<sd> - Storage Domain (To get the status of Storage Doamin, show it from I
˓→<data center> it attached to.
B<nw> - Network
B<tpl> - Template
286
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To list the data center ‘Default’, enter:
lsve -t dc -m <FQDN of rhev manager> -o Default
Output is similar to:
datacenters: [Default]
description: The default Data Center
state: up
storageformat: v1
storagetype: nfs
clusters: [Default]
cpu: Intel Westmere Family
description: The default server cluster
memory_hugepage: true
memory_overcommit: 100
storagedomains: [image]
available: 55834574848
committed: 13958643712
ismaster: true
status: active
storage_add: <Address of storage domain>
storage_format: v1
storage_path: /vfsimg
storage_type: nfs
type: data
used: 9663676416
networks: [rhevm2]
description:
state: operational
stp: false
networks: [rhevm]
description: Management Network
state: operational
stp: false
templates: [Blank]
bootorder: hd
cpucore: 1
cpusocket: 1
creation_time: 2008-04-01T00:00:00.000-04:00
display: spice
memory: 536870912
state: ok
stateless: false
type: desktop
2. To list the cluster ‘Default’, enter:
1.3. Admin Guide
287
xCAT3 Documentation, Release 2.12
lsve -t cl -m <FQDN of rhev manager> -o Default
Output is similar to:
cpu: Intel Westmere Family
description: The default server cluster
memory_hugepage: true
memory_overcommit: 10
3. To list the Storage Domain ‘image’, enter:
lsve -t sd -m <FQDN of rhev manager> -o image
Output is similar to:
storagedomains: [image]
available: 55834574848
committed: 13958643712
ismaster: true
status:
storage_add: <Address of storage domain>
storage_format: v1
storage_path: /vfsimg
storage_type: nfs
type: data
used: 9663676416
4. To list the network ‘rhevm’, enter:
lsve -t nw -m <FQDN of rhev manager> -o rhevm
Output is similar to:
networks: [rhevm]
description: Management Network
state: operational
stp: false
5. To list the template ‘tpl01’, enter:
lsve -t tpl -m <FQDN of rhev manager> -o tpl01
Output is similar to:
templates: [tpl01]
bootorder: network
cpucore: 2
cpusocket: 2
creation_time: 2012-08-22T23:52:35.953-04:00
display: vnc
memory: 1999634432
state: ok
stateless: false
type: server
288
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
FILES
/opt/xcat/bin/lsve
SEE ALSO
cfgve(1)|cfgve.1
lsvlan.1
NAME
lsvlan - It lists the existing vlans for the cluster.
SYNOPSIS
lsvlan
lsvlan [vlanid]
lsvlan [-h | --help]
lsvlan [-v | --version]
DESCRIPTION
The lsvlan command lists all the vlans for the cluster. If vlanid is specifined it will list more details about this vlan
including the nodes in the vlan.
PARAMETERS
vlanid is a unique vlan number. If it is omitted, all vlans will be listed.
OPTIONS
-h|--help Display usage message.
-v|--version Command Version.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
1.3. Admin Guide
289
xCAT3 Documentation, Release 2.12
EXAMPLES
1. To list all the vlans in the cluster
lsvlan
Output is similar to:
vlan 3:
subnet 10.3.0.0
netmask 255.255.0.0
vlan 4:
subnet 10.4.0.0
netmask 255.255.0.0
2. To list the details for vlan3
lsvlan 3
Output is similar to:
vlan 3
subnet 10.3.0.0
netmask 255.255.0.0
hostname
v3n1
v3n2
v3n3
v3n4
v3n5
ip address
10.3.0.1
10.3.0.2
10.3.0.3
10.3.0.4
10.3.0.5
node
c68m4hsp06
x3455n01
x3650n01
x3650n01kvm1
x3650n01kvm2
vm host
x3650n01
x3650n01
FILES
/opt/xcat/bin/lsvlan
SEE ALSO
mkvlan(1)|mkvlan.1, rmvlan(1)|rmvlan.1, chvlan(1)|chvlan.1
lsvm.1
NAME
lsvm - Lists partition profile information for HMC-, DFM-, IVM-, KVM-, VMware- and zVM-managed nodes. For
Power 775, it lists the LPARs’ I/O slots information and CEC configuration.
SYNOPSIS
lsvm [-h| --help]
290
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
lsvm [-v| --version]
lsvm [-V| --verbose] noderange
lsvm [-a| --all] noderange
For PPC (using Direct FSP Management):
lsvm [-l| --long] --p775 noderange
lsvm noderange
For KVM and VMware
lsvm noderange
For zVM:
lsvm noderange
DESCRIPTION
The lsvm command lists all partition profiles defined for the partitions specified in noderange. If noderange is a CEC,
all the partitions associated with that CEC are displayed.
For PPC (using Direct FSP Management):
For Power 775 (use option --p775 to specify), lsvm lists all partition I/O slots information for the partitions specified
in noderange. If noderange is a CEC, it gets the CEC’s pump mode value, octant’s memory interleaving value, the all
the octants configure value, and all the I/O slots information.
For DFM-managed (short for Direct FSP Management mode) normal power machine, lsvm lists the processor, memory, physical I/O slots, hugepage and BSR info for the specified partitions or CEC.
The pump mode value has the valid options: 1 - Node Pump Mode 2 - Chip Pump Mode
The Memory Interleaving Mode has 3 valid options: 0 - not Applicable 1 - interleaved 2 - non-interleaved
More information about this part, refer to the section Using the *vm commands to define partitions in xCAT DFM in the doc be
XCAT_Power_775_Hardware_Management
For KVM and VMware
If noderange is a hypervisor, virtual machines defined on that hypervisor will be displayed. If noderange is a VM,
details for that VM will be displayed.
Note: Only the virtual machine which is in power on state can be listed by lsvm command.
For zVM:
Show the directory entry for a given virtual machine.
1.3. Admin Guide
291
xCAT3 Documentation, Release 2.12
OPTIONS
-h
Display usage message.
-v
Command version.
-V
Verbose output.
-a
List all the profiles for one partition
--p775
Specify the operation is for Power 775 machines.
-l
Show lparnames for lpars. It shall work with option --p775.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To list all partition profiles defined for HMC-managed partition lpar3, enter:
lsvm lpar3
Output is similar to:
lpar3: name=lpar3,lpar_name=lpar3,lpar_id=4,lpar_env=aixlinux,all_resources=0,min_
˓→mem=512, desired_mem=2048, max_mem=3072,min_num_huge_pages=0,desired_num_huge_
˓→pages=0,max_num_huge_pages=0,proc_mode=shared, min_proc_units=0.5,desired_proc_
˓→units=0.5,max_proc_units=0.5,min_procs=1,desired_procs=1,max_procs=1, sharing_
˓→mode=uncap,uncap_weight=128,shared_proc_pool_id=0,shared_proc_pool_name=DefaultPool,
˓→io_slots=none, lpar_io_pool_ids=none,max_virtual_slots=10, "virtual_serial_
˓→adapters=1/server/1/any//any/1,0/server/1/any//any/1", virtual_scsi_adapters=2/
˓→client/1/p6vios/4/1,virtual_eth_adapters=3/0/1//0/1,hca_adapters=none,boot_
˓→mode=norm,conn_monitoring=0,auto_start=0,power_ctrl_lpar_ids=none,work_group_
˓→id=none,redundant_err_path_reporting=0, bsr_arrays=0,lhea_logical_ports=none,lhea_
˓→capabilities=none,lpar_proc_compat_mode=default,electronic_err_reporting=null
2. To list all IVM-managed partitions associated with CEC cec01, enter:
lsvm cec01
Output is similar to:
292
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
cec01: name=10-B7D1G,lpar_name=10-B7D1G,lpar_id=1,os_type=vioserver,all_resources=0,
˓→min_mem=512, desired_mem=2048,max_mem=2048,proc_mode=shared,min_proc_units=0.10,
˓→desired_proc_units=0.40, max_proc_units=4.00,min_procs=1,desired_procs=4,max_
˓→procs=4,sharing_mode=uncap,uncap_weight=128, "io_slots=21010002/none/0,21010003/
˓→none/0,21010004/none/0,21020003/none/0,21020004/none/0,21030003/none/0,21030004/
˓→none/0,21040003/none/0,21040004/none/0",lpar_io_pool_ids=none,max_virtual_slots=48,
˓→"virtual_serial_adapters=0/server/1/any//any/1,1/server/1/any//any/1,10/client/0/2/
˓→lp2/0/0,12/client/0/3/lp3/0/0,14/client/0/4/lp4/0/0","virtual_scsi_adapters=11/
˓→server/2/lp2/2/0,13/server/3/lp3/2/0,15/server/4/lp4/2/0","virtual_eth_adapters=3/0/
˓→1//1/0,4/0/2//1/0,5/0/3//1/0,6/0/4//1/0",boot_mode=norm,conn_monitoring=0,auto_
˓→start=0,power_ctrl_lpar_ids=none
name=lp2,lpar_name=lp2,lpar_id=2,os_type=aixlinux,all_resources=0,min_mem=128,
˓→desired_mem=1024,max_mem=1024,proc_mode=shared,min_proc_units=0.10,desired_proc_
˓→units=0.10,max_proc_units=4.00,min_procs=1,desired_procs=1,max_procs=4,sharing_
˓→mode=uncap,uncap_weight=128,io_slots=none,lpar_io_pool_ids=none,max_virtual_slots=6,
˓→ "virtual_serial_adapters=0/server/1/any//any/1,1/server/1/any//any/1",virtual_scsi_
˓→adapters=2/client/1/10-7D1G/11/1,virtual_eth_adapters=4/0/1//0/0,boot_mode=norm,
˓→conn_monitoring=0,auto_start=0,power_ctrl_lpar_ids=none
name=lp3,lpar_name=lp3,lpar_id=3,os_type=aixlinux,all_resources=0,min_mem=128,
˓→desired_mem=128,max_mem=128,proc_mode=shared,min_proc_units=0.10,desired_proc_
˓→units=0.10,max_proc_units=4.00,min_procs=1,desired_procs=1,max_procs=4,sharing_
˓→mode=uncap,uncap_weight=128,io_slots=none,lpar_io_pool_ids=none,max_virtual_slots=6,
˓→ "virtual_serial_adapters=0/server/1/any//any/1,1/server/1/any//any/1",virtual_scsi_
˓→adapters=2/client/1/10-B7D1G/13/1,virtual_eth_adapters=4/0/1//0/0,boot_mode=of,conn_
˓→monitoring=0,auto_start=1, power_ctrl_lpar_ids=none
3. For Power 775, to list the I/O slot information of lpar1, enter:
lsvm lpar1 --p775
Output is similar to:
1: 514/U78A9.001.0123456-P1-C17/0x21010202/2/1
1: 513/U78A9.001.0123456-P1-C15/0x21010201/2/1
1: 512/U78A9.001.0123456-P1-C16/0x21010200/2/1
4. To list the lparname of lpars, enter:
lsvm lpar1 -l --p775
Output is similar to:
lpar1: 1: 514/U78A9.001.0123456-P1-C17/0x21010202/2/1
lpar1: 1: 513/U78A9.001.0123456-P1-C15/0x21010201/2/1
lpar1: 1: 512/U78A9.001.0123456-P1-C16/0x21010200/2/1
5. For Power 775, to list the I/O slot information and octant configuration of cec1, enter:
lsvm cec1 --p775
Output is similar to:
1: 514/U78A9.001.0123456-P1-C17/0x21010202/2/1
1: 513/U78A9.001.0123456-P1-C15/0x21010201/2/1
1: 512/U78A9.001.0123456-P1-C16/0x21010200/2/1
13: 537/U78A9.001.0123456-P1-C9/0x21010219/2/13
13: 536/U78A9.001.0123456-P1-C10/0x21010218/2/13
17: 545/U78A9.001.0123456-P1-C7/0x21010221/2/17
1.3. Admin Guide
293
xCAT3 Documentation, Release 2.12
17: 544/U78A9.001.0123456-P1-C8/0x21010220/2/17
21: 553/U78A9.001.0123456-P1-C5/0x21010229/2/21
21: 552/U78A9.001.0123456-P1-C6/0x21010228/2/21
25: 569/U78A9.001.0123456-P1-C1/0x21010239/2/25
25: 561/U78A9.001.0123456-P1-C3/0x21010231/2/25
25: 560/U78A9.001.0123456-P1-C4/0x21010230/2/25
29: 568/U78A9.001.0123456-P1-C2/0x21010238/2/29
5: 521/U78A9.001.0123456-P1-C13/0x21010209/2/5
5: 520/U78A9.001.0123456-P1-C14/0x21010208/2/5
9: 529/U78A9.001.0123456-P1-C11/0x21010211/2/9
9: 528/U78A9.001.0123456-P1-C12/0x21010210/2/9
cec1: PendingPumpMode=1,CurrentPumpMode=1,OctantCount=8:
OctantID=0,PendingOctCfg=5,CurrentOctCfg=1,PendingMemoryInterleaveMode=2,
˓→CurrentMemoryInterleaveMode=2;
OctantID=1,PendingOctCfg=1,CurrentOctCfg=1,PendingMemoryInterleaveMode=2,
˓→CurrentMemoryInterleaveMode=2;
OctantID=2,PendingOctCfg=1,CurrentOctCfg=1,PendingMemoryInterleaveMode=2,
˓→CurrentMemoryInterleaveMode=2;
OctantID=3,PendingOctCfg=1,CurrentOctCfg=1,PendingMemoryInterleaveMode=2,
˓→CurrentMemoryInterleaveMode=2;
OctantID=4,PendingOctCfg=1,CurrentOctCfg=1,PendingMemoryInterleaveMode=2,
˓→CurrentMemoryInterleaveMode=2;
OctantID=5,PendingOctCfg=1,CurrentOctCfg=1,PendingMemoryInterleaveMode=2,
˓→CurrentMemoryInterleaveMode=2;
OctantID=6,PendingOctCfg=1,CurrentOctCfg=1,PendingMemoryInterleaveMode=2,
˓→CurrentMemoryInterleaveMode=2;
OctantID=7,PendingOctCfg=1,CurrentOctCfg=1,PendingMemoryInterleaveMode=2,
˓→CurrentMemoryInterleaveMode=2;
6. To list the lparname of lpars, enter:
lsvm cec1 -l --p775
Output is similar to:
lpar1: 1: 514/U78A9.001.0123456-P1-C17/0x21010202/2/1: 32: 0/3/3
lpar1: 1: 513/U78A9.001.0123456-P1-C15/0x21010201/2/1: 32: 0/3/3
lpar1: 1: 512/U78A9.001.0123456-P1-C16/0x21010200/2/1: 32: 0/3/3
lpar13: 13: 537/U78A9.001.0123456-P1-C9/0x21010219/2/13: 32: 0/3/3
lpar13: 13: 536/U78A9.001.0123456-P1-C10/0x21010218/2/13: 32: 0/3/3
lpar17: 17: 545/U78A9.001.0123456-P1-C7/0x21010221/2/17: 32: 0/0/0
lpar17: 17: 544/U78A9.001.0123456-P1-C8/0x21010220/2/17: 32: 0/0/0
lpar21: 21: 553/U78A9.001.0123456-P1-C5/0x21010229/2/21: 32: 0/0/0
lpar21: 21: 552/U78A9.001.0123456-P1-C6/0x21010228/2/21: 32: 0/0/0
lpar24: 25: 569/U78A9.001.0123456-P1-C1/0x21010239/2/25: 32: 0/0/0
lpar25: 25: 561/U78A9.001.0123456-P1-C3/0x21010231/2/25: 32: 0/0/0
lpar25: 25: 560/U78A9.001.0123456-P1-C4/0x21010230/2/25: 32: 0/0/0
lpar29: 29: 568/U78A9.001.0123456-P1-C2/0x21010238/2/29: 32: 0/0/0
lpar5: 5: 521/U78A9.001.0123456-P1-C13/0x21010209/2/5: 32: 0/3/3
lpar5: 5: 520/U78A9.001.0123456-P1-C14/0x21010208/2/5: 32: 0/3/3
lpar9: 9: 529/U78A9.001.0123456-P1-C11/0x21010211/2/9: 32: 0/3/3
lpar9: 9: 528/U78A9.001.0123456-P1-C12/0x21010210/2/9: 32: 0/3/3
cec1: PendingPumpMode=1,CurrentPumpMode=1,OctantCount=8:
OctantID=0,PendingOctCfg=5,CurrentOctCfg=1,PendingMemoryInterleaveMode=2,
˓→CurrentMemoryInterleaveMode=2;
OctantID=1,PendingOctCfg=1,CurrentOctCfg=1,PendingMemoryInterleaveMode=2,
˓→CurrentMemoryInterleaveMode=2;
OctantID=2,PendingOctCfg=1,CurrentOctCfg=1,PendingMemoryInterleaveMode=2,
˓→CurrentMemoryInterleaveMode=2;
294
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
OctantID=3,PendingOctCfg=1,CurrentOctCfg=1,PendingMemoryInterleaveMode=2,
˓→CurrentMemoryInterleaveMode=2;
OctantID=4,PendingOctCfg=1,CurrentOctCfg=1,PendingMemoryInterleaveMode=2,
˓→CurrentMemoryInterleaveMode=2;
OctantID=5,PendingOctCfg=1,CurrentOctCfg=1,PendingMemoryInterleaveMode=2,
˓→CurrentMemoryInterleaveMode=2;
OctantID=6,PendingOctCfg=1,CurrentOctCfg=1,PendingMemoryInterleaveMode=2,
˓→CurrentMemoryInterleaveMode=2;
OctantID=7,PendingOctCfg=1,CurrentOctCfg=1,PendingMemoryInterleaveMode=2,
˓→CurrentMemoryInterleaveMode=2;
Number of BSR arrays: 256,Bytes per BSR array: 4096,Available BSR array: 0;
Available huge page memory(in pages):
0
Configurable huge page memory(in pages): 12
Page Size(in GB):
16
Maximum huge page memory(in pages):
24
Requested huge page memory(in pages):
15
Number of BSR arrays: 256,Bytes per BSR array: 4096,Available BSR array: 0;
Available huge page memory(in pages):
0
Configurable huge page memory(in pages): 12
Page Size(in GB):
16
Maximum huge page memory(in pages):
24
Requested huge page memory(in pages):
15
7. To list the virtual machine’s directory entry:
lsvm gpok3
Output is similar to:
gpok3: USER LNX3 PWD 512M 1G G
gpok3: INCLUDE LNXDFLT
gpok3: COMMAND SET VSWITCH VSW2 GRANT LNX3
8. For DFM-managed normal power machine, list out the detailed resource information:
lsvm cec
Output is similar to:
cec: HYP Configurable Processors: 16, Avail Processors: 16.
HYP Configurable Memory:32.00 GB(128 regions).
HYP Available Memory:
31.25 GB(125 regions).
HYP Memory Region Size: 0.25 GB(256 MB).
cec: All Physical I/O info:
65535,519,U78AA.001.WZSGVU7-P1-C7,0x21010207,0xffff(Empty Slot)
65535,518,U78AA.001.WZSGVU7-P1-C6,0x21010206,0xffff(Empty Slot)
65535,517,U78AA.001.WZSGVU7-P1-C5,0x21010205,0xffff(Empty Slot)
65535,516,U78AA.001.WZSGVU7-P1-C4,0x21010204,0xffff(Empty Slot)
65535,514,U78AA.001.WZSGVU7-P1-C19,0x21010202,0xffff(Empty Slot)
65535,513,U78AA.001.WZSGVU7-P1-T7,0x21010201,0xc03(USB Controller)
65535,512,U78AA.001.WZSGVU7-P1-T9,0x21010200,0x104(RAID Controller)
cec: Huge Page Memory
Available huge page memory(in pages):
2
Configurable huge page memory(in pages): 2
Page Size(in GB):
16
Maximum huge page memory(in pages):
4
Requested huge page memory(in pages):
2
cec: Barrier Synchronization Register(BSR)
1.3. Admin Guide
295
xCAT3 Documentation, Release 2.12
Number of BSR arrays: 256
Bytes per BSR array: 4096
Available BSR array: 256
Note: The lines listed in “All Physical I/O info” section represent all the physical I/O resource information. The format
is like “owner_lparid,slot_id,physical resource name,drc_index,slot_class_code(class description)”. The ‘drc index’
is short for Dynamic Resource Configuration Index, it uniquely indicates a physical I/O resource in a normal power
machine.
9. For DFM-managed partition on normal power machine, list out the detailed information:
lsvm lpar1
Output is similar to:
lpar1: Lpar Processor Info:
Curr Processor Min: 1.
Curr Processor Req: 16.
Curr Processor Max: 16.
lpar1: Lpar Memory Info:
Curr Memory Min: 0.25 GB(1 regions).
Curr Memory Req: 30.75 GB(123 regions).
Curr Memory Max: 32.00 GB(128 regions).
lpar1: 1,519,U78AA.001.WZSGVU7-P1-C7,0x21010207,0xffff(Empty Slot)
lpar1: 1,518,U78AA.001.WZSGVU7-P1-C6,0x21010206,0xffff(Empty Slot)
lpar1: 1,517,U78AA.001.WZSGVU7-P1-C5,0x21010205,0xffff(Empty Slot)
lpar1: 1,516,U78AA.001.WZSGVU7-P1-C4,0x21010204,0xffff(Empty Slot)
lpar1: 1,514,U78AA.001.WZSGVU7-P1-C19,0x21010202,0xffff(Empty Slot)
lpar1: 1,513,U78AA.001.WZSGVU7-P1-T7,0x21010201,0xc03(USB Controller)
lpar1: 1,512,U78AA.001.WZSGVU7-P1-T9,0x21010200,0x104(RAID Controller)
lpar1: 1/2/2
lpar1: 256.
FILES
/opt/xcat/bin/lsvm
SEE ALSO
mkvm(1)|mkvm.1, chvm(1)|chvm.1, rmvm(1)|rmvm.1
lsxcatd.1
NAME
lsxcatd - lists xCAT daemon information.
SYNOPSIS
lsxcatd [-h | --help | -v | --version | -d | --database | -t | --nodetype | -a | --all ]
296
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
DESCRIPTION
The lsxcat command lists important xCAT daemon (xcatd) information.
OPTIONS
-v|--version
Command Version.
-h|--help
Display usage message.
-d|--database
Displays information about the current database being used by xCAT.
-t|--nodetype
Displays whether the node is a Management Node or a Service Node.
-a|--all
Displays all information about the daemon supported by the command.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To display information about the current database:
lsxcatd -d
Output is similar to:
cfgloc=Pg:dbname=xcatdb;host=7.777.47.250|xcatadm
dbengine=Pg
dbname=xcatdb
dbhost=7.777.47.250
dbadmin=xcatadm
2. To display all information:
lsxcatd -a
Output is similar to:
Version 2.8.5 (git commit 0d4888af5a7a96ed521cb0e32e2c918a9d13d7cc, built Tue Jul
˓→29 02:22:47 EDT 2014)
This is a Management Node
cfgloc=mysql:dbname=xcatdb;host=9.114.34.44|xcatadmin
dbengine=mysql
1.3. Admin Guide
297
xCAT3 Documentation, Release 2.12
dbname=xcatdb
dbhost=9.114.34.44
dbadmin=xcatadmin
FILES
/opt/xcat/bin/lsxcatd
SEE ALSO
makentp.1
SYNOPSIS
makentp [-h|--help]
makentp [-v|--version]
makentp [-a|--all] [-V|--verbose]
DESCRIPTION
makentp command sets up the NTP server on the xCAT management node and the service node.
By default, it sets up the NTP server for xCAT management node. If -a flag is specified, the command will setup the
ntp servers for management node as well as all the service nodes that have servicenode.ntpserver set. It honors the site
table attributes extntpservers and ntpservers described below:
site.extntpservers – the NTP servers for the management node to sync with. If it is empty then the NTP server will use
the management node’s own hardware clock to calculate the system date and time.
site.ntpservers – the NTP servers for the service node and compute node to sync with. The keyword <xcatmaster>
means that the node’s NTP server is the node that is managing it (either its service node or the management node).
To setup NTP on the compute node, add setupntp postscript to the postscripts table and run updatenode node -P
setupntp command.
OPTIONS
-a|--all
Setup NTP servers for both management node and the service node. If management node has SLES
installed and used as ntpservers, it is recommanded to use the setupntp postscript to set up NTP server
for service nodes.
-h|--help
Display usage message.
-v|--version
Command Version.
-V|--verbose
298
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
Verbose output.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To setup NTP server on the management node:
makentp
2. To setup NTP servers on both management node and the service node:
makentp -a
FILES
/opt/xcat/bin/makentp
SEE ALSO
mkdef.1
NAME
mkdef - Use this command to create xCAT data object definitions.
SYNOPSIS
mkdef [-h | --help] [-t object-types]
mkdef [-V | --verbose] [-t object-types] [--template template-object-name] [-o object-names] [-z | --stanza]
[-d | --dynamic] [-f | --force] [[-w attr==val] [-w attr=~val] ...] [noderange] [attr=val [attr=val...]] [-u
provmethod={install | netboot | statelite} profile= xxx [osvers= value] [osarch= value]]
DESCRIPTION
This command is used to create xCAT object definitions which are stored in the xCAT database. If the definition
already exists it will return an error message. The force option may be used to re-create a definition. In this case the
old definition will be remove and the new definition will be created.
1.3. Admin Guide
299
xCAT3 Documentation, Release 2.12
OPTIONS
attr=val [attr=val ...]
Specifies one or more “attribute equals value” pairs, separated by spaces. Attr=val pairs must be specified
last on the command line. Use the help option to get a list of valid attributes for each object type.
Note: when creating node object definitions, the ‘groups’ attribute is required.
-d|--dynamic
Use the dynamic option to create dynamic node groups. This option must be used with -w option.
-f|--force
Use the force option to re-create object definitions. This option removes the old definition before creating
the new one.
-h|--help
Display usage message.
noderange
A set of comma delimited node names and/or group names. (must be the first parameter) See the
“noderange” man page for details on supported formats.
-o object-names
A set of comma delimited object names.
-t object-types
A set of comma delimited object types. Use the help option to get a list of valid object types.
--template template-object-name
Name of the xCAT shipped object definition template or an existing object, from which the new object
definition will be created. The newly created object will inherit the attributes of the template definition
unless the attribute is specified in the arguments of mkdef command. If there are a template and an
existing object with the same name template-object-name, the template object takes precedence over the
existing object. For the details of xCAT shipped object definition templates, refer to the manpage of
--template option in lsdef(1)|lsdef.1.
-V|--verbose
Verbose mode.
-w attr==val -w attr=~val ...
Use one or multiple -w flags to specify the selection string that can be used to select objects. The operators
==, !=, =~ and !~ are available. For mkdef command, the -w flag only makes sense for creating dynamic
node group. Use the help option to get a list of valid attributes for each object type.
Operator descriptions: == Select nodes where the attribute value is exactly this value. != Select nodes
where the attribute value is not this specific value. =~ Select nodes where the attribute value matches
this regular expression. !~ Select nodes where the attribute value does not match this regular expression.
Note: if the “val” fields includes spaces or any other characters that will be parsed by shell, the
“attr<operator>val” needs to be quoted. If the operator is ”!~”, the “attr<operator>val” needs to be quoted
using single quote.
-z|--stanza
300
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
Indicates that the file being piped to the command is in stanza format. See the xcatstanzafile man page for
details on using xCAT stanza files.
-u
Fill in the attributes such as template file, pkglist file and otherpkglist file of osimage object based on
the specified parameters. It will search “/install/custom/” directory first, and then “/opt/xcat/share/”. The
provmethod and profile must be specified. If osvers or osarch is not specified, the corresponding value of
the management node will be used.
Note: this option only works for objtype osimage.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To create a site definition.
mkdef -t site -o clustersite
installdir=/xcatinstall
2. To create a basic node definition.
mkdef -t node -o node01 groups="all,aix"
3. To re-create the current definition of “node01”.
mkdef -f -t node -o node01
nodetype=osi groups="linux"
(The group definitions are also created if they don’t already exist.)
4. To create a set of different types of definitions based on information contained in a stanza file.
cat defstanzafile | mkdef -z
5. To create a group definition called LinuxNodes containing the nodes clstrn01 and clstrn02.
mkdef -t group -o LinuxNodes members="clstrn01,clstrn02"
6. To create a node definition for an FSP node using the attributes provided by the group fspnodes.
mkdef -t node fspn1 groups=fspnodes nodetype=fsp
7. To create node definitions for a set of node host names contained in the node range “node1,node2,node3”
mkdef -t node node1,node2,node3 power=hmc groups="all,aix"
8. To create a dynamic node group definition called HMCMgtNodes containing all the HMC managed nodes”
mkdef -t group -o HMCMgtNodes -d -w mgt==hmc -w cons==hmc
9. To create a dynamic node group definition called SLESNodes containing all the SLES nodes
1.3. Admin Guide
301
xCAT3 Documentation, Release 2.12
mkdef -t group -o SLESNodes -d -w "os=~^sles[0-9]+$"
10. To create a entry (7.0) in the policy table for user admin1
mkdef -t policy -o 7.0 name=admin1 rule=allow
11. To create a node definition with nic attributes
mkdef -t node cn1 groups=all nicips.eth0="1.1.1.1|1.2.1.1" nicnetworks.eth0=
˓→"net1|net2" nictypes.eth0="Ethernet"
12. To create an osimage definition and fill in attributes automatically.
mkdef redhat6img -u profile=compute provmethod=statelite
13. To create a PowerLE kvm node definition with the xCAT shipped template “ppc64lekvmguest-template”.
mkdef -t node cn1 --template ppc64lekvmguest-template ip=1.1.1.1
˓→mac=42:3d:0a:05:27:0b vmhost=1.1.0.1 vmnics=br0
14. To create a node definition from an existing node definition “cn1”
mkdef -t node cn2 --template cn1 ip=1.1.1.2 mac=42:3d:0a:05:27:0c
FILES
$XCATROOT/bin/mkdef
(The XCATROOT environment variable is set when xCAT is installed. The default value is “/opt/xcat”.)
NOTES
This command is part of the xCAT software product.
SEE ALSO
chdef(1)|chdef.1, lsdef(1)|lsdef.1, rmdef(1)|rmdef.1, xcatstanzafile(5)|xcatstanzafile.5
mkdocker.1
NAME
mkdocker - Create docker instance.
SYNOPSIS
mkdocker noderange [image=image_name [command=command]] [dockerflag=flags_to_create_instance]
mkdocker [-h | --help]
mkdocker {-v | --version}
302
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
DESCRIPTION
mkdocker To create docker instances with the specified image, command and/or dockerflags.
OPTIONS
image
The docker image name that the instance will use.
command
The command that the instance will run based on the image specified. The image option must be specified
in order to use this option.
dockerflag
A JSON string which will be used as parameters to create a docker. Reference https://docs.docker.
com/engine/reference/api/docker_remote_api_v1.22/ for more information about which parameters can
be specified.
Some useful flags are:
AttachStdin=true | false
Whether attaches to stdin.
AttachStdout=true | false
Whether attaches to stdout.
AttachStderr=true | false
Whether attaches to stderr.
OpenStdin=true | false
Whether opens stdin.
Tty=true | false
Attach standard streams to a tty, including stdin if it is not closed.
ExposedPorts
An object mapping ports to an empty object in the form of:
"ExposedPorts": { "<port>/\<tcp|udp>: {}" }
HostConfig: {“Binds”}
A list of volume bindings for this docker instance, the form will be:
"HostConfig": {"Binds":["<dir_on_dockerhost>:<dir_in_instance>"]}
EXAMPLES
1. To create a basic docker instance with stdin opened
mkdocker host01c01 image=ubuntu command=/bin/bash dockerflag="{\"AttachStdin\":true,\
˓→"AttachStdout\":true,\"AttachStderr\":true,\"OpenStdin\":true}"
1.3. Admin Guide
303
xCAT3 Documentation, Release 2.12
Output is similar to:
host01c01:
host01c01:
host01c01:
host01c01:
host01c01:
Pull image ubuntu start
Pull image ubuntu done
Remove default network connection
Connecting customized network 'mynet0'
success
2. To create a docker instance which have dir “destdir” in docker instance bind from “srcdir” on dockerhost, and
have “Tty” opened with which the docker instance can be attached after started to check the files under “destdir”.
mkdocker host01c01 image=ubuntu command=/bin/bash dockerflag="{\"AttachStdin\":true,\
˓→"AttachStdout\":true,\"AttachStderr\":true,\"OpenStdin\":true,\"Tty\":true,\
˓→"HostConfig\":{\"Binds\":[\"/srcdir:/destdir\"]}}"
Output is similar to:
host01c01: Remove default network connection
host01c01: Connecting customized network 'mynet0'
host01c01: success
SEE ALSO
rmdocker(1)|rmdocker.1, lsdocker(1)|lsdocker.1
mkdsklsnode.1
NAME
mkdsklsnode - Use this xCAT command to define and initialize AIX/NIM diskless machines.
SYNOPSIS
mkdsklsnode [-h|--help ]
mkdsklsnode [-V|--verbose] [-f|--force] [-n|--newname] [-i osimage_name] [-l location] [-u | --updateSN] [-k | -skipsync] [-p | --primarySN] [-b | --backupSN] [-S | --setuphanfs] noderange [attr=val [attr=val ...]]
DESCRIPTION
This xCAT command can be used to define and/or initialize AIX/NIM diskless machines. Once this step is completed
you can use either the xCAT rnetboot command or the rbootseq/rpower commands to initiate a network boot of the
nodes.
The command can be used to define and initialize a new NIM machine object or it can be used to reinitialize an existing
machine to use a different operating system image.
This command will also create a NIM resolv_conf resource to be used when installing the node. If a resolv_conf
resource is not already included in the xCAT osimage definition and if the “domain” and “nameservers” values are set
then a new NIM resolv_conf resource will be created and allocated to the nodes.
The “domain” and “nameservers” attributes can be set in either the xCAT “network” definition used by the nodes or in
the xCAT cluster “site” definition. The setting in the “network” definition will take priority.
304
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
The “search” field of the resolv.conf file will contain a list all the domains listed in the xCAT network definitions and
the xCAT site definition.
The “nameservers” value can either be set to a specific IP address or the “<xcatmaster>” key word. The “<xcatmaster>” key word means that the value of the “xcatmaster” attribute of the node definition will be used in the
/etc/resolv.conf file. (I.e. The name of the install server as known by the node.)
You can set the “domain” and “nameservers” attributes by using the chdef command. For example:
chdef -t network -o clstr_net domain=cluster.com nameservers=<xcatmaster>
If the “domain” and “nameservers” attributes are not set in either the nodes “network” definition or the “site” definition
then no new NIM resolv_conf resource will be created.
If you are using xCAT service nodes the mkdsklsnode command will automatically determine the correct server(s)
for the node and create the NIM definitions on that server(s).
When creating a new NIM machine definition the default is to use the same name as the xCAT node name that is
provided.
You can use the “-n” option of the mkdsklsnode command to create and initialize an alternate NIM machine definition
for the same physical nodes. This option allows you to set up a new image to use when a node is next rebooted while
the node is currently running. This is possible because the NIM name for a machine definition does not have to be the
hostname of the node. This allows you to have multiple NIM machine definitions for the same physical node. The
naming convention for the new NIM machine name is “<xcat_node_name>_<image_name>”, (Ex. “node01_61spot”).
Since all the NIM initialization can be done while the node is running the downtime for for the node is reduced to the
time it takes to reboot.
Note: When using the “-n” option make sure that the new osimage you specify and all the NIM resources that are
used are different than what are currently being used on the nodes. The NIM resources should not be shared between
the old osimage and the new osimage.
You can use the force option to reinitialize a node if it already has resources allocated or it is in the wrong NIM state.
This option will reset the NIM node and deallocate resources before reinitializing. Use this option with caution since
reinitializing a node will stop the node if it is currently running.
After the mkdsklsnode command completes you can use the lsnim command to check the NIM node definition to see
if it is ready for booting the node. (“lsnim -l <nim_node_name>”).
You can supply your own scripts to be run on the management node or on the service node (if their is hierarchy)
for a node during the mkdsklsnode command. Such scripts are called prescripts. They should be copied to /install/prescripts directory. A table called prescripts is used to specify the scripts and their associated actions. The
scripts to be run at the beginning of the mkdsklsnode command are stored in the ‘begin’ column of prescripts table. The scripts to be run at the end of the mkdsklsnode command are stored in the ‘end’ column of prescripts
table. Run ‘tabdump prescripts -d’ command for details. An example for the ‘begin’ or the ‘end’ column is: diskless:myscript1,myscript2. The following two environment variables will be passed to each script: NODES contains
all the names of the nodes that need to run the script for and ACTION contains the current nodeset action, in this case
“diskless”. If #xCAT setting:MAX_INSTANCE=number is specified in the script, the script will get invoked for each
node in parallel, but no more than number of instances will be invoked at a time. If it is not specified, the script will
be invoked once for all the nodes.
OPTIONS
attr=val [attr=val ...]
Specifies one or more “attribute equals value” pairs, separated by spaces. Attr= val pairs must be specified
last on the command line. These are used to specify additional values that can be passed to the underlying
NIM commands.
1.3. Admin Guide
305
xCAT3 Documentation, Release 2.12
Valid values:
duplex
Specifies the duplex setting (optional). Used when defining the NIM machine. Use this setting
to configure the client’s network interface. This value can be full or half. The default is full.
(ex. “duplex=full”)
speed
Specifies the speed setting (optional). Used when defining the NIM machine. This is the
communication speed to use when configuring the client’s network interface. This value can
be 10, 100, or 1000. The default is 100. (ex. “speed=100”)
psize
Specifies the size in Megabytes of the paging space for the diskless node.(optional) Used when
initializing the NIM machine. The minimum and default size is 64 MB of paging space. (ex.
“psize=256”)
sparse_paging
Specifies that the paging file should be created as an AIX sparse file,
“sparse_paging=yes”). The default is “no”.
(ex.
dump_iscsi_port
The tcpip port number to use to communicate dump images from the client to the dump resource server. Normally set by default. This port number is used by a dump resource server.
configdump
Specifies the type dump to be collected from the client. The values are “selective”, “full”, and
“none”. If the configdump attribute is set to “full” or “selective” the client will automatically
be configured to dump to an iSCSI target device. The “selective” memory dump will avoid
dumping user data. The “full” memory dump will dump all the memory of the client partition.
Selective and full memory dumps will be stored in subdirectory of the dump resource allocated
to the client. This attribute is saved in the xCAT osimage definition.
-b |--backupSN
When using backup service nodes only update the backup. The default is to update both the primary and
backup service nodes.
-f |--force
Use the force option to reinitialize the NIM machines.
-h |--help
Display usage message.
-i image_name
The name of an existing xCAT osimage definition. If this information is not provided on the command
line the code checks the node definition for the value of the “provmethod” attribute. If the “-i” value is
provided on the command line then that value will be used to set the “provmethod” attribute of the node
definitions.
-k|--skipsync
Use this option to have the mkdsklsnode command skip the NIM sync_roots operation. This option should
only be used if you are certain that the shared_root resource does not have to be updated from the SPOT.
Normally, when the SPOT is updated, you should do a sync_roots on the shared_root resource.
306
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
-l|--location
The directory location to use when creating new NIM resolv_conf resources. The default location is
/install/nim.
-n|--newname
Create a new NIM machine object name for the xCAT node.
Use the naming convention
“<xcat_node_name>_<image_name>” for the new NIM machine definition.
-p|--primarySN
When using backup service nodes only update the primary. The default is to update both the primary and
backup service nodes.
-S|--setuphanfs
Setup NFSv4 replication between the primary service nodes and backup service nodes to provide high
availability NFS for the compute nodes. This option only exports the /install directory with NFSv4 replication settings, the data synchronization between the primary service nodes and backup service nodes
needs to be taken care of through some mechanism.
-u|--updateSN
Use this option if you wish to update the osimages but do not want to define or initialize the NIM client
definitions. This option is only valid when the xCAT “site” definition attribute “sharedinstall” is set to
either “sns” or “all”.
noderange
A set of comma delimited node names and/or group names. See the “noderange” man page for details on
additional supported formats.
-V |--verbose
Verbose mode.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. Initialize an xCAT node named “node01” as an AIX diskless machine. The xCAT osimage named “61spot”
should be used to boot the node.
mkdsklsnode -i 61spot node01
2. Initialize all AIX diskless nodes contained in the xCAT node group called “aixnodes” using the image definitions
pointed to by the “provmethod” attribute of the xCAT node definitions.
mkdsklsnode aixnodes
3. Initialize diskless node “clstrn29” using the xCAT osimage called “61dskls”. Also set the paging size to be
128M and specify the paging file be an AIX sparse file.
mkdsklsnode -i 61dskls clstrn29 psize=128 sparse_paging=yes
1.3. Admin Guide
307
xCAT3 Documentation, Release 2.12
4. Initialize an xCAT node called “node02” as an AIX diskless node. Create a new NIM machine definition name
with the osimage as an extension to the xCAT node name.
mkdsklsnode -n -i 61spot node02
FILES
/opt/xcat/bin/mkdsklsnode
NOTES
This command is part of the xCAT software product.
SEE ALSO
rmdsklsnode(1)|rmdsklsnode.1
mkflexnode.1
NAME
mkflexnode - Create a flexible node.
SYNOPSIS
mkflexnode [-h | --help]
mkflexnode [-v | --version]
mkflexnode noderange
DESCRIPTION
A flexible node is a Partition in a complex. Creating a flexible node is to create a partition which including all the
slots defined in the xCAT blade node.
Before creating a flexible node, a general xCAT blade node should be defined. The id attribute of this node should be
a node range like ‘a-b’, it means the blades installed in slots ‘a-b’ need to be assigned to the partition. ‘a’ is the start
slot, ‘b’ is the end slot. If this partition only have one slot, the slot range can be ‘a’.
The action of creating flexible node will impact the hardware status. Before creating it, the blades in the slot range
should be in power off state.
After the creating, use the lsflexnode to check the status of the node.
The noderange only can be a blade node.
308
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
OPTIONS
-h | --help
Display the usage message.
-v | --version
Display the version information.
EXAMPLES
1. Create a flexible node base on the xCAT node blade1.
The blade1 should belong to a complex, the id attribute should be set correctly and all the slots should be
in power off state.
mkflexnode blade1
FILES
/opt/xcat/bin/mkflexnode
SEE ALSO
lsflexnode(1)|lsflexnode.1, rmflexnode(1)|rmflexnode.1
mkhwconn.1
NAME
mkhwconn - Sets up connections for CEC and Frame nodes to HMC nodes or hardware server.
SYNOPSIS
mkhwconn [-h| --help]
mkhwconn [-v| --version]
PPC (with HMC) specific:
mkhwconn [-V| --verbose] noderange -t [--port port_value]
mkhwconn [-V| --verbose] noderange -s [hmcnode --port port_value]
mkhwconn [-V| --verbose] noderange -p hmc [-P passwd] [--port port_value]
PPC (using Direct FSP Management) specific:
mkhwconn noderange -t [-T tooltype] [--port port_value]
1.3. Admin Guide
309
xCAT3 Documentation, Release 2.12
DESCRIPTION
For PPC (with HMC) specific:
This command is used to set up connections for CEC and Frame nodes to HMC nodes. (If the connection already
exists, it will not break it.) This command is useful when you have multiple HMCs, each of which will manage a
subset of the CECs/Frames. Use mkhwconn to tell each HMC which CECs/Frames it should manage. When using
this, you should turn off the self-discovery on each HMC. You also need to put all the HMCs and all the Frames on a
single flat service network.
When -t is specified, this command reads the connection information from the xCAT ppc table (e.g. the parent
attribute), and read the user/password from the ppcdirect table. Then this command will assign CEC nodes and Frame
nodes to HMC nodes.
When -p is specified, this command gets the connection information from command line arguments. If -P is not
specified, the default password for CEC and Frame nodes is used.
The flag -s is used to make the connection between the frame and its Service focal point(HMC). Makehwconn will
also set the connections between the CECs within this Frame and the HMC. The sfp of the frame/CEC can either be
defined in ppc table beforehand or specified in command line after the flag -s. If the user use mkhwconn noderange -s
HMC_name, it will not only make the connections but also set the sfp attributes for these nodes in PPC table.
In any case, before running this command, the CEC and Frame nodes need be defined with correct nodetype.nodetype
value (cec or frame) and nodehm.mgt value (hmc).
Note: If a CEC belongs to a frame, which has a BPA installed, this CEC should not be assigned to an HMC individually.
Instead, the whole frame should be assigned to the HMC.
For PPC (using Direct FSP Management) specific:
It is used to set up connections for CEC and Frame node to Hardware Server on management node (or service node ). It
only could be done according to the node definition in xCAT DB. And this command will try to read the user/password
from the ppcdirect table first. If fails, then read them from passwd table. Commonly , the username is HMC. If using
the ppcdirect table, each CEC/Frame and user/password should be stored in ppcdirect table. If using the passwd
table, the key should be “cec” or “frame”, and the related user/password are stored in passwd table.
When --port is specified, this command will create the connections for CECs/Frames whose side in vpd table is equal
to port value.
OPTIONS
-h|--help
Display usage message.
-t
Read connection information from xCAT DB (ppc and ppcdirect tables). Use this option if you need to
connect multiple CECs/Frames to multiple HMCs in a single command.
-p
The HMC node name. Only one HMC nodes can be specified by this flag. To setup connection for
multiple HMC nodes, use flag -t.
-P
The password of HMC based CEC/Frame login user(Default user name is ‘HMC’). This flag is optional.
-T
310
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
The tooltype is used to communicate to the CEC/Frame. The value could be lpar or fnm. The tooltype
value lpar is for xCAT and fnm is for CNM. The default value is “lpar”.
--port
The port value specifies which special side will be used to create the connection to the CEC/Frame. The
value could only be specified as “0” or “1” and the default value is “0,1”. If the user wants to use all ports
to create the connection, he should not specify this value. If the port value is specified as “0”, in the vpd
table, the side column should be A-0 and B-0; If the port value is specified as “1”, the side column should
be A-1 and B-1. When making hardware connection between CEC/Frame and HMC, the value is used to
specify the fsp/bpa port of the cec/frame and will be organized in order of “A-0,A-1,B-0,B-1”. If any side
does not exist, the side would simply be ignored. Generally, only one port of a fsp/bap can be connected
while another port be used as backup.
-s
The flag -s is used to make the connection between the frame and its Service Focal Point(HMC). -s flag is
not supposed to work with other functional flags.
-V|--verbose
Verbose output.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To setup the connection for all CEC nodes in node group cec to HMC node, according to the definition in xCAT
DB:
mkhwconn cec -t
2. To setup the connection for Frame nodes in node group frame to HMC node hmc1, with password ‘abc123’:
mkhwconn frame -p hmc1 -P abc123
3. To setup the connections for all CEC nodes in node group cec to hardware server, and the tooltype value is lpar:
mkhwconn cec -t -T lpar
4. To setup the connections for all cecs nodes in node group cec to hardware server, and the tooltype value is lpar,
and the port value is 1:
mkhwconn cec -t -T lpar --port 1
5. To setup the connection between the frame and it’s SFP node. This command will also set the connections
between the CECs within this frame and their SFP node. User need to define HMC_name in the database in
advance, but no need to set the sfp attribute for these node, xCAT will set the HMC_name as ppc.sfp for these
nodes. The CECs within this frame should have the same sfp attribute as the frame.
mkhwconn cec -s HMC_name -P HMC_passwd
1.3. Admin Guide
311
xCAT3 Documentation, Release 2.12
FILES
$XCATROOT/bin/mkhwconn
(The XCATROOT environment variable is set when xCAT is installed. The default value is “/opt/xcat”.)
NOTES
This command is part of the xCAT software product.
SEE ALSO
lshwconn(1)|lshwconn.1, rmhwconn(1)|rmhwconn.1
mknimimage.1
NAME
mknimimage - Use this xCAT command to create xCAT osimage definitions and related AIX/NIM resources. The
command can also be used to update an existing AIX diskless image(SPOT).
SYNOPSIS
mknimimage [-h | --help ]
mknimimage [-V] -u osimage_name [attr=val [attr=val ...]]
mknimimage [-V] [-f|--force] [-r|--sharedroot] [-D|--mkdumpres] [-l location] [-c | --completeosimage] [-s image_source] [-i current_image] [-p | --cplpp] [-t nimtype] [-m nimmethod] [-n mksysbnode] [-b mksysbfile] osimage_name [attr=val [attr=val ...]]
DESCRIPTION
This command will create both an xCAT osimage definition and the corresponding NIM resource definitions. The
command can also be used to update an existing AIX diskless image(SPOT).
The command will also install the NIM master software and configure NIM if needed.
The naming convention for the NIM SPOT resource definition is to use the same name as the xCAT osimage. The
naming convention for any other NIM resources that are created is “<osimage_name>_<resource_type>”. (ex. “61image_lpp_source” )
When creating a mksysb image definition you must specify either the “-n” or the “-b” option. The “-n” option can
be used to create a mksysb image from an existing NIM client machine. The “-b” option can be used to specify an
existing mksysb backup file.
Adding software and configuration files to the osimage.
When creating a diskless osimage definition you also have the option of automatically updating the NIM SPOT resource. You can have additional software installed or you can have configuration files added or updated. To have
312
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
software installed you must provide either the names of NIM installp_bundle resources or fileset names on the command line using the “attr=val” option. You may also supply the installp flags, RPM flags, emgr flags to use when
installing the software.
To have configuration files updated you must provide the full path name of a “synclists” file which contains the list
of actual files to update. The xCAT osimage definition that is created will contain the installp_bundle, otherpkgs, and
synclists files that are provided on the command line.
Updating an existing xCAT osimage
If you wish to update an existing diskless image after it has already been created you can use the “-u” (update) option.
In this case the xCAT osimage definition will not be updated.
There are two ways to use the update feature.
You can update the osimage definition and run the mknimimage command with no “installp_bundle”, “otherpkgs”,
or “synclists” command line values. The information for updating the SPOT will come from the osimage definition
only. This has the advantage of keeping a record of any changes that were made to the SPOT.
Or, you could do a more ad hoc update by providing one or more of the “installp_bundle”, “otherpkgs”, or “synclists”
values on the command line. If any of these values are provided the mknimimage command will use those values
only. The osimage definition will not be used or updated.
WARNING: Installing random RPM packages in a SPOT may have unpredictable consequences. The SPOT is a very
restricted environment and some RPM packages may corrupt the SPOT or even hang your management system. Try
to be very careful about the packages you install. When installing RPMs, if the mknimimage command hangs or if
there are file systems left mounted after the command completes you may need to reboot your management node to
recover. This is a limitation of the current AIX support for diskless systems
Copying an xCAT osimage.
You can use the “-i” and “-p” options to copy an existing diskless osimage. To do this you must supply the name of an
existing xCAT osimage definition and the name of the new osimage you wish to create. The mknimimage command
will do the following:
• create a new xCAT osimage definition using the new name that was specified.
• copy the NIM SPOT resource to a new location and define it to NIM using a new name.
• if the original osimage included a NIM “shared_root” resource then a new shared_root resource will be created
for the new SPOT.
• any other resources (or attributes) included in the original osimage will be included in the new osimage definition.
• if the “-p” option is specified then the original NIM lpp_source resource will be copied to a new location and
redefined to NIM. (The default would be to use the original lpp_source - to save file system space.)
Additional information
IMPORTANT: The NIM lpp_source and SPOT resources can get quite large. Always make sure that you have sufficient file system space available before running the mknimimage command.
To list the contents of the xCAT osimage definition use the xCAT lsdef command (“lsdef -t osimage -l -o <osimage_name>”).
To check the validity of a SPOT or lpp_source resource
To remove an xCAT osimage definition along with the associated NIM resource definitions use the rmnimimage
command. Be careful not to accidently remove NIM resources if they are still needed.
To list a NIM resource definition use the AIX lsnim command (“lsnim -l <resource_name>”).
To check the validity of a SPOT or lpp_source resource use the AIX nim command (“nim -o check <resourec-name>”).
1.3. Admin Guide
313
xCAT3 Documentation, Release 2.12
To remove specific NIM resource definitions use the AIX nim command. (“nim -o remove <resource-name>”).
OPTIONS
attr=val [attr=val ...]
Specifies one or more “attribute equals value” pairs, separated by spaces. Attr=val pairs must be specified
last on the command line.
Currently supported attributes:
bosinst_data
The name of a NIM bosinst_data resource.
dump
The name of the NIM dump resource.
fb_script
The name of a NIM fb_script resource.
home
The name of the NIM home resource.
installp_bundle
One or more comma separated NIM installp_bundle resources.
lpp_source
The name of the NIM lpp_source resource.
mksysb
The name of a NIM mksysb resource.
otherpkgs
One or more comma separated installp, emgr, or rpm packages. The packages must have
prefixes of ‘I:’, ‘E:’, or ‘R:’, respectively. (ex. R:foo.rpm)
paging
The name of the NIM paging resource.
resolv_conf
The name of the NIM resolv_conf resource.
root
The name of the NIM root resource.
script
The name of a NIM script resource.
shared_home
The name of the NIM shared_home resource.
shared_root
A shared_root resource represents a directory that can be used as a / (root) directory by one or
more diskless clients.
314
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
spot
The name of the NIM SPOT resource.
synclists
The fully qualified name of a file containing a list of files to synchronize on the nodes.
tmp
The name of the NIM tmp resource.
installp_flags
The alternate flags to be passed along to the AIX installp command. (The default for installp_flags is “-abgQXY”.)
rpm_flags
The alternate flags to be passed along to the AIX rpm command. (The default for rpm_flags is
“-Uvh ”.) The mknimimage command will check each rpm to see if it is installed. It will not
be reinstalled unless you specify the appropriate rpm option, such as ‘–replacepkgs’.
emgr_flags
The alternate flags to be passed along to the AIX emgr command. (There is no default flags
for the emgr command.)
dumpsize
The maximum size for a single dump image the dump resource will accept. Space is not
allocated until a client starts to dump. The default size is 50GB. The dump resource should be
large enough to hold the expected AIX dump and snap data.
max_dumps
The maximum number of archived dumps for an individual client. The default is one.
snapcollect
Indicates that after a dump is collected then snap data should be collected. The snap data will
be collected in the clients dump resource directory. Values are “yes” or “no”. The default is
“no”.
nfs_vers
Value Specifies the NFS protocol version required for NFS access.
nfs_sec
Value Specifies the security method required for NFS access.
Note that you may specify multiple “script”, “otherpkgs”, and “installp_bundle” resources by using a
comma separated list. (ex. “script=ascript,bscript”). RPM names may be included in the “otherpkgs”
list by using a “R:” prefix(ex. “R:whatever.rpm”). epkg (AIX interim fix package) file names may be
included in the “otherpkgs” using the ‘E:’ prefix. (ex. “otherpkgs=E:IZ38930TL0.120304.epkg.Z”).
-b mksysbfile
Used to specify the path name of a mksysb file to use when defining a NIM mksysb resource.
-c|--completeosimage
Complete the creation of the osimage definition passed in on the command line. This option will use any
additional values passed in on the command line and/or it will attempt to create required resources in order
to complete the definition of the xCAT osimage. For example, if the osimage definition is missing a spot
or shared_root resource the command will create those resources and add them to the osimage definition.
1.3. Admin Guide
315
xCAT3 Documentation, Release 2.12
-f|--force
Use the force option to re-create xCAT osimage definition. This option removes the old definition before
creating the new one. It does not remove any of the NIM resource definitions named in the osimage definition. Use the rmnimimage command to remove the NIM resources associated with an xCAT osimage
definition.
-h |--help
Display usage message.
osimage_name
The name of the xCAT osimage definition. This will be used as the name of the xCAT osimage definition
as well as the name of the NIM SPOT resource.
-D|--mkdumpres
Create a diskless dump resource.
-i current_image
The name of an existing xCAT osimage that should be copied to make a new xCAT osimage definition.
Only valid when defining a “diskless” or “dataless” type image.
-l location
The directory location to use when creating new NIM resources. The default location is /install/nim.
-m nimmethod
Used to specify the NIM installation method to use. The possible values are “rte” and “mksysb”. The
default is “rte”.
-n mksysbnode
The xCAT node to use to create a mksysb image. The node must be a defined as a NIM client machine.
-p|--cplpp
Use this option when copying existing diskless osimages to indicate that you also wish to have the
lpp_resource copied. This option is only valid when using the “-i” option.
-r|--sharedroot
Use this option to specify that a NIM “shared_root” resource be created for the AIX diskless nodes. The
default is to create a NIM “root” resource. This feature is only available when using AIX version 6.1.4 or
beyond. See the AIX/NIM documentation for a description of the “root” and “shared_root” resources.
-s image_source
The source of software to use when creating the new NIM lpp_source resource. This could be a source
directory or a previously defined NIM lpp_source resource name.
-t nimtype
Used to specify the NIM machine type. The possible values are “standalone”, “diskless” or “dataless”.
The default is “standalone”.
-u
Used to update an AIX/NIM SPOT resource with additional software and configuration files. This option
is only valid for xCAT diskless osimage objects. The SPOT resource associated with the xCAT osimage
definition will be updated. This option can also be used to update the nfs_vers attribute from NFSv3 to
NFSv4 for the NIM resources associated with diskful or diskless image.
-V |--verbose
316
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
Verbose mode.
RETURN VALUE
0. The command completed successfully.
1. An error has occurred.
EXAMPLES
1. Create an osimage definition and the basic NIM resources needed to do a NIM “standalone” “rte” installation of node “node01”. Assume the software contained on the AIX product media has been copied to the
/AIX/instimages directory.
mknimimage -s /AIX/instimages
61image
2. Create an osimage definition that includes some additional NIM resources.
mknimimage -s /AIX/instimages 61image installp_bundle=mybndlres,addswbnd
This command will create lpp_source, spot, and bosinst_data resources using the source specified by the “-s” option. The installp_bundle information will also be included in the osimage definition. The mybndlres and addswbnd
resources must be created before using this osimage definition to install a node.
3. Create an osimage definition that includes a mksysb image and related resources.
mknimimage -m mksysb -n node27 newsysb spot=myspot bosinst_data=mybdata
This command will use node27 to create a mksysb backup image and use that to define a NIM mksysb resource. The
osimage definition will contain the name of the mksysb resource as well as the spot and bosinst_data resource.
4. Create an osimage definition using a mksysb image provided on the command line.
mknimimage -m mksysb -b /tmp/backups/mysysbimage newsysb spot=myspot bosinst_
˓→data=mybdata
This command defines a NIM mksysb resource using mysysbimage.
5. Create an osimage definition and create the required spot definition using the mksysb backup file provided on
the command line.
mknimimage -m mksysb -b /tmp/backups/mysysbimage newsysb bosinst_data=mybdata
This command defines a NIM mksysb resource and a spot definition using mysysbimage.
6. Create a diskless image called 61dskls using the AIX source files provided in the /AIX/instimages directory.
mknimimage -t diskless -s /AIX/instimages 61dskls
7. Create a diskless image called “614dskls” that includes a NIM “shared_root” and a “dump” resource. Use the
existing NIM lpp_resource called “614_lpp_source”. Also specify verbose output.
mknimimage -V -r -D -t diskless -s 614_lpp_source 614dskls snapcollect=yes
The “snapcollect” attribute specifies that AIX “snap” data should be include when a system dump is initiated.
8. Create a new diskless image by copying an existing image.
1.3. Admin Guide
317
xCAT3 Documentation, Release 2.12
mknimimage -t diskless -i 61cosi 61cosi_updt1
Note: If you also wish to have the original lpp_source copied and defined use the -p option.
mknimimage -t diskless -i 61cosi -p 61cosi_updt1
9. Create a diskless image using an existing lpp_source resource named “61cosi_lpp_source” and include NIM
tmp and home resources. This assumes that the “mytmp” and “myhome” NIM resources have already been
created by using NIM commands.
mknimimage -t diskless -s 61cosi_lpp_source 611cosi tmp=mytmp home=myhome
10. Create a diskless image and update it with additional software using rpm flags and configuration files.
mknimimage -t diskless -s 61cosi_lpp_source 61dskls otherpkgs=I:fset1,R:foo.rpm,
˓→E:IZ38930TL0.120304.epkg.Z synclists=/install/mysyncfile rpm_flags="-i --nodeps"
The xCAT osimage definition created by this command will include the “otherpkgs” and “synclists” values. The NIM
SPOT resource associated with this osimage will be updated with the additional software using rpm flags “-i –nodeps”
and configuration files.
11. Update an existing diskless image (AIX/NIM SPOT) using the information saved in the xCAT “61dskls” osimage definition. Also specify verbose messages.
mknimimage -V -u 61dskls
12. Update an existing diskless image called “61dskls”. Install the additional software specified in the NIM “bndres1” and “bndres2” installp_bundle resources using the installp flags “-agcQX”. (The NIM “bndres1” and
“bndres2” definitions must be created before using them in this command.)
mknimimage -u 61dskls installp_bundle=bndres1,bndres2 installp_flags="-agcQX"
Note that when “installp_bundle”, “otherpkgs”, or “synclists” values are specified with the “-u” option then the xCAT
osimage definition is not used or updated.
13. Update an existing image to support NFSv4. Also specify verbose messages.
mknimimage -V -u 61dskls nfs_vers=4
FILES
/opt/xcat/bin/mknimimage
NOTES
This command is part of the xCAT software product.
SEE ALSO
rmnimimage(1)|rmnimimage.1
318
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
mkvlan.1
NAME
mkvlan - It takes a list of nodes and create a private tagged vlan for them.
SYNOPSIS
mkvlan [vlanid] -n | --nodes noderange [-t | --net subnet] [-m | --mask netmask] [-p | --prefix hostname_prefix] [-i |
--interface nic]
mkvlan [-h | --help]
mkvlan [-v | --version]
DESCRIPTION
The mkvlan command takes a list of nodes and move them to a private vlan.
This command will configure the switch to create a new tagged vlan on the given nic. The primary nic will be used if
the nic is not specified. The new vlan ID is given by the command. However, if it is omitted, xCAT will automatically
generate the new vlan ID by querying all the switches involved and finding out the smallest common number that is
not used by any existing vlans. The subnet and the netmask for the vlan will be derived from the value of “vlannets”
and “vlanmasks” from the site table if -t and -m are not specified. The following are the default site table entires:
vlannets="|(\d+)|10.($1+0).0.0|";
vlanmask="255.255.0.0";
The vlan network will be entered in the networks table. The nodes will be added to the vlan using the vlan tagging
technique. And the new IP addresses and new hostnames will be assigned to the nodes. The -p flag specifies the node
hostname prefix for the nodes. If it is not specified, by default, the hostnames for the nodes are having the following
format:
v<vlanid>nY where Y is the node number. For example, the hostname for node 5 on vlan 10 is v10n5.
The switch.vlan will be updated with the new vlan id for the node for standaline nodes. For KVM guests, the vm.nics
identifies which vlan this node belongs to. For example: vl3 means this node is in vlan 3.
If there are more than one switches involved in the vlan, the ports that connect to the switches need to entered in
switches.linkports with the following format:
<port numner>:switch,<port number>:switch....
For example:
"42:switch1,43:switch2"
This command will automatically configure the cross-over ports if the given nodes are on different switches.
For added security, the root guard and bpdu guard will be enabled for the ports in this vlan. However, the guards will
not be disabled if the ports are removed from the vlan using chvlan or rmvlan commands. To disable them, you need
to use the switch command line interface. Refer to the switch command line interface manual to see how to disable
the root guard and bpdu guard for a port.
1.3. Admin Guide
319
xCAT3 Documentation, Release 2.12
PARAMETERS
vlanid is a unique vlan number. If it is omitted, xCAT will automatically generate the new vlan ID by querying all the
switches involved and finding out the smallest common number that is not used by any existing vlans. Use lsvlan to
find out the existing vlan ids used by xCAT.
OPTIONS
-n|--nodes The nodes or groups to be included in the vlan. It can be stand alone nodes or KVM guests. It takes the
noderange format. Check the man page for noderange for details.
-t|--net The subnet for the vlan.
-m|--mask The netmask for the vlan
-p|--prefix The prefix the the new hostnames for the nodes in the vlan.
-i|--interface The interface name where the vlan will be tagged on. If omitted, the xCAT management network will be
assumed. For FVM, this is the interface name on the host.
-h|--help Display usage message.
-v|--version The Command Version.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
To start, the xCAT switches and switches table needs to be filled with switch and port info for the nodes. For example,
the swith table will look like this:
#node,switch,port,vlan,interface,comments,disable
“node1”,”switch1”,”10”„„
“node1”,”switch2”,”1”„”eth1”„
“node2”,”switch1”,”11”„”primary”„
“node2”,”switch2”,”2”„”eth1”„
“node3”,”switch1”,”12”„”primary:eth0”„
“node3”,”switch2”,”3”„”eth1”„
Note that the interface value for the management (primary) network can be empty, the word “primary” or “primary:ethx”. For other networks, the interface attribute must be specified.
The following is an example of the switches table
#switch,snmpversion,username,password,privacy,auth,linkports,sshusername,sshpassword,switchtype,comments,disable
“switch1”,”3”,”username”,”passw0rd”„”sha”,”48:switch2”„„, “switch2”,”2”„„,”43:switch1”„„,
1. To make a private vlan for node1, node2 and node3
mkvlan -n node1,node2,node3
The vlan will be created on eth0 for the nodes.
2. To make a private vlan for node1, node2 and node3 on eth1,
mkvlan -n node1,node2,node3 -i eth1
320
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
3. To make a private vlan for node1, node2 with given subnet and netmask.
mkvlan -n node1,node2,node3 -t 10.3.2.0 -m 255.255.255.0
4. To make a private vlan for KVM guests node1 and node2
chtab key=usexhrm site.vlaue=1
mkdef node1 arch=x86_64 groups=kvm,all installnic=mac primarynic=mac mgt=kvm
˓→netboot=pxe nfsserver=10.1.0.204 os=rhels6 profile=compute provmethod=install
˓→serialport=0 serialspeed=115200 vmcpus=1 vmhost=x3650n01 vmmemory=512
˓→vmnics=br0 vmstorage=nfs://10.1.0.203/vms
mkdef node2 arch=x86_64 groups=kvm,all installnic=mac primarynic=mac mgt=kvm
˓→netboot=pxe nfsserver=10.1.0.204 os=rhels6 profile=compute provmethod=install
˓→serialport=0 serialspeed=115200 vmcpus=1 vmhost=x3650n01 vmmemory=512
˓→vmnics=br0 vmstorage=nfs://10.1.0.203/vms
mkvlan -n node1,node2
mkvm node1,node2 -s 20G
rpower node1,node2 on
rinstall node1,node2
FILES
/opt/xcat/bin/mkvlan
SEE ALSO
chvlan(1)|chvlan.1, rmvlan(1)|rmvlan.1, lsvlan(1)|lsvlan.1
mkvm.1
NAME
mkvm - Creates HMC-, DFM-, IVM-, and zVM-managed partitions or other virtual machines.
SYNOPSIS
Common:
mkvm [-h| --help]
mkvm [-v| --version]
1.3. Admin Guide
321
xCAT3 Documentation, Release 2.12
For PPC (with HMC) specific:
mkvm [-V| --verbose] noderange -i id -l singlenode
mkvm [-V| --verbose] noderange -c destcec -p profile
mkvm [-V| --verbose] noderange --full
For PPC (using Direct FSP Management) specific:
mkvm noderange [--full]
mkvm noderange [vmcpus= min/req/max] [vmmemory= min/req/max] [vmphyslots= drc_index1,drc_index2...]
[vmothersetting= hugepage:N,bsr:N] [vmnics= vlan1[,vlan2..]] [vmstorage= N|viosnode:slotid] [--vios]
For KVM:
mkvm noderange [-s|--size disksize] [--mem memsize] [--cpus cpucount] [-f|--force]
For VMware:
mkvm noderange [-s | --size disksize] [--mem memsize] [--cpus cpucount]
For zVM:
mkvm noderange [directory_entry_file_path]
mkvm noderange [source_virtual_machine] [pool= disk_pool]
DESCRIPTION
For PPC (with HMC) specific:
The first form of mkvm command creates new partition(s) with the same profile/resources as the partition specified
by singlenode. The -i and noderange specify the starting numeric partition number and the noderange for the newly
created partitions, respectively. The LHEA port numbers and the HCA index numbers will be automatically increased
if they are defined in the source partition.
The second form of this command duplicates all the partitions from the source specified by profile to the destination
specified by destcec. The source and destination CECs can be managed by different HMCs.
Make sure the nodes in the noderange is defined in the nodelist table and the mgt is set to ‘hmc’ in the nodehm table
before running this command.
Note that the mkvm command currently only supports creating standard LPARs, not virtual LPARs working with
VIOS server.
322
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
For PPC (using Direct FSP Management) specific:
With option full, a partition using all the resources on a normal power machine will be created.
If no option is specified, a partition using the parameters specified with attributes such as ‘vmcpus’, ‘vmmory’, ‘vmphyslots’, ‘vmothersetting’, ‘vmnics’, ‘vmstorage’ will be created. Those attributes can either be specified with ‘*def’
commands running before or be specified with this command.
For KVM and VMware:
The mkvm command creates new virtual machine(s) with the disksize size of hard disk, memsize size of memory and
cpucount number of cpu.
For zVM:
The first form of mkvm creates a new virtual machine based on a directory entry.
The second form of this creates a new virtual machine with the same profile/resources as the specified node (cloning).
OPTIONS
-h|--help
Display usage message.
-c
The cec (fsp) name for the destination.
--cpus
The cpu count which will be created for the kvm/vmware virtual machine.
--full
Request to create a new full system partition for each CEC.
vmcpus= value vmmemory= value vmphyslots= value vmothersetting= value vmnics= value vmstorage= value
[--vios]
To specify the parameters which are used to create a partition. The vmcpus, vmmemory are necessary, and
the value specified with this command have a more high priority. If the value of any of the three options
is not specified, the corresponding value specified for the node object will be used. If any of the three
attributes is neither specified with this command nor specified with the node object, error information will
be returned. To reference to lsvm(1)|lsvm.1 for more information about ‘drc_index’ for vmphyslots.
The option vios is used to specify the partition that will be created is a VIOS partition. If specified, the
value for vmstorage shall be number which indicate the number of vSCSI server adapter will be created,
and if no value specified for vmphyslots, all the physical slot of the power machine will be asigned to
VIOS partition. If not specified, it shall be in form of vios_name:server_slotid to specify the vios and the
virtual slot id of the vSCSI server adapter that will be connected from the Logical partition.
-f|--force
If the storage already exists, remove it before creating a new virtual machine.
-i
Starting numeric id of the newly created partitions.
1.3. Admin Guide
323
xCAT3 Documentation, Release 2.12
-l
The partition name of the source.
--mem
The memory size which will be used for the new created kvm/vmware virtual machine. Unit is Megabyte.
-p
The file that contains the profiles for the source partitions.
-s|--size
The size of storage which will be created for the kvm/vmware virtual machine.
-v|--version
Command Version.
-V|--verbose
Verbose output.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To create a new HMC-managed partition lpar5 based on the profile/resources of lpar4, enter:
mkdef -t node -o lpar5 mgt=hmc groups=all
then:
mkvm lpar5 -i 5 -l lpar4
Output is similar to:
lpar5: Success
2. To create new HMC-managed partitions lpar5-lpar8 based on the profile/resources of lpar4, enter:
mkdef -t node -o lpar5-lpar8 mgt=hmc groups=all
then:
mkvm lpar5-lpar8 -i 5 -l lpar4
Output is similar to:
lpar5:
lpar6:
lpar7:
lpar8:
324
Success
Success
Success
Success
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
3. To duplicate all the HMC-managed partitions associated with cec01 on cec02, first save the lpars from cec01 to
a file:
lsvm lpar01-lpar04 > /tmp/myprofile
then create lpars on cec02:
mkvm lpar05-lpar08 -c cec02 -p /tmp/myprofile
Output is similar to:
lpar5:
lpar6:
lpar7:
lpar8:
Success
Success
Success
Success
4. To duplicate all the HMC-managed partitions associated with cec01 on cec02, one is for cec01, the other is for
cec02:
mkdef -t node -o lpar5,lpar6 mgt=hmc groups=all
chtab node=lpar5 ppc.parent=cec01
chtab node=lpar6 ppc.parent=cec02
then create lpars on cec01 and cec02:
mkvm lpar5,lpar6 --full
Output is similar to:
lpar5: Success
lpar6: Success
5. To create a new zVM virtual machine (gpok3) based on a directory entry:
mkvm gpok3 /tmp/dirEntry.txt
Output is similar to:
gpok3: Creating user directory entry for LNX3... Done
6. To clone a new zVM virtual machine with the same profile/resources as the specified node:
mkvm gpok4 gpok3 pool=POOL1
Output is similar to:
gpok4:
gpok4:
gpok4:
gpok4:
gpok4:
gpok4:
gpok4:
gpok4:
gpok4:
gpok4:
gpok4:
gpok4:
Cloning gpok3
Linking source disk (0100) as (1100)
Linking source disk (0101) as (1101)
Stopping LNX3... Done
Creating user directory entry
Granting VSwitch (VSW1) access for gpok3
Granting VSwitch (VSW2) access for gpok3
Adding minidisk (0100)
Adding minidisk (0101)
Disks added (2). Disks in user entry (2)
Linking target disk (0100) as (2100)
Copying source disk (1100) to target disk (2100) using FLASHCOPY
1.3. Admin Guide
325
xCAT3 Documentation, Release 2.12
gpok4:
gpok4:
gpok4:
gpok4:
gpok4:
gpok4:
gpok4:
gpok4:
Mounting /dev/dasdg1 to /mnt/LNX3
Setting network configuration
Linking target disk (0101) as (2101)
Copying source disk (1101) to target disk (2101) using FLASHCOPY
Powering on
Detatching source disk (0101) at (1101)
Detatching source disk (0100) at (1100)
Starting LNX3... Done
7. To create a new kvm/vmware virtual machine with 10G storage, 2048M memory and 2 cpus.
mkvm vm1 -s 10G --mem 2048 --cpus 2
8. To create a full partition on normal power machine.
First, define a node object:
mkdef -t node -o lpar1 mgt=fsp cons=fsp nodetype=ppc,osi id=1 hcp=cec parent=cec
˓→hwtype=lpar groups=lpar,all
Then, create the partition on the specified cec.
mkvm lpar1 --full
The output is similar to:
lpar1: Done
To query the resources allocated to node ‘lpar1’
lsvm lpar1
The output is similar to:
lpar1: Lpar Processor Info:
Curr Processor Min: 1.
Curr Processor Req: 16.
Curr Processor Max: 16.
lpar1: Lpar Memory Info:
Curr Memory Min: 0.25 GB(1 regions).
Curr Memory Req: 30.75 GB(123 regions).
Curr Memory Max: 32.00 GB(128 regions).
lpar1: 1,519,U78AA.001.WZSGVU7-P1-C7,0x21010207,0xffff(Empty Slot)
lpar1: 1,518,U78AA.001.WZSGVU7-P1-C6,0x21010206,0xffff(Empty Slot)
lpar1: 1,517,U78AA.001.WZSGVU7-P1-C5,0x21010205,0xffff(Empty Slot)
lpar1: 1,516,U78AA.001.WZSGVU7-P1-C4,0x21010204,0xffff(Empty Slot)
lpar1: 1,514,U78AA.001.WZSGVU7-P1-C19,0x21010202,0xffff(Empty Slot)
lpar1: 1,513,U78AA.001.WZSGVU7-P1-T7,0x21010201,0xc03(USB Controller)
lpar1: 1,512,U78AA.001.WZSGVU7-P1-T9,0x21010200,0x104(RAID Controller)
lpar1: 1/2/2
lpar1: 256.
Note: The ‘parent’ attribute for node ‘lpar1’ is the object name of physical power machine that the full partition will
be created on.
9. To create a partition using some of the resources on normal power machine.
Option 1:
326
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
After a node object is defined, the resources that will be used for the partition shall be specified like this:
chdef lpar1 vmcpus=1/4/16 vmmemory=1G/4G/32G vmphyslots=0x21010201,0x21010200
˓→vmothersetting=bsr:128,hugepage:2
Then, create the partion on the specified cec.
mkvm lpar1
Option 2:
mkvm lpar1 vmcpus=1/4/16 vmmemory=1G/4G/32G vmphyslots=0x21010201,0x21010200
˓→vmothersetting=bsr:128,hugepage:2
The outout is similar to:
lpar1: Done
Note: The ‘vmplyslots’ specify the drc index of the physical slot device. Every drc index shall be delimited with ‘,’.
The ‘vmothersetting’ specify two kinds of resource, bsr(Barrier Synchronization Register) specified the num of BSR
arrays, hugepage(Huge Page Memory) specified the num of huge pages.
To query the resources allocated to node ‘lpar1’
lsvm lpar1
The output is similar to:
lpar1: Lpar Processor Info:
Curr Processor Min: 1.
Curr Processor Req: 4.
Curr Processor Max: 16.
lpar1: Lpar Memory Info:
Curr Memory Min: 1.00 GB(4 regions).
Curr Memory Req: 4.00 GB(16 regions).
Curr Memory Max: 32.00 GB(128 regions).
lpar1: 1,513,U78AA.001.WZSGVU7-P1-T7,0x21010201,0xc03(USB Controller)
lpar1: 1,512,U78AA.001.WZSGVU7-P1-T9,0x21010200,0x104(RAID Controller)
lpar1: 1/2/2
lpar1: 128.
10. To create a vios partition using some of the resources on normal power machine.
mkvm viosnode vmcpus=1/4/16 vmmemory=1G/4G/32G vmphyslots=0x21010201,0x21010200
˓→vmnics=vlan1 vmstorage=5 --vios
The resources for the node is similar to:
viosnode: Lpar Processor Info:
Curr Processor Min: 1.
Curr Processor Req: 4.
Curr Processor Max: 16.
viosnode: Lpar Memory Info:
Curr Memory Min: 1.00 GB(4 regions).
Curr Memory Req: 4.00 GB(16 regions).
Curr Memory Max: 32.00 GB(128 regions).
viosnode: 1,513,U78AA.001.WZSGVU7-P1-T7,0x21010201,0xc03(USB Controller)
viosnode: 1,512,U78AA.001.WZSGVU7-P1-T9,0x21010200,0x104(RAID Controller)
1.3. Admin Guide
327
xCAT3 Documentation, Release 2.12
viosnode: 1,0,U8205.E6B.0612BAR-V1-C,0x30000000,vSerial Server
viosnode: 1,1,U8205.E6B.0612BAR-V1-C1,0x30000001,vSerial Server
viosnode: 1,3,U8205.E6B.0612BAR-V1-C3,0x30000003,vEth (port_vlanid=1,mac_
˓→addr=4211509276a7)
viosnode: 1,5,U8205.E6B.0612BAR-V1-C5,0x30000005,vSCSI Server
viosnode: 1,6,U8205.E6B.0612BAR-V1-C6,0x30000006,vSCSI Server
viosnode: 1,7,U8205.E6B.0612BAR-V1-C7,0x30000007,vSCSI Server
viosnode: 1,8,U8205.E6B.0612BAR-V1-C8,0x30000008,vSCSI Server
viosnode: 1,9,U8205.E6B.0612BAR-V1-C9,0x30000009,vSCSI Server
viosnode: 0/0/0
viosnode: 0.
FILES
/opt/xcat/bin/mkvm
SEE ALSO
chvm(1)|chvm.1, lsvm(1)|lsvm.1, rmvm(1)|rmvm.1
mkzone.1
NAME
mkzone - Defines a new zone in the cluster.
SYNOPSIS
mkzone zonename [--defaultzone] [-k full path to the ssh RSA private key] [-a noderange] [-g] [-f] [-s {yes|no}] [-V]
mkzone [-h | -v]
DESCRIPTION
The mkzone command is designed to divide the xCAT cluster into multiple zones. The nodes in each zone will share
common root ssh keys. This allows the nodes in a zone to be able to as root ssh to each other without password, but
cannot do the same to any node in another zone. All zones share a common xCAT Management Node and database
including the site table, which defines the attributes of the entire cluster. The mkzone command is only supported on
Linux ( No AIX support). The nodes are not updated with the new root ssh keys by mkzone. You must run updatenode
-k or xdsh -K to the nodes to update the root ssh keys to the new generated zone keys. This will also sync any service
nodes with the zone keys, if you have a hierarchical cluster. Note: if any zones in the zone table, there must be one
and only one defaultzone. Otherwise, errors will occur.
OPTIONS
-h | --help
Displays usage information.
328
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
-v | --version
Displays command version and build date.
-k | --sshkeypath full path to the ssh RSA private key
This is the path to the id_rsa key that will be used to build root’s ssh keys for the zone. If -k
is used, it will generate the ssh public key from the input ssh RSA private key and store both in
/etc/xcat/sshkeys/<zonename>/.ssh directory. If -f is not used, then it will generate a set of root ssh
keys for the zone and store them in /etc/xcat/sshkeys/<zonename>/.ssh.
--default
if –defaultzone is input, then it will set the zone defaultzone attribute to yes; otherwise it will set to no.
if –defaultzone is input and another zone is currently the default, then the -f flag must be used to force a
change to the new defaultzone. If -f flag is not use an error will be returned and no change made. Note: if
any zones in the zone table, there must be one and only one defaultzone. Otherwise, errors will occur.
-a | --addnoderange noderange
For each node in the noderange, it will set the zonename attribute for that node to the input zonename.
If the -g flag is also on the command, then it will add the group name “zonename” to each node in the
noderange.
-s| --sshbetweennodes yes|no
If -s entered, the zone sshbetweennodes attribute will be set to yes or no. It defaults to yes. When this is
set to yes, then ssh will be setup to allow passwordless root access between nodes. If no, then root will be
prompted for a password when running ssh between the nodes in the zone.
-f | --force
Used with the (–defaultzone) flag to override the current default zone.
-g | --assigngroup
Used with the (-a) flag to create the group zonename for all nodes in the input noderange.
-V | --Verbose
Verbose mode.
EXAMPLES
1. To make a new zone1 using defaults, enter:
mkzone zone1
Note: with the first mkzone, you will automatically get the xcatdefault zone created as the default zone.
This zone uses ssh keys from <roothome>/.ssh directory.
2. To make a new zone2 using defaults and make it the default zone enter:
mkzone> zone2 --defaultzone -f
3. To make a new zone2A using the ssh id_rsa private key in /root/.ssh:
mkzone zone2A -k /root/.ssh
4. To make a new zone3 and assign the noderange compute3 to the zone enter:
1.3. Admin Guide
329
xCAT3 Documentation, Release 2.12
mkzone zone3 -a compute3
5. To make a new zone4 and assign the noderange compute4 to the zone and add zone4 as a group to each node
enter:
mkzone zone4 -a compute4 -g
6. To make a new zone5 and assign the noderange compute5 to the zone and add zone5 as a group to each node
but not allow passwordless ssh between the nodes enter:
mkzone zone5 -a compute5 -g -s no
Files
/opt/xcat/bin/mkzone/
Location of the mkzone command.
SEE ALSO
chzone(1)|chzone.1, rmzone(1)|rmzone.1, xdsh(1)|xdsh.1, updatenode(1)|updatenode.1
monadd.1
NAME
monadd - Registers a monitoring plug-in to the xCAT cluster.
SYNOPSIS
monadd [-h| --help]
monadd [-v| --version]
monadd name [-n|--nodestatmon] [-s|--settings settings]
DESCRIPTION
This command is used to register a monitoring plug-in module to monitor the xCAT cluster. The plug-in module will
be added to the xCAT monitoring database table and the configuration scripts for the monitoring plug-in, if any, will
be added to the postscripts table. A monitoring plug-in module acts as a bridge that connects a 3rd party monitoring
software and the xCAT cluster. A configuration script is used to configure the 3rd party software. Once added to the
<postscripts> table, it will be invoked on the nodes during node deployment stage.
Parameters
name is the name of the monitoring plug-in module. For example, if the name is called xxx, then the actual file name
that the xcatd looks for is /opt/xcat/lib/perl/xCAT_monitoring/xxx.pm. Use monls -a command to list all the monitoring
plug-in modules that can be used.
330
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
settings is the monitoring plug-in specific settings. It is used to customize the behavior of the plug-in or configure the
3rd party software. Format: -s key-value -s key=value ... Note that the square brackets are needed here. Use monls
name -d command to look for the possible setting keys for a plug-in module.
OPTIONS
-h | --help
Display usage message.
-n | --nodestatmon
Indicate that this monitoring plug-in will be used for feeding the node liveness status to the xCAT nodelist
table.
-s | --settings
Specifies the plug-in specific settings. These settings will be used by the plug-in to customize certain
entities for the plug-in or the third party monitoring software. e.g. -s mon_interval=10 -s toggle=1.
-v | --version
Command Version.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To register gangliamon plug-in module (which interacts with Ganglia monitoring software) to monitor the xCAT
cluster, enter:
monadd gangliamon
2. To register rmcmon plug-in module (which interacts with IBM’s RSCT monitoring software) to monitor the
xCAT cluster and have it feed the node liveness status to xCAT’s nodelist table, enter:
monadd rmcmon -n
This will also add the configrmcnode to the postscripts table. To view the content of the postscripts table, enter:
tabdump postscritps
#node,postscripts,comments,disable
"service","servicenode",,
"xcatdefaults","syslog,remoteshell,configrmcnode",,
3. To register xcatmon plug-in module to feed the node liveness status to xCAT’s nodelist table, enter:
monadd xcatmon -n -s ping-interval=2
where 2 is the number of minutes between the pings.
1.3. Admin Guide
331
xCAT3 Documentation, Release 2.12
FILES
/opt/xcat/bin/monadd
SEE ALSO
monls(1)|monls.1, monrm(1)|monrm.1, monstart(1)|monstart.1, monstop(1)|monstop.1, moncfg(1)|moncfg.1, mondecfg(1)|mondecfg.1
moncfg.1
NAME
moncfg - Configures a 3rd party monitoring software to monitor the xCAT cluster.
SYNOPSIS
moncfg [-h| --help]
moncfg [-v| --version]
moncfg name [noderange] [-r|--remote]
DESCRIPTION
This command is used to configure a 3rd party monitoring software to monitor the xCAT cluster. For example, it
modifies the configuration file for the monitoring software so that the nodes can be included in the monitoring domain.
The operation is performed on the management node and the service nodes of the given nodes. The operation will
also be performed on the nodes if the -r option is specified, though the configuration of the nodes is usually performed
during the node deployment stage.
Parameters
name is the name of the monitoring plug-in module. For example, if the name is called xxx, then the actual file name
that the xcatd looks for is /opt/xcat/lib/perl/xCAT_monitoring/xxx.pm. Use monls -a command to list all the monitoring
plug-in modules that can be used.
noderange specifies the nodes to be monitored. If omitted, all nodes will be monitored.
OPTIONS
-h | --help Display usage message.
-r | --remote Specifies that the operation will also be performed on the nodes.
-v | --version Command Version.
332
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To configure the management node and the service nodes for ganglia monitoring, enter:
moncfg gangliamon
2. To configure the management node, nodes and their service nodes for ganglia monitoring, enter:
moncfg gangliamon -r
FILES
/opt/xcat/bin/moncfg
SEE ALSO
monls(1)|monls.1, mondecfg(1)|mondecfg.1, monadd(1)|monadd.1, monrm(1)|monrm.1, monstart(1)|monstart.1,
monstop(1)|monstop.1
mondecfg.1
NAME
mondecfg - Deconfigures a 3rd party monitoring software from monitoring the xCAT cluster.
SYNOPSIS
mondecfg [-h| --help]
mondecfg [-v| --version]
mondecfg name [noderange] [-r|--remote]
DESCRIPTION
This command is used to deconfigure a 3rd party monitoring software from monitoring the xCAT cluster. The operation
is performed on the management node and the service nodes of the given nodes. The operation will also be performed
on the nodes if the -r option is specified. The deconfiguration operation will remove the nodes from the 3rd party
software’s monitoring domain.
1.3. Admin Guide
333
xCAT3 Documentation, Release 2.12
PARAMETERS
name is the name of the monitoring plug-in module. Use monls command to list all the monitoring plug-in modules
that can be used.
noderange specified the nodes to be deconfigured. If omitted, all nodes will be deconfigured.
OPTIONS
-h | --help Display usage message.
-r | --remote Specifies that the operation will also be performed on the nodes.
-v | --version Command Version.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To deconfigure the management node and the service nodes from the ganglia monitoring, enter:
mondecfg gangliamon
2. To deconfigure the management node, nodes and their service nodes from the ganglia monitoring, enter:
mondecfg gangliamon -r
FILES
/opt/xcat/bin/mondecfg
SEE ALSO
monls(1)|monls.1, moncfg(1)|moncfg.1, monadd(1)|monadd.1, monrm(1)|monrm.1, monstart(1)|monstart.1, monstop(1)|monstop.1
monls.1
NAME
monls - Lists monitoring plug-in modules that can be used to monitor the xCAT cluster.
334
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
SYNOPSIS
monls [-h| --help]
monls [-v| --version]
monls name [-d|--description]
monls [-a|--all] [-d|--description]
DESCRIPTION
This command is used to list the status, description, the configuration scripts and the settings of one or all of the
monitoring plug-in modules.
Parameters
name is the name of the monitoring plug-in module.
OPTIONS
-a | --all Searches the XCATROOT/lib/perl/xCAT_monitoring directory and reports all the monitoring plug-in modules.
If nothing is specified, the list is read from the monitoring table.
-d | --description Display the description of the plug-in modules. The description usually contains the possible
settings.
-h | --help Display usage message.
-v | --version Command Version.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To list the status of all the monitoring plug-in modules from the monitoring table, enter:
monls
The output looks like this:
xcatmon
snmpmon
monitored
not-monitored
node-status-monitored
2. To list the status of all the monitoring plug-in modules including the ones that are not in the monitoring table,
enter
monls -a
1.3. Admin Guide
335
xCAT3 Documentation, Release 2.12
The output looks like this:
xcatmon
snmpmon
gangliamon
rmcmon
nagiosmon
monitored
not-monitored
not-monitored
monitored
not-monitored
node-status-monitored
3. To list the status and the description for snmpmon module, enter:
monls snmpmon -d
FILES
/opt/xcat/bin/monls
SEE ALSO
monadd(1)|monadd.1, monrm(1)|monrm.1, monstart(1)|monstart.1, monstop(1)|monstop.1, moncfg(1)|moncfg.1,
mondecfg(1)|mondecfg.1
monrm.1
NAME
monrm - Unregisters a monitoring plug-in module from the xCAT cluster.
SYNOPSIS
monrm [-h| --help]
monrm [-v| --version]
monrm name
DESCRIPTION
This command is used to unregister a monitoring plug-in module from the monitoring table. It also removes any
configuration scripts associated with the monitoring plug-in from the postscripts table. A monitoring plug-in module
acts as a bridge that connects a 3rd party monitoring software and the xCAT cluster. A configuration script is used
to configure the 3rd party software. Once added to the postscripts table, it will be invoked on the nodes during node
deployment stage.
PARAMETERS
name is the name of the monitoring plug-in module in the monitoring table. Use monls command to list all the
monitoring plug-in modules that can be used.
336
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
OPTIONS
-h | --help Display usage message.
-v | --version Command Version.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1.To unregister gangliamon plug-in module (which interacts with Ganglia monitoring software) from the xCAT cluster,
enter:
monrm gangliamon
Note that gangliamon must have been registered in the xCAT monitoring table. For a list of registered plug-in modules,
use command monls.
FILES
/opt/xcat/bin/monrm
SEE ALSO
monls(1)|monls.1, monadd(1)|monadd.1, monstart(1)|monstart.1, monstop(1)|monstop.1, moncfg(1)|moncfg.1, mondecfg(1)|mondecfg.1
monshow.1
NAME
monshow - Shows event data for monitoring.
SYNOPSIS
monshow [-h| --help]
monshow [-v| --version]
monshow name [noderange] [-s] [-t time] [-a attributes] [-w attr < operator > val [-w attr < operator > val] ... ][-o
{p|e}]
1.3. Admin Guide
337
xCAT3 Documentation, Release 2.12
DESCRIPTION
This command displays the events that happened on the given nodes or the monitoring data that is collected from the
given nodes for a monitoring plugin.
PARAMETERS
name is the name of the monitoring plug-in module to be invoked.
noderange is a list of nodes to be showed for. If omitted, the data for all the nodes will be displayed.
OPTIONS
-h | --help Display usage message.
-v | --version Command Version.
-s shows the summary data.
-t specifies a range of time for the data, The default is last 60 minutes. For example -t 6-4, it will display the data from
last 6 minutes to 4 minutes; If it is -t 6, it will display the data from last 6 minutes until now.
-a specifies a comma-separated list of attributes or metrics names. The default is all.
-w specify one or multiple selection string that can be used to select events. The operators ==, !=, =,!,>,<,>=,<=
are available. Wildcards % and _ are supported in the pattern string. % allows you to match any string of any
length(including zero length) and _ allows you to match on a single character. The valid attributes are eventtype, monitor, monnode, application, component, id, severity, message, rawdata, comments. Valid severity are: Informational,
Warning, Critical.
Operator descriptions:
== Select event where the attribute value is exactly this value.
!= Select event where the attribute value is not this specific value.
=~ Select event where the attribute value matches this pattern string. Not work with severity.
!~> Select event where the attribute value does not match this pattern string. Not work with severity.
> Select event where the severity is higher than this value. Only work with severity.
< Select event where the severity is lower than this value. Only work with severity.
>= Select event where the severity is higher than this value(include). Only work with severity.
<= Select event where the severity is lower than this value(include). Only work with severity.
Note: if the “val” or “operator” fields includes spaces or any other characters that will be parsed by shell, the
“attr<operator>val” needs to be quoted. If the operator is ”!~”, the “attr<operator>val” needs to be quoted using
single quote.
-o specifies montype, it can be p or e. p means performance, e means events.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
338
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
EXAMPLES
1. To show summary data about PctRealMemFree and PctTotalTimeIdle of cluster in last 6 minutes, enter:
monshow rmcmon -s -a PctRealMemFree,PctTotalTimeIdle -t 6
2. To show all data of node1 and node2, enter:
monshow rmcmon node1,node2
3. To show summary data of nodes which managed by servicenode1, enter:
monshow rmcmon servicenode1 -s
4. To show RMC event with severity Critical, enter:
monshow rmcmon -w severity==Critical
FILES
/opt/xcat/bin/monshow
SEE ALSO
monls(1)|monls.1, monstart(1)|monstart.1, monstop(1)|monstop.1, monadd(1)|monadd.1, monrm(1)|monrm.1, moncfg(1)|moncfg.1, mondecfg(1)|mondecfg.1
monstart.1
NAME
monstart - Starts a plug-in module to monitor the xCAT cluster.
SYNOPSIS
monstart [-h| --help]
monstart [-v| --version]
monstart name [noderange] [-r|--remote]
DESCRIPTION
This command is used to start a 3rd party software, (for example start the daemons), to monitor the xCAT cluster. The
operation is performed on the management node and the service nodes of the given nodes. The operation will also be
performed on the nodes if the -r option is specified.
1.3. Admin Guide
339
xCAT3 Documentation, Release 2.12
PARAMETERS
name is the name of the monitoring plug-in module. For example, if the name is called xxx, then the actual file
name that the xcatd looks for is /opt/xcat/lib/perl/xCAT_monitoring/xxx.pm. Use monls -a command to list all the
monitoring plug-in modules that can be used.
noderange is the nodes to be monitored. If omitted, all nodes will be monitored.
OPTIONS
-h | --help Display usage message.
-r | --remote Specifies that the operation will also be performed on the nodes. For example, the3rd party monitoring
software daemons on the nodes will also be started.
-v | --version Command Version.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To start gangliamon plug-in module (which interacts with Ganglia monitoring software) to monitor the xCAT
cluster, enter:
monstart gangliamon -r
2. To start xcatmon plug-in module to feed the node liveness status to xCAT’s nodelist table, enter:
monstart rmcmon
FILES
/opt/xcat/bin/monstart
SEE ALSO
monls(1)|monls.1, monstop(1)|monstop.1, monadd(1)|monadd.1, monrm(1)|monrm.1, moncfg(1)|moncfg.1, mondecfg(1)|mondecfg.1
monstop.1
NAME
monstop - Stops a monitoring plug-in module to monitor the xCAT cluster.
340
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
SYNOPSIS
monstop [-h| --help]
monstop [-v| --version]
monstop name [noderange] [-r|--remote]
DESCRIPTION
This command is used to stop a 3rd party software, (for example stop the daemons), from monitoring the xCAT cluster.
The operation is performed on the management node and the service nodes of the given nodes. The operation will also
be performed on the nodes if the -r option is specified.
PARAMETERS
name is the name of the monitoring plug-in module in the monitoring table. Use monls command to list all the
monitoring plug-in modules that can be used.
noderange is the nodes to be stopped for monitoring. If omitted, all nodes will be stopped.
OPTIONS
-h | -help Display usage message.
-r | --remote Specifies that the operation will also be performed on the nodes. For example, the3rd party monitoring
software daemons on the nodes will also be stopped.
-v | -version Command Version.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1.To stop gangliamon plug-in module (which interacts with Ganglia monitoring software) to monitor the xCAT cluster,
enter:
monstop gangliamon
Note that gangliamon must have been registered in the xCAT monitoring table. For a list of registered plug-in modules,
use command monls.
FILES
/opt/xcat/bin/monstop
1.3. Admin Guide
341
xCAT3 Documentation, Release 2.12
SEE ALSO
monls(1)|monls.1, monstart(1)|monstart.1, monadd(1)|monadd.1, monrm(1)|monrm.1, moncfg(1)|moncfg.1, mondecfg(1)|mondecfg.1
mysqlsetup.1
NAME
mysqlsetup - Sets up the MySQL or MariaDB database for xCAT to use.
SYNOPSIS
mysqlsetup {-h | --help}
mysqlsetup {-v | --version}
mysqlsetup {-i | --init} [-f | --hostfile] [-o | --odbc] [-L | --LL] [-V | --verbose]
mysqlsetup {-u | --update} [-f | --hostfile] [-o | --odbc] [-L | --LL] [-V | --verbose]
mysqlsetup {-o | --odbc} [-V | --verbose]
mysqlsetup {-L | --LL} [-V | --verbose]
DESCRIPTION
mysqlsetup - Sets up the MySQL or MariaDB database (linux only for MariaDB) for xCAT to use. The mysqlsetup
script is run on the Management Node as root after the MySQL code or MariaDB code has been installed. Before
running the init option, the MySQL server should be stopped, if it is running. The xCAT daemon, xcatd, must be
running, do not stop it. No xCAT commands should be run during the init process, because we will be migrating
the xCAT database to MySQL or MariaDB and restarting the xcatd daemon as well as the MySQL daemon. For
full information on all the steps that will be done, read the “Configure MySQL and Migrate xCAT Data to MySQL”
sections in
Setting_Up_MySQL_as_the_xCAT_DB
Two passwords must be supplied for the setup, a password for the xcatadmin id and a password for the root id in
the MySQL database. These will be prompted for interactively, unless the environment variables XCATMYSQLADMIN_PW and XCATMYSQLROOT_PW are set to the passwords for the xcatadmin id and root id in the database,resp.
Note below we refer to MySQL but it works the same for MariaDB.
OPTIONS
-h|--help
Displays the usage message.
-v|--version
Displays the release version of the code.
-V|--verbose
Displays verbose messages.
342
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
-i|--init
The init option is used to setup a xCAT database on an installed MySQL or MariaDB server for xCAT to
use. The mysqlsetup script will check for the installed MariaDB server rpm first and will use MariaDB if
it is installed. This involves creating the xcatdb database, the xcatadmin id, allowing access to the xcatdb
database by the Management Node. It customizes the my.cnf configuration file for xcat and starts the
MySQL server. It also backs up the current xCAT database and restores it into the newly setup xcatdb
MySQL database. It creates the /etc/xcat/cfgloc file to point the xcatd daemon to the MySQL database
and restarts the xcatd daemon using the database. On AIX, it additionally setup the mysql id and group
and corrects the permissions in the MySQL install directories. For AIX, you should be using the MySQL
rpms available from the xCAT website. For Linux, you should use the MySQL or MariaDB rpms shipped
with the OS. You can chose the -f and/or the -o option, to run after the init.
-u|--update
To run the update option, you must first have run the -i option and have xcat successfully running on the
MySQL database. You can chose the -f and/or the -o option, to update.
-f|--hostfile
This option runs during update, it will take all the host from the input file (provide a full path) and give
them database access to the xcatdb in MySQL for the xcatadmin id. Wildcards and ipaddresses may be
used. xCAT must have been previously successfully setup to use MySQL. xcatadmin and MySQL root
password are required.
-o|--odbc
This option sets up the ODBC /etc/../odbcinst.ini, /etc/../odbc.ini and the .odbc.ini file in roots home
directory will be created and initialized to run off the xcatdb MySQL database. See “Add ODBC Support”
in Setting_Up_MySQL_as_the_xCAT_DB
-L|--LL
Additional database configuration specifically for the LoadLeveler product. See “Add ODBC Support” in
Setting_Up_MySQL_as_the_xCAT_DB
ENVIRONMENT VARIABLES
* XCATMYSQLADMIN_PW - the password for the xcatadmin id that will be assigned in the MySQL database.
* XCATMYSQLROOT_PW - the password for the root id that will be assigned to the MySQL root id, if the script
creates it. The password to use to run MySQL command to the database as the MySQL root id. This password may
be different than the unix root password on the Management Node.
EXAMPLES
1. To setup MySQL for xCAT to run on the MySQL xcatdb database :
mysqlsetup -i
2. Add hosts from /tmp/xcat/hostlist that can access the xcatdb database in MySQL:
mysqlsetup -u -f /tmp/xcat/hostlist
Where the file contains a host per line, for example:
1.3. Admin Guide
343
xCAT3 Documentation, Release 2.12
node1
1.115.85.2
10.%.%.%
nodex.cluster.net
3. To setup the ODBC for MySQL xcatdb database access :
mysqlsetup -o
4. To setup MySQL for xCAT and add hosts from /tmp/xcat/hostlist and setup the ODBC in Verbose mode:
mysqlsetup -i -f /tmp/xcat/hostlist -o -V
nimnodecust.1
NAME
nimnodecust - Use this xCAT command to customize AIX/NIM standalone machines.
SYNOPSIS
nimnodecust [-h|--help ]
nimnodecust [-V] -s lpp_source_name [-p packages] [-b installp_bundles] noderange [attr=val [attr=val ...]]
DESCRIPTION
This xCAT command can be used to customize AIX/NIM standalone machines.
The software packages that you wish to install on the nodes must be copied to the appropriate directory locations in
the NIM lpp_source resource provided by the “-s” option. For example, if the location of your lpp_source resource is
“/install/nim/lpp_source/61lpp/” then you would copy RPM packages to “/install/nim/lpp_source/61lpp/RPMS/ppc”
and you would copy your installp packages to “/install/nim/lpp_source/61lpp/installp/ppc”. Typically you would want
to copy the packages to the same lpp_source that was used to install the node. You can find the location for an
lpp_source with the AIX lsnim command. (Ex. “lsnim -l <lpp_source_name>”)
The packages you wish to install on the nodes may be specified with either a comma-separated list of package names
or by a comma-separated list of installp_bundle names. The installp_bundle names are what were used when creating
the corresponding NIM installp_bundle definitions. The installp_bundle definitions may also be used when installing
the nodes.
A bundle file contains a list of package names. The RPMs must have a prefix of “R:” and the installp packages must
have a prefix of “I:”. For example, the contents of a simple bundle file might look like the following.
# RPM
R:expect-5.42.1-3.aix5.1.ppc.rpm
R:ping-2.4b2_to-1.aix5.3.ppc.rpm
#installp
I:openssh.base
I:openssh.license
344
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
To create a NIM installp_bundle definition you can use the “nim -o define” operation. For example, to create a
definition called “mypackages” for a bundle file located at “/install/nim/mypkgs.bnd” you could issue the following
command.
nim -o define -t installp_bundle -a server=master -a location=/install/nim/mypkgs.bnd
˓→mypackages
See the AIX documentation for more information on using installp_bundle files.
The xCAT nimnodecust command will automatically handle the distribution of the packages to AIX service nodes
when using an xCAT hierarchical environment.
OPTIONS
attr=val [attr=val ...]
Specifies one or more “attribute equals value” pairs, separated by spaces. Attr=val pairs must be specified
last on the command line. These are used to specify additional values that can be passed to the underlying
NIM commands, (“nim -o cust...”). See the NIM documentation for valid “nim” command line options.
-b installp_bundle_names
A comma separated list of NIM installp_bundle names.
-h |--help
Display usage message.
-p package_names
A comma-separated list of software packages to install. Packages may be RPM or installp.
noderange
A set of comma delimited node names and/or group names. See the “noderange” man page for details on
additional supported formats.
-V |--verbose
Verbose mode.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. Install the installp package “openssh.base.server” on an xCAT node named “node01”. Assume that the package
has been copied to the NIM lpp_source resource called “61lppsource”.
nimnodecust -s 61lppsource -p openssh.base.server node01
2. Install the product software contained in the two bundles called “llbnd” and “pebnd” on all AIX nodes contained
in the xCAT node group called “aixnodes”. Assume that all the software packages have been copied to the NIM
lpp_source resource called “61lppsource”.
1.3. Admin Guide
345
xCAT3 Documentation, Release 2.12
nimnodecust -s 61lppsource -b llbnd,pebnd
aixnodes
FILES
/opt/xcat/bin/nimnodecust
NOTES
This command is part of the xCAT software product.
nimnodeset.1
NAME
nimnodeset - Use this xCAT command to initialize AIX/NIM standalone machines.
SYNOPSIS
nimnodeset [-h|--help ]
nimnodeset [-V|--verbose] [-f|--force] [-i osimage_name] [-l location] [-p|--primarySN] [-b | --backupSN]
noderange [attr=val [attr=val ...]]
DESCRIPTION
This xCAT command can be used to initialize AIX/NIM standalone machines. Once this step is completed the either
the xCAT rnetboot command or the rbootseq/rpower commands to initiate a network boot of the nodes.
If you are using xCAT service nodes the nimnodeset command will automatically determine the correct server(s) for
the node and do the initialization on that server(s).
The osimage_name is the name of an xCAT osimage definition that contains the list of NIM resources to use when
initializing the nodes. If the osimage_name is not provided on the command line the code checks the node definition
for the value of the “provmethod” attribute (which is the name of an osimage definition). If the osimage_image is
provided on the command line then the code will also set the “provmethod” attribute of the node definitions.
This command will also create a NIM resolv_conf resource to be used when installing the node. If a resolv_conf
resource is not already included in the xCAT osimage definition and if the “domain” and “nameservers” values are set
then a new NIM resolv_conf resource will be created and allocated to the nodes.
The “domain” and “nameservers” attributes can be set in either the xCAT “network” definition used by the nodes or in
the xCAT cluster “site” definition. The setting in the “network” definition will take priority.
The “search” field of the resolv.conf file will contain a list all the domains listed in the xCAT network definitions and
the xCAT site definition.
The “nameservers” value can either be set to a specific IP address or the “<xcatmaster>” key word. The “<xcatmaster>” key word means that the value of the “xcatmaster” attribute of the node definition will be used in the
/etc/resolv.conf file. (I.e. The name of the install server as known by the node.)
You can set the “domain” and “nameservers” attributes by using the chdef command. For example:
346
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
chdef -t network -o clstr_net domain=cluster.com nameservers=<xcatmaster>
If the “domain” and “nameservers” attributes are not set in either the nodes “network” definition or the “site” definition
then no new NIM resolv_conf resource will be created.
You can specify additional attributes and values using the “attr=val” command line option. This information will be
passed on to the underlying call to the NIM “nim -o bos_inst” command. See the NIM documentation for information
on valid command line options for the nim command. The “attr” must correspond to a NIM attribute supported for the
NIM “bos_inst” operation. Information provided by the “attr=val” option will take precedence over the information
provided in the osimage definition.
The force option can be used to reinitialize a node if it already has resources allocated or it is in the wrong NIM state.
This option will reset the NIM node and deallocate resources before reinitializing.
This command will also create a NIM script resource to enable the xCAT support for user-provided customization
scripts.
After the nimnodeset command completes you can use the lsnim command to check the NIM node definition to see
if it is ready for booting the node. (“lsnim -l <nim_node_name>”).
You can supply your own scripts to be run on the management node or on the service node (if their is hierarchy) for a
node during the nimnodeset command. Such scripts are called prescripts. They should be copied to /install/prescripts
directory. A table called prescripts is used to specify the scripts and their associated actions. The scripts to be run at
the beginning of the nimnodeset command are stored in the ‘begin’ column of prescripts table. The scripts to be run
at the end of the nimnodeset command are stored in the ‘end’ column of prescripts table. Run ‘tabdump prescripts
-d’ command for details. An example for the ‘begin’ or the ‘end’ column is: standalone:myscript1,myscript2. The
following two environment variables will be passed to each script: NODES contains all the names of the nodes that
need to run the script for and ACTION contains the current nodeset action, in this case “standalone”. If #xCAT
setting:MAX_INSTANCE=number is specified in the script, the script will get invoked for each node in parallel, but
no more than number of instances will be invoked at at a time. If it is not specified, the script will be invoked once for
all the nodes.
OPTIONS
attr=val [attr=val ...]
Specifies one or more “attribute equals value” pairs, separated by spaces. Attr= val pairs must be specified
last on the command line. These are used to specify additional values that can be passed to the underlying
NIM commands, (“nim -o bos_inst ...”). See the NIM documentation for valid “nim” command line
options. Note that you may specify multiple “script” and “installp_bundle” values by using a comma
separated list. (ex. “script=ascript,bscript”).
-b|--backupSN
When using backup service nodes only update the backup. The default is to update both the primary and
backup service nodes
-f |--force
Use the force option to reinitialize the NIM machines.
-h |--help
Display usage message.
-i image_name
The name of an existing xCAT osimage definition.
-l|--location
1.3. Admin Guide
347
xCAT3 Documentation, Release 2.12
The directory location to use when creating new NIM resolv_conf resources. The default location is
/install/nim.
-p|--primarySN
When using backup service nodes only update the primary. The default is to update both the primary and
backup service nodes.
noderange
A set of comma delimited node names and/or group names. See the “noderange” man page for details on
additional supported formats.
-V |--verbose
Verbose mode.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. Initialize an xCAT node named “node01”. Use the xCAT osimage named “61gold” to install the node.
nimnodeset -i 61gold node01
2. Initialize all AIX nodes contained in the xCAT node group called “aixnodes” using the image definitions pointed
to by the “provmethod” attribute of the xCAT node definitions.
nimnodeset aixnodes
3. Initialize an xCAT node called “node02”. Include installp_bundle resources that are not included in the osimage
definition. This assumes the NIM installp_bundle resources have already been created.
nimnodeset -i 611image node02 installp_bundle=sshbundle,addswbundle
FILES
/opt/xcat/bin/nimnodeset
NOTES
This command is part of the xCAT software product.
SEE ALSO
mknimimage(1)|mknimimage.1, rnetboot(1)|rnetboot.1
348
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
nodeaddunmged.1
NAME
nodeaddunmged - Create a unmanaged node.
SYNOPSIS
nodeaddunmged [-h| --help | -v | --version]
nodeaddunmged hostname=node-name ip=ip-address
DESCRIPTION
The nodeaddunmged command adds an unmanaged node to the __Unmanaged group. You can specify the node
name and IP address of the node.
OPTIONS
-h|--help
Display usage message.
-v|--version
Command Version.
hostname=node-name
Sets the name of the new unmanaged node, where <node-name> is the name of the node.
ip=ip-address
Sets the IP address of the unmanaged node, where ip-address is the IP address of the new node in the form
xxx.xxx.xxx.xxx
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
To add an unmanaged node, use the following command:
nodeaddunmged hostname=unmanaged01 ip=192.168.1.100
1.3. Admin Guide
349
xCAT3 Documentation, Release 2.12
SEE ALSO
nodech.1
NAME
nodech - Changes nodes’ attributes in the xCAT cluster database.
SYNOPSIS
nodech noderange table.column=value [...]
nodech {-d | --delete} noderange table [...]
nodech {-v | --version}
nodech [-? | -h | --help]
DESCRIPTION
The nodech command changes the specified attributes for the given nodes. Normally, the given value will completely
replace the current attribute value. But if ”,=” is used instead of “=”, the specified value will be prepended to the
attribute’s comma separated list, if it is not already there. If “^=” is used, the specified value will be removed from
the attribute’s comma separated list, if it is there. You can also use “^=” and ”,=” in the same command to essentially
replace one item in the list with another. (See the Examples section.)
Additionally, as in nodels, boolean expressions can be used to further limit the scope of nodech from the given
noderange. The operators supported are the same as nodels (=~, !~, ==, and !=).
With these operators in mind, the unambiguous assignment operator is ‘=@’. If you need, for example, to set the
nodelist.comments to =foo, you would have to do nodech n1 nodelist.comments=@=foo.
See the xcatdb man page for an overview of each table.
The nodech command also supports some short cut names as aliases to common attributes. See the nodels man page
for details.
OPTIONS
-d|--delete
Delete the nodes’ row in the specified tables.
-v|--version
Command Version.
-?|-h|--help
Display usage message.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
350
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
EXAMPLES
1. To update nodes in noderange node1-node4 to be in only group all:
nodech node1-node4 groups=all
2. To put all nodes with nodepos.rack value of 2 into a group called rack2:
nodech all nodepos.rack==2 groups,=rack2
3. To add nodes in noderange node1-node4 to the nodetype table with os=rhel5:
nodech node1-node4 groups=all,rhel5 nodetype.os=rhel5
4. To add node1-node4 to group1 in addition to the groups they are already in:
nodech node1-node4 groups,=group1
5. To put node1-node4 in group2, instead of group1:
nodech node1-node4 groups^=group1 groups,=group2
FILES
/opt/xcat/bin/nodech
SEE ALSO
nodels(1)|nodels.1, nodeadd(8)|nodeadd.8, noderange(3)|noderange.3
nodechmac.1
NAME
nodechmac - Updates the MAC address for a node.
SYNOPSIS
nodechmac [-h | --help | -v | --version]
nodechmac node-name mac=mac-address
DESCRIPTION
The nodechmac command changes the MAC address for provisioned node’s network interface.
You can use this command to keep an existing node configuration. For example, if an existing node has hardware
problems, the replacement node can use the old configurations. By using the nodechmac command, the node name
and network settings of the old node can be used by the new node.
1.3. Admin Guide
351
xCAT3 Documentation, Release 2.12
OPTIONS
-h|--help
Display usage message.
-v|--version
Command Version.
node-name
Specifies the name of the node you want to update, where <node-name> is the node that is updated.
mac=mac-address
Sets the new MAC address for the NIC used by the provisioning node, where <mac-address> is the NICs new MAC
address.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
You can update the MAC address for a node, by using the following command:
nodechmac compute-000 mac=2F:3C:88:98:7E:01
SEE ALSO
nodechprofile.1
NAME
nodechprofile - updates a profile used by a node
SYNOPSIS
nodechprofile [-h| --help | -v | --version]
nodechprofile noderange [imageprofile= image-profile] [networkprofile= network-profile] [hardwareprofile=
hardware-profile]
DESCRIPTION
The nodechprofile command updates the profiles used by a node, including: the image profile, network profile, and
hardware management profile.
If you update the image profile for a node, the operating system and provisioning settings for the node are updated.
352
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
If you update the network profile, the IP address and network settings for the node are updated.
If you update the hardware management profile, the hardware settings for the node are updated.
After nodes’ hardware profile or image profile are updated, the status for each node is changed to “defined”. A node
with a “defined” status must be reinstalled
After nodes’ network profile updated, the status for nodes is not changed. You’ll need to run noderegenips to regenerate the nodes’ IP address and nodes’ status may also be updated at this stage.
OPTIONS
-h|--help
Display usage message.
-v|--version
Command Version.
noderange
The nodes to be removed.
imageprofile= image-profile
Sets the new image profile name used by the node, where <image-profile> is the new image profile. An image profile
defines the provisioning method, OS information, kit information, and provisioning parameters for a node. If the
“__ImageProfile_imgprofile” group already exists in the nodehm table, then “imgprofile” is used as the image profile
name.
networkprofile= network-profile
Sets the new network profile name used by the node, where <network-profile> is the new network profile. A network
profile defines the network, NIC, and routes for a node. If the “__NetworkProfile_netprofile” group already exists in
the nodehm table, then “netprofile” is used as the network profile name.
hardwareprofile= hardware-profile
Sets the new hardware profile name used by the node, where <hardware-profile> is the new hardware management
profile used by the node. If a “__HardwareProfile_hwprofile” group exists, then “hwprofile” is the hardware profile
name. A hardware profile defines hardware management related information for imported nodes, including: IPMI,
HMC, CEC, CMM.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To change the image profile to rhels6.3_packaged for compute nodes compute-000 and compute-001, use the
following command:
nodechprofile compute-000,compute-001 imageprofile=rhels6.3_packaged
2. To change all of the profiles for compute node compute-000, enter the following command:
1.3. Admin Guide
353
xCAT3 Documentation, Release 2.12
nodechprofile compute-000 imageprofile=rhels6.3_packaged
˓→networkprofile=default_cn hardwareprofile=default_ipmi
SEE ALSO
nodepurge(1)|nodepurge.1, noderefresh(1)|noderefresh.1, nodeimport(1)|nodeimport.1, noderange(3)|noderange.3
nodediscoverdef.1
NAME
nodediscoverdef - Define the undefined discovery request to a predefined xCAT node, or clean up the discovery entries
from the discoverydata table (which can be displayed by nodediscoverls command)
SYNOPSIS
nodediscoverdef -u uuid -n node
nodediscoverdef -r -u uuid
nodediscoverdef -r -t {seq | profile | switch | blade | manual | undef | all}
nodediscoverdef [-h | --help | -v | --version]
DESCRIPTION
The nodediscoverdef command defines the discovery entry from the discoverydata table to a predefined xCAT node.
The discovery entry can be displayed by nodediscoverls command.
The options -u and -n have to be used together to define a discovery request to a node.
The nodediscoverdef command also can be used to clean up the discovery entries from the discoverydata table.
The option -r is used to remove discovery entries. If working with -u, the specific entry which uuid specified by -u
will be removed.
You also can use the -r -t option to limit that only remove the nodes that were discovered in a particular method of
discovery.
OPTIONS
-t seq|profile|switch|blade|manual|undef|all
Specify the nodes that have been discovered by the specified discovery method:
* seq - Sequential discovery (started via nodediscoverstart noderange=<noderange> ...).
* profile - Profile discovery (started via nodediscoverstart networkprofile=<network-profile> ...).
* switch - Switch-based discovery (used when the switch and switches tables are filled in).
* blade - Blade discovery (used for IBM Flex blades).
* manual - Manually discovery (used when defining node by nodediscoverdef command).
354
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
* undef - Display the nodes that were in the discovery pool, but for which xCAT has not yet received a
discovery request.
* all - All discovered nodes.
-n node
The xCAT node that the discovery entry will be defined to.
-r
Remove the discovery entries from discoverydata table.
-u uuid
The uuid of the discovered entry.
-h|--help
Display usage message.
-v|--version
Command version.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. Define the discovery entry which uuid is 51E5F2D7-0D59-11E2-A7BC-3440B5BEDBB4 to node node1
nodediscoverdef -u 51E5F2D7-0D59-11E2-A7BC-3440B5BEDBB4 -n node1
Output is similar to:
Defined [51E5F2D7-0D59-11E2-A7BC-3440B5BEDBB4] to node node1.
2. Remove the discovery entry which uuid is 51E5F2D7-0D59-11E2-A7BC-3440B5BEDBB4 from the discoverydata table
nodediscoverdef -r -u 51E5F2D7-0D59-11E2-A7BC-3440B5BEDBB4
Output is similar to:
Removing discovery entries finished.
3. Remove the discovery entries which discover type is seq from the discoverydata table
nodediscoverdef -r -t seq
Output is similar to:
Removing discovery entries finished.
1.3. Admin Guide
355
xCAT3 Documentation, Release 2.12
SEE ALSO
nodediscoverstart(1)|nodediscoverstart.1,
nodediscoverstatus(1)|nodediscoverstatus.1,
stop(1)|nodediscoverstop.1, nodediscoverls(1)|nodediscoverls.1
nodediscover-
nodediscoverls.1
NAME
nodediscoverls - List the discovered nodes
SYNOPSIS
nodediscoverls [-t seq | profile | switch | blade | manual | undef | all] [-l]
nodediscoverls [-u uuid] [-l]
nodediscoverls [-h | --help | -v | --version]
DESCRIPTION
The nodediscoverls command lists nodes that have recently been discovered. If discovery is currently in progress (i.e.
nodediscoverstart has been run, but nodediscoverstop has not been), then nodediscoverls will list the nodes that
have been discovered so far in this session. If discovery is not currently in progress, nodediscoverls will list all of the
nodes that were discovered in the last discovery session.
You can use the -t option to limit the output to just the nodes that were discovered in a particular method of discovery.
OPTIONS
-t seq|profile|switch|blade|manual|undef|all
Display the nodes that have been discovered by the specified discovery method:
* seq - Sequential discovery (started via nodediscoverstart noderange=<noderange> ...).
* profile - Profile discovery (started via nodediscoverstart networkprofile=<network-profile> ...).
* switch - Switch-based discovery (used when the switch and switches tables are filled in).
* blade - Blade discovery (used for IBM Flex blades).
* manual - Manually discovery (used when defining node by nodediscoverdef command).
* undef - Display the nodes that were in the discovery pool, but for which xCAT has not yet received a
discovery request.
* all - All discovered nodes.
-l
Display more detailed information about the discovered nodes.
-u uuid
Display the discovered node that has this uuid.
356
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
-h|--help
Display usage message.
-v|--version
Command version.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. Display the discovered nodes when sequential discovery is running:
nodediscoverls
Output is similar to:
UUID
˓→MTM
SERIAL
51E5F2D7-0D59-11E2-A7BC-3440B5BEDBB2
˓→786310X
1052EF2
51E5F2D7-0D59-11E2-A7BC-3440B5BEDBB3
˓→786310X
1052EF3
NODE
METHOD
distest1
sequential
distest2
sequential
2. Display the nodes that were in the discovery pool, but for which xCAT has not yet received a discovery request:
nodediscoverls -t undef
Output is similar to:
UUID
˓→MTM
SERIAL
51E5F2D7-0D59-11E2-A7BC-3440B5BEDBB0
˓→786310X
1052EF0
51E5F2D7-0D59-11E2-A7BC-3440B5BEDBB1
˓→786310X
1052EF1
NODE
METHOD
undef
undef
undef
undef
NODE
METHOD
undef
undef
undef
undef
distest1
sequential
distest2
sequential
3. Display all the discovered nodes:
nodediscoverls -t all
Output is similar to:
UUID
˓→MTM
SERIAL
51E5F2D7-0D59-11E2-A7BC-3440B5BEDBB0
˓→786310X
1052EF0
51E5F2D7-0D59-11E2-A7BC-3440B5BEDBB1
˓→786310X
1052EF1
51E5F2D7-0D59-11E2-A7BC-3440B5BEDBB2
˓→786310X
1052EF2
51E5F2D7-0D59-11E2-A7BC-3440B5BEDBB3
˓→786310X
1052EF3
1.3. Admin Guide
357
xCAT3 Documentation, Release 2.12
4. Display the discovered node whose uuid is 51E5F2D7-0D59-11E2-A7BC-3440B5BEDBB2, with detailed information:
nodediscoverls -u 51E5F2D7-0D59-11E2-A7BC-3440B5BEDBB2 -l
Output is similar to:
Object uuid: 51E5F2D7-0D59-11E2-A7BC-3440B5BEDBB2
node=distest1
method=sequential
discoverytime=03-31-2013 17:05:12
arch=x86_64
cpucount=32
cputype=Intel(R) Xeon(R) CPU E5-2690 0 @ 2.90GHz
memory=198460852
mtm=786310X
serial=1052EF2
nicdriver=eth0!be2net,eth1!be2net
nicipv4=eth0!10.0.0.212/8
nichwaddr=eth0!34:40:B5:BE:DB:B0,eth1!34:40:B5:BE:DB:B4
nicpci=eth0!0000:0c:00.0,eth1!0000:0c:00.1
nicloc=eth0!Onboard Ethernet 1,eth1!Onboard Ethernet 2
niconboard=eth0!1,eth1!2
nicfirm=eth0!ServerEngines BE3 Controller,eth1!ServerEngines BE3
˓→Controller
switchname=eth0!c909f06sw01
switchaddr=eth0!192.168.70.120
switchdesc=eth0!IBM Flex System Fabric EN4093 10Gb Scalable Switch, flash
˓→image: version 7.2.6, boot image: version 7.2.6
switchport=eth0!INTA2
SEE ALSO
nodediscoverstart(1)|nodediscoverstart.1,
nodediscoverstatus(1)|nodediscoverstatus.1,
stop(1)|nodediscoverstop.1, nodediscoverdef(1)|nodediscoverdef.1
nodediscover-
nodediscoverstart.1
NAME
nodediscoverstart - starts the node discovery process
SYNOPSIS
nodediscoverstart [-h | --help | -v | --version]
Sequential Discovery Specific:
nodediscoverstart noderange=noderange [hostiprange=imageprofile] [bmciprange=bmciprange] [groups=groups]
[rack=rack] [chassis=chassis] [height=height] [unit=unit] [osimage= osimagename>] [-n | --dns] [-s | -skipbmcsetup] [-V|--verbose]
Profile Discovery Specific:
358
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
nodediscoverstart networkprofile=network-profile imageprofile=image-profile hostnameformat=nost-nameformat [hardwareprofile=hardware-profile] [groups=node-groups] [rack=rack-name] [chassis=chassis-name]
[height=rack-server-height] [unit=rack-server-unit-location] [rank=rank-num]
DESCRIPTION
The nodediscoverstart command starts either the Sequential Discovery or Profile Discovery process. They can not
both be running at the same time.
Sequential Discovery Specific:
This is the simplest discovery approach. You only need to specify the noderange, hostiprange and bmciprange that
should be given to nodes that are discovered. (If you pre-define the nodes (via nodeadd or mkdef) and specify their
host and BMC IP addresses, then you only need to specify the noderange to the nodediscoverstart command.) Once
you have run nodediscoverstart, then physically power on the nodes in the sequence that you want them to receive
the node names and IPs, waiting a short time (e.g. 30 seconds) between each node.
Profile Discovery Specific:
This is the PCM discovery approach. networkprofile, imageprofile, hostnameformat arguments must be specified
to start the Profile Discovery. All nodes discovered by this process will be associated with specified profiles and
rack/chassis/unit locations.
When the nodes are discovered, PCM updates the affected configuration files on the management node automatically.
Configuration files include the /etc/hosts service file, DNS configuration, and DHCP configuration. Kit plug-ins are
automatically triggered to update kit related configurations and services.
When you power on the nodes, they PXE boot and DHCP/TFTP/HTTP on the management node give each node the
xCAT genesis boot image, which inventories the node hardware and sends data to the management node. There, either
the sequential discovery process or the profile discovery process assigns node attributes and defines the node in the
database.
OPTIONS
noderange=noderange
The set of node names that should be given to nodes that are discovered via the Sequential Discovery
method. This argument is required to Sequential Discovery. Any valid xCAT noderange is allowed, e.g.
node[01-10].
hostiprange=ip range
The ip range which will be assigned to the host of new discovered nodes in the Sequential Discovery method. The format can be: start_ip-end_ip or noderange, e.g. 192.168.0.1-192.168.0.10 or
192.168.0.[1-10].
bmciprange=ip range
The ip range which will be assigned to the bmc of new discovered nodes in the Sequential Discovery method. The format can be: start_ip-end_ip or noderange, e.g. 192.168.1.1-192.168.1.10 or
192.168.1.[1-10].
imageprofile=image-profile
Sets the new image profile name used by the discovered nodes in the Profile Discovery method. An image
profile defines the provisioning method, OS information, kit information, and provisioning parameters for
a node. If the “__ImageProfile_imgprofile” group already exists in the nodehm table, then “imgprofile” is
used as the image profile name.
1.3. Admin Guide
359
xCAT3 Documentation, Release 2.12
networkprofile=network-profile
Sets the new network profile name used by the discovered nodes in the Profile Discovery method. A
network profile defines the network, NIC, and routes for a node. If the “__NetworkProfile_netprofile”
group already exists in the nodehm table, then “netprofile” is used as the network profile name.
hardwareprofile=hardware-profile
Sets the new hardware profile name used by the discovered nodes in the Profile Discovery method. If a
“__HardwareProfile_hwprofile” group exists, then “hwprofile” is the hardware profile name. A hardware
profile defines hardware management related information for imported nodes, including: IPMI, HMC,
CEC, CMM.
hostnameformat=nost-name-format
Sets the node name format for all discovered nodes in the Profile Discovery method. The two types of
formats supported are prefix#NNNappendix and prefix#RRand#NNappendix, where wildcard #NNN and
#NN are replaced by a system generated number that is based on the provisioning order. Wildcard #RR
represents the rack number and stays constant.
For example, if the node name format is compute-#NN, the node name is generated as: compute-00,
compute-01, ..., compute-99. If the node name format is blade#NNN-x64, the node name is generated as:
blade001-x64, blade002-x64, ..., blade999-x64
For example, if the node name format is compute-#RR-#NN and the rack number is 2, the node name is
generated as: compute-02-00, compute-02-01, ..., compute-02-99. If node name format is node-#NN-in#RR and rack number is 1, the node name is generated as: node-00-in-01, node-01-in-01, ..., node-99-in01
groups=node-groups
Sets the node groups that the discovered nodes should be put in for either the Sequential Discovery or
Profile Discovery methods, where node-group is a comma-separated list of node groups.
rack=rack-name>
Sets the rack name where the node is located for either the Sequential Discovery or Profile Discovery
methods.
chassis=chassis-name
Sets the chassis name that the Blade server or PureFlex blade is located in, for either the Sequential
Discovery or Profile Discovery methods. This option is used for the Blade server and PureFlex system
only. You cannot specify this option with the rack option.
height=rack-server-height
Sets the height of a rack-mounted server in U units for either the Sequential Discovery or Profile Discovery
methods. If the rack option is not specified, the default value is 1.
unit=rack-server-unit-location
Sets the start unit value for the node in the rack, for either the Sequential Discovery or Profile Discovery
methods. This option is for a rack server only. If the unit option is not specified, the default value is 1
rank=rank-num
Specifies the starting rank number that is used in the node name format, for the Profile Discovery method.
The rank number must be a valid integer between 0 and 254. This option must be specified with nodenameformat option. For example, if your node name format is compute-#RR-#NN. The rack’s number is
2 and rank is specified as 5, the node name is generated as follows: compute-02-05, compute-02-06, ...,
compute-02-99.
osimage=osimagename
360
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
Specifies the osimage name that will be associated with the new discovered node, the os provisioning will
be started automatically at the end of the discovery process.
-n|--dns
Specifies to run makedns <nodename> for any new discovered node. This is useful mainly for nonpredefined configuration, before running the “nodediscoverstart -n”, the user needs to run makedns -n to
initialize the named setup on the management node.
-s|--skipbmcsetup
Specifies to skip the bmcsetup during the sequential discovery process, if the bmciprange is specified with
nodediscoverstart command, the BMC will be setup automatically during the discovery process, if the user
does not want to run bmcsetup, could specify the “-s|–skipbmcsetup” with nodediscoverstart command to
skip the bmcsetup.
-V|--verbose
Enumerates the free node names and host/bmc ips that are being specified in the ranges given. Use this
option with Sequential Discovery to ensure that you are specifying the ranges you intend.
-h|--help
Display usage message.
-v|--version
Command Version.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. Sequential Discovery: To discover nodes with noderange and host/bmc ip range:
nodediscoverstart noderange=n[1-10] hostiprange='172.20.101.1-172.20.101.10'
˓→bmciprange='172.20.102.1-172.20.102.10' -V
Output is similar to:
Sequential Discovery: Started:
Number of free node names: 10
Number of free host ips: 10
Number of free bmc ips: 10
------------------------------------Free Nodes------------------------------˓→----NODE
HOST IP
BMC IP
n01
172.20.101.1
172.20.102.1
n02
172.20.101.2
172.20.102.2
...
...
...
2. Profile Discovery: To discover nodes using the default_cn network profile and the rhels6.3_packaged image
profile, use the following command:
1.3. Admin Guide
361
xCAT3 Documentation, Release 2.12
nodediscoverstart networkprofile=default_cn imageprofile=rhels6.3_packaged
˓→hostnameformat=compute#NNN
SEE ALSO
nodediscoverstop(1)|nodediscoverstop.1,
tus(1)|nodediscoverstatus.1
nodediscoverls(1)|nodediscoverls.1,
nodediscoversta-
nodediscoverstatus.1
NAME
nodediscoverstatus - gets the node discovery process status
SYNOPSIS
nodediscoverstatus [-h | --help | -v | --version]
DESCRIPTION
The nodediscoverstatus command detects if the sequential or profile node discovery process is currently running, i.e.
nodediscoverstarthas been run, but nodediscoverstop has not.
OPTIONS
-h|--help
Display usage message.
-v|--version
Command Version.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
To determine if there are some nodes discovered and the discovered nodes’ status, enter the following command:
nodediscoverstatus
362
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
SEE ALSO
nodediscoverstart(1)|nodediscoverstart.1,
tus(1)|nodediscoverstop.1
nodediscoverls(1)|nodediscoverls.1,
nodediscoversta-
nodediscoverstop.1
NAME
nodediscoverstop - stops the node discovery process.
SYNOPSIS
nodediscoverstop [-h | --help | -v | --version]
DESCRIPTION
The nodediscoverstop command stops the sequential or profile node discovery process. Once this command has been
run, newly discovered nodes will not be assigned node names and attributes automatically via the sequential or profile
discovery process.
OPTIONS
-h|--help
Display usage message.
-v|--version
Command Version.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
nodediscoverstop
SEE ALSO
nodediscoverstart(1)|nodediscoverstart.1,
tus(1)|nodediscoverstatus.1
1.3. Admin Guide
nodediscoverls(1)|nodediscoverls.1,
nodediscoversta-
363
xCAT3 Documentation, Release 2.12
nodegrpch.1
NAME
nodegrpch - Changes attributes at the group level in the xCAT cluster database.
SYNOPSIS
nodegrpch group1,group2,... table.column=value [...]
nodegrpch {-v | --version}
nodegrpch [-? | -h | --help]
DESCRIPTION
The nodegrpch command is similar to the nodech command, but ensures that the parameters are declared at the group
level rather than the node specific level, and clears conflicting node specific overrides of the specified groups. Using
table.column=value will do a verbatim assignment. If ”,=” is used instead of “=”, the specified value will be prepended
to the attribute’s comma separated list, if it is not already there. If “^=” is used, the specified value will be removed
from the attribute’s comma separated list, if it is there. You can also use “^=” and ”,=” in the same command to
essentially replace one item in the list with another. (See the Examples section.)
With these operators in mind, the unambiguous assignment operator is ‘=@’. If you need, for example, to set the
nodehm.comments to =foo, you would have to do nodegrpch group1 nodehm.comments=@=foo.
See the xcatdb man page for an overview of each table.
The nodegrpch command also supports some short cut names as aliases to common attributes. See the nodels man
page for details.
OPTIONS
-v|--version
Command Version.
-?|-h|--help
Display usage message.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To declare all members of ipmi group to have nodehm.mgt be ipmi
nodegrpch ipmi nodehm.mgt=ipmi
364
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
FILES
/opt/xcat/bin/nodegrpch
SEE ALSO
nodech(1)|nodech.1, nodels(1)|nodels.1, nodeadd(8)|nodeadd.8, noderange(3)|noderange.3
nodeimport.1
NAME
nodeimport - Create profiled nodes by importing hostinfo file.
SYNOPSIS
nodeimport [-h | --help | -v | --version]
nodeimport file= hostinfo-filename networkprofile= network-profile imageprofile= image-profile hostnameformat=
node-name-format [hardwareprofile= hardware-profile] [groups= node-groups]
DESCRIPTION
The nodeimport command creates nodes by importing a hostinfo file which is following stanza format. In this hostinfo
file, we can define node’s hostname, ip, mac, switch name, switch port and host location information like rack, chassis,
start unit, server height...etc
After nodes imported, the configuration files related with these nodes will be updated automatically. For example:
/etc/hosts, dns configuration, dhcp configuration. And the kits node plugins will also be triggered automatically to
update kit related configuration/services.
OPTIONS
-h|--help
Display usage message.
-v|--version
Command Version.
file= nodeinfo-filename
Specifies the node information file, where <nodeinfo-filename> is the full path and file name of the node information
file.
imageprofile= image-profile
Sets the new image profile name used by the node, where <image-profile> is the new image profile. An image profile
defines the provisioning method, OS information, kit information, and provisioning parameters for a node. If the
“__ImageProfile_imgprofile” group already exists in the nodehm table, then “imgprofile” is used as the image profile
name.
1.3. Admin Guide
365
xCAT3 Documentation, Release 2.12
networkprofile= network-profile
Sets the new network profile name used by the node, where <network-profile> is the new network profile. A network
profile defines the network, NIC, and routes for a node. If the “__NetworkProfile_netprofile” group already exists in
the nodehm table, then “netprofile” is used as the network profile name.
hardwareprofile= hardware-profile
Sets the new hardware profile name used by the node, where <hardware-profile> is the new hardware management
profile used by the node. If a “__HardwareProfile_hwprofile” group exists, then “hwprofile” is the hardware profile
name. A hardware profile defines hardware management related information for imported nodes, including: IPMI,
HMC, CEC, CMM.
hostnameformat= host-name-format
Sets the node name format for all nodes discovered, where <node-name-format> is a supported format. The two types
of formats supported are prefix#NNNappendix and prefix#RRand#NNappendix, where wildcard #NNN and #NN are
replaced by a system generated number that is based on the provisioning order. Wildcard #RR represents the rack
number and stays constant.
For example, if the node name format is compute-#NN, the node name is generated as: compute-00, compute-01, ... ,
compute-99. If the node name format is blade#NNN-x64, the node name is generated as: blade001-x64, blade002-x64,
... , blade999-x64
For example, if the node name format is compute-#RR-#NN and the rack number is 2, the node name is generated as:
compute-02-00, compute-02-01, ..., compute-02-99. If node name format is node-#NN-in-#RR and rack number is 1,
the node name is generated as: node-00-in-01, node-01-in-01, ... , node-99-in-01
groups= node-groups
Sets the node groups that the imported node belongs to, where <node-group> is a comma-separated list of node groups.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred while validating parameters.
2 An error has occurred while parsing hostinfo file.
EXAMPLES
To import nodes using a profile, follow the following steps:
1. Find all node groups and profiles, run the following command “tabdump nodegroups”. For detailed profile
information run “lsdef -t group <groupname>”. Example of detailed profile information:
# tabdump nodegroup
#groupname,grouptype,members,membergroups,wherevals,comments,disable
"compute","static",,,,,
"__HardwareProfile_default_ipmi","static","static",,,,
"__NetworkProfile_default_mn","static","static",,,,
"__NetworkProfile_default_cn","static",,,,,
"__ImageProfile_rhels6.2-x86_64-install-compute","static","static",,,,
# lsdef -t group __NetworkProfile_default_cn
Object name: __NetworkProfile_default_cn
grouptype=static
installnic=eth0
366
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
members=compute-000,compute-001
netboot=xnba
nichostnamesuffixes=eth0:-eth0
nicnetworks=eth0:provision
nictypes=eth0:Ethernet
primarynic=eth0
2. Prepare a node information file.
Example of a node information file, a blade and a rack server defined:
# hostinfo begin
# This entry defines a blade.
__hostname__:
mac=b8:ac:6f:37:59:24
ip=192.168.1.20
chassis=chassis01
# This entry defines a rack server.
__hostname__:
mac=b8:ac:6f:37:59:25
ip=192.168.1.20
rack=rack01
height=1
unit=2
# hostinfo end.
Another example of a node information file, a PureFlex X/P node defined:
# hostinfo begin
# To define a PureFlex P/X node, chassis and slot id must be specified.
# The chassis must be a PureFlex chassis.
__hostname__:
mac=b8:ac:6f:37:59:25
chassis=cmm01
slotid=1
# hostinfo end.
Example of a node information file, a switch auto discovery node defined:
# hostinfo begin
# This entry defines a blade.
__hostname__:
switches=eth0!switch1!1,eth0!switch2!1!eth1
Example of a node information file that specifies a CEC-based rack-mounted Power node
˓→that uses direct FSP management:
# Node information file begins
# This entry defines a Power rack-mount node.
__hostname__:
mac=b8:ac:6f:37:59:28
cec=mycec
__hostname__:
mac=b8:ac:6f:37:59:28
cec=mycec
lparid=2
# Node information file ends.
Example of a node information file that specifies a PowerKVM Guest node that uses KVM
˓→management:
1.3. Admin Guide
367
xCAT3 Documentation, Release 2.12
# Node information file begins
# This entry defines a PowerKVM Guest node.
# Make sure the node 'vm01' is already created on Hypervisor
vm01:
mac=b8:ef:3f:28:31:15
vmhost=pkvm1
# Node information file ends.
The node information file includes the following items:
__hostname__: This is a mandatory item.
Description: The name of the node, where __hostname__ is automatically generated by the node name format. You
can also input a fixed node name, for example “compute-node”.
mac=<mac-address> This is a mandatory item.
Description: Specify the MAC address for the NIC used by the provisioning node, where <mac-address> is the NICs
MAC address.
switches=<nic-name!switch-name!switch-port> This is a mandatory item, when define switch, switchport and node
nic name relationship.
Description: Specify nic name, switch name and switch port to define node and switch relationship. We can define
multi nic-switch-port relations here, looks like: switches=eth0!switch1!1,eth1!switch1,2
slotid=<slot-id> This is a mandatory item while define a PureFlex node.
Description: The node position in the PureFlex Chassis.
cec=<cec-name> This is a mandatory option for defining Power rack-mounted nodes.
Description: Specifies the name of a Power rack-mount central electronic complex (CEC).
lparid=<lpar-id> This is a optional option for defining Power rack-mounted nodes.
Description: Specifies the LPAR ID of a Power rack-mounted node, where <lpar-id> is the ID number. The default
value is 1 if it is not defined.
ip=<ip-address> This is an optional item.
Description: Specify the IP address used for provisioning a node, where <ip-address> is in the form xxx.xxx.xxx.xxx.
If this item is not included, the IP address used to provision the node is generated automatically according to the
Network Profile used by the node.
nicips=<nics-ip> This is an optional item.
Description: Lists the IP address for each network interface configuration (NIC) used by the node, excluding the provisioning network, where <nics-ip> is in the form <nic1>!<nic-ip1>,<nic2>!<nic-ip2>,.... For example, if you have 2 network interfaces configured, the nicips attribute should list both network interfaces:
nicips=eth1!10.10.10.11,bmc!192.168.10.3. If the nicips attribute is not specified, the IP addresses are generated
automatically according to the network profile.
rack=<rack-name> This is an optional item.
Description: node location info. Specify the rack name which this node will be placed into. If not specify this item,
there will be no node location info set for this node. this item must be specified together with height + unit.
chassis=<chassis-name> This is an optional item.
Description: node location info, for blade(or PureFlex) only. Specify the chassis name which this blade will be placed
into. this item can not be specified together with rack.
368
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
height=<chassis-height> This is an optional item.
Description: node location info, for rack server only. Specify the server height number, in U. this item must be
specified together with rack and unit.
unit=<rack-server-unit-location> This is an optional item.
Description: node location info, for rack server only. Specify the node’s start unit number in rack, in U. this item must
be specified together with rack and height.
vmhost=<PowerKVM Hypervisior Host Name> This is a mandatory option for defining PowerKVM Guest nodes.
Description: Specifies the vmhost of a Power KVM Guest node, where <vmhost> is the host name of PowerKVM
Hypervisior.
3. Import the nodes, by using the following commands. Note: If we want to import PureFlex X/P nodes, hardware
profile must be set to a PureFlex hardware type.
nodeimport file=/root/hostinfo.txt networkprofile=default_cn imageprofile=rhels6.3_
˓→packaged hostnameformat=compute-#NNN
4. After importing the nodes, the nodes are created and all configuration files used by the nodes are updated,
including: /etc/hosts, DNS, DHCP.
5. Reboot the nodes. After the nodes are booted they are provisioned automatically.
SEE ALSO
nodepurge(1)|nodepurge.1, nodechprofile(1)|nodechprofile.1, noderefresh(1)|noderefresh.1
nodels.1
NAME
nodels - lists the nodes, and their attributes, from the xCAT database.
SYNOPSIS
nodels [noderange] [-b | --blame] [-H | --with-fieldname] [-S] [table.column | shortname] [...]
nodels [noderange] [-H | --with-fieldname] [table]
nodels [-? | -h | --help | -v | --version]
DESCRIPTION
The nodels command lists the nodes specified in the node range. If no noderange is provided, then all nodes are listed.
Additional attributes of the nodes will also be displayed if the table names and attribute names are specified after the
noderange in the form: table.column . A few shortcut names can also be used as aliases to common attributes:
groups
nodelist.groups
tags
1.3. Admin Guide
369
xCAT3 Documentation, Release 2.12
nodelist.groups
mgt
nodehm.mgt
nodels can also select based on table value criteria. The following operators are available:
==
Select nodes where the table.column value is exactly a certain value.
!=
Select nodes where the table.column value is not a given specific value.
=~
Select nodes where the table.column value matches a given regular expression.
!~
Select nodes where the table.column value does not match a given regular expression.
The nodels command with a specific node and one or more table.attribute parameters is a good substitute for grep’ing
through the tab files, as was typically done in xCAT 1.x. This is because nodels will translate any regular expression
rows in the tables into their meaning for the specified node. The tab* commands will not do this, instead they will just
display the regular expression row verbatim.
OPTIONS
-v|--version
Command Version.
-H|--with-fieldname
Force display of table name and column name context for each result
-b|--blame
For values inherited from groups, display which groups provided the inheritance
-S
List all the hidden nodes (FSP/BPA nodes) with other ones.
-?|-h|--help
Display usage message.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To list all defined nodes, enter:
370
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
nodels
Output is similar to:
node1
node2
node3
2. To list all defined attributes in a table for a node or noderange, enter:
nodels rra001a noderes
Output is similar to:
rra001a:
rra001a:
rra001a:
rra001a:
rra001a:
rra001a:
noderes.primarynic: eth0
noderes.xcatmaster: rra000
noderes.installnic: eth0
noderes.netboot: pxe
noderes.servicenode: rra000
noderes.node: rra001a
3. To list nodes in node group ppc, enter:
nodels ppc
Output is similar to:
ppcnode1
ppcnode2
ppcnode3
4. To list the groups each node is part of:
nodels all groups
Output is similar to:
node1: groups: all
node2: groups: all,storage
node3: groups: all,blade
5. To list the groups each node is part of:
nodels all nodehm.power
Output is similar to:
node1: nodehm.power: blade
node2: nodehm.power: ipmi
node3: nodehm.power: ipmi
6. To list the out-of-band mgt method for blade1:
nodels blade1 nodehm.mgt
Output is similar to:
1.3. Admin Guide
371
xCAT3 Documentation, Release 2.12
blade1: blade
7. Listing blades managed through an AMM named ‘amm1’
nodels all mp.mpa==amm1
Output is similar to:
blade1
blade10
blade11
blade12
blade13
blade2
blade3
blade4
blade5
blade6
blade7
blade8
blade9
8. Listing the switch.switch value for nodes in the second rack:
nodels all nodepos.rack==2 switch.switch
Output is similar to:
n41:
n42:
n43:
n44:
n45:
n46:
n47:
n55:
n56:
n57:
n58:
n59:
n60:
switch.switch:
switch.switch:
switch.switch:
switch.switch:
switch.switch:
switch.switch:
switch.switch:
switch.switch:
switch.switch:
switch.switch:
switch.switch:
switch.switch:
switch.switch:
switch2
switch2
switch2
switch2
switch2
switch2
switch2
switch2
switch2
switch2
switch2
switch2
switch2
9. Listing the blade slot number for anything managed through a device with a name beginning with amm:
nodels all mp.mpa=~/^amm.*/ mp.id
Output looks like:
blade1: mp.id: 1
blade10: mp.id: 10
blade11: mp.id: 11
blade12: mp.id: 12
blade13: mp.id: 13
blade2: mp.id: 2
blade3: mp.id: 3
blade4: mp.id: 4
blade5: mp.id: 5
blade6: mp.id: 6
372
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
blade7: mp.id: 7
blade8: mp.id: 8
blade9: mp.id: 9
10. To list the hidden nodes that can’t be seen with other flags. The hidden nodes are FSP/BPAs.
lsdef -S
FILES
/opt/xcat/bin/nodels
SEE ALSO
noderange(3)|noderange.3, tabdump(8)|tabdump.8, lsdef(1)|lsdef.1
nodepurge.1
NAME
nodepurge - Removes nodes.
SYNOPSIS
nodepurge [-h| --help | -v | --version]
nodepurge noderange
DESCRIPTION
The nodepurge automatically removes all nodes from the database and any related configurations used by the node.
After the nodes are removed, the configuration files related to these nodes are automatically updated, including the
following files: /etc/hosts, DNS, DHCP. Any kits that are used by the nodes are triggered to automatically update kit
configuration and services.
OPTIONS
-h|--help
Display usage message.
-v|--version
Command Version
noderange
The nodes to be removed.
1.3. Admin Guide
373
xCAT3 Documentation, Release 2.12
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
To remove nodes compute-000 and compute-001, use the following command:
nodepurge compute-000,compute-001
SEE ALSO
nodeimport(1)|nodeimport.1,
noderange(3)|noderange.3
nodechprofile(1)|nodechprofile.1,
noderefresh(1)|noderefresh.1,
noderefresh.1
NAME
noderefresh - Update nodes configurations by running associated kit plugins.
SYNOPSIS
noderefresh [-h| --help | -v | --version]
noderefresh noderange
DESCRIPTION
The noderefresh command will update nodes settings, it will call all associated kit plug-in configurations and also
services
OPTIONS
-h|--help
Display usage message.
-v|--version
Command Version.
noderange
The nodes to be updated.
374
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
noderefresh compute-000,compute-001
SEE ALSO
nodeimport(1)|nodeimport.1, nodechprofile(1)|nodechprofile.1, nodepurge(1)|nodepurge.1, noderange(3)|noderange.3
noderm.1
NAME
noderm -Removes the nodes in the noderange from all database table.
SYNOPSIS
noderm [-h| --help]
noderm noderange
DESCRIPTION
The noderm command removes the nodes in the input node range.
OPTIONS
-h|--help Display usage message.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To remove the nodes in noderange node1-node4, enter:
noderm node1-node4
1.3. Admin Guide
375
xCAT3 Documentation, Release 2.12
FILES
/opt/xcat/bin/noderm
SEE ALSO
nodels(1)|nodels.1, nodeadd(8)|nodeadd.8, noderange(3)|noderange.3
nodestat.1
Name
nodestat - display the running status of each node in a noderange
Synopsis
nodestat [noderange] [-m | --usemon] [-p | --powerstat] [-f] [-u | --updatedb]
nodestat [-h | --help | -v | --version]
Description
nodestat displays and optionally updates the database the running status of a single or range of nodes or groups. See
noderange(3)|noderange.3.
By default, it works as following:
1. gets the sshd,pbs_mom,xend port status;
2. if none of them are open, it gets the fping status;
3. for pingable nodes that are in the middle of deployment, it gets the deployment status;
4. for non-pingable nodes, it shows ‘noping’.
When -m is specified and there are settings in the monsetting table, it displays the status of the applications specified
in the monsetting table. When -p is specified it shows the power status for the nodes that are not pingable. When -u is
specified it saves the status info into the xCAT database. Node’s pingable status and deployment status is saved in the
nodelist.status column. Node’s application status is saved in the nodelist.appstatus column.
To specify settings in the monsetting table, use ‘xcatmon’ as the name, ‘apps’ as the key and the value will be a list of
comma separated list of application names. For each application, you can specify the port number that can be queried
on the nodes to get the running status. Or you can specify a command that can be called to get the node status from.
The command can be a command that can be run locally at the management node or the service node for hierarchical
cluster, or a command that can be run remotely on the nodes.
The following is an example of the settings in the monsetting table:
name key value
xcatmon apps ssh,ll,gpfs,someapp
xcatmon gpfs cmd=/tmp/mycmd,group=compute,group=service
xcarmon ll port=9616,group=compute
xcatmon someapp dcmd=/tmp/somecmd
376
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
Keywords to use:
apps -- a list of comma separated application names whose status will be queried. For
˓→how to get the status of each app, look for app name in the key field in a
˓→different row.
port -- the application daemon port number, if not specified, use internal list, then
˓→/etc/services.
group -- the name of a node group that needs to get the application status from. If
˓→not specified, assume all the nodes in the nodelist table. To specify more than one
˓→groups, use group=a,group=b format.
cmd -- the command that will be run locally on mn or sn.
lcmd -- the command that will be run the mn only.
dcmd -- the command that will be run distributed on the nodes using xdsh <nodes> ....
For commands specified by ‘cmd’ and ‘lcmd’, the input of is a list of comma separated node names, the output must
be in the following format:
node1:string1
node2:string2
...
For the command specified by ‘dcmd’, no input is needed, the output can be a string.
Options
-f
Uses fping instead of nmap even if nmap is available. If you seem to be having a problem with false
negatives, fping can be more forgiving, but slower.
-m | --usemon
Uses the settings from the monsetting table to determine a list of applications that need to get status for.
-p | --powerstat
Gets the power status for the nodes that are ‘noping’.
-u | --updatedb
Updates the status and appstatus columns of the nodelist table with the returned running status from the
given nodes.
-v | --version
Print version.
-h | --help
Print help.
Examples
1. nodestat compute
Output is similar to:
1.3. Admin Guide
377
xCAT3 Documentation, Release 2.12
node1
node2
node3
node4
node5
sshd
sshd
ping
pbs
noping
2. nodestat compute -p
Output is similar to:
node1
node2
node3
node4
node5
sshd
sshd
ping
pbs
noping(Shutting down)
3. nodestat compute -u
Output is similar to:
node1
node2
node3
node4
node5
sshd
sshd
ping
netboot
noping
4. nodestat compute -m
Output is similar to:
node1
node2
node3
node4
ping,sshd,ll,gpfs=ok
ping,sshd,ll,gpfs=not ok,someapp=something is wrong
netboot
noping
See Also
noderange(3)|noderange.3, nodels(1)|nodels.1, nodeset(8)|nodeset.8
packimage.1
NAME
packimage - Packs the stateless image from the chroot file system.
SYNOPSIS
packimage [-h| --help]
packimage [-v| --version]
packimage [-m | --method cpio|tar] [-c | --compress gzip|pigz|xz] imagename
378
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
DESCRIPTION
Packs the stateless image from the chroot file system into a file to be sent to the node for a diskless boot.
Note: For an osimage that is deployed on a cluster, running packimage will overwrite the existing rootimage file and
be unavailable to the compute nodes while packimage is running.
PARAMETERS
imagename specifies the name of a OS image definition to be used. The specification for the image is stored in the
osimage table and linuximage table.
OPTIONS
-h Display usage message.
-v Command Version.
-m| --method Archive Method (cpio,tar,squashfs, default is cpio)
-c| --compress Compress Method (pigz,gzip,xz, default is pigz/gzip)
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To pack the osimage ‘rhels7.1-x86_64-netboot-compute’:
packimage rhels7.1-x86_64-netboot-compute
2. To pack the osimage ‘rhels7.1-x86_64-netboot-compute’ with “tar” to archive and “pigz” to compress:
packimage -m tar -c pigz rhels7.1-x86_64-netboot-compute
FILES
/opt/xcat/sbin/packimage
NOTES
This command is part of the xCAT software product.
SEE ALSO
genimage(1)|genimage.1
1.3. Admin Guide
379
xCAT3 Documentation, Release 2.12
pasu.1
NAME
pasu - run the ASU to many nodes in parallel
SYNOPSIS
pasu [-V] [-d] [-l user] [-p passwd] [-f fanout] [-i hostname-suffix] noderange command
pasu [-V] [-d] [-l user] [-p passwd] [-f fanout] [-i hostname-suffix] -b batchfile noderange
pasu [-h | --help]
DESCRIPTION
The pasu command runs the ASU command in out-of-band mode in parallel to multiple nodes. Out-of-band mode
means that ASU connects from the xCAT management node to the IMM (BMC) of each node to set or query the ASU
settings. To see all of the ASU settings available on the node, use the “show all” command. To query or set multiple
values, use the -b (batch) option. To group similar output from multiple nodes, use xcoll(1)|xcoll.1.
Before running pasu, you must install the ASU RPM from IBM. You can download it from the IBM Fix Central site.
You also must configure the IMMs properly according to xCAT documentation. Run “rpower noderange stat” to
confirm that the IMMs are configured properly.
OPTIONS
-l|--loginname username
The username to use to connect to the IMMs. If not specified, the row in the xCAT passwd table with key
“ipmi” will be used to get the username.
-p|--passwd passwd
The password to use to connect to the IMMs. If not specified, the row in the xCAT passwd table with key
“ipmi” will be used to get the password.
-f|--fanout
How many processes to run in parallel simultaneously. The default is 64. You can also set the XCATPSHFANOUT environment variable.
-b|--batch -batchfile
A simple text file that contains multiple ASU commands, each on its own line.
-d|--donotfilter
By default, pasu filters out (i.e. does not display) the standard initial output from ASU:
IBM Advanced Settings Utility version 9.30.79N
Licensed Materials - Property of IBM
(C) Copyright IBM Corp. 2007-2012 All Rights Reserved
Connected to IMM at IP address node2-imm
If you want this output to be displayed, use this flag.
380
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
-i|--interface hostname-suffix
The hostname suffix to be appended to the node names.
-V|--verbose
Display verbose messages.
-h|--help
Display usage message.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To display the Com1ActiveAfterBoot setting on 2 nodes:
pasu node1,node2 show DevicesandIOPorts.Com1ActiveAfterBoot
Output is similar to:
node1: DevicesandIOPorts.Com1ActiveAfterBoot=Enable
node2: DevicesandIOPorts.Com1ActiveAfterBoot=Enable
2. To display the Com1ActiveAfterBoot setting on all compute nodes:
pasu compute show DevicesandIOPorts.Com1ActiveAfterBoot | xcoll
Output is similar to:
====================================
compute
====================================
DevicesandIOPorts.Com1ActiveAfterBoot=Enable
3. To set several settings on all compute nodes, create a batch file called (for example) asu-settings with contents:
set
set
set
set
DevicesandIOPorts.Com1ActiveAfterBoot Enable
DevicesandIOPorts.SerialPortSharing Enable
DevicesandIOPorts.SerialPortAccessMode Dedicated
DevicesandIOPorts.RemoteConsole Enable
Then run:
pasu -b asu-settings compute | xcoll
Output is similar to:
====================================
compute
====================================
Batch mode start.
[set DevicesandIOPorts.Com1ActiveAfterBoot Enable]
1.3. Admin Guide
381
xCAT3 Documentation, Release 2.12
DevicesandIOPorts.Com1ActiveAfterBoot=Enable
[set DevicesandIOPorts.SerialPortSharing Enable]
DevicesandIOPorts.SerialPortSharing=Enable
[set DevicesandIOPorts.SerialPortAccessMode Dedicated]
DevicesandIOPorts.SerialPortAccessMode=Dedicated
[set DevicesandIOPorts.RemoteConsole Enable]
DevicesandIOPorts.RemoteConsole=Enable
Beginning intermediate batch update.
Waiting for command completion status.
Command completed successfully.
Completed intermediate batch update.
Batch mode competed successfully.
4. To confirm that all the settings were made on all compute nodes, create a batch file called (for example) asu-show
with contents:
show
show
show
show
DevicesandIOPorts.Com1ActiveAfterBoot
DevicesandIOPorts.SerialPortSharing
DevicesandIOPorts.SerialPortAccessMode
DevicesandIOPorts.RemoteConsole
Then run:
pasu -b asu-show compute | xcoll
Output is similar to:
====================================
compute
====================================
Batch mode start.
[show DevicesandIOPorts.Com1ActiveAfterBoot]
DevicesandIOPorts.Com1ActiveAfterBoot=Enable
[show DevicesandIOPorts.SerialPortSharing]
DevicesandIOPorts.SerialPortSharing=Enable
[show DevicesandIOPorts.SerialPortAccessMode]
DevicesandIOPorts.SerialPortAccessMode=Dedicated
[show DevicesandIOPorts.RemoteConsole]
DevicesandIOPorts.RemoteConsole=Enable
Batch mode competed successfully.
FILES
/opt/xcat/bin/pasu
382
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
SEE ALSO
noderange(3)|noderange.3, rpower(1)|rpower.1, xcoll(1)|xcoll.1
pcons.1
SYNOPSIS
pcons noderange command
pcons[-h | --help]
pcons[-v | --version]
DESCRIPTION
Runs the a command to the noderange using the console.
EXAMPLES
pcons 1,3 stat
pcons all,-129-256 stat
SEE ALSO
psh(1)|psh.1
pgsqlsetup.1
NAME
pgsqlsetup - Sets up the PostgreSQL database for xCAT to use.
SYNOPSIS
pgsqlsetup {-h | --help}
pgsqlsetup {-v | --version}
pgsqlsetup {-i | --init} [-N | --nostart] [-P | --PCM] [-o | --odbc] [-V | --verbose]
pgsqlsetup {-o | --setupODBC} [-V | --verbose]
1.3. Admin Guide
383
xCAT3 Documentation, Release 2.12
DESCRIPTION
pgsqlsetup - Sets up the PostgreSQL database for xCAT to use. The pgsqlsetup script is run on the Management Node
as root after the PostgreSQL code has been installed. The xcatd daemon will be stopped during migration. No xCAT
commands should be run during the init process, because we will be migrating the xCAT database to PostgreSQL and
restarting the xcatd daemon as well as the PostgreSQL daemon. For full information on all the steps that will be done
reference One password must be supplied for the setup, a password for the xcatadm unix id and the same password for
the xcatadm database id. The password will be prompted for interactively or you can set the XCATPGPW environment
variable to the password and then there will be no prompt.
OPTIONS
-h|--help
Displays the usage message.
-v|--version
Displays the release version of the code.
-V|--verbose
Displays verbose messages.
-i|--init
The init option is used to setup an installed PostgreSQL database so that xCAT can use the database.
This involves creating the xcat database, the xcat admin id, allowing access to the xcatdb database by
the Management Node. It customizes the postgresql.conf configuration file, adds the management server
to the pg_hba.conf and starts the PostgreSQL server. It also backs up the current xCAT database and
restores it into the newly setup xcatdb PostgreSQL database. It creates the /etc/xcat/cfgloc file to point
the xcatd daemon to the PostgreSQL database and restarts the xcatd daemon using the database. On AIX,
it additionally setup the xcatadm unix id and the postgres id and group. For AIX, you should be using
the PostgreSQL rpms available from the xCAT website. For Linux, you should use the PostgreSQL rpms
shipped with the OS. You can chose the -o option, to run after the init. To add additional nodes to access
the PostgreSQL server, setup on the Management Node, edit the pg_hba.conf file.
For more documentation see:Setting_Up_PostgreSQL_as_the_xCAT_DB
-N|--nostart
This option with the -i flag will create the database, but will not backup and restore xCAT tables into the
database. It will create the cfgloc file such that the next start of xcatd will try and contact the database.
This can be used to setup the xCAT PostgreSQL database during or before install.
-P|--PCM
This option sets up PostgreSQL database to be used with xCAT running with PCM.
-o|--odbc
This option sets up the ODBC /etc/../odbcinst.ini, /etc/../odbc.ini and the .odbc.ini file in roots home
directory will be created and initialized to run off the xcatdb PostgreSQL database.
ENVIRONMENT VARIABLES
XCATPGPW
The password to be used to setup the xCAT admin id for the database.
384
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
EXAMPLES
1. To setup PostgreSQL for xCAT to run on the PostgreSQL xcatdb database :
pgsqlsetup -i
2. To setup the ODBC for PostgreSQL xcatdb database access :
pgsqlsetup -o
piflash.1
SYNOPSIS
piflash <noderange> -****-package <filename>
DESCRIPTION
piflash Remotely applies firmware updates to servers.
pping.1
SYNOPSIS
pping [-i | --interface interfaces] [-f | --use_fping] noderange
pping [-h | --help]
pping {-v | --version}
DESCRIPTION
pping is a utility used to ping a list of nodes in parallel. pping will return an unsorted list of nodes with a ping or
noping status. pping front-ends nmap or fping if available.
This command does not support the xcatd client/server communication. It must be run on the management node.
OPTIONS
-i | --interface interfaces
A comma separated list of network interface names that should be pinged instead of the interface represented by the nodename/hostname. The following name resolution convention is assumed: an interface
is reachable by the hostname <nodename>-<interface>. For example, the ib2 interface on node3 has a
hostname of node3-ib2.
If more than one interface is specified, each interface will be combined with the nodenames as described
above and will be pinged in turn.
-f | --use_fping
Use fping instead of nmap
1.3. Admin Guide
385
xCAT3 Documentation, Release 2.12
-h | --help
Show usage information.
-v | --version
Display the installed version of xCAT.
EXAMPLES
1. pping all
Output is similar to:
node1: ping
node2: ping
node3: noping
2. pping all -i ib0,ib1
Output is similar to:
node1-ib0:
node2-ib0:
node3-ib0:
node1-ib1:
node2-ib1:
node3-ib1:
ping
ping
noping
ping
ping
noping
SEE ALSO
psh(1)|psh.1, noderange(3)|noderange.3
ppping.1
SYNOPSIS
ppping [-i | --interface interfaces] [-d | --debug] [-V | --verbose] [-q | --quiet] [-s | --serial] noderange
ppping [-h | --help]
pping {-v | --version}
DESCRIPTION
ppping is a utility used to test the connectivity between nodes in the noderange using ping. By default, ppping will
return an unsorted list of the node pairs that are not able to ping each other, or a message that all nodes are pingable.
More or less output can be controlled by the -V and -q options. ppping front-ends pping and xdsh.
386
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
OPTIONS
-s
Ping serially instead of in parallel.
-i | --interface interfaces
A comma separated list of network interface names that should be pinged instead of the interface represented by the nodename/hostname. The following name resolution convention is assumed: an interface
is reachable by the hostname <nodename>-<interface>. For example, the ib2 interface on node3 has a
hostname of node3-ib2.
If more than one interface is specified, each interface will be combined with the nodenames as described
above and will be pinged in turn.
-V | --verbose
Display verbose output. The result of every ping attempt from every node will be displayed. Without this
option, just a summary of the successful pings are displayed, along with all of the unsuccessful pings.
-q | --quiet
Display minimum output: just the unsuccessful pings. This option has the effect that if all pings are
successful, nothing is displayed. But it also has the performance benefit that each node does not have to
send successful ping info back to the management node.
-d | --debug
Print debug information.
-h | --help
Show usage information.
-v | --version
Display the installed version of xCAT.
EXAMPLES
1. ppping all -q
Output is similar to:
blade7: node2: noping
blade8: node2: noping
blade9: node2: noping
devmaster: node2: noping
node2: noping
2. ppping node1,node2 -i ib0,ib1,ib2,ib3
Output is similar to:
node1:
node1:
node1:
node1:
node2:
pinged
pinged
pinged
pinged
pinged
1.3. Admin Guide
all
all
all
all
all
nodes
nodes
nodes
nodes
nodes
successfully
successfully
successfully
successfully
successfully
on
on
on
on
on
interface
interface
interface
interface
interface
ib0
ib1
ib2
ib3
ib0
387
xCAT3 Documentation, Release 2.12
node2: pinged all nodes successfully on interface ib1
node2: pinged all nodes successfully on interface ib2
node2: pinged all nodes successfully on interface ib3
SEE ALSO
psh(1)|psh.1, pping(1)|pping.1
prsync.1
Name
prsync - parallel rsync
Synopsis
prsync filename [filename ...] noderange:destinationdirectory
prsync [-o rsyncopts] [-f fanout] [filename filename ...] [directory directory ...] noderange:destinationdirectory
prsync {-h | --help | -v | --version}
Description
prsync is a front-end to rsync for a single or range of nodes and/or groups in parallel.
Note: this command does not support the xcatd client/server communication and therefore must be run on the management node. It does not support hierarchy, use xdcp -F to run rsync from the management node to the compute node
via a service node
prsync is NOT multicast, but is parallel unicasts.
Options
-o rsyncopts
rsync options. See rsync(1).
-f fanout
Specifies a fanout value for the maximum number of concurrently executing remote shell processes.
filename
A space delimited list of files to rsync.
directory
A space delimited list of directories to rsync.
noderange:destination
A noderange(3)|noderange.3 and destination directory. The : is required.
-h | --help
388
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
Print help.
-v | --version
Print version.
XCATPSHFANOUT
Specifies the fanout value. This variable is overridden by the -f flag. Default is 64.
Examples
1. cd /install; prsync -o "crz" post stage:/install
2. prsync passwd group rack01:/etc
See Also
noderange(3)|noderange.3, pscp(1)|pscp.1, pping(1)|pping.1, psh(1)|psh.1
pscp.1
Name
pscp - parallel remote copy
Synopsis
pscp [-i suffix] [scp options ...] [-f fanout] filename [filename ...] noderange:destinationdirectory
pscp {-h | --help | -v | --version}
Description
pscp is a utility used to copy a single or multiple set of files and/or directories to a single or range of nodes and/or
groups in parallel.
pscp is a front-end to the remote copy scp.
Note: this command does not support the xcatd client/server communication and therefore must be run on the management node. It does not support hierarchy, use xdcp to run remote copy command from the management node to the
compute node via a service node.
pscp is NOT multicast, but is parallel unicasts.
Options
-f fanout
Specifies a fanout value for the maximum number of concur- rently executing remote shell processes.
-i suffix
1.3. Admin Guide
389
xCAT3 Documentation, Release 2.12
Interfaces to be used.
scp options
See scp(1)
filename
A space delimited list of files to copy. If -r is passed as an scp option, directories may be specified as well.
noderange:destination
A noderange(3)|noderange.3 and destination directory. The : is required.
-h | --help
Print help.
-v | --version
Print version.
XCATPSHFANOUT
Specifies the fanout value. This variable is overridden by the -f flag. Default is 64.
Examples
1. pscp -r /usr/local node1,node3:/usr/local
2. pscp passwd group rack01:/etc
See Also
noderange(3)|noderange.3, pping(1)|pping.1, prsync(1)|prsync.1, psh(1)|psh.1
psh.1
Name
psh - parallel remote shell
Synopsis
psh [-i interface] [-f fanout] [-l user] noderange command
psh {-h | --help | -v | --version}
Description
psh is a utility used to run a command across a list of nodes in parallel.
ssh must be set up to allow no prompting for psh to work.
Note:
390
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
This command does not run through xcatd like most xCAT commands do. This means you must either run it on the
management node, or have a network connection between your machine and the nodes. It does not support hierarchy,
use xdsh to run remote command from the management node to the compute node via a service node.
psh arguments need to precede noderange, otherwise, you will get unexpected errors.
Options
-i interface
The NIC on the node that psh should communicate with. For example, if interface is eth1, then psh will
concatenate -eth1 to the end of every node name before ssh’ing to it. This assumes those host names have
been set up to resolve to the IP address of each of the eth1 NICs.
-f fanout
Specifies a fanout value for the maximum number of concur- rently executing remote shell processes.
-l user
Log into the nodes as the specified username. The default is to use the same username as you are running
the psh command as.
-n|--nonodecheck
Do not send the noderange to xcatd to expand it into a list of nodes. Instead, use the noderange exactly
as it is specified. In this case, the noderange must be a simple list of comma-separated hostnames of the
nodes. This allows you to run psh even when xcatd is not running.
noderange
See noderange(3)|noderange.3.
command
Command to be run in parallel. If no command is give then pshenters interactive mode. In interactive mode a “>” prompt is displayed. Any command entered is executed in parallel to the nodes in the
noderange. Use “exit” or “Ctrl-D” to end the interactive session.
-h | --help
Print help.
Environment Variables
XCATPSHFANOUT
Specifies the fanout value. This variable is overridden by the -f flag. Default is 64.
Examples
1. Run uptime on 3 nodes:
psh node4-node6 uptime
Output is similar to:
1.3. Admin Guide
391
xCAT3 Documentation, Release 2.12
node4: Sun Aug
node5: Sun Aug
node6: Sun Aug
5 17:42:06 MDT 2001
5 17:42:06 MDT 2001
5 17:42:06 MDT 2001
2. Run a command on some BladeCenter management modules:
psh amm1-amm5 'info -T mm[1]'
3. Remove the tmp files on the nodes in the 1st frame:
psh rack01 'rm -f /tmp/*'
Notice the use of ‘’ to forward shell expansion. This is not necessary in interactive mode.
See Also
noderange(3)|noderange.3, pscp(1)|pscp.1, pping(1)|pping.1, prsync(1)|prsync.1
pushinitrd.1
NAME
pushinitrd - queries your SoftLayer account and gets attributes for each server.
SYNOPSIS
pushinitrd [-v | --verbose] [-w waittime] [noderange]
pushinitrd [-? | -h | --help]
DESCRIPTION
The pushinitrd command copies the initrd, kernel, params, and static IP info to nodes, so they can be net installed
even across vlans (w/o setting up pxe/dhcp broadcast relay). This assumes a working OS is on the nodes. Before
running this command, you must run nodeset for these nodes. All of the nodes given to one invocation of pushinitrd
must be using the same osimage.
Before using this command, if will be most convenient if you exchange the ssh keys using:
xdsh <noderange> -K
OPTIONS
-w waittime
The number of seconds the initrd should wait before trying to communicate over the network. The default
is 75. This translates into the netwait kernel parameter and is usually needed in a SoftLayer environment
because it can take a while for a NIC to be active after changing state.
-?|-h|--help
392
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
Display usage message.
-v|--version
Command Version.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. Configure nodes for net installing in a SoftLayer environment:
pushinitrd <noderange>
FILES
/opt/xcat/bin/pushinitrd
SEE ALSO
getslnodes(1)|getslnodes.1
rbeacon.1
SYNOPSIS
rbeacon noderange {on | blink | off | stat}
rbeacon [-h | --help]
rbeacon {-v | --version}
DESCRIPTION
rbeacon Turns beacon (a light on the front of the physical server) on/off/blink or gives status of a node or noderange.
EXAMPLES
rbeacon
rbeacon
rbeacon
rbeacon
1,3 off
14-56,70-203 on
1,3,14-56,70-203 blink
all,-129-256 stat
1.3. Admin Guide
393
xCAT3 Documentation, Release 2.12
SEE ALSO
noderange(3)|noderange.3, rpower(1)|rpower.1
rbootseq.1
SYNOPSIS
rbootseq [-h | --help | -v | --version]
Blade specific:
rbootseq noderange {hd0 | hd1 | hd2 | hd3 | net | iscsi | iscsicrit | cdrom | usbflash | floppy | none | list | stat},...
HP Blade specific:
rbootseq noderange {hd | net1 | net2 | net3 | net4 | cdrom | usbflash | floppy | none},...
PPC (using Direct FSP Management) specific:
rbootseq noderange [hfi|net]
DESCRIPTION
For Blade specific:
rbootseq sets the boot sequence (the order in which boot devices should be tried) for the specified blades. Up to four
different medium/devices can be listed, separated by commas. The boot sequence will remain in effect for these blades
until set differently.
For PPC (using Direct FSP Management) specific:
rbootseq sets the ethernet (net) or hfi device as the first boot device for the specified PPC LPARs. The rbootseq
command requires that the ethernet or hfi mac address is stored in the mac table, and that the network information is
correct in the networks table.
OPTIONS
hd0 | harddisk0 | hd | harddisk
The first hard disk.
hd1 | harddisk1
The second hard disk.
hd2 | harddisk2
The third hard disk.
hd3 | harddisk3
394
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
The fourth hard disk.
n | net | network
Boot over the ethernet network, using a PXE or BOOTP broadcast.
n | net | network | net1 | nic1 (HP Blade Only)
Boot over the first ethernet network, using a PXE or BOOTP broadcast.
net2 | nic2 (HP Blade Only)
Boot over the second ethernet network, using a PXE or BOOTP broadcast.
net3 | nic3 (HP Blade Only)
Boot over the third ethernet network, using a PXE or BOOTP broadcast.
net3 | nic3 (HP Blade Only)
Boot over the fourth ethernet network, using a PXE or BOOTP broadcast.
hfi
Boot p775 nodes over the HFI network, using BOOTP broadcast.
iscsi
Boot to an iSCSI disk over the network.
iscsicrit
??
cd | cdrom
The CD or DVD drive.
usbflash | usb | flash
A USB flash drive.
floppy
The floppy drive.
none
If it gets to this part of the sequence, do not boot. Can not be specified 1st, or before any real boot devices.
list | stat
Display the current boot sequence.
EXAMPLES
1. Set blades 14-56 and 70-203 to try to boot first from the CD drive, then the floppy drive, then the network, and
finally from the 1st hard disk:
rbootseq blade[14-56],blade[70-203] c,f,n,hd0
SEE ALSO
rsetboot(1)|rsetboot.1
1.3. Admin Guide
395
xCAT3 Documentation, Release 2.12
rcons.1
Name
rcons - remotely accesses the serial console of a node
Synopsis
rcons singlenode [conserver-host] [-f] [-s]
rcons [-h | --help | -v | --version]
Description
rcons provides access to a single remote node serial console, using the out-of-band infrastructure for the node (e.g.
BMC, Management Module, HMC, KVM, etc.). It uses the conserver open source package to provide one read-write
and multiple read-only instances of the console, plus console logging.
If conserver-host is specified, the conserver daemon on that host will be contacted, instead of on the local host.
To exit the console session, enter: <ctrl>e c .
Options
-f
If another console for this node is already open in read-write mode, force that console into read-only (spy)
mode, and open this console in read-write mode. If -f is not specified, this console will be put in spy mode
if another console is already open in read-write mode. The -f flag can not be used with the -s flag.
-s
Open the console in read-only (spy) mode, in this mode all the escape sequences work, but all other
keyboard input is discarded. The -s flag can not be used with the -f flag.
-h | --help
Print help.
-v | --version
Print version.
Files
nodehm table - xCAT node hardware management table. See nodehm(5)|nodehm.5 for further details. This is used to
determine the console access method.
Examples
rcons node5
396
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
See Also
wcons(1)|wcons.1
regnotif.1
NAME
regnotif - Registers a Perl module or a command that will get called when changes occur in the desired xCAT database
tables.
SYNOPSIS
regnotif [-h| --help]
regnotif [-v| --version]
regnotif filename tablename[,tablename]... [-o | --operation actions]
DESCRIPTION
This command is used to register a Perl module or a command to the xCAT notification table. Once registered, the
module or the command will get called when changes occur in the xCAT database tables indicated by tablename. The
changes can be row addition, deletion and update which are specified by actions.
PARAMETERS
filename is the path name of the Perl module or command to be registered. tablename is the name of the table that the
user is interested in.
OPTIONS
-h | --help Display usage message.
-v | --version Command Version.
-V | --verbose Verbose output.
-o | --operation specifies the database table actions that the user is interested in. It is a comma separated list. ‘a’ for
row addition, ‘d’ for row deletion and ‘u’ for row update.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
1.3. Admin Guide
397
xCAT3 Documentation, Release 2.12
EXAMPLES
1. To register a Perl module that gets invoked when rows get added or deleted. in the nodelist and the nodehm
tables, enter:
regnotif /opt/xcat/lib/perl/xCAT_monitoring/mycode.pm nodelist,nodhm -o a,d
2. To register a command that gets invoked when rows get updated in the switch table, enter:
regnotif /usr/bin/mycmd switch
-o u
FILES
/opt/xcat/bin/regnotif
SEE ALSO
unregnotif(1)|unregnotif.1
renergy.1
NAME
renergy - remote energy management tool
SYNOPSIS
renergy [-h | --help]
renergy [-v | --version]
Power 6 server specific :
renergy noderange [-V] {all | [savingstatus] [cappingstatus] [cappingmaxmin] [cappingvalue] [cappingsoftmin]
[averageAC] [averageDC] [ambienttemp] [exhausttemp] [CPUspeed] [syssbpower] [sysIPLtime]}
renergy noderange [-V] {savingstatus={on | off} | cappingstatus={on | off} | cappingwatt=watt | cappingperc=percentage}
Power 7 server specific :
renergy noderange [-V] {all | [savingstatus] [dsavingstatus] [cappingstatus] [cappingmaxmin] [cappingvalue]
[cappingsoftmin] [averageAC] [averageDC] [ambienttemp] [exhausttemp] [CPUspeed] [syssbpower] [sysIPLtime] [fsavingstatus] [ffoMin] [ffoVmin] [ffoTurbo] [ffoNorm] [ffovalue]}
renergy noderange [-V] {savingstatus={on | off} | dsavingstatus={on-norm | on-maxp | off} | fsavingstatus={on |
off} | ffovalue=MHZ | cappingstatus={on | off} | cappingwatt=watt | cappingperc=percentage}
Power 8 server specific :
renergy noderange [-V] {all | [savingstatus] [dsavingstatus] [averageAC] [averageAChistory] [averageDC] [averageDChistory] [ambienttemp] [ambienttemphistory] [exhausttemp] [exhausttemphistory]
[fanspeed] [fanspeedhistory] [CPUspeed] [CPUspeedhistory] [syssbpower] [sysIPLtime] [fsavingstatus]
[ffoMin] [ffoVmin] [ffoTurbo] [ffoNorm] [ffovalue]}
398
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
renergy noderange [-V] {savingstatus={on | off} | dsavingstatus={on-norm | on-maxp | off} | fsavingstatus={on |
off} | ffovalue=MHZ }
NOTE: The setting operation for Power 8 server is only supported for the server which is running in PowerVM mode.
Do NOT run the setting for the server which is running in OPAL mode.
BladeCenter specific :
For Management Modules:
renergy noderange [-V] {all | pd1all | pd2all | [pd1status] [pd2status] [pd1policy] [pd2policy]
[pd1powermodule1] [pd1powermodule2] [pd2powermodule1] [pd2powermodule2] [pd1avaiablepower]
[pd2avaiablepower] [pd1reservedpower] [pd2reservedpower] [pd1remainpower] [pd2remainpower]
[pd1inusedpower] [pd2inusedpower] [availableDC] [averageAC] [thermaloutput] [ambienttemp] [mmtemp]}
For a blade server nodes:
renergy noderange [-V] {all | [averageDC] [capability] [cappingvalue] [CPUspeed] [maxCPUspeed] [savingstatus] [dsavingstatus]}
renergy noderange [-V] {savingstatus={on | off} | dsavingstatus={on-norm | on-maxp | off}}
Flex specific :
For Flex Management Modules:
renergy noderange [-V] {all | [powerstatus] [powerpolicy] [powermodule] [avaiablepower] [reservedpower] [remainpower] [inusedpower] [availableDC] [averageAC] [thermaloutput] [ambienttemp] [mmtemp]}
For Flex node (power and x86):
renergy noderange [-V] {all | [averageDC] [capability] [cappingvalue] [cappingmaxmin] [cappingmax] [cappingmin] [cappingGmin] [CPUspeed] [maxCPUspeed] [savingstatus] [dsavingstatus]}
renergy noderange [-V] {cappingstatus={on | off} | cappingwatt=watt | cappingperc=percentage | savingstatus={on | off} | dsavingstatus={on-norm | on-maxp | off}}
iDataPlex specific :
renergy noderange [-V] [{cappingmaxmin | cappingmax | cappingmin}] [cappingstatus] [cappingvalue] [relhistogram]}
renergy noderange [-V] {cappingstatus={on | enable | off | disable} | {cappingwatt|cappingvalue}=watt}
OpenPOWER server specific :
renergy noderange {powerusage | temperature}
DESCRIPTION
This renergy command can be used to manage the energy consumption of IBM servers which support IBM EnergyScale technology. Through this command, user can query and set the power saving and power capping status, and
also can query the average consumed energy, the ambient and exhaust temperature, the processor frequency for a
server.
renergy command supports IBM POWER6, POWER7 and POWER8 rack-mounted servers, BladeCenter management
modules, blade servers, and iDataPlex servers. For Power6 and Power7 rack-mounted servers, the following specific
hardware types are supported: 8203-E4A, 8204-E8A, 9125-F2A, 8233-E8B, 8236-E8C. For Power8 server, there’s no
hardware type restriction.
The parameter noderange needs to be specified for the renergy command to get the target servers. The noderange
should be a list of CEC node names, blade management module node names or blade server node names. Lpar name
is not acceptable here.
1.3. Admin Guide
399
xCAT3 Documentation, Release 2.12
renergy command can accept multiple of energy attributes to query or one of energy attribute to set. If only the
attribute name is specified, without the ‘=’, renergygets and displays the current value. Otherwise, if specifying the
attribute with ‘=’ like ‘savingstatus=on’, renergy will set the attribute savingstatus to value ‘on’.
The attributes listed in the SYNOPSIS section are which ones can be handled by renergy command. But for each
specific type of server, there are some attributes that are not supported. If user specifies an attribute which is not
supported by a specific server, the return value of this attribute will be ‘na’.
Note: the options powerusage and temperature are only supported for OpenPOWER servers.
The supported attributes for each specific system p hardware type is listed as follows:
8203-E4A, 8204-E8A
Supported attributes:
Query: savingstatus,cappingstatus,cappingmin,cappingmax, cappingvalue,cappingsoftmin,averageAC,averageDC,ambienttemp,
exhausttemp,CPUspeed,syssbpower,sysIPLtime
Set: savingstatus,cappingstatus,cappingwatt,cappingperc
9125-F2A
Supported attributes:
Query: savingstatus,averageAC,ambienttemp,exhausttemp, CPUspeed
Set: savingstatus
8233-E8B, 8236-E8C
Supported attributes:
Query: savingstatus,dsavingstatus,cappingstatus,cappingmin, cappingmax,cappingvalue,cappingsoftmin,averageAC,averageDC,
ambienttemp,exhausttemp,CPUspeed,syssbpower,sysIPLtime
Set: savingstatus,dsavingstatus,cappingstatus,cappingwatt, cappingperc
9125-F2C, 9119-FHB
Supported attributes:
Query: savingstatus,dsavingstatus,cappingstatus,cappingmin, cappingmax,cappingvalue,cappingsoftmin,averageAC,averageDC,
ambienttemp,exhausttemp,CPUspeed,syssbpower,sysIPLtime, fsavingstatus,ffoMin,ffoVmin,ffoTurbo,ffoNorm,ffovalue
Set: savingstatus,dsavingstatus,cappingstatus,cappingwatt, cappingperc,fsavingstatus,ffovalue
Non of Above
For the machine type which is not in the above list, the following attributes can be tried but not guaranteed:
Query: savingstatus,dsavingstatus,cappingstatus,cappingmin, cappingmax„cappingvalue,cappingsoftmin,averageAC,averageDC,
ambienttemp,exhausttemp,CPUspeed,syssbpower,sysIPLtime
Set: savingstatus,dsavingstatus,cappingstatus,cappingwatt, cappingperc
Note: For system P CEC nodes, each query operation for attribute CPUspeed, averageAC or averageDC needs about
30 seconds to complete. The query for others attributes will get response immediately.
PREREQUISITES
For the Power6 and Power7 nodes, the renergy command depends on the Energy Management Plugin xCAT-pEnergy
to communicate with server. xCAT-pEnergy can be downloaded from the IBM web site: http://www.ibm.com/
support/fixcentral/. (Other Software -> EM)
400
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
NOTE: Power8 nodes don’t need this specific energy management package.
For iDataPlex nodes, the renergy command depends on the Energy Management Plugin xCAT-xEnergy to communicate with server. This plugin must be requested from IBM.
(The support for BladeCenter energy management is built into base xCAT, so no additional plugins are needed for
BladeCenter.)
OPTIONS
-h | --help
Display the usage message.
-v | --version
Display the version information.
-V
Verbose output.
all
Query all energy attributes which supported by the specific type of hardware.
For Power8 machines, will not display the attributes for historical records.
pd1all
Query all energy attributes of the power domain 1 for blade management module node.
pd2all
Query all energy attributes of the power domain 2 for blade management module node.
ambienttemp
Query the current ambient temperature. (Unit is centigrade)
ambienttemphistory
Query the historical records which were generated in last one hour for ambienttemp.
availableDC
Query the total DC power available for the entire blade center chassis.
averageAC
Query the average power consumed (Input). (Unit is watt)
Note: For 9125-F2A,9125-F2C server, the value of attribute averageAC is the aggregate for all of the
servers in a rack.
Note: For Blade Center, the value of attribute averageAC is the total AC power being consumed by all
modules in the chassis. It also includes power consumed by the Chassis Cooling Devices for BCH chassis.
averageAChistory
Query the historical records which were generated in last one hour for averageAC.
averageDC
Query the average power consumed (Output). (Unit is watt)
averageDChistory
1.3. Admin Guide
401
xCAT3 Documentation, Release 2.12
Query the historical records which were generated in last one hour for averageDC.
capability
Query the Power Capabilities of the blade server.
staticPowerManagement: the module with the static worst case power values.
fixedPowermanagement: the module with the static power values but ability to throttle.
dynamicPowerManagement: the module with power meter capability, measurement enabled, but capping
disabled.
dynamicPowerMeasurement1: the module with power meter capability, measurement enabled, phase 1
only
dynamicPowerMeasurement2: the module with power meter capability, measurement enabled, phase 2 or
higher
dynamicPowerMeasurementWithPowerCapping: the module with power meter capability, both measurement and capping enabled, phase 2 or higher
cappingGmin
Query the Guaranteed Minimum power capping value in watts.
cappingmax
Query the Maximum of power capping value in watts.
cappingmaxmin
Query the Maximum and Minimum of power capping value in watts.
cappingmin
Query the Minimum of power capping value in watts.
cappingperc=percentage
Set the power capping value base on the percentage of the max-min of capping value which getting from
cappingmaxmim attribute. The valid value must be from 0 to 100.
cappingsoftmin
Query the minimum value that can be assigned to power capping without guaranteed enforceability. (Unit
is watt)
cappingstatus
Query the power capping status. The result should be ‘on’ or ‘off’.
cappingstatus={on | off}
Set the power capping status. The value must be ‘on’ or ‘off’. This is the switch to turn on or turn off the
power capping function.
cappingvalue
Query the current power capping value. (Unit is watt)
cappingwatt=watt
Set the power capping value base on the watt unit.
If the ‘watt’ > maximum of cappingmaxmin or ‘watt’ < cappingsoftmin, the setting operation will be
failed. If the ‘watt’ > cappingsoftmin and ‘watt’ < minimum of cappingmaxmin, the value can NOT be
guaranteed.
402
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
CPUspeed
Query the effective processor frequency. (Unit is MHz)
CPUspeedhistory
Query the historical records which were generated in last one hour for CPUspeed
dsavingstatus
Query the dynamic power saving status. The result should be ‘on-norm’, ‘on-maxp’ or ‘off’.
If turning on the dynamic power saving, the processor frequency and voltage will be dropped dynamically
based on the core utilization. It supports two modes for turn on state:
on-norm - means normal, the processor frequency cannot exceed the nominal value;
on-maxp - means maximum performance, the processor frequency can exceed the nominal value.
dsavingstatus={on-norm | on-maxp | off}
Set the dynamic power saving. The value must be ‘on-norm’, ‘on-maxp’ or ‘off’.
The dsavingstatus setting operation needs about 2 minutes to take effect. (The used time depends on the
hardware type)
The dsavingstatus only can be turned on when the savingstatus is in turn off status.
exhausttemp
Query the current exhaust temperature. (Unit is centigrade)
exhausttemphistory
Query the historical records which were generated in last one hour for exhausttemp
fanspeed
Query the fan speed for all the fans which installed in this node. (Unit is RPM - Rotations Per Minute))
If there are multiple fans for a node, multiple lines will be output. And a fan name in bracket will be
appended after fanspped attribute name.
fanspeedhistory
Query the historical records which were generated in last one hour for fanspeed.
ffoMin
Query the minimum cpu frequency which can be set for FFO. (Fixed Frequency Override)
ffoNorm
Query the maximum cpu frequency which can be set for FFO.
ffoTurbo
Query the advertised maximum cpu frequency (selling point).
ffoVmin
Query the minimum cpu frequency which can be set for dropping down the voltage to save power. That
means when you drop the cpu frequency from the ffoVmin to ffoVmin, the voltage won’t change, then
there’s no obvious power to be saved.
ffovalue
Query the current value of FFO.
ffovalue=MHZ
1.3. Admin Guide
403
xCAT3 Documentation, Release 2.12
Set the current value of FFO. The valid value of ffovalue should be between the ffoMin and ffoNorm.
Note1: Due to the limitation of firmware, the frequency in the range 3501 MHz - 3807 MHz can NOT be
set to ffovalue. This range may be changed in future.
Note2: The setting will take effect only when the fsavingstatus is in ‘on’ status. But you need to set the
ffovalue to a valid value before enabling the fsavingstatus. (It’s a limitation of the initial firmware and
will be fixed in future.)
The ffovalue setting operation needs about 1 minute to take effect.
fsavingstatus
Query the status of FFO. The result should be ‘on’ or ‘off’. ‘on’ - enable; ‘off’ - disable.
fsavingstatus={on | off}
Set the status of FFO. The value must be ‘on’ or ‘off’.
‘on’ - enable. It will take effect only when the ffovaluehas been set to a valid value.
‘off’ -disable. It will take effect immediately.
Note: See the Note2 of ffovalue=MHZ.
maxCPUspeed
Query the maximum processor frequency. (Unit is MHz)
mmtemp
Query the current temperature of management module. (Unit is centigrade)
pd1status | powerstatus
Query the status of power domain 1 for blade management module node.
Note: for the attribute without the leading ‘pd1’ which means there’s only one power doamin in the
chassis.
pd1policy | powerpolicy
Query the power management policy of power domain 1.
pd1powermodule1 | powermodule
Query the First Power Module capacity in power domain 1.
pd1powermodule2 | powermodule
Query the Second Power Module capacity in power domain 1.
pd1avaiablepower | avaiablepower
Query the total available power in power domain 1.
pd1reservedpower | reservedpower
Query the power that has been reserved for power domain 1.
pd1remainpower | remainpower
Query the remaining power available in power domain 1.
pd1inusedpower | inusedpower
Query the total power being used in power domain 1.
pd2status
404
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
Query the status of power domain 2 for blade management module node.
pd2policy
Query the power management policy of power domain 2.
pd2powermodule1
Query the First Power Module capacity in power domain 2.
pd2powermodule2
Query the Second Power Module capacity in power domain 2.
pd2avaiablepower
Query the total available power in power domain 2.
pd2reservedpower
Query the power that has been reserved for power domain 2.
pd2remainpower
Query the remaining power available in power domain 2.
pd2inusedpower
Query the total power being used in power domain 2.
relhistogram
Query histogram data for wattage information
savingstatus
Query the static power saving status. The result should be ‘on’ or ‘off’. ‘on’ - enable; ‘off’ - disable.
savingstatus={on | off}
Set the static power saving. The value must be ‘on’ or ‘off’.
If turning on the static power saving, the processor frequency and voltage will be dropped to a fixed value
to save energy.
The savingstatus setting operation needs about 2 minutes to take effect. (The used time depends on the
hardware type)
The savingstatus only can be turned on when the dsavingstatus is in turn off status.
sysIPLtime
Query the time used from FSP standby to OS standby. (Unit is Second)
syssbpower
Query the system power consumed prior to power on. (Unit is Watt)
thermaloutput
Query the thermal output (load) in BTUs per hour for the blade center chassis.
powerusage
Query System Power Statistics with DCMI (Data Center Manageability Interface).
temperature
Query the temperature from DCMI (Data Center Manageability Interface) Temperature sensor. Currently,
only CPU temperature and baseboard temperature sensor available for OpenPOWER servers.
1.3. Admin Guide
405
xCAT3 Documentation, Release 2.12
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. Query all attributes which CEC1,CEC2 supported.
renergy CEC1,CEC2 all
The output of the query operation:
CEC1:
CEC1:
CEC1:
CEC1:
CEC1:
CEC1:
CEC1:
CEC1:
CEC1:
CEC1:
CEC1:
CEC1:
CEC1:
CEC1:
CEC2:
CEC2:
CEC2:
CEC2:
CEC2:
CEC2:
CEC2:
CEC2:
CEC2:
CEC2:
CEC2:
savingstatus: off
dsavingstatus: off
cappingstatus: off
cappingmin: 1953 W
cappingmax: 2358 W
cappingvalue: 2000 W
cappingsoftmin: 304 W
averageAC: na
averageDC: na
ambienttemp: na
exhausttemp: na
CPUspeed: na
syssbpower: 40 W
sysIPLtime: 900 S
savingstatus: off
cappingstatus: off
cappingmin: 955 W
cappingmax: 1093 W
cappingvalue: 1000 W
cappingsoftmin: 226 W
averageAC: 627 W
averageDC: 531 W
ambienttemp: 25 C
exhausttemp: 40 C
CPUspeed: 4695 MHz
2. Query the fanspeed attribute for Power8 CEC.
renergy CEC1 fanspeed
The output of the query operation:
CEC1:
CEC1:
CEC1:
CEC1:
CEC1:
CEC1:
CEC1:
CEC1:
fanspeed
fanspeed
fanspeed
fanspeed
fanspeed
fanspeed
fanspeed
fanspeed
(Fan
(Fan
(Fan
(Fan
(Fan
(Fan
(Fan
(Fan
U78CB.001.WZS00MA-A1
U78CB.001.WZS00MA-A2
U78CB.001.WZS00MA-A3
U78CB.001.WZS00MA-A4
U78CB.001.WZS00MA-A5
U78CB.001.WZS00MA-A6
U78CB.001.WZS00MA-E1
U78CB.001.WZS00MA-E2
00002101):
00002103):
00002105):
00002107):
00002109):
0000210B):
0000210C):
0000210D):
5947
6081
6108
6000
6013
6013
4992
5016
RPM
RPM
RPM
RPM
RPM
RPM
RPM
RPM
3. Query the historical records for the CPUspeed attribute. (Power8 CEC)
406
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
renergy CEC1 CPUspeedhistory
The output of the query operation:
CEC1:
CEC1:
CEC1:
CEC1:
CEC1:
CEC1:
CEC1:
CEC1:
CEC1:
CEC1:
...
CPUspeedhistory:
CPUspeedhistory:
CPUspeedhistory:
CPUspeedhistory:
CPUspeedhistory:
CPUspeedhistory:
CPUspeedhistory:
CPUspeedhistory:
CPUspeedhistory:
CPUspeedhistory:
2027
2027
2244
2393
2393
2393
2393
2393
2393
2393
MHZ:
MHZ:
MHZ:
MHZ:
MHZ:
MHZ:
MHZ:
MHZ:
MHZ:
MHZ:
20141226042900
20141226042930
20141226043000
20141226043030
20141226043100
20141226043130
20141226043200
20141226043230
20141226043300
20141226043330
4
Query all the attirbutes for management module node MM1. (For chassis)
renergy MM1 all
The output of the query operation:
mm1: availableDC: 5880W
mm1: frontpaneltmp: 18.00 Centigrade
mm1: inusedAC: 2848W
mm1: mmtmp: 28.00 Centigrade
mm1: pd1avaiablepower: 2940W
mm1: pd1inusedpower: 848W
mm1: pd1policy: redundantWithoutPerformanceImpact
mm1: pd1powermodule1: Bay 1: 2940W
mm1: pd1powermodule2: Bay 2: 2940W
mm1: pd1remainpower: 1269W
mm1: pd1reservedpower: 1671W
mm1: pd1status: 1 - Power domain status is good.
mm1: pd2avaiablepower: 2940W
mm1: pd2inusedpower: 1490W
mm1: pd2policy: redundantWithoutPerformanceImpact
mm1: pd2powermodule1: Bay 3: 2940W
mm1: pd2powermodule2: Bay 4: 2940W
mm1: pd2remainpower: 51W
mm1: pd2reservedpower: 2889W
mm1: pd2status: 2 - Warning: Power redundancy does not exist in this power
˓→domain.
mm1: thermaloutput: 9717.376000 BTU/hour
5. Query all the attirbutes for blade server node blade1.
renergy blade1 all
The output of the query operation:
blade1:
blade1:
blade1:
blade1:
blade1:
CPUspeed: 4204MHZ
averageDC: 227W
capability: dynamicPowerMeasurement2
cappingvalue: 315W
dsavingstatus: off
1.3. Admin Guide
407
xCAT3 Documentation, Release 2.12
blade1: maxCPUspeed: 4204MHZ
blade1: savingstatus: off
6. Query the attributes savingstatus, cappingstatus and CPUspeed for server CEC1.
renergy CEC1 savingstatus cappingstatus CPUspeed
The output of the query operation:
CEC1: savingstatus: off
CEC1: cappingstatus: on
CEC1: CPUspeed: 3621 MHz
7. Turn on the power saving function of CEC1.
renergy CEC1 savingstatus=on
The output of the setting operation:
CEC1: Set savingstatus succeeded.
CEC1: This setting may need some minutes to take effect.
8. Set the power capping value base on the percentage of the max-min capping value. Here, set it to 50%.
renergy CEC1 cappingperc=50
If the maximum capping value of the CEC1 is 850w, and the minimum capping value of the CEC1 is
782w, the Power Capping value will be set as ((850-782)*50% + 782) = 816w.
The output of the setting operation:
CEC1: Set cappingperc succeeded.
CEC1: cappingvalue: 816
9. Query powerusage and temperature for OpenPOWER servers.
renergy ops01 powerusage temperature
The output will be like this:
ops01:
ops01:
ops01:
ops01:
ops01:
ops01:
ops01:
ops01:
ops01:
Current Power
Minimum Power over sampling duration
Maximum Power over sampling duration
Average Power over sampling duration
Time Stamp
Statistics reporting time period
Power Measurement
CPU Temperature Instance 0
Baseboard temperature Instance 0
:
:
:
:
:
:
:
:
:
591W
558W
607W
572W
11/18/2015 - 1:4:1
10000 milliseconds
Active
+39 Centigrade
+28 Centigrade
REFERENCES
1. For more information on ‘Power System Energy Management’:
http://www-03.ibm.com/systems/power/software/energy/index.html
2. EnergyScale white paper for Power6:
408
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
http://www-03.ibm.com/systems/power/hardware/whitepapers/energyscale.html
3. EnergyScale white paper for Power7:
http://www-03.ibm.com/systems/power/hardware/whitepapers/energyscale7.html
FILES
/opt/xcat/bin/renergy
replaycons.1
NAME
replaycons - replay the console output for a node
SYNOPSIS
replaycons [node] [bps] [tail_amount]
replaycons [-h | --help | -v | --version]
DESCRIPTION
The replaycons command reads the console log stored by conserver for this node, and displays it in a way that
simulates the original output of the console. Using the bps value, it will throttle the speed of the output play back.
(The conserver logs are stored in /var/log/consoles.)
For now, replaycons must be run locally on the system on which the console log is stored. This is normally that
management node, but in a hierarchical cluster will usually be the service node.
OPTIONS
bps
The display rate to use to play back the console output. Default is 19200.
tail_amount
The place in the console log file to start play back, specified as the # of lines from the end.
-v|--version
Command Version.
-h|--help
Display usage message.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
1.3. Admin Guide
409
xCAT3 Documentation, Release 2.12
EXAMPLES
1. To replay the console for node1 at the default rate, starting 2000 lines from the end:
replaycons 19200 2000
FILES
/opt/xcat/bin/replaycons
SEE ALSO
rcons(1)|rcons.1
restartxcatd.1
NAME
restartxcatd - Restart the xCAT daemon (xcatd).
SYNOPSIS
restartxcatd [[-h | --help] | [-v | --version] | [-r | --reload]] [-V | --verbose]
DESCRIPTION
The restartxcatd command restarts the xCAT daemon (xcatd).
Linux Specific
It will perform the xcatd fast restart. The xcatd fast restart is a specific restart which has two advantages compares to the stop an
1. The interval of xcatd out of service is very short.
2. The in processing request which initiated by old xcatd will not be stopped by force. The old xcatd will
hand over the sockets to new xcatd, but old xcat will still be waiting for the in processing request to finish
before the exit.
It does the same thing as ‘service xcatd restart’ on NON-systemd enabled Operating System like rh6.x and sles11.x.
But for the systemd enabled Operating System like rh7 and sles12, the ‘service xcatd restart’ just do the stop and start
instead of xcatd fast restart.
It’s recommended to use restartxcatd command to restart xcatd on systemd enable system like rh7 and sles12 instead
of ‘service xcatd restart’ or ‘systemctl restart xcatd’.
AIX Specific
It runs ‘stopsrc -s xcatd’ to stop xcatd first if xcatd is active, then runs ‘startsrc -s xcatd’ to start xcatd.
If the xcatd subsystem was not created, restartxcatd will create it automatically.
410
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
OPTIONS
-h|--help Display usage message.
-v|--version Command Version.
-r|--reload On a Service Node, services will not be restarted.
-V|--verbose Display the verbose messages.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To restart the xCAT daemon, enter:
restartxcatd
FILES
/opt/xcat/sbin/restartxcatd
restorexCATdb.1
NAME
restorexCATdb - restores the xCAT db tables .
SYNOPSIS
restorexCATdb [-a] [-V] [{-p | --path} path]
restorexCATdb [-b] [-V] [{-t | --timestamp} timestamp] [{-p | --path} path]
restorexCATdb [-h | --help] [-v | --version]
DESCRIPTION
If not using binary restore(-b), the restorexCATdb command restores the xCAT database tables from the *.csv files in
directory given by the -p flag. The site table skiptables attribute can be set to a list of tables not to restore. It will not
restore isnm_perf* tables. See man dumpxCATdb.
If using the binary restore option for DB2 or postgreSQL, the entire database is restored from the binary backup made
with dumpxCATdb. The database will be restored using the database Utilities. For DB2, the timestamp of the correct
DB2 backup file (-t) must be provided. All applications accessing the DB2 database must be stopped before you can
use the binary restore options. See the xCAT DB2 document for more information. For postgreSQL, you do not have
1.3. Admin Guide
411
xCAT3 Documentation, Release 2.12
to stop the applications accessing the database and the complete path to the backup file, must be supplied on the -p
flag.
OPTIONS
-h|--help Display usage message.
-v|--version Command Version.
-V|--verbose Verbose.
-a All,without this flag the eventlog and auditlog will be skipped. These tables are skipped by default because restoring
will generate new indexes
-b Restore from the binary image.
-p|--path Path to the directory containing the database restore files. If restoring from the binary image (-b) and using
postgeSQL, then this is the complete path to the restore file that was created with dumpxCATdb -b.
-t|--timestamp Use with the -b flag to designate the timestamp of the binary image to use to restore for DB2.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To restore the xCAT database from the /dbbackup/db directory, enter:
restorexCATdb -p /dbbackup/db
2. To restore the xCAT database including auditlog and eventlog from the /dbbackup/db directory, enter:
restorexCATdb -a -p /dbbackup/db
3. To restore the xCAT DB2 database from the binary image with timestamp 20111130130239 enter:
restorexCATdb -b -t 20111130130239 -p /dbbackup/db
4. To restore the xCAT postgreSQL database from the binary image file pgbackup.20553 created by dumpxCATdb
enter:
restorexCATdb -b
-p /dbbackup/db/pgbackup.20553
FILES
/opt/xcat/sbin/restorexCATdb
SEE ALSO
dumpxCATdb(1)|dumpxCATdb.1
412
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
reventlog.1
Name
reventlog - retrieve or clear remote hardware event logs
Synopsis
reventlog noderange {number-of-entries [-s]|all [-s] | clear}
reventlog [-h | --help | -v | --version]
Description
reventlog can display any number of remote hardware event log entries or clear them for a range of nodes. Hardware
event logs are stored on each servers service processor.
Options
number-of-entries
Retrieve the specified number of entries from the nodes’ service processors.
all
Retrieve all entries.
-s
To sort the entries from latest (always the last entry in event DB) to oldest (always the first entry in event
DB). If number-of-entries specified, the latest number-of-entries events will be output in the order of
latest to oldest.
clear
Clear event logs.
-h | --help
Print help.
-v | --version
Print version.
Examples
1. reventlog node4,node5 5
Output is similar to:
node4: SERVPROC I 09/06/00 15:23:33 Remote Login Successful User ID = USERID[00]
node4: SERVPROC I 09/06/00 15:23:32 System spn1 started a RS485 connection with
˓→us[00]
node4: SERVPROC I 09/06/00 15:22:35 RS485 connection to system spn1 has ended[00]
1.3. Admin Guide
413
xCAT3 Documentation, Release 2.12
node4: SERVPROC
node4: SERVPROC
˓→us[00]
node5: SERVPROC
node5: SERVPROC
˓→us[00]
node5: SERVPROC
node5: SERVPROC
node5: SERVPROC
˓→us[00]
I 09/06/00 15:22:32 Remote Login Successful User ID = USERID[00]
I 09/06/00 15:22:31 System spn1 started a RS485 connection with
I 09/06/00 15:22:32 Remote Login Successful User ID = USERID[00]
I 09/06/00 15:22:31 System spn1 started a RS485 connection with
I 09/06/00 15:21:34 RS485 connection to system spn1 has ended[00]
I 09/06/00 15:21:30 Remote Login Successful User ID = USERID[00]
I 09/06/00 15:21:29 System spn1 started a RS485 connection with
2. reventlog node4,node5 clear
Output is similar to:
node4: clear
node5: clear
SEE ALSO
rpower(1)|rpower.1, monstart(1)|monstart.1
rflash.1
Name
rflash - Performs Licensed Internal Code (LIC) update support for HMC-attached POWER5 and POWER6 Systems,
and POWER7 systems using Direct FSP management. rflash is also able to update firmware for NextScale Fan Power
Controllers (FPC).
Synopsis
rflash [-h | --help | -v | --version]
PPC (with HMC) specific:
rflash noderange -p directory [--activate {concurrent | disruptive}] [-V | --verbose]
rflash noderange {--commit | --recover} [-V | --verbose]
PPC (without HMC, using Direct FSP Management) specific:
rflash noderange -p directory [--activate {disruptive | deferred}] [-d data_directory]
rflash noderange {--commit | --recover}
NeXtScale FPC specific:
rflash noderange http_directory
414
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
OpenPOWER BMC specific:
rflash noderange hpm_file_path [-c | --check] [--retry=count] [-V]
Description
The rflash command initiates Firmware updates on supported xCAT nodes. Licensed Internal Code (also known as
microcode) updates are performed on supported HMC-attached POWER5 and POWER6 pSeries nodes, and POWER7
systems using Direct FSP management.
The command scans the specified directory structure for Firmware update package files applicable to the given nodes
and components. And then it will automatically select the latest version for the upgrade. The firmware update files
include the Microcode update package and associated XML file. They can be downloaded from the IBM Web site:
http://www-933.ibm.com/support/fixcentral/.
The POWER5 and POWER6 systems contain several components that use Licensed Internal Code. The rflash command supports two of these components: the managed system (also known as the Central Electronics Complex, or
CEC) and the power subsystem (also known as the Bulk Power Assembly (BPA) or Bulk Power Controller (BPC)).
Some POWER5 managed systems can be attached to a power subsystem. These power subsystems can support multiple managed systems. When the rflash command is invoked, xCAT will determine the managed system or power
subsystem associated with that CEC and perform the update.
The noderange can be an CEC or CEC list, a Lpar or Lpar list and a Frame or Frame list. But CEC (or Lpar) and
Frame can’t be used at the same time. When the noderange is an CEC or CEC list, rflash will upgrade the firmware
of the CEC or CECs in the cec list. If noderange is a Lpar or Lpar list, rflash will update Licensed Internal Code (LIC)
on HMC-attached POWER5 and POWER6 pSeries nodes, and POWER7 systems using Direct FSP management. If
noderange is a Frame or Frame list, rflash will update Licensed Internal Code (LIC) of the power subsystem on
HMC-attached POWER5 and POWER6 pSeries nodes. The noderange can also be the specified node groups. You
can specify a comma or space-separated list of node group ranges. See the noderange man page for detailed usage
information.
The command will update firmware for NeXtScale FPC when given an FPC node and the http information needed to
access the firmware.
PPC (with HMC) specific:
The rflash command uses the xdsh command to connect to the HMC controlling the given managed system and
perform the updates. Before running rflash, use rspconfig to check if the related HMC ssh is enabled. To enable a
HMC ssh connection, use rspconfig command.
Warning! This command may take considerable time to complete, depending on the number of systems being updated
and the workload on the target HMC. In particular, power subsystem updates may take an hour or more if there are
many attached managed systems.
Depending on the Licensed Internal Code update that is installed, the affected HMC-attached POWER5 and POWER6
systems may need to be recycled. The --activate flag determines how the affected systems activate the new code. The
concurrent option activates code updates that do not require a system recycle (known as a “concurrent update”). If
this option is given with an update that requires a system recycle (known as a “disruptive update”), a message will be
returned, and no activation will be performed. The disruptive option will cause any affected systems that are powered
on to be powered down before installing and activating the update. Once the update is complete, the command will
attempt to power on any affected systems that it powered down. Those systems that were powered down when the
command was issued will remain powered down when the update is complete.
The flash chip of a POWER5 and POWER6 managed system or power subsystem stores firmware in two locations,
referred to as the temporary side and the permanent side. By default, most POWER5 and POWER6 systems boot from
1.3. Admin Guide
415
xCAT3 Documentation, Release 2.12
the temporary side of the flash. When the rflash command updates code, the current contents of the temporary side
are written to the permanent side, and the new code is written to the temporary side. The new code is then activated.
Therefore, the two sides of the flash will contain different levels of code when the update has completed.
The --commit flag is used to write the contents of the temporary side of the flash to the permanent side. This flag
should be used after updating code and verifying correct system operation. The --recover flag is used to write the
permanent side of the flash chip back to the temporary side. This flag should be used to recover from a corrupt flash
operation, so that the previously running code can be restored.
NOTE:When the --commit or --recover two flags is used, the noderange cannot be BPA. It only can be CEC or
LPAR, and will take effect for both managed systems and power subsystems.
xCAT recommends that you shutdown your Operating System images and power off your managed systems before
applying disruptive updates to managed systems or power subsystems.
Any previously activated code on the affected systems will be automatically accepted into permanent flash by this
procedure.
IMPORTANT! If the power subsystem is recycled, all of its attached managed systems will be recycled.
If it outputs “Timeout waiting for prompt” during the upgrade, set the “ppctimeout” larger in the site table. After
the upgrade, remeber to change it back. If run the “rflash” command on an AIX management node, need to make sure
the value of “useSSHonAIX” is “yes” in the site table.
PPC (using Direct FSP Management) specific:
In currently Direct FSP/BPA Management, our rflash doesn’t support concurrent value of --activate flag, and supports disruptive and deferred. The disruptive option will cause any affected systems that are powered on to be
powered down before installing and activating the update. So we require that the systems should be powered off
before do the firmware update.
The deferred option will load the new firmware into the T (temp) side, but will not activate it like the disruptive
firmware. The customer will continue to run the Frames and CECs working with the P (perm) side and can wait for a
maintenance window where they can activate and boot the Frame/CECs with new firmware levels. Refer to the doc to
get more details: XCAT_Power_775_Hardware_Management
In Direct FSP/BPA Management, there is -d data_directory option. The default value is /tmp. When doing firmware
update, rflash will put some related data from rpm packages in <data_directory> directory, so the execution of rflash
will require available disk space in <data_directory> for the command to properly execute:
For one GFW rpm package and one power code rpm package, if the GFW rpm package size is gfw_rpmsize, and
the Power code rpm package size is power_rpmsize, it requires that the available disk space should be more than:
1.5*gfw_rpmsize + 1.5*power_rpmsize
For Power 775, the rflash command takes effect on the primary and secondary FSPs or BPAs almost in parallel.
For more details about the Firmware Update using Direct FSP/BPA Management, refer
XCAT_Power_775_Hardware_Management#Updating_the_BPA_and_FSP_firmware_using_xCAT_DFM
to:
NeXtScale FPC specific:
The command will update firmware for NeXtScale FPC when given an FPC node and the http information needed to
access the firmware. The http information required includes both the MN IP address as well as the directory containing
the firmware. It is recommended that the firmware be downloaded and placed in the /install directory structure as the
xCAT MN /install directory is configured with the correct permissions for http. Refer to the doc to get more details:
XCAT_NeXtScale_Clusters
416
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
OpenPOWER specific:
The command will update firmware for OpenPOWER BMC when given an OpenPOWER node and the hpm1 formatted file path.
Options
-h|--help
Writes the command’s usage statement to standard output.
-c|--check
Check the firmware version of BMC and HPM file.
-p directory
Specifies the directory where the packages are located.
-d data_directory
Specifies the directory where the raw data from rpm packages for each CEC/Frame are located. The
default directory is /tmp. The option is only used in Direct FSP/BPA Management.
--activate {concurrent | disruptive}
Must be specified to activate the new Licensed Internal Code. The “disruptive” option will cause the target
systems to be recycled. Without this flag, LIC updates will be installed only, not activated.
--commit
Used to commit the flash image in the temporary side of the chip to the permanent side for both managed
systems and power subsystems.
--recover
Used to recover the flash image in the permanent side of the chip to the temporary side for both managed
systems and power subsystems.
--retry=count
Specify number of times to retry the update if failure is detected. Default value is 2. Value of 0 can be
used to indicate no retries.
-v|--version
Displays the command’s version.
-V|--verbose
Verbose output.
Exit Status
0 The command completed successfully.
1 An error has occurred.
1.3. Admin Guide
417
xCAT3 Documentation, Release 2.12
Examples
1. To update only the power subsystem attached to a single HMC-attached pSeries CEC(cec_name), and recycle
the power subsystem and all attached managed systems when the update is complete, and the Microcode update
package and associated XML file are in /tmp/fw, enter:
rflash cec_name -p /tmp/fw --activate disruptive
2. To update only the power subsystem attached to a single HMC-attached pSeries node, and recycle the power
subsystem and all attached managed systems when the update is complete, and the Microcode update package
and associated XML file are in /tmp/fw, enter:
rflash bpa_name -p /tmp/fw --activate disruptive
3. To commit a firmware update to permanent flash for both managed system and the related power subsystems,
enter:
rflash cec_name --commit
4. To update the firmware on a NeXtScale FPC specify the FPC node name and the HTTP location of the file
including the xCAT MN IP address and the directory on the xCAT MN containing the firmware as follows:
rflash fpc01 http://10.1.147.169/install/firmware/fhet17a/ibm_fw_fpc_fhet17a˓→2.02_anyos_noarch.rom
5. To update the firmware on OpenPOWER machine specify the node name and the file path of the HPM firmware
file as follows:
rflash fs3 /firmware/8335_810.1543.20151021b_update.hpm
Print verbose message to rflash log file (/var/log/xcat/rflash/fs3.log) when updading firmware:
rflash fs3 /firmware/8335_810.1543.20151021b_update.hpm -V
Location
/opt/xcat/bin/rflash
NOTES
This command is part of the xCAT software product.
SEE ALSO
rinv(1)|rinv.1, rspconfig(1)|rspconfig.1
rinv.1
Name
rinv - Remote hardware inventory
418
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
Synopsis
rinv [-h | --help | -v | --version]
BMC/MPA specific:
rinv noderange [model | serial | asset | vpd | deviceid | guid | firm | dimm | mprom | all]
OpenPOWER (IPMI) server specific:
rinv noderange [model | serial | deviceid | uuid | guid | vpd | mprom | firm | all]
OpenPOWER (OpenBMC) server specific:
rinv noderange [model | serial | firm | cpu | dimm | all] [-V | --verbose]
PPC (with HMC) specific:
rinv noderange {bus | config | serial | model | firm | all}
PPC (using Direct FSP Management) specific:
rinv noderange {firm}
rinv noderange {deconfig [-x]}
Blade specific:
rinv noderange {mtm | serial | mac | bios | diag | mprom | mparom | firm | all}
VMware specific:
rinv noderange [-t]
pdu specific:
rinv noderange
zVM specific:
rinv noderange [config | all]
rinv noderange [--diskpoolspace]
rinv noderange [--diskpool pool space]
rinv noderange [--fcpdevices state details]
1.3. Admin Guide
419
xCAT3 Documentation, Release 2.12
rinv noderange [--diskpoolnames]
rinv noderange [--networknames]
rinv noderange [--network name]
rinv noderange [--ssi]
rinv noderange [--smapilevel]
rinv noderange [--wwpns fcp_channel]
rinv noderange [--zfcppool pool space]
rinv noderange [--zfcppoolnames]
Description
rinv retrieves hardware configuration information from the on-board Service Processor for a single or range of nodes
and groups.
Calling rinv for VMware will display the UUID/GUID, number of CPUs, amount of memory, the MAC address and
a list of Hard disks. The output for each Hard disk includes the label, size and backing file location.
Options
bus
List all buses for each I/O slot.
config
Retrieves number of processors, speed, total memory, and DIMM locations.
model
Retrieves model number.
serial
Retrieves serial number.
firm
Retrieves firmware versions.
deconfig
Retrieves deconfigured resources. Deconfigured resources are hw components (cpus, memory, etc.) that
have failed so the firmware has automatically turned those components off. This option is only capable of
listing some of the deconfigured resources and should not be the only method used to check the hardware
status.
-x
To output the raw information of deconfigured resources for CEC.
asset
Retrieves asset tag. Usually it’s the MAC address of eth0.
vpd
Same as specifying model, serial, deviceid, and mprom.
420
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
diag
Diagnostics information of firmware.
mprom
Retrieves mprom firmware level.
dimm
Retrieves dual in-line memory module information.
deviceid
Retrieves device identification. Usually device, manufacturing and product IDs.
uuid
Retrieves the universally unique identifier.
guid
Retrieves the global unique identifier .
all
All of the above.
-h | --help
Print help.
-v | --version
Print version.
-V | --verbose
Prints verbose output, if available.
-t
Set the values in the vm table to what vCenter has for the indicated nodes.
zVM specific :
--diskpoolspace
Calculates the total size of every known storage pool.
--diskpool pool space
Lists the storage devices (ECKD and FBA) contained in a disk pool. Space can be: all, free, or used.
--fcpdevices state details
Lists the FCP device channels that are active, free, or offline. State can be: active, free, or offline.
--diskpoolnames
Lists the known disk pool names.
--networknames
Lists the known network names.
--network name
Shows the configuration of a given network device.
--ssi
1.3. Admin Guide
421
xCAT3 Documentation, Release 2.12
Obtain the SSI and system status.
--smapilevel
Obtain the SMAPI level installed on the z/VM system.
--wwpns fcp_channel
Query a given FCP device channel on a z/VM system and return a list of WWPNs.
--zfcppool pool space
List the SCSI/FCP devices contained in a zFCP pool. Space can be: free or used.
--zfcppoolnames
List the known zFCP pool names.
Examples
1. To retrieve all information available from blade node4, enter:
rinv node5 all
Output is similar to:
node5:
node5:
node5:
node5:
node5:
node5:
node5:
node5:
node5:
node5:
node5:
node5:
node5:
node5:
node5:
node5:
node5:
node5:
node5:
node5:
Machine Type/Model 865431Z
Serial Number 23C5030
Asset Tag 00:06:29:1F:01:1A
PCI Information
Bus VendID DevID
RevID Description
Slot Pass/Fail
0
1166
0009
06
Host Bridge
0
PASS
0
1166
0009
06
Host Bridge
0
PASS
0
5333
8A22
04
VGA Compatible Controller0
PASS
0
8086
1229
08
Ethernet Controller
0
PASS
0
8086
1229
08
Ethernet Controller
0
PASS
0
1166
0200
50
ISA Bridge
0
PASS
0
1166
0211
00
IDE Controller
0
PASS
0
1166
0220
04
Universal Serial Bus
0
PASS
1
9005
008F
02
SCSI Bus Controller
0
PASS
1
14C1
8043
03
Unknown Device Type
2
PASS
Machine Configuration Info
Number of Processors:
Processor Speed: 866 MHz
Total Memory:
512 MB
Memory DIMM locations: Slot(s) 3 4
2. To output the raw information of deconfigured resources for CEC cec01, enter:
rinv cec01 deconfig -x
Output is similar to:
cec01:
<SYSTEM>
<System_type>IH</System_type>
<NODE>
<Location_code>U78A9.001.0123456-P1</Location_code>
<RID>800</RID>
422
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
</NODE>
</SYSTEM>
3. To retrieve ‘config’ information from the HMC-managed LPAR node3, enter:
rinv node3 config
Output is similar to:
node5: Machine Configuration Info
node5: Number of Processors: 1
node5: Total Memory (MB): 1024
4. To retrieve information about a VMware node vm1, enter:
rinv vm1
Output is similar to:
vm1: UUID/GUID: 42198f65-d579-fb26-8de7-3ae49e1790a7
vm1: CPUs: 1
vm1: Memory: 1536 MB
vm1: Network adapter 1: 36:1b:c2:6e:04:02
vm1: Hard disk 1 (d0): 9000 MB @ [nfs_192.168.68.21_vol_rc1storage_vmware] vm1_3/
˓→vm1.vmdk
vm1: Hard disk 2 (d4): 64000 MB @ [nfs_192.168.68.21_vol_rc1storage_vmware] vm1_3/
˓→vm1_5.vmdk
zVM specific :
5. To list the defined network names available for a given node:
rinv pokdev61 --getnetworknames
Output is similar to:
pokdev61:
pokdev61:
pokdev61:
pokdev61:
pokdev61:
pokdev61:
pokdev61:
pokdev61:
LAN:QDIO SYSTEM GLAN1
LAN:HIPERS SYSTEM GLAN2
LAN:QDIO SYSTEM GLAN3
VSWITCH SYSTEM VLANTST1
VSWITCH SYSTEM VLANTST2
VSWITCH SYSTEM VSW1
VSWITCH SYSTEM VSW2
VSWITCH SYSTEM VSW3
6. To list the configuration for a given network:
rinv pokdev61 --getnetwork GLAN1
Output is similar to:
pokdev61: LAN SYSTEM GLAN1
Type: QDIO
Connected: 1
Maxconn: INFINITE
pokdev61:
PERSISTENT UNRESTRICTED IP
Accounting: OFF
pokdev61:
IPTimeout: 5
MAC Protection: Unspecified
pokdev61:
Isolation Status: OFF
7. To list the disk pool names available:
1.3. Admin Guide
423
xCAT3 Documentation, Release 2.12
rinv pokdev61 --diskpoolnames
Output is similar to:
pokdev61: POOL1
pokdev61: POOL2
pokdev61: POOL3
8. List the configuration for a given disk pool:
rinv pokdev61 --diskpool POOL1 free
Output is similar to:
pokdev61: #VolID DevType StartAddr Size
pokdev61: EMC2C4 3390-09 0001 10016
pokdev61: EMC2C5 3390-09 0001 10016
9. List the known zFCP pool names.
rinv pokdev61 --zfcppoolnames
Output is similar to:
pokdev61: zfcp1
pokdev61: zfcp2
pokdev61: zfcp3
10. List the SCSI/FCP devices contained in a given zFCP pool:
rinv pokdev61 --zfcppool zfcp1
Output is similar to:
pokdev61: #status,wwpn,lun,size,range,owner,channel,tag
pokdev61: used,500512345678c411,4014412100000000,2g,3B40-3B7F,ihost13,3b77,
pokdev61: used,500512345678c411,4014412200000000,8192M,3B40-3B7F,ihost13,3b77,
˓→replace_root_device
pokdev61: free,500512345678c411,4014412300000000,8g,3B40-3B7F,,,
pokdev61: free,5005123456789411,4014412400000000,2g,3B40-3B7F,,,
pokdev61: free,5005123456789411;5005123456789411,4014412600000000,2G,3B40-3B7F,,,
SEE ALSO
rpower(1)|rpower.1
rmdef.1
NAME
rmdef - Use this command to remove xCAT data object definitions.
424
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
SYNOPSIS
rmdef [-h | --help] [-t object-types]
rmdef [-V | --verbose] [-a | --all] [-t object-types] [-o object-names] [-f | --force] [-C | --cleanup] [noderange]
DESCRIPTION
This command is used to remove xCAT object definitions that are stored in the xCAT database.
OPTIONS
-a|--all
Clear the whole xCAT database. A backup of the xCAT definitions should be saved before using this
option as the xCAT daemons will no longer work once cleared.
To restore:
1. export XCATBYPASS=1 and run the restorexCATdb command.
or
2. Run xcatconfig -d which initializes the database the same as when xCAT was installed.
-f|--force
Use this with the --all option as an extra indicator that ALL definitions are to be removed.
-h|--help
Display a usage message.
noderange
A set of comma delimited node names and/or group names. See the “noderange” man page for details on
supported formats.
-o object-names
A set of comma delimited object names.
-t object-types
A set of comma delimited object types.
-C|--cleanup
Perform additional cleanup by running nodeset offline on the objects specified in the noderange.
-V|--verbose
Verbose mode.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
1.3. Admin Guide
425
xCAT3 Documentation, Release 2.12
EXAMPLES
1. To remove a range of node definitions.
rmdef -t node node1-node4
2. To remove all node definitions for the nodes contained in the group bpcnodes.
rmdef -t node -o bpcnodes
3. To remove the group called bpcnodes.
rmdef -t group -o bpcnodes
(This will also update the values of the “groups” attribute of the member nodes.)
FILES
$XCATROOT/bin/rmdef
(The XCATROOT environment variable is set when xCAT is installed. The default value is “/opt/xcat”.)
NOTES
This command is part of the xCAT software product.
SEE ALSO
mkdef(1)|mkdef.1, lsdef(1)|lsdef.1, chdef(1)|chdef.1, xcatstanzafile(5)|xcatstanzafile.5
rmdocker.1
SYNOPSIS
rmdocker noderange [-f | --force]
rmdocker [-h | --help]
rmdocker {-v | --version}
DESCRIPTION
rmdocker To remove docker instances with the specified node name
OPTIONS
-f|--force
Force to removal of a running container or failed to disconnect customized network
426
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
EXAMPLES
rmdocker host01c01
host01c01: Disconnect customized network 'mynet0' done
host01c01: success
SEE ALSO
mkdocker(1)|mkdocker.1, lsdocker(1)|lsdocker.1
rmdsklsnode.1
NAME
rmdsklsnode - Use this xCAT command to remove AIX/NIM diskless machine definitions.
SYNOPSIS
rmdsklsnode [-h | --help ]
rmdsklsnode [-V|--verbose] [-f|--force] [-r|--remdef] [-i image_name] [-p|--primarySN] [-b|--backupSN]
noderange
DESCRIPTION
Use this command to remove all NIM client machine definitions that were created for the specified xCAT nodes.
The xCAT node definitions will not be removed. Use the xCAT rmdef command to remove xCAT node definitions.
If you are using xCAT service nodes the rmdsklsnode command will automatically determine the correct server(s)
for the node and remove the NIM definitions on that server(s).
If the node you are trying to remove is currently running the rmdsklsnode command will not remove the definitions.
You can use the “-f” option to shut down the node and remove the definition.
Removing alternate NIM client definitions
If you used the “-n” option when you created the NIM client definitions with the mkdsklsnode command then the
NIM client machine names would be a combination of the xCAT node name and the osimage name used to initialize
the NIM machine. To remove these definitions, you must provide the name of the osimage that was used using the “-i”
option.
In most cases you would most likely want to remove the old client definitions without disturbing the nodes that you
just booted with the new alternate client definition. The rmdsklsnode -r option can be used to remove the old alternate
client definitions without stopping the running node.
However, if you have NIM dump resources assign to your nodes be aware that when the old NIM alternate client
definitions are removed it will leave the nodes unable to produce a system dump. This is a current limitation in the
NIM support for alternate client definitions. For this reason, it is recommended that you wait to do this cleanup until
right before you do your next upgrade.
1.3. Admin Guide
427
xCAT3 Documentation, Release 2.12
OPTIONS
-f |--force
Use the force option to stop and remove running nodes. This handles the situation where a NIM machine
definition indicates that a node is still running even though it is not.
-b |--backupSN
When using backup service nodes only update the backup. The default is to update both the primary and
backup service nodes.
-h |--help
Display usage message.
-i image_name
The name of an xCAT image definition.
noderange
A set of comma delimited node names and/or group names. See the “noderange” man page for details on
additional supported formats.
-p|--primarySN
When using backup service nodes only update the primary. The default is to update both the primary and
backup service nodes.
-r|--remdef
Use this option to reset, deallocate, and remove NIM client definitions. This option will not attempt to
shut down running nodes. This option should be used when remove alternate NIM client definitions that
were created using mkdsklsnode -n.
-V |--verbose
Verbose mode.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. Remove the NIM client definition for the xCAT node named “node01”. Give verbose output.
rmdsklsnode -V node01
2. Remove the NIM client definitions for all the xCAT nodes in the group “aixnodes”. Attempt to shut down the
nodes if they are running.
rmdsklsnode -f aixnodes
3. Remove the NIM client machine definition for xCAT node “node02” that was created with the mkdsklsnode -n
option and the image “AIXdskls”. (i.e. NIM client machine name “node02_AIXdskls”.)
428
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
rmdsklsnode -i AIXdskls node02
This assume that node02 is not currently running.
4. Remove the old alternate client definition “node27_olddskls”.
rmdsklsnode -r -i olddskls node27
Assuming the node was booted using an new alternate NIM client definition then this will leave the node running.
FILES
/opt/xcat/bin/rmdsklsnode
NOTES
This command is part of the xCAT software product.
SEE ALSO
mkdsklsnode(1)|mkdsklsnode.1
rmflexnode.1
NAME
rmflexnode - Delete a flexible node.
SYNOPSIS
rmflexnode [-h | --help]
rmflexnode [-v | --version]
rmflexnode noderange
DESCRIPTION
Delete a flexible node which created by the mkflexnode command.
The rmflexnode command will delete the Partition which the slots in id attribute assigned to.
The action of deleting flexible node will impact the hardware status. Before deleting it, the blades in the slot range
should be in power off state.
After the deleting, use the lsflexnode to check the status of the node.
The noderange only can be a blade node.
1.3. Admin Guide
429
xCAT3 Documentation, Release 2.12
OPTIONS
-h | --help
Display the usage message.
-v | --version
Display the version information.
EXAMPLES
1 Delete a flexible node base on the xCAT node blade1.
The blade1 should belong to a complex, the id attribute should be set correctly and all the slots should be
in power off state.
rmflexnode blade1
FILES
/opt/xcat/bin/rmflexnode
SEE ALSO
lsflexnode(1)|lsflexnode.1, mkflexnode(1)|mkflexnode.1
rmhwconn.1
NAME
rmhwconn - Use this command to remove connections from CEC and Frame nodes to HMC nodes.
SYNOPSIS
rmhwconn [-h| --help]
rmhwconn [-v| --version]
PPC (with HMC) specific:
rmhwconn [-V| --verbose] noderange
PPC (without HMC, using FSPAPI) specific:
rmhwconn noderange -T tooltype
430
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
PPC (use HMC as SFP) specific:
rmhwconn -s
DESCRIPTION
For PPC (with HMC) specific:
This command is used to disconnect CEC and Frame nodes from HMC nodes, according to the connection information
defined in ppc table in xCAT DB.
Note: If a CEC belongs to a frame with a BPA installed, this CEC cannot be disconnected individually. Instead, the
whole frame should be disconnected.
For PPC (without HMC, using FSPAPI) specific:
It’s used to disconnection CEC and Frame nodes from hardware server.
For PPC (use HMC as SFP) specific:
It is used to disconnect Frame nodes from HMC nodes.
OPTIONS
-h|--help
Display usage message.
-V|--verbose
Verbose output.
-T
The tooltype is used to communicate to the CEC/Frame. The value could be lpar or fnm. The tooltype
value lpar is for xCAT and fnm is for CNM.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To disconnect all CEC nodes in node group cec from their HMC nodes:
rmhwconn cec
2. To remove the connection for Frame node frame1:
rmhwconn frame1
3. To disconnect all CEC nodes in node group cec from their related hardware serveri, using lpar tooltype:
1.3. Admin Guide
431
xCAT3 Documentation, Release 2.12
rmhwconn cec -T lpar
FILES
$XCATROOT/bin/rmhwconn
(The XCATROOT environment variable is set when xCAT is installed. The default value is “/opt/xcat”.)
NOTES
This command is part of the xCAT software product.
SEE ALSO
lshwconn(1)|lshwconn.1, mkhwconn(1)|mkhwconn.1
rmhypervisor.1
NAME
rmhypervisor - Remove the virtualization hosts.
SYNOPSIS
RHEV specific :
rmhypervisor noderange [-f]
DESCRIPTION
The rmhypervisor command can be used to remove the virtualization host.
OPTIONS
-f
If -f is specified, the host will be deactivated to maintenance before the removing.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
432
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
EXAMPLES
1. To remove the host ‘host1’, enter:
rmhypervisor host1
FILES
/opt/xcat/bin/rmhypervisor
rmigrate.1
Name
rmigrate - Execute migration of a guest VM between hosts/hypervisors
Synopsis
rmigrate noderange target_host
For zVM:
rmigrate
noderange
[destination=target_host]
[max_total=total] [max_quiesce=quiesce]
[action=action]
[force=force]
[immediate=yes_no]
Description
rmigrate requests that a guest VM to be moved from the current hypervisor to another. It will request a live migration
if possible. The vmstorage directory should be shared between the source and destination hypervisors.
Note: Make sure SELINUX is disabled and firewall is turned off on source and destination hypervisors.
For zVM:
rmigrate migrates a VM from one z/VM member to another in an SSI cluster (only in z/VM 6.2).
OPTIONS
zVM specific:
destination= The name of the destination z/VM system to which the specified virtual machine will be relocated.
action= It can be: (MOVE) initiate a VMRELOCATE MOVE of the VM, (TEST) determine if VM is eligible to be
relocated, or (CANCEL) stop the relocation of VM.
force= It can be: (ARCHITECTURE) attempt relocation even though hardware architecture facilities or CP features
are not available on destination system, (DOMAIN) attempt relocation even though VM would be moved outside of its
1.3. Admin Guide
433
xCAT3 Documentation, Release 2.12
domain, or (STORAGE) relocation should proceed even if CP determines that there are insufficient storage resources
on destination system.
immediate= It can be: (YES) VMRELOCATE command will do one early pass through virtual machine storage and
then go directly to the quiesce stage, or (NO) specifies immediate processing.
max_total= The maximum wait time for relocation to complete.
max_quiesce= The maximum quiesce time a VM may be stopped during a relocation attempt.
Files
vm table - Table governing VM parameters. See vm(5)|vm.5 for further details. This is used to determine the current
host to migrate from.
Examples
1. To migrate kvm guest “kvm1” from hypervisor “hyp01” to hypervisor “hyp02”, run:
rmigrate kvm1 hyp02
2. To migrate kvm guest “kvm1” to hypervisor “hyp02”, which is already on host “hyp02”, enter:
rmigrate kvm1 hyp02
zVM specific:
rmigrate ihost123 destination=pokdev62
rmimage.1
NAME
rmimage - Removes the Linux stateless or statelite image from the file system.
SYNOPSIS
rmimage [-h | --help]
rmimage [-V | --verbose] imagename [--xcatdef]
DESCRIPTION
Removes the Linux stateless or statelite image from the file system. The install dir is setup by using “installdir”
attribute set in the site table.
If imagename is specified, this command uses the information in the imagenameto calculate the image root directory;
otherwise, this command uses the operating system name, architecture and profile name to calculate the image root
directory.
434
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
The osimage definition will not be removed from the xCAT tables by default, specifying the flag --xcatdef will remove
the osimage definition, or you can use rmdef -t osimage to remove the osimage definition.
The statelite image files on the diskful service nodes will not be removed, remove the image files on the service nodes
manually if necessary, for example, use command “rsync -az –delete /install <sn>:/” to remove the image files on the
service nodes, where the <sn> is the hostname of the service node.
Parameters
imagename specifies the name of an os image definition to be used. The specification for the image is stored in the
osimage table and linuximage table.
OPTIONS
-h | --help Display usage message.
-V | --verbose Verbose mode.
--xcatdef Remove the xCAT osimage definition
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To remove a RHEL 7.1 stateless image for a compute node architecture x86_64, enter:
rmimage rhels7.1-x86_64-netboot-compute
2. To remove a rhels5.5 statelite image for a compute node architecture ppc64 and the osimage definition, enter:
rmimage rhels5.5-ppc64-statelite-compute --xcatdef
FILES
/opt/xcat/sbin/rmimage
NOTES
This command is part of the xCAT software product.
SEE ALSO
genimage(1)|genimage.1, packimage(1)|packimage.1
1.3. Admin Guide
435
xCAT3 Documentation, Release 2.12
rmkit.1
NAME
rmkit - Remove Kits from xCAT
SYNOPSIS
rmkit [-? | -h | --help] [-v | --version]
rmkit [-V | --verbose] [-f | --force] [-t | --test] kitlist
DESCRIPTION
The rmkit command removes kits on the xCAT management node from kit names.
Note: The xCAT support for Kits is only available for Linux operating systems.
OPTIONS
-h|--help
Display usage message.
-V|--verbose
Verbose mode.
-v|--version
Command version.
-f|--force
Remove this kit even there is any component in this kit is listed by osimage.kitcomponents. If this option
is not specified, this kit will not be removed if any kit components listed in an osimage.kitcomponents
-t|--test
Test if kitcomponents in this kit are used by osimage
kitlist
A comma delimited list of kits that are to be removed from the xCAT cluster. Each entry can be a kitname
or kit basename. For kit basename, rmkit command will remove all the kits that have that kit basename.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
436
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
EXAMPLES
1. To remove two kits from tarball files.
rmkit kit-test1,kit-test2
Output is similar to:
Kit kit-test1-1.0-Linux,kit-test2-1.0-Linux was successfully removed.
2. To remove two kits from tarball files even the kit components in them are still being used by osimages.
rmkit kit-test1,kit-test2 --force
Output is similar to:
Kit kit-test1-1.0-Linux,kit-test2-1.0-Linux was successfully removed.
3. To list kitcomponents in this kit used by osimage
rmkit kit-test1,kit-test2 -t
Output is similar to:
kit-test1-kitcomp-1.0-Linux is being used by osimage osimage-test
Following kitcomponents are in use: kit-test1-kitcomp-1.0-Linux
SEE ALSO
lskit(1)|lskit.1,
addkit(1)|addkit.1,
comp(1)|chkkitcomp.1
addkitcomp(1)|addkitcomp.1,
rmkitcomp(1)|rmkitcomp.1,
chkkit-
rmkitcomp.1
NAME
rmkitcomp - Remove Kit components from an xCAT osimage.
SYNOPSIS
rmkitcomp [-? | -h | --help] [-v | --version]
rmkitcomp [-V | --verbose] [-u | --uninstall] [-f | --force] [--noscripts] -i osimage kitcompname_list
DESCRIPTION
The rmkitcomp command removes kit components from an xCAT osimage. All the kit component attribute values that
are contained in the osimage will be removed, and the kit component meta rpm and package rpm could be uninstalled
by -u|--uninstall option.
Note: The xCAT support for Kits is only available for Linux operating systems.
1.3. Admin Guide
437
xCAT3 Documentation, Release 2.12
OPTIONS
-u|--uninstall
All the kit component meta rpms and package rpms in otherpkglist will be uninstalled during genimage
for stateless image and updatenode for stateful nodes.
-h|--help
Display usage message.
-V|--verbose
Verbose mode.
-v|--version
Command version.
-f|--force
Remove this kit component from osimage no matter it is a dependency of other kit components.
--noscripts
Do not remove kitcomponent’s postbootscripts from osimage
-i osimage
osimage name that include this kit component.
kitcompname_list
A comma-delimited list of valid full kit component names or kit component basenames that are to be
removed from the osimage. If a basename is specified, all kitcomponents matching that basename will be
removed from the osimage.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To remove a kit component from osimage
rmkitcomp -i rhels6.2-ppc64-netboot-compute comp-test1-1.0-1-rhels-6.2-ppc64
Output is similar to:
kitcomponents comp-test1-1.0-1-rhels-6.2-ppc64 were removed from osimage rhels6.2˓→ppc64-netboot-compute successfully
2. To remove a kit component even it is still used as a dependency of other kit component.
rmkitcomp -f -i rhels6.2-ppc64-netboot-compute comp-test1-1.0-1-rhels-6.2-ppc64
Output is similar to:
438
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
kitcomponents comp-test1-1.0-1-rhels-6.2-ppc64 were removed from osimage rhels6.2˓→ppc64-netboot-compute successfully
3. To remove a kit component from osimage and also remove the kit component meta RPM and package RPM.
So in next genimage for statelss image and updatenode for stateful nodes, the kit component meta RPM and
package RPM will be uninstalled.
rmkitcomp -u -i rhels6.2-ppc64-netboot-compute comp-test1-1.0-1-rhels-6.2-ppc64
Output is similar to:
kitcomponents comp-test1-1.0-1-rhels-6.2-ppc64 were removed from osimage rhels6.2˓→ppc64-netboot-compute successfully
SEE ALSO
lskit(1)|lskit.1, addkit(1)|addkit.1, rmkit(1)|rmkit.1, addkitcomp(1)|addkitcomp.1, chkkitcomp(1)|chkkitcomp.1
rmnimimage.1
NAME
rmnimimage - Use this xCAT command to remove NIM resources specified in an xCAT osimage definition.
SYNOPSIS
rmnimimage [-h|--help]
rmnimimage [-V|--verbose] [-f|--force] [-d|--delete] [-x|--xcatdef] [-M|--managementnode] [-s servicenoderange]
osimage_name
DESCRIPTION
Use this xCAT command to remove the AIX resources specified in an xCAT osimage definition.
To list the contents of the xCAT osimage definition use the xCAT lsdef command (“lsdef -t osimage -l -o <osimage_name>”). Before running the rmnimimage command you should be absolutely certain that you really want
to remove the NIM resources specified in the xCAT osimage definition!
The default behavior of this command is to remove all the NIM resources, except the lpp_source, on the xCAT
management node in addition to the resources that were replicated on any xCAT service nodes.
This command may also be used to clean up individual xCAT service nodes and remove the xCAT osimage definitions.
The “nim -o remove” operation is used to remove the NIM resource definitions. If you wish to completely remove all
the files and directories (left behind by the NIM command) you must specify the “-d” option when you run rmnimimage. The “-d” option will also remove the lpp_source resource.
If you wish to remove the NIM resource from one or more xCAT service nodes without removing the resources from
the management node you can use the “-s <servicenoderange>” option. In this case the NIM resources specified in
the xCAT osimage definition will be removed from the service nodes ONLY. The NIM resources on the management
node will not be removed.
1.3. Admin Guide
439
xCAT3 Documentation, Release 2.12
If you wish to remove NIM resources on the management node only, you can specify the “-M” option.
If you wish to also remove the xCAT osimage definition you must specify the “-x” option.
This command will not remove NIM resources if they are currently being used in another xCAT osimage definition.
To see which resources are common between osimages you can specify the “-V” option. You can override this check
by specifying the “-f” option.
This command will not remove NIM resources if they are currently allocated. You must deallocate the resources
before they can be removed. See the xcat2nim and rmdsklsnode commands for information on how to deallocate and
remove NIM machine definitions for standalone and diskless nodes.
See the AIX NIM documentation for additional details on how to deallocate and remove unwanted NIM objects.
OPTIONS
-h |--help
Display usage message.
-d|--delete
Delete any files or directories that were left after the “nim -o remove” command was run. This option
will also remove the lpp_source resouce and all files contained in the lpp_source directories. When this
command completes all definitions and files will be completely erased so use with caution!
-f|--force
Override the check for shared resources when removing an xCAT osimage.
-M|--managementnode
Remove NIM resources from the xCAT management node only.
-s servicenoderange
Remove the NIM resources on these xCAT service nodes only. Do not remove the NIM resources from
the xCAT management node.
osimage_name
The name of the xCAT osimage definition.
-V|--verbose
Verbose mode. This option will display the underlying NIM commands that are being called.
-x|--xcatdef
Remove the xCAT osimage definition.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
440
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
EXAMPLES
1. Remove all NIM resources specified in the xCAT “61image” definition.
rmnimimage 61image
The “nim -o remove” operation will be used to remove the NIM resource definitions on the management node as well
as any service nodes where the resource has been replicated. This NIM operation does not completely remove all files
and directories associated with the NIM resources.
2. Remove all the NIM resources specified by the xCAT “61rte” osimage definition. Delete ALL files and directories associated with the NIM resources. This will also remove the lpp_source resource.
rmnimimage -d 61rte
3. Remove all the NIM resources specified by the xCAT “614img” osimage definition and also remove the xCAT
definition.
rmnimimage -x -d 614img
Note: When this command completes all definitions and files will be completely erased, so use with caution!
4. Remove the NIM resources specified in the “614dskls” osimage definition on the xcatsn1 and xcatsn2 service
nodes. Delete all files or directories associated with the NIM resources.
rmnimimage -d -s xcatsn1,xcatsn2 614dskls
5. Remove the NIM resources specified in the “614old” osimage definition on the xCAT management node only.
rmnimimage -M -d 614old
FILES
/opt/xcat/bin/rmnimimage
NOTES
This command is part of the xCAT software product.
SEE ALSO
mknimimage(1)|mknimimage.1
rmvlan.1
NAME
rmvlan - It remves the vlan from the cluster.
1.3. Admin Guide
441
xCAT3 Documentation, Release 2.12
SYNOPSIS
rmvlan vlanid
rmvlan [-h | --help]
rmvlan [-v | --version]
DESCRIPTION
The rmvlan command removes the given vlan ID from the cluster. It removes the vlan id from all the swithces
involved, deconfigures the nodes so that vlan adaptor (tag) will be remved, cleans up /etc/hosts, DNS and database
tables for the given vlan.
For added security, the root guard and bpdu guard were enabled for the ports in this vlan by mkvlan and chvlan
commands. However, the guards will not be disabled by this command. To disable them, you need to use the switch
command line interface. Refer to the switch command line interface manual to see how to disable the root guard and
bpdu guard for a port.
Parameters
vlanid is a unique vlan number.
OPTIONS
-h|--help Display usage message.
-v|--version The Command Version.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To remove vlan 3
rmvlan 3
If the nodes are KVM guest then the do the following after the vlan is removed: rpower node1,node2 off
rmvm node1,node2
FILES
/opt/xcat/bin/rmvlan
442
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
SEE ALSO
mkvlan(1)|mkvlan.1, chvlan(1)|chvlan.1, lsvlan(1)|lsvlan.1
rmvm.1
NAME
rmvm - Removes HMC-, DFM-, IVM-, KVM-, VMware- and zVM-managed partitions or virtual machines.
SYNOPSIS
rmvm [-h| --help]
rmvm [-v| --version]
rmvm [-V| --verbose] noderange [-r] [--service]
For KVM and VMware:
rmvm [-p] [-f] noderange
PPC (using Direct FSP Management) specific:
rmvm [-p] noderange
DESCRIPTION
The rmvm command removes the partitions specified in noderange. If noderange is an CEC, all the partitions associated with that CEC will be removed. Note that removed partitions are automatically removed from the xCAT database.
For IVM-managed systems, care must be taken to not remove the VIOS partition, or all the associated partitions will
be removed as well.
For DFM-managed (short For Direct FSP Management mode) normal power machines, only partitions can be removed.
No options is needed.
OPTIONS
-h|--help Display usage message.
-v|--version Command Version.
-V|--verbose Verbose output.
-r Retain the data object definitions of the nodes.
--service Remove the service partitions of the specified CECs.
-p KVM: Purge the existence of the VM from persistent storage. This will erase all storage related to the VM in
addition to removing it from the active virtualization configuration. PPC: Remove the specified partition on normal
power machine.
1.3. Admin Guide
443
xCAT3 Documentation, Release 2.12
-f Force remove the VM, even if the VM appears to be online. This will bring down a live VM if requested.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To remove the HMC-managed partition lpar3, enter:
rmvm lpar3
Output is similar to:
lpar3: Success
2. To remove all the HMC-managed partitions associated with CEC cec01, enter:
rmvm cec01
Output is similar to:
lpar1: Success
lpar2: Success
lpar3: Success
3. To remove the HMC-managed service partitions of the specified CEC cec01 and cec02, enter:
rmvm cec01,cec02 --service
Output is similar to:
cec01: Success
cec02: Success
4. To remove the HMC-managed partition lpar1, but retain its definition, enter:
rmvm lpar1 -r
Output is similar to:
lpar1: Success
5. To remove a zVM virtual machine:
rmvm gpok4
Output is similar to:
gpok4: Deleting virtual server LNX4... Done
6. To remove a DFM-managed partition on normal power machine:
444
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
rmvm lpar1
Output is similar to:
lpar1: Done
FILES
/opt/xcat/bin/rmvm
SEE ALSO
mkvm(1)|mkvm.1, lsvm(1)|lsvm.1, chvm(1)|chvm.1
rmzone.1
NAME
rmzone - Removes a zone from the cluster.
SYNOPSIS
rmzone zonename [-g] [-f]
rmzone [-h | -v]
DESCRIPTION
The rmzone command is designed to remove a previously defined zone from the cluster. It will remove the zone entry
in the zone table. It will remove the zone from the zonename attributes on the nodes that were assigned to the zone.
Optionally, it will remove the zonename group from the nodes that were assigned to the zone. It will also remove the
root ssh keys that were created for that zone on the Management Node. The rmzone command is only supported on
Linux ( No AIX support). The nodes are not automatically updated with new root ssh keys by rmzone. You must run
updatenode -k or xdsh -K to the nodes to update the root ssh keys. The nodes new ssh key will be assigned from the
defaultzone in the zone table, or if no entries in the zone table, the keys will come from /root/.ssh. Note: if any zones
in the zone table, there must be one and only one defaultzone. Otherwise, errors will occur.
OPTIONS
-h | --help
Displays usage information.
-v | --version
Displays command version and build date.
-f | --force
1.3. Admin Guide
445
xCAT3 Documentation, Release 2.12
Used to remove a zone that is defined as current default zone. This should only be done if you are
removing all zones, or you will adding a new zone or changing an existing zone to be the default zone.
-g | --assigngroup
Remove the assigned group named zonename from all nodes assigned to the zone being removed.
-V | --Verbose
Verbose mode.
Examples
1. To remove zone1 from the zone table and the zonename attribute on all it’s assigned nodes , enter:
rmzone zone1
2. To remove zone2 from the zone table, the zone2 zonename attribute, and the zone2 group assigned to all nodes
that were in zone2, enter:
rmzone zone2 -g
3. To remove zone3 from the zone table, all the node zone attributes and override the fact it is the defaultzone,
enter:
rmzone zone3 -g -f
Files
/opt/xcat/bin/rmzone/
Location of the rmzone command.
SEE ALSO
L <mkzone(1)|mkzone.1>,L <chzone(1)|chzone.1>,L <xdsh(1)|xdsh.1>, updatenode(1)|updatenode.1
rnetboot.1
NAME
rnetboot - Cause the range of nodes to boot to network.
SYNOPSIS
rnetboot [-V | --verbose] [-s boot_device_order] [-F] [-f] noderange [-m table.column==expectedstatus [-m table.column=~expectedstatus]] [-t timeout] [-r retrycount]
rnetboot [-h | --help] [-v | --version]
zVM specific:
rnetboot noderange [ipl= address]
446
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
DESCRIPTION
The rnetboot command will do what is necessary to make each type of node in the given noderange boot from the
network. This is usually used to boot the nodes stateless or to network install system p nodes.
OPTIONS
-s
Set the boot device order. Accepted boot devices are hd and net.
-F
Force reboot the system no matter what state the node is. By default, rnetboot will not reboot the node if node is in
‘boot’ state.
-f
Force immediate shutdown of the partition.
-m
Use one or multiple -m flags to specify the node attributes and the expected status for the node installation monitoring
and automatic retry mechanism. The operators ==, !=, =~ and !~ are valid. This flag must be used with -t flag.
Note: if the “val” fields includes spaces or any other characters that will be parsed by shell, the “attr<oper-ator>val”
needs to be quoted. If the operator is ”!~”, the “attr<operator>val” needs to be quoted using single quote.
-r
specify the number of retries that the monitoring process will perform before declaring the failure. The default value is
3. Setting the retrycount to 0 means only monitoring the os installation progress and will not re-initiate the installation
if the node status has not been changed to the expected value after timeout. This flag must be used with -m flag.
-t
Specify the timeout, in minutes, to wait for the expectedstatus specified by -m flag. This is a required flag if the -m
flag is specified.
-V|--verbose
Verbose output.
-h|--help
Display usage message.
-v|--version
Command Version.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
1.3. Admin Guide
447
xCAT3 Documentation, Release 2.12
EXAMPLES
rnetboot 1,3
rnetboot 14-56,70-203
rnetboot 1,3,14-56,70-203
rnetboot all,-129-256
rnetboot all -s hd,net
rnetboot all ipl=00c
SEE ALSO
nodeset(8)|nodeset.8
rollupdate.1
NAME
rollupdate - performs cluster rolling update
SYNOPSIS
cat stanza-file | rollupdate [-V | --verbose] [-t| --test]
rollupdate [-? | -h | --help | -v | --version]
DESCRIPTION
The rollupdate command creates and submits scheduler reservation jobs that will notify xCAT to shutdown a group of
nodes, run optional out-of-band commands from the xCAT management node, and reboot the nodes. Currently, only
LoadLeveler is supported as a job scheduler with rollupdate.
Input to the rollupdate command is passed in as stanza data through STDIN. Information such as the
sets of nodes that will be updated, the name of the job scheduler, a template for generating job command files, and other control data are required. See /opt/xcat/share/xcat/rollupdate/rollupdate.input.sample and
/opt/xcat/share/xcat/rollupdate/rollupdate_all.input.sample for stanza keywords, usage, and examples.
The rollupdate command will use the input data to determine each set of nodes that will be managed together as an
update group. For each update group, a job scheduler command file is created and a reservation request is submitted.
When the group of nodes becomes available and the scheduler activates the reservation, the xcatd daemon on the
management node will be notified to begin the update process for all the nodes in the update group. If specified,
prescripts will be run, an operating system shutdown command will be sent to each node, out-of-band operations can
be run on the management node, and the nodes are powered back on.
The rollupdate command assumes that, if the update is to include rebooting stateless or statelite nodes to a new
operating system image, the image has been created and tested, and that all relevant xCAT commands have been run
for the nodes such that the new image will be loaded when xCAT reboots the nodes.
448
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
OPTIONS
-v|--version
Command Version.
-V|--verbose
Display additional progress and error messages. Output is also logged in /var/log/xcat/rollupdate.log.
-t|--test
Run the rollupdate command in test mode only to verify the output files that are created. No scheduler
reservation requests will be submitted.
-?|-h|--help
Display usage message.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To run a cluster rolling update based on the information you have created in the file
/u/admin/rolling_updates/update_all.stanza enter:
cat /u/admin/rolling_updates/update_all.stanza | rollupdate
FILES
/opt/xcat/bin/rollupdate
/opt/xcat/share/xcat/rollupdate/rollupdate.input.sample
/opt/xcat/share/xcat/rollupdate/ll.tmpl
/opt/xcat/share/xcat/rollupdate/rollupdate_all.input.sample
/opt/xcat/share/xcat/rollupdate/llall.tmpl
/var/log/xcat/rollupdate.log
SEE ALSO
rpower.1
NAME
rpower - remote power control of nodes
1.3. Admin Guide
449
xCAT3 Documentation, Release 2.12
SYNOPSIS
rpower noderange [--nodeps] [on | onstandby | off | suspend | stat | state | reset | boot] [-m table.column==expectedstatus [-m table.column=~expectedstatus]] [-t timeout] [-r retrycount]
rpower [-h | --help | -v | --version]
BMC (using IPMI):
rpower noderange [on | off | softoff | reset | boot | stat | state | status | wake | suspend [-w timeout] [-o] [-r]]
rpower noderange [pduon | pduoff | pdustat | pdureset]
OpenPOWER BMC (using IPMI):
rpower noderange [on | off | reset | boot | stat | state | status]
rpower noderange [pduon | pduoff | pdustat | pdureset]
OpenPOWER OpenBMC:
rpower noderange [off | on | reset | boot | stat | state | status]
PPC (with IVM or HMC) specific:
rpower noderange [--nodeps] {of}
CEC (with HMC) specific:
rpower noderange [on | off | reset | boot | onstandby]
LPAR (with HMC) specific:
rpower noderange [on | off | stat | state | reset | boot | of | sms | softoff]
CEC (using Direct FSP Management) specific:
rpower noderange [onstandby | stat | state] [-T tooltype]
rpower noderange [on | off | resetsp]
Frame (using Direct FSP Management) specific:
rpower noderange [rackstandby | exit_rackstandby | stat | state | resetsp]
450
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
LPAR (using Direct FSP Management) specific:
rpower noderange [on | off | stat | state | reset | boot | of | sms]
Blade (using Direct FSP Management) specific:
rpower noderange [on | onstandby | off | stat | state | sms]
Blade specific:
rpower noderange [cycle | softoff]
Lenovo High-Density Server specific:
rpower noderange [on | off | reset | boot | reseat]
zVM specific:
rpower noderange [on | off | reset | stat | softoff]
docker specific:
rpower noderange [start | stop | restart | pause | unpause | state]
pdu specific:
rpower noderange [stat | off | on | reset]
DESCRIPTION
rpower controls the power for a single or range of nodes, via the out-of-band path.
OPTIONS
on
Turn power on.
onstandby
Turn power on to standby state
-T
The value could be lpar or fnm. The tooltype value lpar is for xCAT and fnm is for CNM. The default
value is “lpar”. For cold start in the large cluster, it will save a lot of time if the admins use “rpower
noderange onstandby -T fnm” to power on all the CECs from the management node through the fnm
connections.
1.3. Admin Guide
451
xCAT3 Documentation, Release 2.12
rackstandby
Places the rack in the rack standby state. It requires that all CECs and DE be powered off before it will
run.
exit_rackstandby
Exit Rack standby will be the default state that a rack goes into when power is initially applied to the rack.
It simply moves the BPA from Rack standby to both bpa’s in standby state.
resetsp
Reboot the service processor. If there are primary and secondary FSPs/BPAs of one cec/frame, it will
reboot them almost at the same time.
softoff
Attempt to request clean shutdown of OS (may not detect failures in completing command)
off
Turn power off.
suspend
Suspend the target nodes execution.
The suspend action could be run together with -w -o -r.
Refer to the following steps to enable the suspend function:
1. Add the ‘acpid’ and ‘suspend’(the suspend package is not needed on RHEL) package to the .pkglist
of your osimage so that the required package could be installed correctly to your target system.
2. Add two configuration files for the base function:
/etc/pm/config.d/suspend
S2RAM_OPTS="--force --vbe_save --vbe_post --vbe_mode"
/etc/acpi/events/suspend_event
event=button/sleep.*
action=/usr/sbin/pm-suspend
3. Add the hook files for your specific applications which need specific action before or after the
suspend action.
Refer to the ‘pm-utils’ package for how to create the specific hook files.
wake
Wake up the target nodes which is in suspend state.
Don’t try to run wake against the ‘on’ state node, it would cause the node gets to ‘off’ state.
For some of xCAT hardware such as NeXtScale, it may need to enable S3 before using wake. The
following steps can be used to enable S3. Reference pasu(1)|pasu.1 for “pasu” usage.
[root@xcatmn home]# echo "set Power.S3Enable Enable" > power-setting
[root@xcatmn home]# pasu -b power-setting node01
node01: Batch mode start.
node01: [set Power.S3Enable Enable]
node01: Power.S3Enable=Enable
node01:
node01: Beginning intermediate batch update.
node01: Waiting for command completion status.
452
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
node01: Command completed successfully.
node01: Completed intermediate batch update.
node01: Batch mode completed successfully.
[root@xcatmn home]# pasu node01 show all|grep -i s3
node01: IMM.Community_HostIPAddress3.1=
node01: IMM.Community_HostIPAddress3.2=
node01: IMM.Community_HostIPAddress3.3=
node01: IMM.DNS_IP_Address3=0.0.0.0
node01: IMM.IPv6DNS_IP_Address3=::
node01: Power.S3Enable=Enable
stat | state
Print the current power state/status.
reset
Send a hard reset.
boot
If off, then power on. If on, then hard reset. This option is recommended over cycle.
cycle
Power off, then on.
reseat
For Lenovo high-density servers, simulates unplugging and replugging the node into the chassis.
of
Boot the node to open firmware console mode.
sms
Boot the node to open firmware SMS menu mode.
-m table.column==expectedstatus -m table.column=~expectedstatus
Use one or multiple -m flags to specify the node attributes and the expected status for the node installation
monitoring and automatic retry mechanism. The operators ==, !=, =~ and !~ are valid. This flag must be
used with -t flag.
Note: if the “val” fields includes spaces or any other characters that will be parsed by shell, the “attr<operator>val” needs to be quoted. If the operator is ”!~”, the “attr<operator>val” needs to be quoted using
single quote.
--nodeps
Do not use dependency table (default is to use dependency table). Valid only with on|off|boot|reset|cycle
for blade power method and on|off|reset|softoff for hmc/fsp power method.
-r retrycount
specify the number of retries that the monitoring process will perform before declaring the failure. The
default value is 3. Setting the retrycount to 0 means only monitoring the os installation progress and will
not re-initiate the installation if the node status has not been changed to the expected value after timeout.
This flag must be used with -m flag.
-t timeout
1.3. Admin Guide
453
xCAT3 Documentation, Release 2.12
Specify the timeout, in minutes, to wait for the expectedstatus specified by -m flag. This is a required flag
if the -m flag is specified.
Power off, then on.
-w timeout
To set the timeout for the suspend action to wait for the success.
-o
To specify that the target node will be power down if suspend action failed.
-r
To specify that the target node will be reset if suspend action failed.
start
To start a created docker instance.
stop
To stop a created docker instance.
restart
To restart a created docker instance.
pause
To pause all processes in the instance.
unpause
To unpause all processes in the instance.
state
To get state of the instance.
-h | --help
Prints out a brief usage message.
-v | --version
Display the version number.
EXAMPLES
1. To display power status of nodes4 and note5
rpower node4,node5 stat
Output is similar to:
node4: on
node5: off
2. To power on node5
rpower node5 on
Output is similar to:
454
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
node5: on
SEE ALSO
noderange(3)|noderange.3, rcons(1)|rcons.1, rinv(1)|rinv.1, rvitals(1)|rvitals.1, rscan(1)|rscan.1
rscan.1
NAME
rscan - Collects node information from one or more hardware control points.
SYNOPSIS
rscan [-h|--help]
rscan [-v|--version]
rscan [-V|--verbose] noderange [-u][-w][-x|-z]
DESCRIPTION
The rscan command lists hardware information for each node managed by the hardware control points specified in
noderange.
For the management module of blade, if the blade server is a Flex system P node, the fsp belongs to the blade server
also will be scanned.
For the KVM host, all the KVM guests on the specified KVM host will be scanned.
Note: The first line of the output always contains information about the hardware control point. When using the rscan
command to generate output for HMC or IVM hardware control points, it provides the FSPs and BPAs as part of the
output. The only exception is the rscan -u flag which provides updates made hardware control point in the xCAT
database.
OPTIONS
-h|--help Display usage message.
-v|--version Command Version.
-V|--verbose Verbose output.
-u Updates and then prints out node definitions in the xCAT database for CEC/BPA. It updates the existing nodes that
contain the same mtms and serial number for nodes managed by the specified hardware control point. This primarily
works with CEC/FSP and frame/BPA nodes when the node name is not the same as the managed system name on
hardware control point (HMC), This flag will update the BPA/FSP node name definitions to be listed as the managed
system name in the xCAT database.
For the Flex system manager, both the blade server and fsp object of xCAT will be updated if the mpa and slot id are
matched to the object which has been defined in the xCAT database.
For KVM host, the information of the KVM guests which have been defined in xCAT database will be updated.
1.3. Admin Guide
455
xCAT3 Documentation, Release 2.12
Note: only the matched object will be updated.
-n For KVM host, the information of the KVM guests, which are not defined in xCAT database yet, will be written
into xCAT database.
-w Writes output to xCAT database.
For KVM host, updates the information of the KVM guests which have been defined in xCAT database with the same
node name and KVM host, creates the definition of the KVM guests which do not exist in xCAT database , and notifies
user about the conflicting KVM guests that the name exist in xCAT database but the kvm host is different.
-x XML format.
-z Stanza formated output.
Note: For KVM host, -z is not a valid option for rscan.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To list all nodes managed by HMC hmc01 in tabular format, enter:
rscan hmc01
Output is similar to:
type
name
hmc
fsp
lpar
lpar
lpar
lpar
hmc01
Server-9117-MMA-SN10F6F3D
lpar3
lpar2
lpar1
p6vios
id
4
3
2
1
type-model
serial-number
address
7310-C05
9117-MMA
9117-MMA
9117-MMA
9117-MMA
9117-MMA
10F426A
10F6F3D
10F6F3D
10F6F3D
10F6F3D
10F6F3D
hmc01
3.3.3.197
2. To list all nodes managed by IVM ivm02 in XML format and write the output to the xCAT database, enter:
rscan ivm02 -x -w
Output is similar to:
<Node>
<cons></cons>
<profile></profile>
<parent></parent>
<serial>10B7D1G</serial>
<model>9133-55A</model>
<node>Server-9133-55A-10B7D1G</node>
<mgt>ivm</mgt>
<nodetype>fsp</nodetype>
<hcp>ivm02</hcp>
<groups>fsp,all</groups>
<id>10</id>
456
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
</Node>
<Node>
<cons>ivm</cons>
<profile>lpar01</profile>
<parent>Server-9133-55A-10B7D1G</parent>
<serial></serial>
<model></model>
<node>lpar01</node>
<mgt>ivm</mgt>
<nodetype>lpar,osi</nodetype>
<hcp>ivm02</hcp>
<groups>lpar,all</groups>
<id>1</id>
<Node>
</Node>
<cons>ivm</cons>
<profile>lpar02</profile>
<parent>Server-9133-55A-10B7D1G</parent>
<serial></serial>
<model></model>
<node>lpar02</node>
<mgt>ivm</mgt>
<nodetype>lpar,osi</nodetype>
<hcp>ivm02</hcp>
<groups>lpar,all</groups>
<id>2</id>
</Node>
3. To list all nodes managed by HMC hmc02 in stanza format and write the output to the xCAT database, enter:
rscan hmc02 -z -w
Output is similar to:
Server-9458-100992001Y_B:
objtype=node
nodetype=bpa
id=2
model=9458-100
serial=992001Y
hcp=hmc02
profile=
parent=
groups=bpa,all
mgt=hmc
cons=
Server-9119-590-SN02C5F9E:
objtype=node
type=fsp
id=10
model=9119-590
serial=02C5F9E
hcp=hmc02
profile=
parent=Server-9458-100992001Y_B
1.3. Admin Guide
457
xCAT3 Documentation, Release 2.12
groups=fsp,all
mgt=hmc
cons=
lpar01:
objtype=node
nodetype=lpar,osi
id=1
model=
serial=
hcp=hmc02
profile=lpar01
parent=Server-9119-590-SN02C5F9E
groups=lpar,all
mgt=hmc
cons=hmc
lpar02:
objtype=node
nodetype=lpar,osi
id=2
model=
serial=
hcp=hmc02
profile=lpar02
parent=Server-9119-590-SN02C5F9E
groups=lpar,all
mgt=hmc
cons=hmc
4. To update definitions of nodes, which is managed by hmc03, enter:
rscan hmc03 -u
Output is similar to:
#Updated following nodes:
type
name
fsp
Server-9125-F2A-SN0262672-B
˓→243
id
3
type-model
9125-F2A
serial-number
0262672
address
192.168.200.
5. To collects the node information from one or more hardware control points on zVM AND populate the database
with details collected by rscan:
rscan gpok2 -w
Output is similar to:
gpok2:
objtype=node
arch=s390x
os=sles10sp3
hcp=gpok3.endicott.ibm.com
userid=LINUX2
nodetype=vm
parent=POKDEV61
groups=all
mgt=zvm
458
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
6. To scan the Flex system cluster:
rscan cmm01
Output is similar to:
type
cmm
blade
xblade
blade
name
AMM680520153
SN#YL10JH184067
SN#YL10JH184068
SN#YL10JH184079
id
0
1
2
3
type-model
789392X
789542X
789542X
789542X
serial-number
100048A
10F752A
10F652A
10F697A
mpa
cmm01
cmm01
cmm01
cmm01
address
cmm01
12.0.0.9
12.0.0.10
12.0.0.11
7. To update the Flex system cluster:
rscan cmm01 -u
Output is similar to:
cmm
blade
blade
[AMM680520153]
[SN#YL10JH184067]
[SN#YL10JH184079]
Matched To =>[cmm01]
Matched To =>[cmm01node01]
Matched To =>[cmm01node03]
8. To scan the KVM host “hyp01”, write the KVM guest information into xCAT database:
rscan hyp01 -w
9. To update definitions of kvm guest, which is managed by hypervisor hyp01, enter:
rscan hyp01 -u
Output is similar to:
type
name
kvm
kvm2
˓→hda.qcow2
kvm
kvm1
˓→hda.qcow2
hypervisor
hyp01
id
12
cpu
2
memory
1024
nic
virbr0
disk
/install/vms/kvm2.
hyp01
10
1
1024
virbr0
/install/vms/kvm1.
FILES
/opt/xcat/bin/rscan
SEE ALSO
lsslp(1)|lsslp.1
rsetboot.1
SYNOPSIS
rsetboot noderange {hd | net | cd | default | stat} [-u] [-p]
rsetboot [-h | --help | -v | --version]
1.3. Admin Guide
459
xCAT3 Documentation, Release 2.12
DESCRIPTION
rsetboot sets the boot media and boot mode that should be used on the next boot of the specified nodes. After the
nodes are booted with the specified device and boot mode (e.g. via rpower(1)|rpower.1), the nodes will return to using
the default boot device specified in the BIOS. Currently this command is only supported for IPMI nodes.
OPTIONS
hd
Boot from the hard disk.
net
Boot over the network, using a PXE or BOOTP broadcast.
cd
Boot from the CD or DVD drive.
def | default
Boot using the default set in BIOS.
stat
Display the current boot setting.
-u
To specify the next boot mode to be “UEFI Mode”.
-p
To make the specified boot device and boot mode settings persistent.
EXAMPLES
1. Set nodes 1 and 3 to boot from the network on the next boot:
rsetboot node1,node3 net
2. Display the next-boot value for nodes 14-56 and 70-203:
rsetboot node[14-56],node[70-203] stat
3. Restore the next-boot value for these nodes back to their default set in the BIOS:
rsetboot node1,node3,node[14-56],node[70-203] default
SEE ALSO
rbootseq(1)|rbootseq.1
460
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
rspconfig.1
NAME
rspconfig - Configures nodes’ service processors
SYNOPSIS
rspconfig [-h | --help | -v | --version]
BMC/MPA specific:
rspconfig noderange {alert | snmpdest | community}
rspconfig noderange alert={on | enable | off | disable}
rspconfig noderange snmpdest=snmpmanager-IP
rspconfig noderange community={public | string}
BMC specific:
rspconfig noderange {ip | netmask | gateway | backupgateway | garp | vlan}
rspconfig noderange garp=time
OpenBMC specific:
rspconfig noderange {ip | netmask | gateway | vlan}
MPA specific:
rspconfig noderange {sshcfg | snmpcfg | pd1 | pd2 | network | swnet | ntp | textid | frame}
rspconfig noderange USERID={newpasswd} updateBMC={y | n}
rspconfig noderange sshcfg={enable | disable}
rspconfig noderange snmpcfg={enable | disable}
rspconfig noderange solcfg={enable | disable}
rspconfig noderange pd1={nonred | redwoperf | redwperf}
rspconfig noderange pd2={nonred | redwoperf | redwperf}
rspconfig noderange network={[ip],[host],[gateway],[netmask]|*}
rspconfig noderange initnetwork={[ip],[host],[gateway],[netmask]|*}
rspconfig noderange textid={* | textid}
rspconfig singlenode frame={frame_number}
rspconfig noderange frame={*}
1.3. Admin Guide
461
xCAT3 Documentation, Release 2.12
rspconfig noderange swnet={[ip],[gateway],[netmask]}
rspconfig noderange ntp={[ntpenable],[ntpserver],[frequency],[v3]}
FSP/CEC specific:
rspconfig noderange {autopower | iocap | decfg | memdecfg | procdecfg | time | date | spdump | sysdump | network}
rspconfig noderange autopower={enable | disable}
rspconfig noderange iocap={enable | disable}
rspconfig noderange time=hh:mm:ss
rspconfig noderange date=mm:dd:yyyy
rspconfig noderange decfg={enable|disable:policyname,...}
rspconfig noderange procdecfg={configure|deconfigure:processingunit:id,...}
rspconfig noderange memdecfg={configure|deconfigure:processingunit:unit|bank:id,...>}
rspconfig noderange network={nic,*}
rspconfig noderange network={nic,[IP,][hostname,][gateway,][netmask]}
rspconfig noderange network={nic,0.0.0.0}
rspconfig noderange HMC_passwd={currentpasswd,newpasswd}
rspconfig noderange admin_passwd={currentpasswd,newpasswd}
rspconfig noderange general_passwd={currentpasswd,newpasswd}
rspconfig noderange *_passwd={currentpasswd,newpasswd}
rspconfig noderange {hostname}
rspconfig noderange hostname={* | name}
rspconfig noderange --resetnet
Flex system Specific:
rspconfig noderange sshcfg={enable | disable}
rspconfig noderange snmpcfg={enable | disable}
rspconfig noderange network={[ip],[host],[gateway],[netmask] | *}
rspconfig noderange solcfg={enable | disable}
rspconfig noderange textid={* | textid}
rspconfig noderange cec_off_policy={poweroff | stayon}
BPA/Frame Specific:
rspconfig noderange {network}
rspconfig noderange network={nic,*}
462
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
rspconfig noderange network={nic,[IP,][hostname,][gateway,][netmask]}
rspconfig noderange network={nic,0.0.0.0}
rspconfig noderange HMC_passwd={currentpasswd,newpasswd}
rspconfig noderange admin_passwd={currentpasswd,newpasswd}
rspconfig noderange general_passwd={currentpasswd,newpasswd}
rspconfig noderange *_passwd={currentpasswd,newpasswd}
rspconfig noderange {hostname}
rspconfig noderange hostname={* | name}
rspconfig noderange --resetnet
FSP/CEC (using Direct FSP Management) Specific:
rspconfig noderange HMC_passwd={currentpasswd,newpasswd}
rspconfig noderange admin_passwd={currentpasswd,newpasswd}
rspconfig noderange general_passwd={currentpasswd,newpasswd}
rspconfig noderange *_passwd={currentpasswd,newpasswd}
rspconfig noderange {sysname}
rspconfig noderange sysname={* | name}
rspconfig noderange {pending_power_on_side}
rspconfig noderange pending_power_on_side={temp | perm}
rspconfig noderange {cec_off_policy}
rspconfig noderange cec_off_policy={poweroff | stayon}
rspconfig noderange {BSR}
rspconfig noderange {huge_page}
rspconfig noderange huge_page={NUM}
rspconfig noderange {setup_failover}
rspconfig noderange setup_failover={enable | disable}
rspconfig noderange {force_failover}
rspconfig noderange --resetnet
BPA/Frame (using Direct FSP Management) Specific:
rspconfig noderange HMC_passwd={currentpasswd,newpasswd}
rspconfig noderange admin_passwd={currentpasswd,newpasswd}
rspconfig noderange general_passwd={currentpasswd,newpasswd}
rspconfig noderange *_passwd={currentpasswd,newpasswd}
rspconfig noderange {frame}
1.3. Admin Guide
463
xCAT3 Documentation, Release 2.12
rspconfig noderange frame={* | frame_number}
rspconfig noderange {sysname}
rspconfig noderange sysname={* | name}
rspconfig noderange {pending_power_on_side}
rspconfig noderange pending_power_on_side={temp | perm}
rspconfig noderange --resetnet
HMC Specific:
rspconfig noderange {sshcfg}
rspconfig noderange sshcfg={enable | disable}
rspconfig noderange --resetnet
DESCRIPTION
rspconfig configures various settings in the nodes’ service processors. If only a keyword is specified, without the =, it
displays the current value.
For options autopower | iocap | decfg | memdecfg | procdecfg | time | date | spdump | sysdump | network, user
need to use chdef -t site enableASMI=yes to enable ASMI first.
OPTIONS
alert={on | enable | off | disable}
Turn on or off SNMP alerts.
autopower={enable | disable}
Select the policy for auto power restart. If enabled, the system will boot automatically once power is
restored after a power disturbance.
backupgateway
Get the BMC backup gateway ip address.
community={public | string}
Get or set the SNMP commmunity value. The default is public.
date=mm:dd:yyy
Enter the current date.
decfg={enable | disable:policyname,...}
Enables or disables deconfiguration policies.
frame={framenumber | *}
Set or get frame number. If no framenumber and * specified, framenumber for the nodes will be displayed
and updated in xCAAT database. If framenumber is specified, it only supports single node and the framenumber will be set for that frame. If * is specified, it supports noderange and all the frame numbers
for the noderange will be read from xCAT database and set to frames. Setting the frame number is a
disruptive command which requires all CECs to be powered off prior to issuing the command.
464
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
cec_off_policy={poweroff | stayon}
Set or get cec off policy after lpars are powered off. If no cec_off_policy value specified, the
cec_off_policy for the nodes will be displayed. the cec_off_policy has two values: poweroff and stayon.
poweroff means Power off when last partition powers off. stayon means Stay running after last partition
powers off. If cec_off_policy value is specified, the cec off policy will be set for that cec.
HMC_passwd={currentpasswd,newpasswd}
Change the password of the userid HMC for CEC/Frame. If the CEC/Frame is the factory default, the
currentpasswd should NOT be specified; otherwise, the currentpasswd should be specified to the current
password of the userid HMC for the CEC/Frame.
admin_passwd={currentpasswd,newpasswd}
Change the password of the userid admin for CEC/Frame from currentpasswd to newpasswd. If the
CEC/Frame is the factory default, the currentpasswd should NOT be specified; otherwise, the currentpasswd should be specified to the current password of the userid admin for the CEC/Frame.
general_passwd={currentpasswd,newpasswd}
Change the password of the userid general for CEC/Frame from currentpasswd to newpasswd. If the
CEC/Frame is the factory default, the currentpasswd should NOT be specified; otherwise, the currentpasswd should be specified to the current password of the userid general for the CEC/Frame.
*_passwd={currentpasswd,newpasswd}
Change the passwords of the userids HMC, admin and general for CEC/Frame from currentpasswd
to newpasswd. If the CEC/Frame is the factory default, the currentpasswd should NOT be specified;
otherwise, if the current passwords of the userids HMC, admin and general for CEC/Frame are the same
one, the currentpasswd should be specified to the current password, and then the password will be changed
to the newpasswd. If the CEC/Frame is NOT the factory default, and the current passwords of the userids
HMC, admin and general for CEC/Frame are NOT the same one, this option could NOT be used, and
we should change the password one by one.
frequency
The NTP update frequency (in minutes).
garp=time
Get or set Gratuitous ARP generation interval. The unit is number of 1/2 second.
gateway
The gateway ip address.
hostname
Display the CEC/BPA system names.
BSR
Get Barrier Synchronization Register (BSR) allocation for a CEC.
huge_page
Query huge page information or request NUM of huge pages for CEC. If no value specified, it means
query huge page information for the specified CECs, if a CEC is specified, the specified huge_page value
NUM will be used as the requested number of huge pages for the CEC, if CECs are specified, it means to
request the same NUM huge pages for all the specified CECs.
setup_failover={enable | disable}
Enable or disable the service processor failover function of a CEC or display status of this function.
1.3. Admin Guide
465
xCAT3 Documentation, Release 2.12
force_failover
Force a service processor failover from the primary service processor to the secondary service processor.
hostname={* | name}
Set CEC/BPA system names to the names in xCAT DB or the input name.
iocap={enable | disable}
Select the policy for I/O Adapter Enlarged Capacity. This option controls the size of PCI memory space
allocated to each PCI slot.
vlan
Get or set vlan ID. For get vlan ID, if vlan is not enabled, ‘BMC VLAN disabled’ will be outputed. For
set vlan ID, the valid value are [1-4096].
ip
The ip address.
memdecfg={configure | deconfigure:processingunit:unit|bank:id,...}
Select whether each memory bank should be enabled or disabled. State changes take effect on the next
platform boot.
netmask
The subnet mask.
network={[ip],[host],[gateway],[netmask]|*}
For MPA: get or set the MPA network parameters. If ‘*’ is specified, all parameters are read from the
xCAT database.
For FSP of Flex system P node: set the network parameters. If ‘*’ is specified, all parameters are read
from the xCAT database.
initnetwork={[ip],[host],[gateway],[netmask]|*}
For MPA only. Connecting to the IP of MPA from the hosts.otherinterfaces to set the MPA network
parameters. If ‘*’ is specified, all parameters are read from the xCAT database.
network={nic,{[ip],[host],[gateway],[netmask]}|*}
Not only for FSP/BPA but also for IMM. Get or set the FSP/BPA/IMM network parameters. If ‘*’ is
specified, all parameters are read from the xCAT database. If the value of ip is ‘0.0.0.0’, this nic will be
configured as a DHCP client. Otherwise this nic will be configured with a static IP.
Note that IPs of FSP/BPAs will be updated with this option, user needs to put the new IPs to /etc/hosts
manually or with xCAT command makehosts. For more details, see the man page of makehosts.
nonred
Allows loss of redundancy.
ntp={[ntpenable],[ntpserver],[frequency],[v3]}
Get or set the MPA Network Time Protocol (NTP) parameters.
ntpenable
Enable or disable NTP (enable|disable).
ntpserver
Get or set NTP server IP address or name.
466
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
pd1={nonred | redwoperf | redwperf}
Power Domain 1 - determines how an MPA responds to a loss of redundant power.
pd2={nonred | redwoperf | redwperf}
Power Domain 2 - determines how an MPA responds to a loss of redundant power.
procdecfg={configure|deconfigure:processingunit:id,...}
Selects whether each processor should be enabled or disabled. State changes take effect on the next
platform boot.
redwoperf
Prevents components from turning on that will cause loss of power redundancy.
redwperf
Power throttles components to maintain power redundancy and prevents components from turning on that
will cause loss of power redundancy.
snmpcfg={enable | disable}
Enable or disable SNMP on MPA.
snmpdest=snmpmanager-IP
Get or set where the SNMP alerts should be sent to.
solcfg={enable | disable}
Enable or disable the sol on MPA (or CMM) and blade servers belongs to it.
spdump
Performs a service processor dump.
sshcfg={enable | disable}
Enable or disable SSH on MPA.
swnet={[ip],[gateway],[netmask]}
Set the Switch network parameters.
sysdump
Performs a system dump.
sysname
Query or set sysname for CEC or Frame. If no value specified, means to query sysname of the specified
nodes. If ‘*’ specified, it means to set sysname for the specified nodes, and the sysname values would
get from xCAT datebase. If a string is specified, it means to use the string as sysname value to set for the
specified node.
pending_power_on_side={temp|perm}
List or set pending power on side for CEC or Frame. If no pending_power_on_side value specified, the
pending power on side for the CECs or frames will be displayed. If specified, the pending_power_on_side
value will be set to CEC’s FSPs or Frame’s BPAs. The value ‘temp’ means T-side or temporary side. The
value ‘perm’ means P-side or permanent side.
time=hh:mm:ss
Enter the current time in UTC (Coordinated Universal Time) format.
textid={\*|textid}
1.3. Admin Guide
467
xCAT3 Documentation, Release 2.12
Set the blade or MPA textid. When using ‘*’, the textid used is the node name specified on the commandline. Note that when specifying an actual textid, only a single node can be specified in the noderange.
USERID={newpasswd} updateBMC={y|n}
Change the password of the userid USERID for CMM in Flex system cluster. The option updateBMC can
be used to specify whether updating the password of BMCs that connected to the specified CMM. The
value is ‘y’ by default which means whenever updating the password of CMM, the password of BMCs
will be also updated. Note that there will be several seconds needed before this command complete.
If value “*” is specified for USERID and the object node is Flex System X node, the password used to
access the BMC of the System X node through IPMI will be updated as the same password of the userid
USERID of the CMM in the same cluster.
--resetnet
Reset the network interfaces of the specified nodes.
v3
Enable or disable v3 authentication (enable|disable).
-h | --help
Prints out a brief usage message.
-v | --version
Display the version number.
EXAMPLES
1. To setup new ssh keys on the Management Module mm:
rspconfig mm snmpcfg=enable sshcfg=enable
2. To turn on SNMP alerts for node5:
rspconfig node5 alert=on
Output is similar to:
node5: Alerts: enabled
3. To display the destination setting for SNMP alerts for node4:
rspconfig node4 snmpdest
Output is similar to:
node4: BMC SNMP Destination 1: 9.114.47.227
4. To display the frame number for frame 9A00-10000001
rspconfig> 9A00-10000001 frame
Output is similar to:
9A00-10000001: 1
468
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
5. To set the frame number for frame 9A00-10000001
rspconfig 9A00-10000001 frame=2
Output is similar to:
9A00-10000001: SUCCESS
6. To set the frame numbers for frame 9A00-10000001 and 9A00-10000002
rspconfig 9A00-10000001,9A00-10000002 frame=*
Output is similar to:
9A00-10000001: SUCCESS
9A00-10000002: SUCCESS
7. To display the MPA network parameters for mm01:
rspconfig mm01 network
Output is similar to:
mm01:
mm01:
mm01:
mm01:
MM IP: 192.168.1.47
MM Hostname: MM001125C31F28
Gateway: 192.168.1.254
Subnet Mask: 255.255.255.224
8. To change the MPA network parameters with the values in the xCAT database for mm01:
rspconfig mm01 network=*
Output is similar to:
mm01:
mm01:
mm01:
mm01:
MM IP: 192.168.1.47
MM Hostname: mm01
Gateway: 192.168.1.254
Subnet Mask: 255.255.255.224
9. To change only the gateway parameter for the MPA network mm01:
rspconfig mm01 network=,,192.168.1.1,
Output is similar to:
mm01: Gateway: 192.168.1.1
10. To display the FSP network parameters for fsp01:
rspconfig> fsp01 network
Output is similar to:
fsp01:
eth0:
IP Type: Dynamic
IP Address: 192.168.1.215
Hostname:
1.3. Admin Guide
469
xCAT3 Documentation, Release 2.12
Gateway:
Netmask: 255.255.255.0
eth1:
IP Type: Dynamic
IP Address: 192.168.200.51
Hostname: fsp01
Gateway:
Netmask: 255.255.255.0
11. To change the FSP network parameters with the values in command line for eth0 on fsp01:
rspconfig fsp01 network=eth0,192.168.1.200,fsp01,,255.255.255.0
Output is similar to:
fsp01: Success to set IP address,hostname,netmask
12. To change the FSP network parameters with the values in the xCAT database for eth0 on fsp01:
rspconfig fsp01 network=eth0,*
Output is similar to:
fsp01: Success to set IP address,hostname,gateway,netmask
13. To configure eth0 on fsp01 to get dynamic IP address from DHCP server:
rspconfig fsp01 network=eth0,0.0.0.0
Output is similar to:
fsp01: Success to set IP type to dynamic.
14. To get the current power redundancy mode for power domain 1 on mm01:
rspconfig mm01 pd1
Output is similar to:
mm01: Redundant without performance impact
15. To change the current power redundancy mode for power domain 1 on mm01 to non-redundant:
rspconfig mm01 pd1=nonred
Output is similar to:
mm01: nonred
16. To enable NTP with an NTP server address of 192.168.1.1, an update frequency of 90 minutes, and with v3
authentication enabled on mm01:
rspconfig mm01 ntp=enable,192.168.1.1,90,enable
Output is similar to:
470
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
mm01:
mm01:
mm01:
mm01:
NTP: disabled
NTP Server: 192.168.1.1
NTP: 90 (minutes)
NTP: enabled
17. To disable NTP v3 authentication only on mm01:
rspconfig mm01 ntp=,,,disable
Output is similar to:
mm01: NTP v3: disabled
18. To disable Predictive Failure and L2 Failure deconfiguration policies on mm01:
rspconfig mm01 decfg=disable:predictive,L3
Output is similar to:
mm01: Success
19. To deconfigure processors 4 and 5 of Processing Unit 0 on mm01:
rspconfig mm01 procedecfg=deconfigure:0:4,5
Output is similar to:
mm01: Success
20. To check if CEC sysname set correct on mm01:
rspconfig mm01 sysname
mm01: mm01
rspconfig mm01 sysname=cec01
mm01: Success
rspconfig mm01 sysname
mm01: cec01
21. To check and change the pending_power_on_side value of cec01’s fsps:
rspconfig cec01 pending_power_on_side
cec01: Pending Power On Side Primary: temp
cec01: Pending Power On Side Secondary: temp
rspconfig cec01 pending_power_on_side=perm
cec01: Success
rspconfig cec01 pending_power_on_side
cec01: Pending Power On Side Primary: perm
cec01: Pending Power On Side Secondary: perm
1.3. Admin Guide
471
xCAT3 Documentation, Release 2.12
22. To show the BSR allocation for cec01:
rspconfig cec01 BSR
Output is similar to:
cec01:
cec01:
cec01:
cec01:
cec01:
cec01:
cec01:
cec01:
cec01:
cec01:
cec01:
cec01:
cec01:
Barrier Synchronization Register (BSR)
Number of BSR arrays: 256
Bytes per BSR array : 4096
Available BSR array : 0
Partition name: BSR arrays
lpar01
: 32
lpar02
: 32
lpar03
: 32
lpar04
: 32
lpar05
: 32
lpar06
: 32
lpar07
: 32
lpar08
: 32
23. To query the huge page information for CEC1, enter:
rspconfig CEC1 huge_page
Output is similar to:
CEC1:
CEC1:
CEC1:
CEC1:
CEC1:
CEC1:
CEC1:
CEC1:
CEC1:
CEC1:
CEC1:
CEC1:
CEC1:
CEC1:
CEC1:
Huge Page Memory
Available huge page memory(in pages):
Configurable huge page memory(in pages):
Page Size (in GB):
Maximum huge page memory(in pages):
Requested huge page memory(in pages):
Partition name: Huge pages
lpar1
: 3
lpar5
: 3
lpar9
: 3
lpar13
: 3
lpar17
: 0
lpar21
: 0
lpar25
: 0
lpar29
: 0
0
12
16
24
15
24. To request 10 huge pages for CEC1, enter:
rspconfig CEC1 huge_page=10
Output is similar to:
CEC1: Success
25. To disable service processor failover for cec01, in order to complete this command, the user should power off
cec01 first:
rspconfig cec01 setup_failover
cec01: Failover status: Enabled
rpower cec01 off
472
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
rspconfig cec01 setup_failover=disable
cec01: Success
rspconfig cec01 setup_failover
cec01: Failover status: Disabled
26. To force service processor failover for cec01:
lshwconn cec01
cec01: 192.168.1.1:
cec01: 192.168.2.1:
˓→state=LINE UP
cec01: 192.168.1.2:
˓→state=LINE UP
cec01: 192.168.2.2:
LINE DOWN
sp=primary,ipadd=192.168.2.1,alt_ipadd=unavailable,
sp=secondary,ipadd=192.168.1.2,alt_ipadd=unavailable,
LINE DOWN
rspconfig cec01 force_failover
cec01: Success.
lshwconn> cec01
cec01: 192.168.1.1:
˓→state=LINE UP
cec01: 192.168.2.1:
cec01: 192.168.1.2:
cec01: 192.168.2.2:
˓→state=LINE UP
sp=secondary,ipadd=192.168.1.1,alt_ipadd=unavailable,
LINE DOWN
LINE DOWN
sp=primary,ipadd=192.168.2.2,alt_ipadd=unavailable,
27. To deconfigure memory bank 9 and 10 of Processing Unit 0 on mm01:
rspconfig mm01 memdecfg=deconfigure:bank:0:9,10
Output is similar to:
mm01: Success
28. To reset the network interface of the specified nodes:
rspconfig --resetnet
Output is similar to:
Start to reset network..
Reset network failed nodes:
Reset network succeed nodes:
Server-8233-E8B-SN1000ECP-A,Server-9119-FHA-SN0275995-B,Server-9119-FHA-SN0275995˓→A,
Reset network finished.
29. To update the existing admin password on fsp:
1.3. Admin Guide
473
xCAT3 Documentation, Release 2.12
rspconfig fsp admin_passwd=admin,abc123
Output is similar to:
fsp: Success
30. To set the initial password for user HMC on fsp:
rspconfig fsp HMC_passwd=,abc123
Output is similar to:
fsp: Success
SEE ALSO
noderange(3)|noderange.3, rpower(1)|rpower.1, rcons(1)|rcons.1, rinv(1)|rinv.1, rvitals(1)|rvitals.1, rscan(1)|rscan.1,
rflash(1)|rflash.1
rspreset.1
Name
rspreset - resets the service processors associated with the specified nodes
Synopsis
rspreset noderange
rspreset [-h | --help | -v | --version]
Description
rspreset resets the service processors associated with the specified nodes. It searches the nodehm table and associated
tables to find the service processors associated with the nodes specified. If the node is a BMC-based node, the node’s
BMC will be reset. If the node is a blade, the blade’s on board service processor will be reset.
Options
-h | --help
Print help.
-v | --version
Print version.
474
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
Examples
1. Reset the service processor that controls node5:
rspreset node5
SEE ALSO
rpower(1)|rpower.1, nodehm(5)|nodehm.5
rvitals.1
Name
rvitals - remote hardware vitals
Synopsis
rvitals [-h | --help | -v | --version]
FSP/LPAR (with HMC) specific:
rvitals noderange {temp | voltage | lcds | all}
CEC/LPAR/Frame (using Direct FSP Management) specific:
rvitals noderange {rackenv | lcds | all} [-V| --verbose]
MPA specific:
rvitals noderange {temp | voltage | wattage | fanspeed | power | leds | summary | all}
Blade specific:
rvitals noderange {temp | wattage | fanspeed | leds | summary | all}
BMC specific:
rvitals noderange {temp | voltage | wattage | fanspeed | power | leds | all}
OpenPOWER (IPMI) specific:
rvitals noderange [temp | voltage | wattage | fanspeed | power | leds | chassis | all]
1.3. Admin Guide
475
xCAT3 Documentation, Release 2.12
OpenPOWER (OpenBMC) specific:
rvitals noderange [temp | voltage | wattage | fanspeed | power | altitude | all]
Description
rvitals Retrieves hardware vital information from the on-board Service Processor for a single or range of nodes and
groups.
Options
cputemp
Retrieves CPU temperatures.
disktemp
Retrieves HD back plane temperatures.
ambtemp
Retrieves ambient temperatures.
temp
Retrieves all temperatures.
voltage
Retrieves power supply and VRM voltage readings.
fanspeed
Retrieves fan speeds.
lcds
Retrieves LCDs status.
rackenv
Retrieves rack environmentals.
leds
Retrieves LEDs status.
chassis
Retrieves chassis status.
altitude
Retrieves altitude related attributes.
power
Retrieves power status.
powertime
Retrieves total power uptime. This value only increases, unless the Service Processor flash gets updated.
This option is not valid for x86 architecture systems.
reboot
476
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
Retrieves total number of reboots. This value only increases, unless the Service Processor flash gets
updated. This option is not valid for x86 architecture systems.
state
Retrieves the system state.
all
All of the above.
-h | --help
Print help.
-v | --version
Print version.
Examples
rvitals node5 all
Output is similar to:
node5:
node5:
node5:
node5:
node5:
node5:
node5:
node5:
node5:
node5:
node5:
node5:
node5:
node5:
node5:
node5:
node5:
node5:
node5:
node5:
node5:
CPU 1 Temperature: + 29.00 C (+ 84.2 F)
CPU 2 Temperature: + 19.00 C (+ 66.2 F)
DASD Sensor 1 Temperature: + 32.00 C (+ 89.6 F)
System Ambient Temperature Temperature: + 26.00 C (+ 78.8 F)
+5V Voltage: + 5.01V
+3V Voltage: + 3.29V
+12V Voltage: + 11.98V
+2.5V Voltage: + 2.52V
VRM1 Voltage: + 1.61V
VRM2 Voltage: + 1.61V
Fan 1 Percent of max:
100%
Fan 2 Percent of max:
100%
Fan 3 Percent of max:
100%
Fan 4 Percent of max:
100%
Fan 5 Percent of max:
100%
Fan 6 Percent of max:
100%
Current Power Status On
Current LCD1: SuSE Linux
Power On Seconds 11855915
Number of Reboots
930
System State Booting OS or in unsupported OS
SEE ALSO
rpower(1)|rpower.1, rinv(1)|rinv.1
sinv.1
NAME
sinv - Checks the software configuration of the nodes in the cluster.
1.3. Admin Guide
477
xCAT3 Documentation, Release 2.12
SYNOPSIS
sinv [-o output] -p template path [-t template count] [-s seed node] [-i] [-e] [-r] [-V] [--devicetype type_of_device] [-l
userID] {-f command file | -c command}
sinv [-h | -v]
DESCRIPTION
The sinv command is designed to check the configuration of the nodes in a cluster. The command takes as input
command line flags, and one or more templates which will be compared against the output of the xdsh command,
designated to be run by the -c or -f flag, on the nodes in the noderange.
The nodes will then be grouped according to the template they match and a report returned to the administrator in the
output file designated by the -o flag, or to stdout.
sinv supports checking the output from the rinv or xdsh command.
The sinv command is an xCAT Distributed Shell Utility.
COMMAND SPECIFICATION:
The xdsh or rinv command to execute on the remote targets is specified by the -c flag, or by the -f flag which is
followed by the fully qualified path to a file containing the command.
Note: do not add | xdshcoll to the command on the command line or in the command file, it is automatically added by
sinv.
The syntax for the -c parameter is as follows:
“command[; command]...”
where command is the command to run on the remote target. Quotation marks are required to ensure that all commands
in the list are executed remotely, and that any special characters are interpreted correctly on the remote target.
The sinv command does not work with any interactive commands, including those that read from standard input.
REMOTE SHELL COMMAND:
For xdsh, support is explicitly provided for AIX Remote Shell and OpenSSH, but any secure remote command that
conforms to the IETF (Internet Engineering Task Force) Secure Remote Command Protocol can be used. See man
xdsh for more details.
OPTIONS
-o | --output report output file
Optional output file. This is the location of the file that will contain the report of the nodes that match,
and do not match, the input templates. If the flag is not used, the output will go to stdout.
-p | --tp template path
This is the path to the template file. The template contains the output of xdsh or rinv command, that has
been run against a “seed” node, a node that contains the configuration that you would like all nodes in
your noderange to match.
The admin can create the template by running the xdsh or rinv command on the seed node, pipe to
xdshcoll (required) and store the output in the template path. See examples.
Note: The admin can also edit the template to remove any lines that they do not want checked.
478
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
An alternative method is to use the [-s seed node] parameter, which will automatically build the template
for you from the seed node named.
If a a seed node is not provided, then command will automatically use the first node in the noderange as
the seed node.
-t | --tc template count
This count is the number of templates that the command will use to check for nodes matches. If the
template in the template path does not match a node, the sinv will check additional templates up to the
template count.
For each node, it will compare the node against each template to see if there is a match. If there is no
match, and we are not over the template count, then a new template will be created from the node output.
This will result in having all nodes that match a given template reported in their group at the end of the
run in the output file. If no template count is specified, 0 is the default, and all nodes will be compared
against the first template.
-s | --seed seed node
This is the node that will be used to build the first template that is stored in template path. You can use
this parameter instead of running the command yourself to build the template.
Note: If no seed node is supplied, the first node in the noderange is automatically selected as a seed node.
-i | --ignorefirst
This flag suppresses the reporting of the nodes matching the first template. In very large systems, you
may not want to show the nodes that have the correct configuration, since the list could contain thousands
of nodes. This allows you to only report the nodes that do not match the required configuration.
-e | --exactmatch
This requires the check of node output against template to be an exact match. If this flag is not set, sinv
checks to see if the return from the xdsh or rinv command to the nodes contain a match for each line in
the input template (except for xdshcoll header and comments). If not in exactmatch mode, there can be
more lines in the xdsh or rinv return from the nodes.
For example, if running a “rpm -qa | grep xCAT” command, without exactmatch set, if the node contains
more xCAT rpms that listed in the template, it would be considered a match, as long as all rpms listed in
the template were on the node. With exactmatch set, the output must be identical to the template.
--devicetype type_of_device
Specify a user-defined device type that references the location of relevant device configuration file. The
devicetype value must correspond to a valid device configuration file. xCAT ships some default configuration files for Ethernet switches and IB switches under /opt/xcat/share/xcat/devicetype directory. If you
want to overwrite any of the configuration files, copy them to /var/opt/xcat/ directory and cutomize. For
example, base/IBSwitch/Qlogic/config is the configuration file location if devicetype is specified as IBSwitch::Qlogic. xCAT will first search config file using /var/opt/xcat/ as the base. If not found, it will
search for it using /opt/xcat/share/xcat/devicetype/ as the base.
-l | --user user_ID
Specifies a remote user name to use for remote command execution.
-c | --command
The xdsh or rinv command that will be run. The command should be enclosed in double quotes to insure
correct shell interpretation. This parameter must only contain, the node range or the image path (Linux)
or spot name for AIX. It cannot be used to set additional input flags to xdsh or rinv (for example -s,-T,-e).
See examples below.
1.3. Admin Guide
479
xCAT3 Documentation, Release 2.12
Note: do not add the | xdshcoll to the command, it is automatically added by sinv. sinv also automatically
sets the -v flag for xdsh.
-f | --file
The file containing the xdsh or rinv command that will be run. This should be the fully qualified name of
the file.
Note: do not add the | xdshcoll to the command in the file, it is automatically added by sinv.
-r | --remove
This flag indicates that generated templates should be removed at the at the end of the sinv command
execution.
If the flag is specified, then all templates that are generated by the sinvcommand, will be removed. If the
first template is created by the admin, it will not be removed.
If the flag is not specified, no templates will be removed. It is up to the admin to cleanup templates.
-h | --help
Displays usage information.
-v | --version
Displays xCAT release version.
-V | --Verbose
Verbose mode.
Examples
1. To setup sinv.template (name optional) for input to the sinv command, enter:
xdsh node1,node2 "rpm -qa | grep ssh " | xdshcoll
> /tmp/sinv.template
Note: when setting up the template the output of xdsh must be piped to xdshcoll, sinv processing depends
on it.
2. To setup rinv.template for input to the sinv command , enter:
rinv node1-node2 serial | xdshcoll
> /tmp/rinv.template
Note: when setting up the template the output of rinv must be piped to xdshcoll, sinv processing depends
on it.
3. To execute sinv using the sinv.template generated above on the nodegroup, testnodes ,possibly generating up to two
new templates, and removing all generated templates in the end, and writing output report to /tmp/sinv.output, enter:
sinv -c "xdsh testnodes rpm -qa | grep ssh" -p /tmp/sinv.template -t 2 -r -o
˓→/tmp/sinv.output
Note: do not add the pipe to xdshcoll on the -c flag, it is automatically added by the sinv.
4. To execute sinv on noderange, node1-node4, using the seed node, node8, to generate the first template, using the
xdsh command (-c), possibly generating up to two additional templates and not removing any templates at the end,
enter:
sinv -c "xdsh node1-node4 lslpp -l | grep bos.adt" -s node8 -p /tmp/sinv.
˓→template -t 2 -o /tmp/sinv.output
480
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
5. To execute sinv on noderange, node1-node4, using the seed node, node8, to generate the first template, using the
rinv command (-c), possibly generating up to two additional templates and removing any generated templates at the
end, enter:
sinv -c "rinv node1-node4 serial" -s node8 -p /tmp/sinv.template -t 2 -r -o /
˓→tmp/rinv.output
6. To execute sinv on noderange, node1-node4, using node1 as the seed node, to generate the sinv.template from the
xdsh command (-c), using the exact match option, generating no additional templates, enter:
sinv -c "xdsh node1-node4 lslpp -l | grep bos.adt" -s node1 -e -p /tmp/sinv.
˓→template
-o /tmp/sinv.output
Note: the /tmp/sinv.template file must be empty, otherwise it will be used as an admin generated template.
7. To execute sinv on the Linux osimage defined for cn1. First build a template from the /etc/hosts on the node.
Then run sinv to compare.
xdsh cn1 "cat /etc/hosts" | xdshcoll > /tmp/sinv2/template"
sinv -c "xdsh -i /install/netboot/rhels6/ppc64/test_ramdisk_statelite/
˓→rootimg cat /etc/hosts" -e -t 1 -p /tmp/sinv.template -o /tmp/sinv.output
8. To execute sinv on the AIX NIM 611dskls spot and compare /etc/hosts to compute1 node, run the following:
xdsh compute1 "cat /etc/hosts" | xdshcoll > /tmp/sinv2/template"
sinv -c "xdsh -i 611dskls cat /etc/hosts" -e -t1 -p /tmp/sinv.template -o /tmp/
˓→sinv.output
9. To execute sinv on the device mswitch2 and compare to mswitch1
sinv -c "xdsh mswitch enable;show version" -s mswitch1 -p /tmp/sinv/template -˓→devicetype IBSwitch::Mellanox -l admin -t 2
Files
/opt/xcat/bin/sinv/
Location of the sinv command.
SEE ALSO
L <xdsh(1)|xdsh.1>, noderange(3)|noderange.3
snmove.1
NAME
snmove - Move xCAT compute nodes to a different xCAT service node.
SYNOPSIS
snmove noderange [-V] [-l | --liteonly] [-d | --dest sn2] [-D | --destn sn2n] [-i | --ignorenodes] [-P | --postscripts
script1,script2... | all]
1.3. Admin Guide
481
xCAT3 Documentation, Release 2.12
snmove [-V] [-l | --liteonly] -s | --source sn1 [-S | --sourcen sn1n] [-d | --dest sn2] [-D | --destn sn2n] [-i | -ignorenodes] [-P | --postscripts script1,script2... | all]
snmove [-h | --help | -v | --version]
DESCRIPTION
The snmove command may be used to move a node or nodes from one service node to another backup service node.
The use of backup service nodes in an xCAT hierarchical cluster can help improve the overall reliability, availability,
and serviceability of the cluster.
Before you run the snmove command it is assumed that the backup service node has been configured properly to manage the new node or nodes. (See the xCAT document named “Using xCAT Service Nodes with AIX” for information
on how to set up backup AIX service nodes.).
The snmove command can use the information stored in the xCAT database or information passed in on the command
line to determine the current service node and the backup service node.
To specify the primary and backup service nodes you can set the “servicenode” attribute of the node definitions.
The servicenode attribute is the hostname of the xCAT service node as it is known by the management node. The
xcatmaster attribute is the hostname of the xCAT service node as known by the node. The servicenode attribute
should be set to a comma-separated list so that the primary service node is first and the backup service node is second.
The xcatmaster attribute must be set to the hostname of the primary service node as it is known by the node.
When the snmove command is run it modifies the xCAT database to switch the primary server to the backup server.
It will also check the other services that are being used for the node (tftpserver, monserver, nfsserver, conserver), and
if they were set to the original service node they will be changed to point to the backup service node.
By default the command will modify the nodes so that they will be able to be managed by the backup service node.
If the -i option is specified, the nodes themselves will not be modified.
You can also have postscripts executed on the nodes by using the -P option if needed.
The xCAT snmove command may also be used to synchronize statelite persistent files from the primary service node
to the backup service node without actually moving the nodes to the backup servers.
If you run the command with the “-l” option it will attempt to use rsync to update the statelite persistent directory on
the backup service node. This will only be done if the server specified in the “statelite” table is the primary service
node.
When the snmove command is executed the new service node must be running but the original service node may be
down.
Note: On a Linux cluster, for NFS statelite nodes that do not use external NFS server, if the original service node is
down, the nodes it manages will be down too. You must run nodeset command and then reboot the nodes after running
snmove. For stateless nodes and RAMDisk statelite nodes, the nodes will be up even if the original service node is
down. However, make sure to run nodeset command if you decide to reboot the nodes later.
OPTIONS
-d|--dest
Specifies the hostname of the new destination service node as known by (facing) the management node.
-D|--destn
Specifies the hostname of the destination service node as known by (facing) the nodes.
482
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
-h|--help
Display usage message.
-i|--ignorenodes
No modifications will be made on the nodes. If not specified, several xCAT postscripts will be run on the
nodes to complete the switch to the new service node.
-l|--liteonly
Use this option to ONLY synchronize any AIX statelite files from the primary server to the backup server
for the nodes. It will not do the actual moving of the nodes to the backup servers.
-P|--postscripts
Specifies a list of extra postscripts to be run on the nodes after the nodes are moved over to the new service
node. If all is specified, all the postscripts defined in the postscripts table will be run for the nodes. The
specified postscripts must be stored under /install/postscripts directory.
-s|--source
Specifies the hostname of the current (source) service node sa known by (facing) the management node.
-S|--sourcen
Specifies the hostname of the current service node adapter as known by (facing) the nodes.
-V|--verbose
Verbose mode.
-v|--version
Command Version.
EXAMPLES
1. Move the nodes contained in group “group1” to the service node named “xcatsn02”.
snmove group1 -d xcatsn02 -D xcatsn02-eth1
2. Move all the nodes that use service node xcatsn01 to service node xcatsn02.
snmove -s xcatsn01 -S xcatsn01-eth1 -d xcatsn02 -D xcatsn02-eth1
3. Move any nodes that have sn1 as their primary server to the backup service node set in the xCAT node definition.
snmove -s sn1
4. Move all the nodes in the xCAT group named “nodegroup1” to their backup SNs.
snmove nodegroup1
5. Move all the nodes in xCAT group “sngroup1” to the service node named “xcatsn2”.
snmove sngroup1 -d xcatsn2
6. Move all the nodes in xCAT group “sngroup1” to the SN named “xcatsn2” and run extra postscripts.
snmove sngroup1 -d xcatsn2 -P test1
1.3. Admin Guide
483
xCAT3 Documentation, Release 2.12
7. Move all the nodes in xCAT group “sngroup1” to the SN named “xcatsn2” and do not run anything on the nodes.
snmove sngroup1 -d xcatsn2 -i
8. Synchronize any AIX statelite files from the primary server for compute03 to the backup server. This will not
actually move the node to it’s backup service node.
snmove compute03 -l -V
FILES
/opt/xcat/sbin/snmove
SEE ALSO
noderange(3)|noderange.3
swapnodes.1
NAME
swapnodes - swap the location info in the db (all the attributes in the ppc table and the nodepos table) between 2
nodes. If swapping within a cec, it will assign the IO adapters that were assigned to the defective node to the available
node.
SYNOPSIS
swapnodes [-h| --help]
swapnodes -c current_node -f fip_node [-o]
DESCRIPTION
This command is only for Power 775 using Direct FSP Management, and used in Power 775 Availability Plus.
The swapnodes command will keep the current_node name in the xCAT table, and use the fip_node‘s hardware
resource. Besides that, the IO adapters will be assigned to the new hardware resource if they are in the same CEC. So
the swapnodes command will do 2 things:
1. Swap the location info in the db between 2 nodes:
All the ppc table attributes (including hcp, id, parent, supernode and so on). All the nodepos table attributes(including
rack, u, chassis, slot, room and so on).
2. Assign the I/O adapters from the defective node(the original current_node) to the available node(the original
fip_node) if the nodes are in the same cec.
The swapnodes command shouldn’t make the decision of which 2 nodes are swapped. It will just received the 2 node
names as cmd line parameters.
484
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
After running swapnodes command, the order of the I/O devices may be changed after IO re-assignment, so the
administrator needs to run rbootseq to set the boot string for the current_node. And then boot the node with the same
image and same postscripts because they have the same attributes.
Without -o option, it’s used to swap the location info in the db between 2 nodes. With -o option, it’s used to move
the current_node definition to fip_node (the 2nd octant), not move the fip_node definition to the 1st octant. If the
two nodes are in a cec, it will assign the IO adapters that were assigned to the defective node to the available node.
Originally, the current_node is a defective non-compute node, and fip_node is a available compute node. After the
swapping, the current_node will be a available node.
OPTIONS
-h|--help
Display usage message.
-c
current_node – the defective non-compute node.
-f
fip_node – a compute node which will be swapped as the non-compute node.
-o
one way. Only move the current_node definition to the fip_node‘s hardware resource, and not move
the fip_node definition to the current_node. And then the current_node will use the fip_node‘s hardware
resource, and the fip_node definition is not changed. if the two nodes are in the same CEC, the I/O adapter
from the original current_node will be assigned to the fip_node.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To swap the service node attributes and IO assignments between sn1 and compute2 which are in the same cec,
all the attributes in the ppc table and nodepos table of the two node will be swapped, and the I/O adapters from
the defective node (the original sn1) will be assigned to the available node (the original compute2). After the
swapping, the sn1 will use the compute2’s hardware resource and the I/O adapters from the original sn1.
swapnodes -c sn1 -f compute2
2. To swap the service node attributes and IO assignments between sn1 and compute2 which are NOT in the same
cec, all the attributes in the ppc table and nodepos table of the two node will be swapped. After the swapping,
the sn1 will use the compute2’s hardware resource.
swapnodes -c sn1 -f compute2
3. Only to move the service node (sn1) definition to the compute node (compute2)’s hardware resource, and not
move the compute2 definition to the sn1. After the swapping, the sn1 will use the compute2’s hardware resource,
and the compute2 definition is not changed.
1.3. Admin Guide
485
xCAT3 Documentation, Release 2.12
swapnodes -c sn1 -f compute2 -o
FILES
$XCATROOT/bin/swapnodes
(The XCATROOT environment variable is set when xCAT is installed. The default value is “/opt/xcat”.)
NOTES
This command is part of the xCAT software product.
SEE ALSO
lsvm(1)|lsvm.1, mkvm(1)|mkvm.1, chvm(1)|chvm.1
switchblade.1
SYNOPSIS
switchblade MM {list | stat}
switchblade node {media | mt | kvm | video | both} [slot_num]
switchblade [-h | --help | -v | --version]
DESCRIPTION
switchblade assigns the BladeCenter media tray and/or KVM to the specified blade, so that they can be used with that
blade. If list or stat are specified instead, switchblade will display the current assignment. You can either specify
a management module or a node (blade) to switchblade. If the latter, switchblade will determine the management
module of the node.
OPTIONS
list | stat
Display which blade the media tray and KVM are currently assigned to.
media | mt
Assign the media tray to the specified blade.
kvm | video
Assign the KVM (video display) to the specified blade.
both
Assign both the media tray and the KVM to the specified blade.
slot_num
486
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
The slot # of the blade that the resources should be assigned to. If not specified, it will use the slot # of
the node specified.
EXAMPLES
1. Switch the media tray to be assigned to the blade in slot 4 (assume it is node4):
switchblade node4 media
Output will be like:
Media Tray slot: 4
SEE ALSO
rbootseq(1)|rbootseq.1
switchdiscover.1
SYNOPSIS
switchdiscover [-h| --help]
switchdiscover [-v| --version]
switchdiscover [noderange | --range ip_ranges] [-V] [-w][-r|-x|-z][-s scan_methods --setup]
DESCRIPTION
The switchdiscover command scans the subnets and discovers all the switches on the subnets. The command takes a
list of subnets as input. The default subnets are the ones that the xCAT management node is on. It uses nmap command
as default to discover the switches. However, you can specify other discovery methods such as lldp or snmp with -s
flag. You can write the discovered switches into xCAT database with -w flag. This command supports may output
formats such as xml(-x), raw(-r) and stanza(-z) in addition to the default format.
--setup flag is for switch-based switch discovery. It will find all the discovered switches on the subnets, then match
them with predefined switches in the xCATDB. Next, it will set discovered switches with static ip address and hostname based on the predefined switch. It will also enable snmpv3 configuration. The details of the process are defined
in the http://xcat-docs.readthedocs.io/en/latest/advanced/networks/switchdiscover/switches_discovery.html.
To view all the switches defined in the xCAT database use lsdef -w “nodetype=switch” command.
For lldp method, make sure that lldpd package is installed and lldpd is running on the xCAT management node. lldpd
comes from xcat-dep package or you can get it from http://vincentbernat.github.io/lldpd/installation.html.
For snmp method, make sure that snmpwalk command is installed and snmp is enabled for switches. To install
snmpwalk, “yum install net-snmp-utils” for redhat and sles, “apt-get install snmp” for Ubuntu.
OPTIONS
noderange
1.3. Admin Guide
487
xCAT3 Documentation, Release 2.12
The switches which the user want to discover. If the user specify the noderange, switchdiscover will just
return the switches in the node range. Which means it will help to add the new switches to the xCAT
database without modifying the existed definitions. But the switches’ name specified in noderange should
be defined in database in advance. The ips of the switches will be defined in /etc/hosts file. This command
will fill the switch attributes for the switches defined.
-h|--help
Display usage message.
--range
Specify one or more IP ranges. Each can be an ip address (10.1.2.3) or an ip range (10.1.2.0/24). If the
range is huge, for example, 192.168.1.1/8, the switch discover may take a very long time to scan. So the
range should be exactly specified.
For nmap and snmp scan method, it accepts multiple formats. For example: 192.168.1.1/24, 40-41.1-2.34.1-100.
If the range is not specified, the command scans all the subnets that the active network interfaces (eth0,
eth1) are on where this command is issued.
-r
Display Raw responses.
-s
It is a comma separated list of methods for switch discovery. The possible switch scan methods are: lldp,
nmap or snmp. The default is nmap.
-v|--version
Command Version.
-V
Verbose output.
-w
Writes output to xCAT database.
-x
XML formatted output.
-z
Stanza formatted output.
--setup
Process switch-based switch discovery. Update discovered switch’s ip address, hostname and enable
snmpv3 configuration based on the predefined switch.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
488
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
EXAMPLES
1. To discover the switches on some subnets:
switchdiscover --range 10.2.3.0/24,192.168.3.0/24,11.5.6.7
2. To do the switch discovery and save them to the xCAT database:
switchdiscover --range 10.2.3.4/24 -w
It is recommended to run makehosts after the switches are saved in the DB.
3. To use lldp method to discover the switches:
switchdiscover -s lldp
4. To process switch-based switch discovery, the core switch has to be configured and top-of-rack (edge) switch
has to be predefine into xCAT databse with attribute switch and switchport to core switch:
switchdiscover --range 192.168.5.150-170 -s snmp --setup
FILES
/opt/xcat/bin/switchdiscover
SEE ALSO
tabgrep.1
NAME
tabgrep - list table names in which an entry for the given node appears.
SYNOPSIS
tabgrep nodename
tabgrep [-? | -h | --help]
DESCRIPTION
The tabgrep command displays the tables that contain a row for the specified node. Note that the row can either have
that nodename as the key or it could have a group that contains the node as the key.
OPTIONS
-?|-h|--help
Display usage message.
1.3. Admin Guide
489
xCAT3 Documentation, Release 2.12
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To display the tables that contain blade1:
tabgrep blade1
The output would be similar to:
nodelist
nodehm
mp
chain
hosts
mac
noderes
nodetype
FILES
/opt/xcat/bin/tabgrep
SEE ALSO
nodels(1)|nodels.1, tabdump(8)|tabdump.8
unregnotif.1
NAME
unregnotif - unregister a Perl module or a command that was watching for the changes of the desired xCAT database
tables.
SYNOPSIS
unregnotif [-h| --help]
unregnotif [-v| --version]
unregnotif filename
DESCRIPTION
This command is used to unregistered a Perl module or a command that was watching for the changes of the desired
xCAT database tables.
490
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
PARAMETERS
filename is the path name of the Perl module or command to be registered.
OPTIONS
-h | -help Display usage message.
-v | -version Command Version.
-V | -verbose Verbose output.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To unregistered a Perl module, enter:
unregnotif /opt/xcat/lib/perl/xCAT_monitoring/mycode.pm
2. To register a command, enter:
unregnotif /usr/bin/mycmd
FILES
/opt/xcat/bin/unregnotif
SEE ALSO
regnotif(1)|regnotif.1
updateSNimage.1
NAME
updateSNimage - Adds the needed Service Node configuration files to the install image.
SYNOPSIS
updateSNimage [-h | --help ]
updateSNimage [-v | --version]
updateSNimage [-n node] [-p path]
1.3. Admin Guide
491
xCAT3 Documentation, Release 2.12
DESCRIPTION
This command is used to add the Service Node configuration files to the install image. It will either copy them locally
or scp them to a remote host.
OPTIONS
-h |--help Display usage message.
-v |--version Display xCAT version.
-n |--node A remote host name or ip address that contains the install image to be updated.
-p |--path Path to the install image.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To update the image on the local host.
updateSNimage -p /install/netboot/fedora8/x86_64/test/rootimg
2. To update the image on a remote host.
updateSNimage -n 9.112.45.6 -p /install/netboot/fedora8/x86_64/test/rootimg
updatenode.1
NAME
updatenode - Update nodes in an xCAT cluster environment.
SYNOPSIS
updatenode noderange [-V | --verbose] [-F | --sync] [-f | --snsync] [-S | --sw] [-l userID] [-P | --scripts
[script1,script2...]] [-s | --sn] [-A | --updateallsw] [-c | --cmdlineonly] [-d alt_source_dir] [--fanout=fanout_value]
[-t timeout} [attr=val [attr=val...]] [-n | --noverify]
updatenode noderange [-k | --security] [-t timeout]
updatenode noderange [-g | --genmypost]
updatenode noderange [-V | --verbose] [-t timeout] [script1,script2...]
updatenode noderange [-V | --verbose] [-f | --snsync]
updatenode [-h | --help] [-v | --version]
492
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
DESCRIPTION
The updatenode command is run on the xCAT management node and can be used to perform the following node
updates:
1. Distribute and synchronize files.
2. Install or update software on diskful nodes.
3. Run postscripts.
4. Update the ssh keys and host keys for the service nodes and compute nodes; Update the ca and credentials for the
service nodes.
The default behavior when no options are input to updatenode will be to run the following options -S, -P and -F options
in this order. If you wish to limit updatenode to specific actions you can use combinations of the -S, -P, and -F flags.
For example, If you just want to synchronize configuration file you could specify the -F flag. If you want to synchronize
files and update software you would specify the -F and -S flags. See the descriptions of these flags and examples below.
The flag -k (--security) can NOT be used together with -S, -P, and -F flags.
The flag -f (--snsync) can NOT be used together with -S, -P, and -F flags.
Note: In a large cluster environment the updating of nodes in an ad hoc manner can quickly get out of hand, leaving
the system administrator with a very confusing environment to deal with. The updatenode command is designed to
encourage users to handle cluster updates in a manner that is recorded and easily repeatable.
To distribute and synchronize files
The basic process for distributing and synchronizing nodes is:
* Create a synclist file.
* Indicate the location of the synclist file.
* Run the updatenode command to update the nodes.
Files may be distributed and synchronized for both diskless and diskful nodes. Syncing files to NFS-based statelite
nodes is not supported.
More information on using the synchronization file function is in the following doc: Using_Updatenode.
Create the synclist file
The synclist file contains the configuration entries that specify where the files should be synced to. In the synclist file,
each line is an entry which describes the location of the source files and the destination location for the files on the
target node.
For more information on creating your synclist files and where to put them, read:
Sync-ing_Config_Files_to_Nodes
Run updatenode to synchronize the files
updatenode <noderange> -F
1.3. Admin Guide
493
xCAT3 Documentation, Release 2.12
To install or update software
updatenode can be use to install or update software on the nodes. See the following documentation for setting up
otherpkgs: Install_Additional_Packages
To install/update the packages, run:
updatenode <noderange> -S
For Linux systems:
It this is equivalent to running the following command:
updatenode noderange -P ospkgs,otherpkgs
It will update all the rpms specified in the .pkglist file and .otherpkgs.pkglist file. ospkgs postscript will normally
remove all the existing rpm repositories before adding server:/install/<os>/<arch/ as the new repository. To preserve
the existing repositories, you can run the following command instead:
updatenode noderange -P "ospkgs --keeprepo,otherpkgs"
For AIX systems:
Note: The updatenode command is used to update AIX diskful nodes only. For updating diskless AIX nodes refer
to the xCAT for AIX update documentation and use the xCAT mknimimage command. For information on updating
software on AIX cluster: For diskful installs, read: XCAT_AIX_RTE_Diskful_Nodes For diskless installs, read:
XCAT_AIX_Diskless_Nodes
updatenode can also be used in Sysclone environment to push delta changes to target node. After capturing the delta
changes from the golden client to management node, just run below command to push delta changes to target nodes.
updatenode <targetnoderange> -S
To run postscripts
The scripts must be copied to the /install/postscripts directory on the xCAT management node. (Make sure they are
executable and world readable.)
To run scripts on a node you must either specify them on the command line or you must add them to the “postscripts”
attribute for the node.
To set the postscripts attribute of the node (or group) definition you can use the xCAT chdef command. Set the value
to be a comma separated list of the scripts that you want to be executed on the nodes. The order of the scripts in the
list determines the order in which they will be run. You can use the lsdef command to check the postscript order.
Scripts can be run on both diskless and diskful nodes.
To run all the customization scripts that have been designated for the nodes, (in the “postscripts and postbootscripts”
attributes), type:
updatenode <noderange> -P
To run the “syslog” script for the nodes, type:
updatenode <noderange> -P syslog
To run a list of scripts, type:
494
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
updatenode <noderange> -P "script1 p1 p2,script2"
where p1 p2 are the parameters for script1.
The flag ‘-P’ can be omitted when only scripts names are specified.
Note: script1,script2 may or may not be designated as scripts to automatically run on the node. However, if
you want script1 and script2 to get invoked next time the nodes are deployed then make sure to add them to the
“postscripts/postbootscripts” attribute in the database for the nodes.
Update security
The basic functions of update security for nodes:
* Setup the ssh keys for the target nodes. It enables the management node and service nodes to ssh to the target nodes
without password.
* Redeliver the host keys to the target nodes.
* Redeliver the ca and certificates files to the service node. These files are used to authenticate the ssl connection
between xcatd’s of management node and service node.
* Remove the entries of target nodes from known_hosts file.
Set up the SSH keys
A password for the user who is running this command is needed to setup the ssh keys. This user must have the same
uid and gid as the userid on the target node where the keys will be setup.
If the current user is root, roots public ssh keys will be put in the authorized_keys* files under roots .ssh directory on
the node(s). If the current user is non-root, the user must be in the policy table and have credential to run the xdsh
command. The non-root users public ssh keys and root’s public ssh keys will be put in the authorized_keys* files
under the non-root users .ssh directory on the node(s ).
Handle the hierarchical scenario
When update security files for the node which is served by a service node, the service node will be updated automatically first, and then the target node.
The certificates files are needed for a service node to authenticate the ssl connections between the xCAT client and
xcatd on the service node, and the xcatd’s between service node and management node. The files in the directories
/etc/xcat/cert/ and ~/.xcat/ will be updated.
Since the certificates have the validity time, the ntp service is recommended to be set up between management node
and service node.
Simply running following command to update the security keys:
updatenode <noderange> -k
PARAMETERS
noderange
A set of comma delimited xCAT node names and/or group names. See the xCAT “noderange” man page
for details on additional supported formats.
script1,script2...
1.3. Admin Guide
495
xCAT3 Documentation, Release 2.12
A comma-separated list of script names. The scripts must be executable and copied to the /install/postscripts directory. Each script can take zero or more parameters. If parameters are specified,
the whole list needs to be quoted by double quotes. For example:
"script1 p1 p2,script2"
[attr=val [attr=val...]]
Specifies one or more “attribute equals value” pairs, separated by spaces. Attr=val pairs must be specified last on the command line. The currently supported attributes are: “installp_bundle”, “otherpkgs”,
“installp_flags”, “emgr_flags” and “rpm_flags”. These attributes are only valid for AIX software maintenance support.
OPTIONS
--fanout=fanout_value
Specifies a fanout value for the maximum number of concurrently executing remote shell processes. Serial
execution can be specified by indicating a fanout value of 1. If --fanout is not specified, a default fanout
value of 64 is used.
-A|--updateallsw
Install or update all software contained in the source directory. (AIX only)
-c|cmdlineonly
Specifies that the updatenode command should only use software maintenance information provided on
the command line. This flag is only valid when using AIX software maintenance support.
-d alt_source_dir
Used to specify a source directory other than the standard lpp_source directory specified in the xCAT
osimage definition. (AIX only)
-F|--sync
Specifies that file synchronization should be performed on the nodes. rsync and ssh must be installed and
configured on the nodes. The function is not supported for NFS-based statelite installations. For NFSbased statelite installations to sync files, you should use the read-only option for files/directories listed in
litefile table with source location specified in the litetree table.
-f|--snsync
Specifies that file synchronization should be performed to the service nodes that service the nodes in the
noderange. This updates the service nodes with the data to sync to the nodes. rsync and ssh must be
installed and configured on the service nodes. For hierarchy, this optionally can be done before syncing
the files to the nodes with the -F flag. If the -f flag is not used, then the -F flag will sync the servicenodes
before the nodes automatically. When installing nodes in a hierarchical cluster, this flag should be used
to sync the service nodes before the install, since the files will be sync’d from the service node by the
syncfiles postscript during the install. The function is not supported for NFS-based statelite installations.
For statelite installations to sync files, you should use the read-only option for files/directories listed in
litefile table with source location specified in the litetree table.
-g|--genmypost
Will generate a new mypostscript file for the nodes in the noderange, if site precreatemypostscripts is 1 or
YES.
-h|--help
496
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
Display usage message.
-k|--security
Update the ssh keys and host keys for the service nodes and compute nodes; Update the ca and credentials
to the service nodes. Never run this command to the Management Node, it will take down xcatd. You
must be running updatenode as root to use the -k flag.
-l | --user user_ID
Specifies a non-root user name to use for remote command execution. This option is only available when
running postscripts (-P) for AIX and Linux and updating software (-S) for Linux only. The non-root userid
must be previously defined as an xCAT user. The userid sudo setup will have to be done by the admin on
the node. This is not supported in a hierarchical cluster, that is the node is serviced by a service node. See
the document Granting_Users_xCAT_privileges for required xcat/sudo setup.
-P|--scripts
Specifies that postscripts and postbootscripts should be run on the nodes. updatenode -P syncfiles is
not supported. The syncfiles postscript can only be run during install. You should use updatenode
<noderange> -F instead.
-S|--sw
Specifies that node software should be updated. In Sysclone environment, specifies pushing the delta
changes to target nodes.
-n|--noverify
Specifies that node network availability verification will be skipped.
-s|--sn
Set the server information stored on the nodes in /opt/xcat/xcatinfo on Linux.
-t timeout
Specifies a timeout in seconds the command will wait for the remote targets to complete. If timeout is not
specified it will wait indefinitely. updatenode -k is the exception that has a timeout of 10 seconds, unless
overridden by this flag.
-v|--version
Command Version.
-V|--verbose
Verbose mode.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To perform all updatenode features for the Linux nodes in the group “compute”:
updatenode compute
1.3. Admin Guide
497
xCAT3 Documentation, Release 2.12
The command will: run any scripts listed in the nodes “postscripts and postbootscripts” attribute, install
or update any software indicated in the /install/custom/install/<ostype>/profile.otherpkgs.pkglist (refer to
the To install or update software part), synchronize any files indicated by the synclist files specified in
the osimage “synclists” attribute.
2. To run postscripts,postbootscripts and file synchronization only on the node “clstrn01”:
updatenode clstrn01 -F -P
3. Running updatenode -P with the syncfiles postscript is not supported. You should use updatenode -F instead.
Do not run:
updatenode clstrno1 -P syncfiles
Run:
updatenode clstrn01 -F
4. To run the postscripts and postbootscripts indicated in the postscripts and postbootscripts attributes on the node
“clstrn01”:
updatenode clstrn01 -P
5. To run the postscripts script1 and script2 on the node “clstrn01”:
cp script1,script2 /install/postscripts
updatenode clstrn01 -P "script1 p1 p2,script2"
Since flag ‘-P’ can be omitted when only script names are specified, the following command is equivalent:
updatenode clstrn01 "script1 p1 p2,script2"
p1 p2 are parameters for script1.
6. To synchronize the files on the node “clstrn01”: Prepare the synclist file. For AIX, set the full path
of synclist in the osimage table synclists attribute. For Linux, put the synclist file into the location: /install/custom/<inst_type>/<distro>/<profile>.<os>.<arch>.synclist Then:
updatenode clstrn01 -F
7.
To perform the software update on the
tra rpm into the /install/post/otherpkgs/<os>/<arch>/*
stall/custom/install/<ostype>/profile.otherpkgs.pkglist . Then:
Linux node “clstrn01”:
Copy
and add the rpm names into
the
the
ex/in-
updatenode clstrn01 -S
8. To update the AIX node named “xcatn11” using the “installp_bundle” and/or “otherpkgs” attribute values stored in
the xCAT database. Use the default installp, rpm and emgr flags.
updatenode xcatn11 -V -S
Note: The xCAT “xcatn11” node definition points to an xCAT osimage definition which contains the
“installp_bundle” and “otherpkgs” attributes as well as the name of the NIM lpp_source resource.
9. To update the AIX node “xcatn11” by installing the “bos.cpr” fileset using the “-agQXY” installp flags. Also
display the output of the installp command.
498
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
updatenode xcatn11 -V -S otherpkgs="I:bos.cpr" installp_flags="-agQXY"
Note: The ‘I:’ prefix is optional but recommended for installp packages.
10. To uninstall the “bos.cpr” fileset that was installed in the previous example.
updatenode xcatn11 -V -S otherpkgs="I:bos.cpr" installp_flags="-u"
11. To update the AIX nodes “xcatn11” and “xcatn12” with the “gpfs.base” fileset and the “rsync” rpm using the
installp flags “-agQXY” and the rpm flags “-i –nodeps”.
updatenode xcatn11,xcatn12 -V -S otherpkgs="I:gpfs.base,R:rsync-2.6.2-1.aix5.
˓→1.ppc.rpm" installp_flags="-agQXY" rpm_flags="-i --nodeps"
Note: Using the “-V” flag with multiple nodes may result in a large amount of output.
12. To uninstall the rsync rpm that was installed in the previous example.
updatenode xcatn11 -V -S otherpkgs="R:rsync-2.6.2-1" rpm_flags="-e"
13. Update the AIX node “node01” using the software specified in the NIM “sslbnd” and “sshbnd” installp_bundle
resources and the “-agQXY” installp flags.
updatenode node01 -V -S installp_bundle="sslbnd,sshbnd" installp_flags="˓→agQXY"
14. To get a preview of what would happen if you tried to install the “rsct.base” fileset on AIX node “node42”. (You
must use the “-V” option to get the full output from the installp command.)
updatenode node42 -V -S otherpkgs="I:rsct.base" installp_flags="-apXY"
15. To check what rpm packages are installed on the AIX node “node09”. (You must use the “-c” flag so updatenode
does not get a list of packages from the database.)
updatenode node09 -V -c -S rpm_flags="-qa"
16. To install all software updates contained in the /images directory.
updatenode node27 -V -S -A -d /images
Note: Make sure the directory is exportable and that the permissions are set correctly for all the files.
(Including the .toc file in the case of installp filesets.)
17. Install the interim fix package located in the /efixes directory.
updatenode node29 -V -S -d /efixes otherpkgs=E:IZ38930TL0.120304.epkg.Z
18. To uninstall the interim fix that was installed in the previous example.
updatenode xcatsn11 -V -S -c emgr_flags="-r -L IZ38930TL0"
19. To update the security keys for the node “node01”
updatenode node01 -k
20. To update the service nodes with the files to be synchronized to node group compute:
1.3. Admin Guide
499
xCAT3 Documentation, Release 2.12
updatenode compute -f
21. To run updatenode with the non-root userid “user1” that has been setup as an xCAT userid with sudo on node1 to
run as root, do the following: See Granting_Users_xCAT_privileges for required sudo setup.
updatenode node1 -l user1 -P syslog
22. In Sysclone environment, after capturing the delta changes from golden client to management node, to run
updatenode to push these delta changes to target nodes.
updatenode target-node -S
FILES
/opt/xcat/bin/updatenode
wcons.1
Name
wcons - windowed remote console
Synopsis
wcons [-t | --tile=n] [xterm-options] noderange
wcons [-h | --help | -v | --version]
Description
wcons provides access to the remote node serial console of a single or range or nodes or groups.
wcons is a simple front-end to rcons in an xterm session for each console.
Options
-t | --tile=n
Tile wcons windows from top left to bottom right. If n is spec- ified then tile n across. If n is not specified
then tile to edge of screen. If tiled wcons windows reach bottom right, then the windows start at top left
overlaying existing wcons windows.
-h | --help
Print help.
-v | --version
Print version.
xterm options
500
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
See xterm(1). Any options other than those listed above are passed directly to xterm. Note: when given
multiple nodes, wcons will override -title and tries to figure out optimal -geometryoptions for the xterms
(however, -geometry can still be specified).
Files
nodehm table - xCAT node hardware management table. See nodehm(5)|nodehm.5 for further details. This is used to
determine the console access method.
Examples
wcons node1-node5
wcons --tile --font=nil2 all
wcons -t 4 node1-node16
wcons -f vs -t 4 node1-node4
Bugs
Tile mode assumes that the width of the left window border is also the width of the right and bottom window border.
Most window managers should not have a problem. If you really need support for a screwy window manager let me
know.
See Also
noderange(3)|noderange.3, rcons(1)|rcons.1, xterm(1)
wkill.1
Name
wkill - kill windowed remote consoles
Synopsis
wkill [noderange]
wkill [-h | --help | -v | --version]
Description
wkill will kill the wcons windows on your $DISPLAY for a single or range or nodes or groups.
wkill was written because I’m too lazy to point and click off 64 windows.
wkill will only kill windows on your display and for only the noderange(3)|noderange.3 you specify.
noderange(3)|noderange.3 is specified, then all wcons windows on your $DISPLAY will be killed.
1.3. Admin Guide
If no
501
xCAT3 Documentation, Release 2.12
Options
-h | --help
Print help.
-v | --version
Print version.
Examples
wkill node1-node5
See Also
noderange(3)|noderange.3, wcons(1)|wcons.1
wvid.1
Name
wvid - windowed remote video console for nodes
Synopsis
wvid noderange
Description
wvid provides access to the remote node video console of a single node, or range of nodes or groups. wvid provides
a simple front-end to the hardware’s remote console capability. Currently this command is supported for: blades,
BMC/IMM, KVM, and Xen
The nodehm.cons attribute of the node determines the method used to open the console. See nodehm(5)|nodehm.5
for further details.
Options
No options are supported at this time.
Examples
1. To open video consoles for the 1st 2 nodes:
wvid node1,node2
502
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
See Also
noderange(3)|noderange.3, rcons(1)|rcons.1, wcons(1)|wcons.1
xCATWorld.1
NAME
xCATWorld - Sample client program for xCAT.
SYNOPSIS
xCATWorld noderange
DESCRIPTION
The xCATWorld program gives you a sample client program that interfaces to the
/opt/xcat/lib/perl/xCAT_plugin/xCATWorld.pm plugin. For debugging purposes we have an Environment Variable XCATBYPASS. If export XCATBYPASS=yes, the client will call the plugin without going through the xcat
daemon, xcatd.
OPTIONS
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1.To run , enter:
xCATWorld nodegrp1
FILES
/opt/xcat/bin/xCATWorld
NOTES
This command is part of the xCAT software product.
1.3. Admin Guide
503
xCAT3 Documentation, Release 2.12
xcat2nim.1
NAME
xcat2nim - Use this command to create and manage AIX NIM definitions based on xCAT node, group and network
object definitions.
SYNOPSIS
xcat2nim [-h|--help]
xcat2nim [-V|--verbose] [-u|--update] [-l|--list] [-r|--remove] [-f|--force] [-t object-types] [-o object-names] [-a|-allobjects] [-p|--primarySN] [-b|--backupSN] [noderange] [attr=val [attr=val...]]
DESCRIPTION
The xcat2nim command uses xCAT node, group and network object definitions to create, update, list, or remove
corresponding NIM definitions.
Before you create or update NIM definitions the xCAT definitions must be created and NIM must be configured.
The xcat2nim command uses xCAT database information, command line input, and default values to run the appropriate NIM commands.
The xCAT node, group and network definition names will correspond to the NIM machine, machine group and network
definitions.
Note: The length of a NIM object name must be no longer than 39 characters.
To create or update a NIM definition you must provide the names of the xCAT definitions to use. The default behavior
is to create new NIM definitions but not apply updates to existing definitions. If you wish to update existing NIM
definitions then you must use the “update” option. If you wish to completely remove the old definition and re-create it
you must use the “force” option.
The xCAT code uses the appropriate NIM commands to create the NIM definitions. To create definitions the “nim
-o define” operation is used. To update definitions the “nim -o change” operation is used. If you wish to specify
additional information to pass to the NIM commands you can use the “attr=val” support. The attribute names must
correspond to the attributes supported by the relevant NIM commands. (For example. “netboot_kernel=mp”)
If the object type you are creating is a node then the object names can be a noderange value.
If you are using xCAT service nodes the xcat2nim command will automatically determine the correct server for the
node and create the NIM definitions on that server.
The xcat2nim command support for NIM networks is limited to creating and listing.
When creating network definitions the command will check to make sure the network definition (or it’s equivalent)
does not exist and then create the required NIM network, route and interface definitions. In some cases the equivalent
network definition may exist using a different name. In this case a new definition WILL NOT be created.
To list the NIM definitions that were created you must specify the “list” option and the names of the xCAT objects that
were used to create the NIM definitions. The xcat2nim command will list the corresponding NIM machine, machine
group or network definitions using the “lsnim -l” command.
To remove NIM definitions you must specify the “remove” option and the names of the xCAT objects that were used
to create the NIM definitions.
The remove(“-r”), force(“-f”) and update(“-u”) options are not supported for NIM network definitions.
504
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
OPTIONS
-a|--all The list of objects will include all xCAT node, group and network objects.
attr=val [attr=val ...] Specifies one or more “attribute equals value” pairs, separated by spaces. Attr=val pairs must
be specified last on the command line. The attribute names must correspond to the attributes supported by the relevant
NIM commands. When providing attr=val pairs on the command line you must not specify more than one object type.
-b|--backupSN When using backup service nodes only update the backup. The default is to update both the primary
and backup service nodes.
-f|--force The force option will remove the existing NIM definition and create a new one.
-h|--help Display the usage message.
-l|--list List NIM definitions corresponding to xCAT definitions.
-o object-names A set of comma delimited xCAT object names. Objects must be of type node, group, or network.
-p|--primarySN When using backup service nodes only update the primary. The default is to update both the primary
and backup service nodes.
-r|--remove Remove NIM definitions corresponding to xCAT definitions.
-t object-types A set of comma delimited xCAT object types. Supported types include: node, group, and network.
Note: If the object type is “group”, it means that the xcat2nim command will operate on a NIM machine group
definition corresponding to the xCAT node group definition. Before creating a NIM machine group, all the NIM client
nodes definition must have been created.
-u|--update Update existing NIM definitions based on xCAT definitions.
-V|--verbose Verbose mode.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To create a NIM machine definition corresponding to the xCAT node “clstrn01”.
xcat2nim -t node -o clstrn01
2. To create NIM machine definitions for all xCAT node definitions.
xcat2nim -t node
3. Update all the NIM machine definitions for the nodes contained in the xCAT “compute” node group and specify
attribute values that will be applied to each definition.
xcat2nim -u -t node -o compute netboot_kernel=mp cable_type="N/A"
4. To create a NIM machine group definition corresponding to the xCAT group “compute”.
xcat2nim -t group -o compute
1.3. Admin Guide
505
xCAT3 Documentation, Release 2.12
5. To create NIM network definitions corresponding to the xCAT “clstr_net” an “publc_net” network definitions.
Also display verbose output.
xcat2nim -V -t network -o "clstr_net,publc_net"
6. To list the NIM definition for node clstrn02.
xcat2nim -l -t node clstrn02
7. To re-create a NIM machine definition and display verbose output.
xcat2nim -V -t node -f clstrn05
8. To remove the NIM definition for the group “AIXnodes”.
xcat2nim -t group -r -o AIXnodes
9. To list the NIM “clstr_net” definition.
xcat2nim -l -t network -o clstr_net
FILES
$XCATROOT/bin/xcat2nim
NOTES
This command is part of the xCAT software product.
SEE ALSO
mkdef(1)|mkdef.1
xcatchroot.1
NAME
xcatchroot - Use this xCAT command to modify an xCAT AIX diskless operating system image.
SYNOPSIS
xcatchroot -h
xcatchroot [-V] -i osimage_name cmd_string
506
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
DESCRIPTION
For AIX diskless images this command will modify the AIX SPOT resource using the chroot command. You must
include the name of an xCAT osimage definition and the command that you wish to have run in the spot.
WARNING:
Be very careful when using this command!!! Make sure you are very clear about exactly what you are changing so
that you do not accidently corrupt the image.
As a precaution it is advisable to make a copy of the original spot in case your changes wind up corrupting the image.
When you are done updating a NIM spot resource you should always run the NIM check operation on the spot.
nim -Fo check <spot_name>
The xcatchroot command will take care of any of the required setup so that the command you provide will be able to
run in the spot chroot environment. It will also mount the lpp_source resource listed in the osimage definition so that
you can access additional software that you may wish to install.
For example, assume that the location of the spot named in the xCAT osimage definition is /install/nim/spot/614spot/usr.
The associated root directory in this spot would be /install/nim/spot/614spot/usr/lpp/bos/inst_root. The chroot is automatically done to this new root directory. The
spot location is mounted on /.../inst_root/usr so that when your command is run in the chroot environment it is actually
running commands from the spot usr location.
Also, the location of the lpp_source resource specified in the osimage definition will be mounted to a subdirectory of
the spot /.../inst_root directory. For example, if the lpp_source location is /install/nim/lpp_source/614lpp_lpp_source
then that would be mounted over /install/nim/spot/614spot/usr/lpp/bos/inst_root/lpp_source.
When you provide a command string to run make sure you give the full paths of all commands and files assuming the
/.../inst_root directory is you root directory.
If you wish to install software from the lpp_source location you would provide a directory location of /lpp_source (or
/lpp_source/installp/ppc or /lpp_source/RPMS/ppc etc.) See the example below.
Always run the NIM check operation after you are done updating your spot. (ex. “nim -o check <spot_name>”)
OPTIONS
cmd_string
The command you wish to have run in the chroot environment. (Use a quoted string.)
-h |--help
Display usage message.
-i osimage_name
The name of the xCAT osimage definition.
-V |--verbose
Verbose mode.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
1.3. Admin Guide
507
xCAT3 Documentation, Release 2.12
EXAMPLES
1) Set the root password to “cluster” in the spot so that when the diskless node boots it will have a root password set.
xcatchroot -i 614spot "/usr/bin/echo root:cluster | /usr/bin/chpasswd -c"
2. Install the bash rpm package.
xcatchroot -i 614spot "/usr/bin/rpm -Uvh /lpp_source/RPMS/ppc bash-3.2-1.aix5.2.ppc.
˓→rpm"
3. To enable system debug.
xcatchroot -i 614spot "bosdebug -D -M"
4. To set the “ipforwarding” system tunable.
xcatchroot -i 614spot "/usr/sbin/no -r -o ipforwarding=1"
FILES
/opt/xcat/bin/xcatchroot
NOTES
This command is part of the xCAT software product.
xcatperftest.1
NAME
xcatperftest - Run xCAT command performance baseline testing on fake nodes.
SYNOPSIS
xcatperftest [-?|-h]
[PERF_DRYRUN=y] [PERF_NOCREATE=y] xcatperftest <total> [command-list-file]
DESCRIPTION
The xcatperftest command runs commandes defined in a command list file and get their execution response time
baseline for performance purpose. The xcatperftest command is part of the xCAT package xCAT-test, and you can run
it standalone or leverage it to build up your automation test cases.
Any commands could be defined in the command list file, however, it is recommended that the one-time initial configuration are well prepared prior to run xcatperftest command. For example, the network object, osdistor and osimage
image objects.
Follow the below steps to run xcatperftest command:
508
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
1, Install xCAT-test on a xCAT management nodes.
2, Prepare a command list in which the commands are what you want to messure.
3, Prepare the initial configuration based on the command list to make sure all commands could be executed in techinal.
4, Run xcatperftest with the total fake nodes number and the above command list file.
Node: It is suggested to run the command in background as it normally takes long time to finish all the performanc
testing with large amount of fake nodes.
OPTIONS
-?|-h
Display usage message.
<command-list-file>
Specifies the command list file with full-path.
/opt/xcat/share/xcat/tools/autotest/perfcmds.lst
xCAT supports an example command file:
<total>
Total number of fake nodes will be defined during the testing.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
COMMAND LIST FILE
The command list file is in flat text format, the testing framework will parse the file line by line, here is an example of
the commannd list file:
#SERIES# 1,50,100,250,500,1000,2500,5000
mkdef -z -f < #STANZ#
lsdef #NODES#
makehosts #NODES#
makedns -n #NODES#
makedhcp #NODES#
makeknownhosts #NODES#
nodech #NODES# groups,=group1
nodels #NODES# noderes
nodeset #NODES# osimage=rhels7.3-GA-ppc64le-install-compute
chdef -t node -o #NODES# postscripts="fake" profile="install" netboot="grub2"
rmdef -t node #PERFGRP#
mkdef -z < #STANZ#
noderm #PERFGRP#
Note: Each line defines one command, and the commands dependency should be handled by the line order. If you
define a node range series line (started with #SERIES#) in this file, xcatperftest will run the command for each node
range defined in series line.
#SERIES# To define a node range series, and the series should be an comma split incremental number sequence.
1.3. Admin Guide
509
xCAT3 Documentation, Release 2.12
#STANZ# It will be replaced with real stanz file path when this command line runs.
#NODES# It will be replaced with real node range defined in #SERIES# line when this command line runs. If no
series line, the node group will be used.
#PERFGRP# It will be replaced with node group when this command line runs.
ENVIRONMENT VARIABLE
The xcatperftest command supports be customized by some environment variables.
FAKE_NODE_PREFIX
Optional, the prefix of the fake compute node name. By default, the value is ‘fake’
FAKE_NODE_GROUP
# Optional, the group name of all the fake compute nodes. By default, the value is ‘perftest’
FAKE_NETWORK_PRO
Mandatory, the Provision network for all the fake compute nodes. By default, the value is ‘192.168’. It must be a
string like ‘A.B’, and be matched with ‘tabdump networks‘
FAKE_NETWORK_BMC
Mandatory, the BMC network for all the fake compute nodes. By default, the value is ‘192.168’. Note: It could not be
the same subnet as ‘FAKE_NETWORK_PRO’ It must be a string like ‘A.B’ and no need to be defined in ‘networks’
table.
PERF_NODETEMPL
Optional, The node template name used for generating fake nodes. By default, it will be auto-detected according to
the current arch.
PERF_DRYRUN
Optional, Indicate no real commands will be executed if the environment variable is set.
PERF_NOCREATE
Optional, Indicate no new fake nodes will be created if the environment variable is set.
EXAMPLES
1. To run the performance testing for the commands defined in /tmp/cmd.lst on 5000 fake nodes:
xcatperftest 5000 /tmp/cmd.lst
2. To generate an xCAT node object stanz file for 10000 nodes in subnet 10.100.0.0:
FAKE_NETWORK_PRO=10.100 FAKE_NETWORK_BMC=10.200 xcatperftest 10000
3. To run the performance testing for the commands defined in /opt/xcat/share/xcat/tools/autotest/perfcmds.lst on
5000 existing fake nodes:
PERF_NOCREATE=y xcatperftest 5000 /opt/xcat/share/xcat/tools/autotest/perfcmds.lst
510
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
FILES
/opt/xcat/bin/xcatperftest
/opt/xcat/share/xcat/tools/autotest/perfcmds.lst
xcattest.1
NAME
xcattest - Run automated xCAT test cases.
SYNOPSIS
xcattest [-?|-h]
xcattest [-f configure file] [-b case bundle list]
xcattest [-f configure file] [-t case list]
xcattest [-f configure file] [-c cmd list]
xcattest [-b case bundle list] [-l]
xcattest [-t case list] [-l]
xcattest [-c cmd list] [-l]
xcattest [-s command]
xcattest [-s bundle]
DESCRIPTION
The xcattest command runs test cases to verify the xCAT functions, it can be used when you want to verify the xCAT
functions for whatever reason, for example, to ensure the code changes you made do not break the existing commands;
to run acceptance test for new build you got; to verify the xCAT snapshot build or development build before putting it
onto your production system. The xcattest command is part of the xCAT package xCAT-test.
The root directory for the xCAT-test package is /opt/xcat/share/xcat/tools/autotest/. All test cases are in the sub directory testcase, indexed by the xCAT command, you can add your own test cases according to the test cases format below.
The subdirectory bundle contains all the test cases bundles definition files, you can customize or create any test cases
bundle file as required. The testing result information will be written into the subdirectory result, the timestamps are
used as the postfixes for all the result files. xCAT-test package ships two configuration files template aix.conf.template
and linux.conf.template for AIX and Linux environment, you can use the template files as the start point of making
your own configuration file.
OPTIONS
-?|-h
Display usage message.
-f configure file
1.3. Admin Guide
511
xCAT3 Documentation, Release 2.12
Specifies the configuration file with full-path.
/opt/xcat/share/xcat/tools/autotest/linux.conf.template
xCAT supports an example config file:
-b case bundle list
Comma separated list of test cases bundle files, each test cases bundle can contain multiple lines and each
line for one test case name. The bundle files should be listed in: /opt/xcat/share/xcat/tools/autotest/bundle.
-t cases list
Comma separated list of test cases that will be run.
-c cmd list
Comma separated list of commands which will be tested, i.e., all the test cases under the command sub
directory will be run.
-l
Display the test cases names specified by the flag -b, -t or -c.
-s
Display the bundle files and command with value: bundle or command.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
TEST CASE FORMAT
The xCAT-test test cases are in flat text format, the testing framework will parse the test cases line by line, here is an
example of the test case:
#required, case name
start:case name
#optional, description of the test case
description: what the test case is for?
#optional, environment requirements
os:AIX/Linux
#optional, environment requirements
arch:ppc/x86
#optional, environment requirements
hcp:hmc/mm/bmc/fsp
#required, command need to run
cmd:comand
#optional, check return code of last executed command
check:rc == or != return code
#optional, check output of last executed command
check:output== or != or =~ or !~ output check string
end
Note: Each test case can have more than one cmd sections and each cmd section can have more than one check:rc
sections and more than one check:output sections, the output check string can include regular expressions.
512
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
EXAMPLES
1. To run all test cases related command rpower:
xcattest -f /tmp/config -c rpower
2. To run customized bundle with /tmp/config file:
xcattest -c lsdef -l > /opt/xcat/share/xcat/tools/autotest/bundle/custom.bundle
Modify custom.bundle
xcattest -f /tmp/config -b custom.bundle
3. To run specified test cases with /tmp/config file:
xcattest -f /tmp/config -t lsdef_t_o_l_z
4. To add a new case to test chvm. In the example, we assume that the min_mem should not be equal to 16 in the
lpar profile of computenode. The case name is chvm_custom. It create a test lpar named testnode firstly, that
change the min_mem of the lpar to 16 using chvm, then check if min_mem have changed correctly. At last, the
testnode be remove to ensure no garbage produced in the cases.
add a new test case file in /opt/xcat/share/xcat/tools/autotest/chvm
edit filename
start:chvm_custom
hcp:hmc
cmd:lsvm $$CN > /tmp/autotest.profile
check:rc==0
cmd:mkdef -t node -o testnode mgt=hmc groups=all
cmd:mkvm testnode -i $$MaxLparID -l $$CN
check:rc==0
cmd:perl -pi -e 's/min_mem=\d+/min_mem=16/g' /tmp/autotest.profile
cmd:cat /tmp/autotest.profile|chvm testnode
check:rc==0
cmd:lsvm testnode
check:output=~min_mem=16
cmd:rmvm testnode
cmd:rm -f /tmp/autotest.profile
end
INLINE FUNCTIONS
The xCAT-test testing framework provides some inline functions. The inline functions can be called in test cases as
__FUNCTIONNAME(PARAMTERLIST)__ to get some necessary attributes defined in the configuration file. The
inline functions can be used in cmd section and the check:output section.
1. GETNODEATTR(nodename, attribute) To get the value of specified node’s attribute
2. INC(digit) To get value of digit+1.
For example, to run rscan command against the hardware control point of compute node specified in the configuration
file:
rscan __GETNODEATTR($$CN, hcp)__ -z
3. B<GETTABLEVALUE(keyname, key, colname, table)> To get the value of column where
˓→keyname == key in specified table.
1.3. Admin Guide
513
xCAT3 Documentation, Release 2.12
FILES
/opt/xcat/bin/xcattest
xcoll.1
NAME
xcoll - Formats and consolidates the output of the psh, rinv commands.
SYNOPSIS
xcoll [-n] [-c]
DESCRIPTION
The xcoll command formats and consolidates output from the psh,rinv command. The xcollcommand takes, as input,
lines in the following format:
groupname: line of output from remote command, will use group name, if defined
The xcoll command formats the lines as follows and writes them to standard output. Assume that the output from
node3 and node4 is identical:
====================================
node1 or nodegroup name
====================================
.
.
lines from psh for node1 with hostnames stripped off
.
.
====================================
node2 or nodegroup name
====================================
.
.
lines from psh for node2 with hostnames stripped off
.
.
====================================
node3, node4 or nodegroup name
====================================
.
.
lines from psh for node 3 with hostnames stripped off
.
.
514
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
OPTIONS
-c
Display a total nodecount for each set of output.
-n
Display output as nodenames instead of groupnames.
EXAMPLES
1. To display the results of a command issued on several nodes, in the format used in the Description, enter:
psh node1,node2,node3 cat /etc/passwd | xcoll
SEE ALSO
psh(1)|psh.1, xdshbak(1)|xdshbak.1 ,xdshcoll(1)|xdshcoll.1
xdcp.1
NAME
xdcp - Concurrently copies files to or from multiple nodes. In addition, provides an option to use rsync to update the
files on the nodes, or to an installation image on the local node.
SYNOPSIS
xdcp noderange [[-B | --bypass] [-f fanout] [-L] [-l userID] [-o node_options] [-p] [-P] [-r node_remote_shell] [-R]
[-t timeout] [-T] [-v] [-q] [-X env_list] sourcefile.... targetpath
xdcp noderange [-F rsync input file]
xdcp computenoderange [-s -F rsync input file]
xdcp [-i path to install image] [-F rsync input file]
xdcp [-h | -V | -q]
DESCRIPTION
The xdcp command concurrently copies files to or from remote target nodes. The command issues a remote copy
command for each node or device specified. When files are pulled from a target, they are placed into the target_path
with the name of the remote node or device appended to the copied source_file name. The /usr/bin/rcp command is
the model for syntax and security. If using hierarchy, then xdcp runs on the service node that is servicing the compute
node. The file will first be copied to the path defined in the site table, SNsyncfiledir attribute, or the default path
/var/xcat/syncfiles on the service node, if the attribute is not defined. The -P flag will not automatically copy the
files from the compute node to the Management node, hierarchically. There is a two step process, see -P flag. If
the Management Node is target node, it must be defined in the xCAT database with nodetype=mn. When the xdcp
command runs the Management Node as the target, it does not use remote commands but uses the local OS copy (cp)
command.
1.3. Admin Guide
515
xCAT3 Documentation, Release 2.12
REMOTE USER:
A user_ID can be specified for the remote copy command. Remote user specification is identical for the xdcp and xdsh
commands. See the xdsh command for more information.
REMOTE COMMAND COPY:
The xdcp command uses a configurable remote copy command to execute remote copies on remote targets. Support
is explicitly provided for Remote Shell rcp command, the OpenSSH scp command and the /usr/bin/rsync command.
For node targets, the remote copy command is determined by the follow- ing order of precedence:
1. The -r flag.
2. The /usr/bin/scp command.
COMMAND EXECUTIONS:
The maximum number of concurrent remote copy command processes (the fanout) can be specified with the -f flag or
the DSH_FANOUT environment variable. The fanout is only restricted by the number of remote shell commands that
can be run in parallel. You can experiment with the DSH_FANOUT value on your management server to see if higher
values are appropriate.
A timeout value for remote copy command execution can be specified with the -t flag or DSH_TIMEOUT environment
variable. If any remote target does not respond within the timeout value, the xdcp command displays an error message
and exits.
The -T flag provides diagnostic trace information for dcp command execution. Default settings and the actual remote
copy commands that are executed to the remote targets are displayed.
The xdcp command can be executed silently using the -Q flag; no target standard output or standard error is displayed.
OPTIONS
sourcefile...
Specifies the complete path for the file to be copied to or from the target. Multiple files can be specified.
When used with the -R flag, only a single directory can be specified. When used with the -P flag, only a
single file can be specified.
targetpath
If one source_file file, then it specifies the file to copy the source_file file to on the target. If multiple
source_file files, it specifies the directory to copy the source_file files to on the target. If the -P flag is
specified, the target_path is the local host location for the copied files. The remote file directory structure
is recreated under target_path and the remote target name is appended to the copied source_file name in
the target_path directory. Note: the targetpath directory must exist.
-B | --bypass
Runs in bypass mode, use if the xcatd daemon is hung.
-f | --fanout fanout_value
Specifies a fanout value for the maximum number of concur- rently executing remote shell processes.
Serial execution can be specified by indicating a fanout value of 1. If -f is not specified, a default fanout
value of 64 is used.
-F | --File rsync input file
Specifies the path to the file that will be used to build the rsync command. The format of the input file is
as follows, each line contains:
516
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
<path to source file1> <path to source file2> ... ->
˓→file/directory>
< path to destination
or
<path to source file> -> <path to destination file>
or
<path to source file> -> <path to destination directory ( must end in /)>
For example:
/etc/password /etc/hosts -> /etc
/tmp/file2
->
/tmp/file2
/tmp/file2
->
/tmp/
/tmp/filex
->
/tmp/source/filey
/etc/*
/etc/
->
Running postscripts after files are sync’d to the nodes:
After you define the files to rsync, you can add an EXECUTE: clause in the synclist file. The EXECUTE:
clause will list all the postscripts that you would like to run after the files are sync’d to the node. The
postscript file must be of the form filename.post, where the <filename> is the <filename> is the from
<filename>, reside in the same directory as filename, and be executable. If the file filename is rsync’d to
the node, then the filename.postwill automatically be run on the node. If the file filename is not updated
on the node, the filename.post will not be run.
Putting the filename.post in the file list to rsync to the node is required for hierarchical clusters. It is
optional for non-hierarchical clusters.
Another option is the EXECUTEALWAYS: clause in the synclist file. The EXECUTEALWAYS: will
list all the postscripts that you would like to run after the files are sync’d to the nodes. These scripts will
run whether or not any files are sync’d to the nodes. The scripts have no special format, but must contain
the fully qualified path.
The scripts must be also added to the file list to rsync to the node for hierarchical clusters. It is optional
for non-hierarchical clusters.
For example, your rsynclist file may look like this:
/tmp/share/file2 -> /tmp/file2
/tmp/share/file2.post -> /tmp/file2.post
/tmp/share/file3 -> /tmp/filex
/tmp/share/file3.post -> /tmp/file3.post
/tmp/myscript -> /tmp/myscript
# the below are postscripts
EXECUTE:
/tmp/share/file2.post
/tmp/share/file3.post
EXECUTEALWAYS:
/tmp/myscript
If /tmp/file2 and /tmp/file3 update /tmp/file2 and /tmp/filex on the node, then the postscripts /tmp/file2.post
and /tmp/file3.post are automatically run on the node. /tmp/myscript will always be run on the node.
1.3. Admin Guide
517
xCAT3 Documentation, Release 2.12
Another option is the APPEND: clause in the synclist file. The APPEND: clause is used to append the
contents of the input file to an existing file on the node. The file to append must already exist on the node
and not be part of the synclist that contains the APPEND: clause.
For example, your rsynclist file may look like this:
/tmp/share/file2 -> /tmp/file2
/tmp/share/file2.post -> /tmp/file2.post
/tmp/share/file3 -> /tmp/filex
/tmp/share/file3.post -> /tmp/file3.post
/tmp/myscript -> /tmp/myscript
# the below are postscripts
EXECUTE:
/tmp/share/file2.post
/tmp/share/file3.post
EXECUTEALWAYS:
/tmp/myscript
APPEND:
/etc/myappenddir/appendfile -> /etc/mysetup/setup
/etc/myappenddir/appendfile2 -> /etc/mysetup/setup2
When you use the append script, the file (left) of the arrow is appended to the file right of the arrow. In this
example, /etc/myappenddir/appendfile is appended to /etc/mysetup/setup file, which must already exist on
the node. The /opt/xcat/share/xcat/scripts/xdcpappend.sh is used to accomplish this.
Another option is the MERGE: clause in the synclist file. The MERGE: clause is used to append the
contents of the input file to /etc/passwd, /etc/group, or /etc/shadow on a Linux node. It is only supported
for those files and only on Linux. You must not use both the APPEND and MERGE funcion for these
three files. The processing could end up not creating the file you desire. The MERGE function is the
preferred method, becuase APPEND only adds to the file. MERGE will add to the file but also insure
there are no duplicate entries.
For example, your rsynclist file may look like this: /tmp/share/file2 -> /tmp/file2 /tmp/share/file2.post
-> /tmp/file2.post /tmp/share/file3 -> /tmp/filex /tmp/share/file3.post -> /tmp/file3.post
/tmp/myscript -> /tmp/myscript # the below are postscripts EXECUTE: /tmp/share/file2.post
/tmp/share/file3.post EXECUTEALWAYS: /tmp/myscript APPEND: /custom/mypasswd ->
/etc/passwd /custom/mygroups -> /etc/group /custom/myshadow -> /etc/shadow
Note: no order can be assumed by the order that the EXECUTE,EXECUTEALWAYS and APPEND
clause fall in the synclist file.
For more information on syncing files to node, read Sync-ing_Config_Files_to_Nodes
On Linux rsync always uses ssh remoteshell.
site.useSSHonAIX attribute.
On AIX, ssh or rsh is used depending on the
-h | --help
Displays usage information.
-i | --rootimg install image
Specifies the path to the install image on the local Linux node.
-o | --node-options node_options
Specifies options to pass to the remote shell command for node targets. The options must be specified
within double quotation marks (“”) to distinguish them from xdsh options.
-p | --preserve
Preserves the source file characteristics as implemented by the configured remote copy command.
518
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
-P | --pull
Pulls (copies) the files from the targets and places them in the target_path directory on the local host. The
target_path must be a directory. Files pulled from remote machines have ._target appended to the file
name to distinguish between them. When the -P flag is used with the -R flag, ._target is appended to the
directory. Only one file per invocation of the xdcp pull command can be pulled from the specified targets.
Hierarchy is not automatically support yet. You must first pull the file to the Service Node and then pull
the file to the Management node.
-q | --show-config
Displays the current environment settings for all DSH Utilities commands. This includes the values of all
environment variables and settings for all currently installed and valid contexts. Each setting is prefixed
with context: to identify the source context of the setting.
-r | --node-rcp node_remote_copy
Specifies the full path of the remote copy command used for remote command execution on node targets.
-R | --recursive recursive
Recursively copies files from a local directory to the remote targets, or when specified with the -P flag,
recursively pulls (copies) files from a remote directory to the local host. A single source directory can be
specified using the source_file parameter.
-s synch service nodes
Will only sync the files listed in the synclist (-F), to the service nodes for the input compute node
list. The files will be placed in the directory defined by the site.SNsyncfiledir attribute, or the default
/var/xcat/syncfiles directory.
-t | --timeout timeout
Specifies the time, in seconds, to wait for output from any currently executing remote targets. If no output
is available from any target in the specified timeout, xdsh displays an error and terminates execution for
the remote targets that failed to respond. If timeout is not specified, xdsh waits indefinitely to continue
processing output from all remote targets. When specified with the -i flag, the user is prompted for an
additional timeout interval to wait for output.
-T | --trace
Enables trace mode. The xdcp command prints diagnostic messages to standard output during execution
to each target.
-v | --verify
Verifies each target before executing any remote commands on the target. If a target is not responding,
execution of remote commands for the target is canceled.
-V | --version
Displays the xdcp command version information.
Environment Variables
DSH_ENVIRONMENT
Specifies a file that contains environment variable definitions to export to the target before executing the
remote command. This variable is overridden by the -E flag.
DSH_FANOUT
Specifies the fanout value. This variable is overridden by the -f flag.
1.3. Admin Guide
519
xCAT3 Documentation, Release 2.12
DSH_NODE_OPTS
Specifies the options to use for the remote shell command with node targets only. This variable is overridden by the -o flag.
DSH_NODE_RCP
Specifies the full path of the remote copy command to use to copy local scripts and local environment
configuration files to node targets.
DSH_NODE_RSH
Specifies the full path of the remote shell to use for remote command execution on node targets. This
variable is overridden by the -r flag.
DSH_NODEGROUP_PATH
Specifies a colon-separated list of directories that contain node group files for the DSH context. When the
-a flag is specified in the DSH context, a list of unique node names is collected from all node group files
in the path.
DSH_PATH
Sets the command path to use on the targets. If DSH_PATH is not set, the default path defined in the
profile of the remote user_ID is used.
DSH_SYNTAX
Specifies the shell syntax to use on remote targets; ksh or csh. If not specified, the ksh syntax is assumed.
This variable is overridden by the -S flag.
DSH_TIMEOUT
Specifies the time, in seconds, to wait for output from each remote target. This variable is overridden by
the -tflag.
Exit Status
Exit values for each remote copy command execution are displayed in messages from the xdcp command, if the remote
copy command exit value is non-zero. A non-zero return code from a remote copy command indicates that an error
was encountered during the remote copy. If a remote copy command encounters an error, execution of the remote copy
on that tar- get is bypassed.
The xdcp command exit code is 0, if the xdcp command executed without errors and all remote copy commands
finished with exit codes of 0. If internal xdcp errors occur or the remote copy commands do not complete successfully,
the xdcp command exit value is greater than 0.
Security
The xdcp command has no security configuration requirements. All remote command security requirements - configuration, authentication, and authorization - are imposed by the underlying remote command configured for xdsh.
The command assumes that authentication and authorization is configured between the local host and the remote targets. Interactive password prompting is not supported; an error is displayed and execution is bypassed for a remote
target if password prompting occurs, or if either authorization or authentication to the remote target fails. Security
configurations as they pertain to the remote environment and remote shell command are userdefined.
520
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
Examples
1. To copy the /etc/hosts file from all nodes in the cluster to the /tmp/hosts.dir directory on the local host, enter:
xdcp all -P /etc/hosts /tmp/hosts.dir
A suffix specifying the name of the target is appended to each file name. The contents of the /tmp/hosts.dir
directory are similar to:
hosts._node1
hosts._node2
hosts._node3
hosts._node4
hosts._node5
hosts._node6
hosts._node7
hosts._node8
2. To copy the directory /var/log/testlogdir from all targets in NodeGroup1 with a fanout of 12, and save each directory
on the local host as /var/log._target, enter:
xdcp NodeGroup1 -f 12 -RP /var/log/testlogdir /var/log
3. To copy /localnode/smallfile and /tmp/bigfile to /tmp on node1 using rsync and input -t flag to rsync, enter:
xdcp node1 -r /usr/bin/rsync -o "-t" /localnode/smallfile /tmp/bigfile /tmp
4. To copy the /etc/hosts file from the local host to all the nodes in the cluster, enter:
xdcp all /etc/hosts /etc/hosts
5. To copy all the files in /tmp/testdir from the local host to all the nodes in the cluster, enter:
xdcp all /tmp/testdir/* /tmp/testdir
6. To copy all the files in /tmp/testdir and it’s subdirectories from the local host to node1 in the cluster, enter:
xdcp node1 -R /tmp/testdir /tmp/testdir
7. To copy the /etc/hosts file from node1 and node2 to the /tmp/hosts.dir directory on the local host, enter:
xdcp node1,node2 -P /etc/hosts /tmp/hosts.dir
8. To rsync the /etc/hosts file to your compute nodes:
Create a rsync file /tmp/myrsync, with this line:
/etc/hosts -> /etc/hosts
or
/etc/hosts -> /etc/ (last / is required)
Run:
xdcp compute -F /tmp/myrsync
9. To rsync all the files in /home/mikev to the compute nodes:
Create a rsync file /tmp/myrsync, with this line:
/home/mikev/* -> /home/mikev/ (last / is required)
Run:
1.3. Admin Guide
521
xCAT3 Documentation, Release 2.12
xdcp compute -F /tmp/myrsync
10. To rsync to the compute nodes, using service nodes, the command will first rsync the files to the /var/xcat/syncfiles
directory on the service nodes and then rsync the files from that directory to the compute nodes. The /var/xcat/syncfiles
default directory on the service nodes, can be changed by putting a directory value in the site table SNsyncfiledir
attribute.
Create a rsync file /tmp/myrsync, with this line:
/etc/hosts /etc/passwd -> /etc
or
/etc/hosts /etc/passwd -> /etc/
Run:
xdcp compute -F /tmp/myrsync
to update the Compute Nodes
11. To rsync to the service nodes in preparation for rsyncing the compute nodes during an install from the service
node.
Create a rsync file /tmp/myrsync, with this line:
/etc/hosts /etc/passwd -> /etc
Run:
xdcp compute -s -F /tmp/myrsync
to sync the service node for compute
12. To rsync the /etc/file1 and file2 to your compute nodes and rename to filex and filey:
Create a rsync file /tmp/myrsync, with these line:
/etc/file1 -> /etc/filex
/etc/file2 -> /etc/filey
Run:
xdcp compute -F /tmp/myrsync
to update the Compute Nodes
13. To rsync files in the Linux image at /install/netboot/fedora9/x86_64/compute/rootimg on the MN:
Create a rsync file /tmp/myrsync, with this line:
/etc/hosts /etc/passwd -> /etc
Run:
xdcp -i /install/netboot/fedora9/x86_64/compute/rootimg -F /tmp/myrsync
14. To define the Management Node in the database so you can use xdcp, run
xcatconfig -m
522
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
Files
SEE ALSO
xdsh(1)|xdsh.1, noderange(3)|noderange.3
xdsh.1
NAME
xdsh - Concurrently runs remote commands on multiple nodes (Management Node, Service Nodes, compute nodes),
or an install image.
SYNOPSIS
xdsh noderange [-B | --bypass] [--devicetype type_of_device] [-e] [-E environment_file] [-f fanout] [-L] [-l userID] [m] [-o node_options] [-Q] [-r node_remote_shell] [-s] [-S {csh | ksh}] [-t timeout] [-T] [-v] [-X env_list] [-z] [--sudo]
command_list
xdsh noderange [-K]
xdsh noderange [-K] [-l userID] --devicetype type_of_device
xdsh [-i image path | nim image name] command_list
xdsh noderange [-c]
xdsh [-h | -V | -q]
DESCRIPTION
The xdsh command runs commands in parallel on remote nodes and/or the Management Node. The xdsh command
issues a remote shell command for each target specified, and returns the output from all targets, formatted so that
command results from all nodes can be managed. If the command is to be executed on the Management Node, it
does not use a remote shell command, but uses the local OS copy or shell command. The Management Node must be
defined in the xCAT database. The best way to do this is to use the xcatconfig -m option. The xdsh command is an
xCAT Distributed Shell Utility.
COMMAND SPECIFICATION:
The commands to execute on the targets are specified by the command_list xdsh parameter, or executing a local script
using the -e flag.
The syntax for the command_list xdsh parameter is as follows:
command[; command]...
where command is the command to run on the remote target. Quotation marks are required to ensure that all commands
in the list are executed remotely, and that any special characters are interpreted correctly on the remote target. A script
file on the local host can be executed on each of the remote targets by using the -e flag. If -e is specified, command_list
is the script name and arguments to the script. For example:
xdsh hostname -e script_filename [arguments]...
The script_filename file is copied to a random filename in the /tmp directory on each remote target and then executed
on the targets.
1.3. Admin Guide
523
xCAT3 Documentation, Release 2.12
The xdsh command does not work with any interactive commands, including those that read from standard input.
REMOTE SHELL COMMAND:
The xdsh command uses a configurable remote shell command to execute remote commands on the remote targets.
Support is explicitly provided for AIX Remote Shell and OpenSSH, but any secure remote command that conforms to
the IETF (Internet Engineering Task Force) Secure Remote Command Protocol can be used.
The remote shell is determined as follows, in order of precedence:
1. The -r flag.
2. The DSH_NODE_RSH environment variable.
3. The default node remote shell as defined by the target context.
4. The /usr/bin/ssh command.
The remote shell options are determined as follows, in order of prece- dence:
1. The -o flag.
2. The DSH_NODE_OPTS environment variable.
REMOTE SHELL ENVIRONMENT:
The shell environment used on the remote target defaults to the shell defined for the user_ID on the remote target. The
command syntax that xdsh uses to form the remote commands can be specified using the -S flag. If -S is not specified,
the syntax defaults to sh syntax.
When commands are executed on the remote target, the path used is determined by the DSH_PATH environment
variable defined in the shell of the current user. If DSH_PATH is not set, the path used is the remote shell default path.
For example, to set the local path for the remote targets, use:
DSH_PATH=$PATH
The -E flag exports a local environment definition file to each remote target. Environment variables specified in this
file are defined in the remote shell environment before the command_list is executed. The file should be executable
and contain one environment variable per line.
COMMAND EXECUTION:
The maximum number of concurrent remote shell command processes (the fanout) can be specified with the -f flag or
with the DSH_FANOUTenvironment variable. The fanout is only restricted by the number of remote shell commands
that can be run in parallel. You can experiment with the DSH_FANOUT value on your management server to see if
higher values are appropriate.
A timeout value for remote command execution can be specified with the -t flag or with the DSH_TIMEOUT environment variable. If any remote target does not provide output to either standard output or standard error within the
timeout value, xdsh displays an error message and exits.
If streaming mode is specified with the -s flag, output is returned as it becomes available from each target, instead of
waiting for the command_list to complete on all targets before returning output. This can improve performance but
causes the output to be unsorted.
The -z flag displays the exit code from the last command issued on the remote node in command_list. Note that
OpenSSH behaves differently; it returns the exit status of the last remote command issued as its exit status. If the
command issued on the remote node is run in the background, the exit status is not displayed.
The -m flag monitors execution of the xdsh command by printing status messages to standard output. Each status
message is preceded by dsh.
The -T flag provides diagnostic trace information for the execution of the xdsh command. Default settings and the
actual remote shell commands executed on the remote targets are displayed.
524
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
No error detection or recovery mechanism is provided for remote targets. The xdsh command output to standard error
and standard output can be analyzed to determine the appropriate course of action.
COMMAND OUTPUT:
The xdsh command waits until complete output is available from each remote shell process and then displays that
output before initiating new remote shell processes. This default behavior is overridden by the -s flag.
The xdsh command output consists of standard error and standard output from the remote commands. The xdsh
standard output is the standard output from the remote shell command. The xdsh standard error is the standard error
from the remote shell command. Each line is prefixed with the host name of the node that produced the output. The
host name is followed by the : character and a command output line. A filter for displaying identical outputs grouped
by node is provided separately. See the xdshbak command for more information.
A command can be run silently using the -Q flag; no output from each target’s standard output or standard error is
displayed.
SIGNALS:
Signal 2 (INT), Signal 3 (QUIT), and Signal 15 (TERM) are propagated to the commands executing on the remote
targets.
Signal 19 (CONT), Signal 17 (STOP), and Signal 18 (TSTP) default to xdsh; the xdsh command responds normally
to these signals, but the signals do not have an effect on remotely executing commands. Other signals are caught by
xdsh and have their default effects on the xdshcommand; all current child processes, through propagation to remotely
running commands, are terminated (SIGTERM).
OPTIONS
-B | --bypass
Runs in bypass mode, use if the xcatd daemon is hung.
-c | --cleanup
This flag will have xdsh remove all files from the subdirectories of the the directory on the servicenodes, where xdcp stages the copy to the compute nodes as defined in the site table SNsyncfiledir and
nodesyncfiledir attribute, when the target is a service node.
It can also be used to remove the nodesyncfiledir directory on the compute nodes, which keeps the backup
copies of files for the xdcp APPEND function support, if a compute node is the target.
-e | --execute
Indicates that command_list specifies a local script filename and arguments to be executed on the remote
targets. The script file is copied to the remote targets and then remotely executed with the given arguments.
The DSH_NODE_RCP environment variables specify the remote copy command to use to copy the script
file to node targets.
-E | --environment environment_file
Specifies that the environment_file contains environment variable definitions to export to the target before
executing the command_list.
--devicetype type_of_device
Specify a user-defined device type that references the location of relevant device configuration file. The
devicetype value must correspond to a valid device configuration file. xCAT ships some default configuration files for Ethernet switches and and IB switches under /opt/xcat/share/xcat/devicetype directory. If
you want to overwrite any of the configuration files, copy them to /var/opt/xcat/ directory and cutomize.
For example, base/IBSwitch/Qlogic/config is the configuration file location if devicetype is specified as
1.3. Admin Guide
525
xCAT3 Documentation, Release 2.12
IBSwitch::Qlogic. xCAT will first search config file using /var/opt/xcat/ as the base. If not found, it will
search for it using /opt/xcat/share/xcat/devicetype/ as the base.
-f | --fanout fanout_value
Specifies a fanout value for the maximum number of concurrently executing remote shell processes. Serial
execution can be specified by indicating a fanout value of 1. If -f is not specified, a default fanout value
of 64 is used.
-h | --help
Displays usage information.
-i | --rootimg install image
For Linux, Specifies the path to the install image on the local node. For AIX, specifies the name of the
osimage on the local node. Run lsnim for valid names. xdsh will chroot (xcatchroot for AIX) to this path
and run the xdsh command against the install image. No other xdsh flags, environment variables apply
with this input. A noderange is not accepted. Only runs on the local host, normally the Management
Node. The command you run must not prompt for input, the prompt will not be returned to you, and it
will appear that xdsh hangs.
-K | --ssh-setup
-K | --ssh-setup -l | --user user_ID --devicetype type_of_device
Set up the SSH keys for the user running the command to the specified node list. The userid must have
the same uid, gid and password as the userid on the node where the keys will be setup.
If the current user is root, roots public ssh keys will be put in the authorized_keys* files under roots .ssh
directory on the node(s). If the current user is non-root, the user must be in the policy table and have
credential to run the xdsh command. The non-root users public ssh keys and root’s public ssh keys will
be put in the authorized_keys* files under the non-root users .ssh directory on the node(s). Other device
types, such as IB switch, are also supported. The device should be defined as a node and nodetype should
be defined as switch before connecting. The xdsh -K command must be run from the Management Node.
-l | --user user_ID
Specifies a remote user name to use for remote command execution.
-L | --no-locale
Specifies to not export the locale definitions of the local host to the remote targets. Local host locale
definitions are exported by default to each remote target.
-m | --monitor
Monitors remote shell execution by displaying status messages during execution on each target.
-o | --node-options node_options
Specifies options to pass to the remote shell command for node targets. The options must be specified
within double quotation marks (“”) to distinguish them from xdsh options.
-q | --show-config
Displays the current environment settings for all DSH Utilities commands. This includes the values of all
environment variables and settings for all currently installed and valid contexts. Each setting is prefixed
with context: to identify the source context of the setting.
-Q | --silent
Specifies silent mode. No target output is written to standard output or standard error. Monitoring messages are written to standard output.
526
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
-r | --node-rsh node_remote_shell
Specifies the path of the remote shell command used for remote command execution on node targets.
-s | --stream
Specifies that output is returned as it becomes available from each target, instead of waiting for the command_listto be completed on a target before returning output.
-S | --syntax {csh | ksh}
Specifies the shell syntax to be used on the remote target. If not specified, the ksh syntax is used.
--sudo
Adding the --sudo flag to the xdsh command will have xdsh run sudo before running the command. This
is particular useful when using the -e option. This is required when you input -l with a non-root user id
and want that id to be able to run as root on the node. The non-root userid will must be previously defined
as an xCAT user, see process for defining non-root ids in xCAT and setting up for using xdsh. The userid
sudo setup will have to be done by the admin on the node. This includes, allowing all commands that
you would like to run with xdsh by using visudo to edit the /etc/sudoers file. You must disabl ssh tty
requirements by commenting out or removing this line in the /etc/sudoes file “#Defaults requiretty”. See
the document Granting_Users_xCAT_privileges for sudo setup requirements. This is not supported in a
hierarical cluster, that is the nodes are serviced by servicenodes.
-t | --timeout timeout
Specifies the time, in seconds, to wait for output from any currently executing remote targets. If no output
is available from any target in the specified timeout, xdshdisplays an error and terminates execution for
the remote targets that failed to respond. If timeout is not specified, xdsh waits indefinitely to continue
processing output from all remote targets. The exception is the -K flag which defaults to 10 seconds.
-T | --trace
Enables trace mode. The xdsh command prints diagnostic messages to standard output during execution
to each target.
-v | --verify
Verifies each target before executing any remote commands on the target. If a target is not responding,
execution of remote commands for the target is canceled. When specified with the -i flag, the user is
prompted to retry the verification request.
-V | --version
Displays the xdsh command version information.
-X env_list
Ignore xdsh environment variables. This option can take an argument which is a comma separated list of
environment variable names that should NOT be ignored. If there is no argument to this option, or the
argument is an empty string, all xdsh environment variables will be ignored. This option is useful when
running xdsh from within other scripts when you don’t want the user’s environment affecting the behavior
of xdsh.
-z | --exit-status
Displays the exit status for the last remotely executed non-asynchronous command on each target. If the
command issued on the remote node is run in the background, the exit status is not displayed.
Exit values for each remote shell execution are displayed in messages from the xdsh command, if the
remote shell exit values are non-zero. A non-zero return code from a remote shell indicates that an error
was encountered in the remote shell. This return code is unrelated to the exit code of the remotely issued
1.3. Admin Guide
527
xCAT3 Documentation, Release 2.12
command. If a remote shell encounters an error, execution of the remote command on that target is
bypassed.
The xdsh command exit code is 0 if the command executed without errors and all remote shell commands
finished with exit codes of 0. If internal xdsh errors occur or the remote shell commands do not complete
successfully, the xdsh command exit value is greater than 0. The exit value is increased by 1 for each
successive instance of an unsuccessful remote command execution. If the remotely issued command is
run in the background, the exit code of the remotely issued command is 0.
Environment Variables
DEVICETYPE
Specify a user-defined device type. See --devicetype flag.
DSH_ENVIRONMENT
Specifies a file that contains environment variable definitions to export to the target before executing the
remote command. This variable is overridden by the -E flag.
DSH_FANOUT
Specifies the fanout value. This variable is overridden by the -f flag.
DSH_NODE_OPTS
Specifies the options to use for the remote shell command with node targets only. This variable is overridden by the -o flag.
DSH_NODE_RCP
Specifies the full path of the remote copy command to use to copy local scripts and local environment
configuration files to node targets.
DSH_NODE_RSH
Specifies the full path of the remote shell to use for remote command execution on node targets. This
variable is overridden by the -r flag.
DSH_PATH
Sets the command path to use on the targets. If DSH_PATH is not set, the default path defined in the
profile of the remote user_ID is used.
DSH_REMOTE_PASSWORD
If DSH_REMOTE_PASSWORD is set to the password of the userid (usually root) that will ssh to the
node, then when you use the -K flag, you will not be prompted for a password.
DSH_SYNTAX
Specifies the shell syntax to use on remote targets; ksh or csh. If not specified, the ksh syntax is assumed.
This variable is overridden by the -S flag.
DSH_TIMEOUT
Specifies the time, in seconds, to wait for output from each remote target. This variable is overridden by
the -t flag.
DSH_VERIFY
Verifies each target before executing any remote commands on the target. If a target is not responding,
execution of remote commands for the target is canceled. This variable is overridden by the -v flag.
528
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
Compatibility with AIX dsh
To provide backward compatibility for scripts written using dsh in AIX and CSM, a tool has been provided groupfiles4dsh, which will build node group files from the xCAT database that can be used by dsh. See man groupfiles4dsh.
SECURITY
The xdsh command has no security configuration requirements. All remote command security requirements - configuration, authentication, and authorization - are imposed by the underlying remote command configured for xdsh.
The command assumes that authentication and authorization is configured between the local host and the remote targets. Interactive password prompting is not supported; an error is displayed and execution is bypassed for a remote
target if password prompting occurs, or if either authorization or authentication to the remote target fails. Security
configurations as they pertain to the remote environment and remote shell command are userdefined.
EXIT STATUS
The xdsh command exit code is 0 if the command executed without errors and all remote shell commands finished
with exit codes of 0. If internal dsh errors occur or the remote shell commands do not complete successfully, the dsh
command exit value is greater than 0. The exit value is increased by 1 for each successive instance of an unsuccessful
remote command execution. If the remotely issued command is run in the background, the exit code of the remotely
issued command is 0.
EXAMPLES
1. To set up the SSH keys for root on node1, run as root:
xdsh node1 -K
2. To run the ps -ef command on node targets node1 and node2, enter:
xdsh node1,node2 "ps -ef"
3. To run the ps command on node targets node1 and run the remote command with the -v and -t flag, enter:
xdsh node1,node2
-o "-v -t" ps
4. To execute the commands contained in myfile in the XCATcontext on several node targets, with a fanout of 1, enter:
xdsh node1,node2 -f 1 -e myfile
5. To run the ps command on node1 and ignore all the dsh environment variable except the DSH_NODE_OPTS, enter:
xdsh node1 -X `DSH_NODE_OPTS' ps
6. To run on Linux, the xdsh command “rpm -qa | grep xCAT” on the service node fedora9 diskless image, enter:
xdsh -i /install/netboot/fedora9/x86_64/service/rootimg "rpm -qa | grep xCAT"
7. To run on AIX, the xdsh command “lslpp -l | grep bos” on the NIM 611dskls spot, enter:
xdsh -i 611dskls "/usr/bin/lslpp -l | grep bos"
8. To cleanup the servicenode directory that stages the copy of files to the nodes, enter:
1.3. Admin Guide
529
xCAT3 Documentation, Release 2.12
xdsh servicenoderange -c
9. To define the QLogic IB switch as a node and to set up the SSH keys for IB switch qswitch with device
configuration file /var/opt/xcat/IBSwitch/Qlogic/config and user name username, enter
chdef -t node -o qswitch groups=all nodetype=switch
xdsh qswitch -K -l username --devicetype IBSwitch::Qlogic
10. To define the Management Node in the database so you can use xdsh, enter
xcatconfig -m
11. To define the Mellanox switch as a node and run a command to show the ssh keys. mswitch with and user name
username, enter
chdef -t node -o mswitch groups=all nodetype=switch
xdsh mswitch -l admin --devicetype IBSwitch::Mellanox
˓→terminal;show ssh server host-keys'
'enable;configure
12. To define a BNT Ethernet switch as a node and run a command to create a new vlan with vlan id 3 on the switch.
chdef myswitch groups=all
tabch switch=myswitch switches.sshusername=admin switches.sshpassword=passw0rd
˓→switches.protocol=[ssh|telnet]
where admin and passw0rd are the SSH user name and password for the switch.
If it is for Telnet, add tn: in front of the user name: tn:admin.
dsh myswitch --devicetype EthSwitch::BNT 'enable;configure terminal;vlan 3;end;
˓→show vlan'
13. To run xdsh with the non-root userid “user1” that has been setup as an xCAT userid and with sudo on node1
and node2 to run as root, do the following, see xCAT doc on Granting_Users_xCAT_privileges:
xdsh node1,node2 --sudo -l user1 "cat /etc/passwd"
Files
SEE ALSO
xdshbak(1)|xdshbak.1, noderange(3)|noderange.3, groupfiles4dsh(1)|groupfiles4dsh.1
xdshbak.1
NAME
xdshbak - Formats the output of the xdsh command.
530
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
SYNOPSIS
xdshbak [-c | -x [ -b ] | -h | -q]
DESCRIPTION
The xdshbak command formats output from the xdsh command. The xdshbakcommand takes, as input, lines in the
following format:
host_name: line of output from remote command
The xdshbak command formats the lines as follows and writes them to standard output. Assume that the output from
node3 and node4 is identical, and the -c (collapse) flag was specified:
HOSTS -------------------------------------------------------node1
-------------------------------------------------------------.
.
lines from xdsh for node1 with hostnames stripped off
.
.
HOSTS -------------------------------------------------------node2
-------------------------------------------------------------.
.
lines from xdsh for node2 with hostnames stripped off
.
.
HOSTS -------------------------------------------------------node3, node4
-------------------------------------------------------------.
.
lines from xdsh for node 3 with hostnames stripped off
.
.
When output is displayed from more than one node in collapsed form, the host names are displayed alphabetically.
When output is not collapsed, output is displayed sorted alphabetically by host name.
If the -q quiet flag is not set then xdshbakcommand writes ”.” for each 1000 lines of output processed (to show
progress), since it won’t display the output until it has processed all of it.
If the -x flag is specified, the extra header lines that xdshbak normally displays for each node will be omitted, and the
hostname at the beginning of each line is not stripped off, but xdshbak still sorts the output by hostname for easier
viewing:
node1: lines from xdsh for node1
.
.
node2: lines from xdsh for node2
.
.
If the -b flag is specified in addition to -x, the hostname at the beginning of each line is stripped.
1.3. Admin Guide
531
xCAT3 Documentation, Release 2.12
Standard Error
When the xdshbak filter is used and standard error messages are generated, all error messages on standard error appear
before all standard output messages. This is true with and without the -c flag.
OPTIONS
-b
Strip the host prefix from the beginning of the lines. This only works with the -x option.
-c
If the output from multiple nodes is identical it will be collapsed and displayed only once.
-x
Omit the extra header lines that xdshbak normally displays for each node. This provides more compact
output, but xdshbak still sorts the output by node name for easier viewing. This option should not be used
with -c.
-h
Displays usage information.
-q
Quiet mode, do not display ”.” for each 1000 lines of output.
EXAMPLES
1. To display the results of a command issued on several nodes, in the format used in the Description, enter:
xdsh node1,node2,node3 cat /etc/passwd | xdshbak
2. To display the results of a command issued on several nodes with identical output displayed only once, enter:
xdsh host1,host2,host3 pwd | xdshbak -c
3. To display the results of a command issued on several nodes with compact output and be sorted alphabetically by
host name, enter:
xdsh host1,host2,host3 date | xdshbak -x
SEE ALSO
xdsh(1)|xdsh.1, xcoll(1)|xcoll.1
xdshcoll.1
NAME
xdshcoll - Formats and consolidates the output of the xdsh,sinv commands.
532
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
SYNOPSIS
xdshcoll
DESCRIPTION
The xdshcoll command formats and consolidates output from the xdsh,sinv command. The xdshcollcommand takes,
as input, lines in the following format:
host_name: line of output from remote command
The xdshcoll command formats the lines as follows and writes them to standard output. Assume that the output from
node3 and node4 is identical:
====================================
node1
====================================
.
.
lines from xdsh for node1 with hostnames stripped off
.
.
====================================
node2
====================================
.
.
lines from xdsh for node2 with hostnames stripped off
.
.
====================================
node3, node4
====================================
.
.
lines from xdsh for node 3 with hostnames stripped off
.
.
EXAMPLES
1. To display the results of a command issued on several nodes, in the format used in the Description, enter:
xdsh node1,node2,node3 cat /etc/passwd> | B<xdshcoll
SEE ALSO
xdshbak(1)|xdshbak.1
1.3. Admin Guide
533
xCAT3 Documentation, Release 2.12
xpbsnodes.1
NAME
xpbsnodes - PBS pbsnodes front-end for a noderange.
SYNOPSIS
xpbsnodes [{noderange}] [{offline | clear | stat | state}]
xpbsnodes [-h | --help] [-v | --version]
DESCRIPTION
xpbsnodes is a front-end to PBS pbsnode but uses xCAT’s noderange to specify nodes.
OPTIONS
-h|--help Display usage message.
-v|--version Command Version.
offline|off Take nodes offline.
clear|online|on Take nodes online.
stat|state Display PBS node state.
RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
EXAMPLES
1. To display status of all PBS nodes, enter:
xpbsnodes all stat
FILES
/opt/torque/x86_64/bin/xpbsnodes
SEE ALSO
noderange(3)|noderange.3
534
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
man3
noderange.3
Name
noderange - syntax for compactly expressing a list of node names
Synopsis
Examples:
node1,node2,node8,node20,group1
node14-node56,node70-node203,group1-group10
node1,node2,node8,node20,node14-node56,node70-node203
node[14-56]
f[1-3]n[1-20]
all,-node129-node256,-frame01-frame03
/node.*
^/tmp/nodes
node10+5
10-15,-13
group1@group2
table.attribute<operator>value
Description
noderange is a syntax that can be used in most xCAT commands to conveniently specify a list of nodes. The result is
that the command will be applied to a range of nodes, often in parallel.
noderange is a comma-separated list. Each token (text between commas) in the list can be any of the forms listed
below:
Individual node or group:
node01
group1
A range of nodes or groups:
node01-node10
node[01-10]
node01:node10
(equivalent to: node01,node02,node03,...node10)
(same as above)
(same as above)
1.3. Admin Guide
535
xCAT3 Documentation, Release 2.12
node[01:10]
(same as above)
f[1-2]n[1-3]
(equivalent to: f1n1,f1n2,f1n3,f2n1,f2n2,f2n3)
group1-group3 (equivalent to: group1,group2,group3)
(all the permutations supported above for nodes are also supported for groups)
nodeRange tries to be intelligent about detecting padding, so you can specify “node001-node200” and it will add the
proper number of zeroes to make all numbers 3 digits.
An incremented range of nodes:
node10+3
(equivalent to: node10,node11,node12,node13)
A node shorthand range of nodes:
10-20
10+3
(equivalent to: node10,node11,node12,...node20)
(equivalent to: node10,node11,node12,node13)
Currently, the prefix that will be prepended for the above syntax is always “node”. Eventually, the prefix and optional suffix will be settable via the environment variables XCAT_NODE_PREFIX and XCAT_NODE_SUFFIX, but
currently this only works in bypass mode.
A regular expression match of nodes or groups:
/node[345].*
/group[12].*
(will match any nodes that start with node3, node4, or node5)
(will match any groups that start with group1 or group2)
The path of a file containing noderanges of nodes or groups:
^/tmp/nodelist
where /tmp/nodelist can contain entries like:
#my node list (this line ignored)
^/tmp/foo #ignored
node01
#node comment
node02
node03
node10-node20
/group[456].*
-node50
Node ranges can contain any combination:
node01-node30,node40,^/tmp/nodes,/node[13].*,2-10,node50+5
Any individual noderange may be prefixed with an exclusion operator (default -) with the exception of the file operator
(default ^). This will cause that individual noderange to be subtracted from the total resulting list of nodes.
The intersection operator @ calculates the intersection of the left and right sides:
group1@group2
˓→common)
(will result in the list of nodes that group1 and group2 have in
Any combination or multiple combinations of inclusive and exclusive ranges of nodes and groups is legal. There is no
precedence implied in the order of the arguments. Exclusive ranges have precedence over inclusive. Parentheses can
be used to explicitly specify precendence of any operators.
Nodes have precedence over groups. If a node range match is made then no group range match will be attempted.
536
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
All node and group names are validated against the nodelist table. Invalid names are ignored and return nothing.
xCAT Node Name Format
Throughout this man page the term xCAT Node Name Format is used. xCAT Node Name Format is defined by the
following regex:
^([A-Za-z-]+)([0-9]+)(([A-Za-z-]+[A-Za-z0-9-]*)*)
In plain English, a node or group name is in xCAT Node Name Format if starting from the begining there are:
* one or more alpha characters of any case and any number of “-” in any combination
* followed by one or more numbers
* then optionally followed by one alpha character of any case or “-“
* followed by any combination of case mixed alphanumerics and “-“
noderange supports node/group names in any format. xCAT Node Name Format is not required, however some
node range methods used to determine range will not be used for non-conformant names.
Example of xCAT Node Name Format node/group names:
NODENAME
node1
node001
node-001
node-foo-001-bar
node-foo-1bar
foo1bar2
rack01unit34
unit34rack01
pos0134
PREFIX
node
node
nodenode-foonode-foofoo
rack
unit
pos
NUMBER
1
001
001
001
1
1
01
34
0134
SUFFIX
-bar
bar
bar2
unit34
rack01
Examples
1. Generates a list of all nodes (assuming all is a group) listed in the nodelist table less node5 through node10:
all,-node5-node10
2. Generates a list of nodes 1 through 10 less nodes 3,4,5. Note that node4 is listed twice, first in the range and
then at the end. Because exclusion has precedence node4 will be excluded.
node1-node10,-node3-node5,node4
3. Generates a list of nodes 1 through 10 less nodes 3 and 5.
node1-node10,-node3,-node5
4. Generates a list of all (assuming ‘all’ is a group) nodes in the nodelist table less 17 through 32.
-node17-node32,all
5. Generates a list of nodes 1 through 128, and user nodes 1 through 4.
1.3. Admin Guide
537
xCAT3 Documentation, Release 2.12
node1-node128,user1-user4
6. Generates a list of all nodes (assuming ‘all’ is a group), less nodes in groups rack1 through rack3 (assuming
groups rack1, rack2, and rack3 are defined), less nodes 100 through 200, less nodes in the storage group. Note
that node150 is listed but is excluded.
all,-rack1-rack3,-node100-node200,node150,-storage
7. Generates a list of nodes matching the regex node[23].\*. That is all nodes that start with node2 or node3 and
end in anything or nothing. E.g. node2, node3, node20, node30, node21234 all match.
/node[23].*
8. Generates a list of nodes which have the value hmc in the nodehm.cons attribute.
nodehm.cons==hmc
nodehm.cons=~hmc
9. Generate a list of nodes in the 1st two frames:
f[1-2]n[1-42]
SEE ALSO
nodels(1)|nodels.1
man5
auditlog.5
NAME
auditlog - a table in the xCAT database.
SYNOPSIS
auditlog Attributes: recid, audittime, userid, clientname, clienttype, command, noderange, args, status, comments,
disable
DESCRIPTION
Audit Data log.
auditlog Attributes:
recid
The record id.
538
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
audittime
The timestamp for the audit entry.
userid
The user running the command.
clientname
The client machine, where the command originated.
clienttype
Type of command: cli,java,webui,other.
command
Command executed.
noderange
The noderange on which the command was run.
args
The command argument list.
status
Allowed or Denied.
comments
Any user-provided notes.
disable
Do not use. tabprune will not work if set to yes or 1
SEE ALSO
nodels(1), chtab(8), tabdump(8), tabedit(8)
bootparams.5
NAME
bootparams - a table in the xCAT database.
SYNOPSIS
bootparams Attributes: node, kernel, initrd, kcmdline, addkcmdline, dhcpstatements, adddhcpstatements, comments,
disable
DESCRIPTION
Current boot settings to be sent to systems attempting network boot for deployment, stateless, or other reasons. Mostly
automatically manipulated by xCAT.
1.3. Admin Guide
539
xCAT3 Documentation, Release 2.12
bootparams Attributes:
node
The node or group name
kernel
The kernel that network boot actions should currently acquire and use. Note this could be a chained boot
loader such as memdisk or a non-linux boot loader
initrd
The initial ramdisk image that network boot actions should use (could be a DOS floppy or hard drive
image if using memdisk as kernel)
kcmdline
Arguments to be passed to the kernel
addkcmdline
User specified one or more parameters to be passed to the kernel. For the kernel options need to be
persistent after installation, specify them with prefix “R::”
dhcpstatements
xCAT manipulated custom dhcp statements (not intended for user manipulation)
adddhcpstatements
Custom dhcp statements for administrator use (not implemneted yet)
comments
Any user-written notes.
disable
Set to ‘yes’ or ‘1’ to comment out this row.
SEE ALSO
nodels(1), chtab(8), tabdump(8), tabedit(8)
boottarget.5
NAME
boottarget - a table in the xCAT database.
SYNOPSIS
boottarget Attributes: bprofile, kernel, initrd, kcmdline, comments, disable
DESCRIPTION
Specify non-standard initrd, kernel, and parameters that should be used for a given profile.
540
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
boottarget Attributes:
bprofile
All nodes with a nodetype.profile value equal to this value and nodetype.os set to “boottarget”, will use
the associated kernel, initrd, and kcmdline.
kernel
The kernel that network boot actions should currently acquire and use. Note this could be a chained boot
loader such as memdisk or a non-linux boot loader
initrd
The initial ramdisk image that network boot actions should use (could be a DOS floppy or hard drive
image if using memdisk as kernel)
kcmdline
Arguments to be passed to the kernel
comments
Any user-written notes.
disable
Set to ‘yes’ or ‘1’ to comment out this row.
SEE ALSO
nodels(1), chtab(8), tabdump(8), tabedit(8)
cfgmgt.5
NAME
cfgmgt - a table in the xCAT database.
SYNOPSIS
cfgmgt Attributes: node, cfgmgr, cfgserver, roles, comments, disable
DESCRIPTION
Configuration management data for nodes used by non-xCAT osimage management services to install and configure
software on a node.
cfgmgt Attributes:
node
The node being managed by the cfgmgr service
cfgmgr
1.3. Admin Guide
541
xCAT3 Documentation, Release 2.12
The name of the configuration manager service. Currently ‘chef’ and ‘puppet’ are supported services.
cfgserver
The xCAT node name of the chef server or puppet master
roles
The roles associated with this node as recognized by the cfgmgr for the software that is to be installed
and configured. These role names map to chef recipes or puppet manifest classes that should be used for
this node. For example, chef OpenStack cookbooks have roles such as mysql-master,keystone, glance,
nova-controller, nova-conductor, cinder-all.
comments
Any user-written notes.
disable
Set to ‘yes’ or ‘1’ to comment out this row.
SEE ALSO
nodels(1), chtab(8), tabdump(8), tabedit(8)
chain.5
NAME
chain - a table in the xCAT database.
SYNOPSIS
chain Attributes: node, currstate, currchain, chain, ondiscover, comments, disable
DESCRIPTION
Controls what operations are done (and it what order) when a node is discovered and deployed.
chain Attributes:
node
The node name or group name.
currstate
The current or next chain step to be executed on this node by xCAT-genesis. Set by xCAT during node
discovery or as a result of nodeset.
currchain
542
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
The chain steps still left to do for this node. This attribute will be automatically adjusted by xCAT while
xCAT-genesis is running on the node (either during node discovery or a special operation like firmware
update). During node discovery, this attribute is initialized from the chain attribute and updated as the
chain steps are executed.
chain
A comma-delimited chain of actions to be performed automatically when this node is discovered for the
first time. (xCAT and the DHCP server do not recognize the MAC address of the node when xCAT initializes the discovery process.) The last step in this process is to run the operations listed in the chain attribute,
one by one. Valid values: boot, runcmd=<cmd>, runimage=<URL>, shell, standby. For example, to have
the genesis kernel pause to the shell, use chain=shell.
ondiscover
This attribute is currently not used by xCAT. The “nodediscover” operation is always done during node
discovery.
comments
Any user-written notes.
disable
Set to ‘yes’ or ‘1’ to comment out this row.
SEE ALSO
nodels(1), chtab(8), tabdump(8), tabedit(8)
deps.5
NAME
deps - a table in the xCAT database.
SYNOPSIS
deps Attributes: node, nodedep, msdelay, cmd, comments, disable
DESCRIPTION
Describes dependencies some nodes have on others. This can be used, e.g., by rpower -d to power nodes on or off in
the correct order.
deps Attributes:
node
The node name or group name.
nodedep
Comma-separated list of nodes or node groups it is dependent on.
1.3. Admin Guide
543
xCAT3 Documentation, Release 2.12
msdelay
How long to wait between operating on the dependent nodes and the primary nodes.
cmd
Comma-separated list of which operation this dependency applies to.
comments
Any user-written notes.
disable
Set to ‘yes’ or ‘1’ to comment out this row.
SEE ALSO
nodels(1), chtab(8), tabdump(8), tabedit(8)
discoverydata.5
NAME
discoverydata - a table in the xCAT database.
SYNOPSIS
discoverydata Attributes: uuid, node, method, discoverytime, arch, cpucount, cputype, memory, mtm, serial, nicdriver, nicipv4, nichwaddr, nicpci, nicloc, niconboard, nicfirm, switchname, switchaddr, switchdesc, switchport, otherdata, comments, disable
DESCRIPTION
Discovery data which sent from genesis.
discoverydata Attributes:
uuid
The uuid of the node which send out the discovery request.
node
The node name which assigned to the discovered node.
method
The method which handled the discovery request. The method could be one of: switch, blade, profile,
sequential.
discoverytime
The last time that xCAT received the discovery message.
arch
544
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
The architecture of the discovered node. e.g. x86_64.
cpucount
The number of cores multiply by threads core supported for the discovered node. e.g. 192.
cputype
The cpu type of the discovered node. e.g. Intel(R) Xeon(R) CPU E5-2690 0 @ 2.90GHz
memory
The memory size of the discovered node. e.g. 198460852
mtm
The machine type model of the discovered node. e.g. 786310X
serial
The serial number of the discovered node. e.g. 1052EFB
nicdriver
The driver of the nic.
eth0!be2net,eth1!be2net
The value should be comma separated <nic name!driver name>.
e.g.
nicipv4
The ipv4 address of the nic. The value should be comma separated <nic name!ipv4 address>. e.g.
eth0!10.0.0.212/8
nichwaddr
The hardware address of the nic. The should will be comma separated <nic name!hardware address>. e.g.
eth0!34:40:B5:BE:DB:B0,eth1!34:40:B5:BE:DB:B4
nicpci
The pic device of the nic. The value should be comma separated <nic name!pci device>.
eth0!0000:0c:00.0,eth1!0000:0c:00.1
e.g.
The location of the nic. The value should be comma separated <nic name!nic location>.
eth0!Onboard Ethernet 1,eth1!Onboard Ethernet 2
e.g.
nicloc
niconboard
The onboard info of the nic. The value should be comma separated <nic name!onboard info>. e.g.
eth0!1,eth1!2
nicfirm
The firmware description of the nic. The value should be comma separated <nic name!fimware description>. e.g. eth0!ServerEngines BE3 Controller,eth1!ServerEngines BE3 Controller
switchname
The switch name which the nic connected to. The value should be comma separated <nic name!switch
name>. e.g. eth0!c909f06sw01
switchaddr
The address of the switch which the nic connected to. The value should be comma separated <nic
name!switch address>. e.g. eth0!192.168.70.120
switchdesc
1.3. Admin Guide
545
xCAT3 Documentation, Release 2.12
The description of the switch which the nic connected to. The value should be comma separated <nic
name!switch description>. e.g. eth0!IBM Flex System Fabric EN4093 10Gb Scalable Switch, flash
image: version 7.2.6, boot image: version 7.2.6
switchport
The port of the switch that the nic connected to. The value should be comma separated <nic name!switch
port>. e.g. eth0!INTA2
otherdata
The left data which is not parsed to specific attributes (The complete message comes from genesis)
comments
Any user-written notes.
disable
Set to ‘yes’ or ‘1’ to comment out this row.
SEE ALSO
nodels(1), chtab(8), tabdump(8), tabedit(8)
domain.5
NAME
domain - a table in the xCAT database.
SYNOPSIS
domain Attributes: node, ou, authdomain, adminuser, adminpassword, type, comments, disable
DESCRIPTION
Mapping of nodes to domain attributes
domain Attributes:
node
The node or group the entry applies to
ou
For an LDAP described machine account (i.e. Active Directory), the organizational unit to place the
system. If not set, defaults to cn=Computers,dc=your,dc=domain
authdomain
If a node should participate in an AD domain or Kerberos realm distinct from domain indicated in site,
this field can be used to specify that
adminuser
546
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
Allow a node specific indication of Administrative user. Most will want to just use passwd table to indicate
this once rather than by node.
adminpassword
Allow a node specific indication of Administrative user password for the domain. Most will want to
ignore this in favor of passwd table.
type
Type, if any, of authentication domain to manipulate. The only recognized value at the moment is activedirectory.
comments
Any user-written notes.
disable
Set to ‘yes’ or ‘1’ to comment out this row.
SEE ALSO
nodels(1), chtab(8), tabdump(8), tabedit(8)
eventlog.5
NAME
eventlog - a table in the xCAT database.
SYNOPSIS
eventlog Attributes: recid, eventtime, eventtype, monitor, monnode, node, application, component, id, severity, message, rawdata, comments, disable
DESCRIPTION
Stores the events occurred.
eventlog Attributes:
recid
The record id.
eventtime
The timestamp for the event.
eventtype
The type of the event.
monitor
1.3. Admin Guide
547
xCAT3 Documentation, Release 2.12
The name of the monitor that monitors this event.
monnode
The node that monitors this event.
node
The node where the event occurred.
application
The application that reports the event.
component
The component where the event occurred.
id
The location or the resource name where the event occurred.
severity
The severity of the event. Valid values are: informational, warning, critical.
message
The full description of the event.
rawdata
The data that associated with the event.
comments
Any user-provided notes.
disable
Do not use. tabprune will not work if set to yes or 1
SEE ALSO
nodels(1), chtab(8), tabdump(8), tabedit(8)
firmware.5
NAME
firmware - a table in the xCAT database.
SYNOPSIS
firmware Attributes: node, cfgfile, comments, disable
DESCRIPTION
Maps node to firmware values to be used for setup at node discovery or later
548
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
firmware Attributes:
node
The node id.
cfgfile
The file to use.
comments
Any user-written notes.
disable
Set to ‘yes’ or ‘1’ to comment out this row.
SEE ALSO
nodels(1), chtab(8), tabdump(8), tabedit(8)
hosts.5
NAME
hosts - a table in the xCAT database.
SYNOPSIS
hosts Attributes: node, ip, hostnames, otherinterfaces, comments, disable
DESCRIPTION
IP addresses and hostnames of nodes. This info is optional and is only used to populate /etc/hosts and DNS via
makehosts and makedns. Using regular expressions in this table can be a quick way to populate /etc/hosts.
hosts Attributes:
node
The node name or group name.
ip
The IP address of the node. This is only used in makehosts. The rest of xCAT uses system name resolution
to resolve node names to IP addresses.
hostnames
Hostname aliases added to /etc/hosts for this node. Comma or blank separated list.
otherinterfaces
Other IP addresses to add for this node. Format: -<ext>:<ip>,<intfhostname>:<ip>,...
1.3. Admin Guide
549
xCAT3 Documentation, Release 2.12
comments
Any user-written notes.
disable
Set to ‘yes’ or ‘1’ to comment out this row.
SEE ALSO
nodels(1), chtab(8), tabdump(8), tabedit(8)
hwinv.5
NAME
hwinv - a table in the xCAT database.
SYNOPSIS
hwinv Attributes: node, cputype, cpucount, memory, disksize, comments, disable
DESCRIPTION
The hareware inventory for the node.
hwinv Attributes:
node
The node name or group name.
cputype
The cpu model name for the node.
cpucount
The number of cpus for the node.
memory
The size of the memory for the node in MB.
disksize
The size of the disks for the node in GB.
comments
Any user-provided notes.
disable
Set to ‘yes’ or ‘1’ to comment out this row.
550
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
SEE ALSO
nodels(1), chtab(8), tabdump(8), tabedit(8)
hypervisor.5
NAME
hypervisor - a table in the xCAT database.
SYNOPSIS
hypervisor Attributes: node, type, mgr, interface, netmap, defaultnet, cluster, datacenter, preferdirect, comments,
disable
DESCRIPTION
Hypervisor parameters
hypervisor Attributes:
node
The node or static group name
type
The plugin associated with hypervisor specific commands such as revacuate
mgr
The virtualization specific manager of this hypervisor when applicable
interface
The definition of interfaces for the hypervisor.
The format is
name:interfacename:bootprotocol:IP:netmask:gateway] that split with | for each interface
[network-
netmap
Optional mapping of useful names to relevant physical ports.
For example,
10ge=vmnic_16.0&vmnic_16.1,ge=vmnic1 would be requesting two virtual switches to be created, one called 10ge with vmnic_16.0 and vmnic_16.1 bonded, and another simply connected to vmnic1.
Use of this allows abstracting guests from network differences amongst hypervisors
defaultnet
Optionally specify a default network entity for guests to join to if they do not specify.
cluster
Specify to the underlying virtualization infrastructure a cluster membership for the hypervisor.
datacenter
Optionally specify a datacenter for the hypervisor to exist in (only applicable to VMWare)
1.3. Admin Guide
551
xCAT3 Documentation, Release 2.12
preferdirect
If a mgr is declared for a hypervisor, xCAT will default to using the mgr for all operations. If this is field
is set to yes or 1, xCAT will prefer to directly communicate with the hypervisor if possible
comments
disable
SEE ALSO
nodels(1), chtab(8), tabdump(8), tabedit(8)
ipmi.5
NAME
ipmi - a table in the xCAT database.
SYNOPSIS
ipmi Attributes: node, bmc, bmcport, taggedvlan, bmcid, username, password, comments, disable
DESCRIPTION
Settings for nodes that are controlled by an on-board BMC via IPMI.
ipmi Attributes:
node
The node name or group name.
bmc
The hostname of the BMC adapter.
bmcport
In systems with selectable shared/dedicated ethernet ports, this parameter can be used to specify the
preferred port. 0 means use the shared port, 1 means dedicated, blank is to not assign.
The following special cases exist for IBM System x servers:
For x3755 M3 systems, 0 means use the dedicated port, 1 means
shared, blank is to not assign.
For certain systems which have a mezzaine or ML2 adapter, there is a second
value to include:
For x3750 M4 (Model 8722):
552
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
0 2
1st 1Gbps interface for LOM
0 0
1st 10Gbps interface for LOM
0 3
2nd 1Gbps interface for LOM
0 1
2nd 10Gbps interface for LOM
For
M4:
x3750 M4 (Model 8752), x3850/3950 X6, dx360 M4, x3550 M4, and x3650
˓→
0
Shared (1st onboard interface)
1
Dedicated
2 0
First interface on ML2 or mezzanine adapter
2 1
Second interface on ML2 or mezzanine adapter
2 2
Third interface on ML2 or mezzanine adapter
2 3
Fourth interface on ML2 or mezzanine adapter
taggedvlan
bmcsetup script will configure the network interface of the BMC to be tagged to the VLAN specified.
bmcid
Unique identified data used by discovery processes to distinguish known BMCs from unrecognized BMCs
username
The BMC userid. If not specified, the key=ipmi row in the passwd table is used as the default.
password
The BMC password. If not specified, the key=ipmi row in the passwd table is used as the default.
comments
Any user-written notes.
disable
Set to ‘yes’ or ‘1’ to comment out this row.
SEE ALSO
nodels(1), chtab(8), tabdump(8), tabedit(8)
iscsi.5
NAME
iscsi - a table in the xCAT database.
1.3. Admin Guide
553
xCAT3 Documentation, Release 2.12
SYNOPSIS
iscsi Attributes: node, server, target, lun, iname, file, userid, passwd, kernel, kcmdline, initrd, comments, disable
DESCRIPTION
Contains settings that control how to boot a node from an iSCSI target
iscsi Attributes:
node
The node name or group name.
server
The server containing the iscsi boot device for this node.
target
The iscsi disk used for the boot device for this node. Filled in by xCAT.
lun
LUN of boot device. Per RFC-4173, this is presumed to be 0 if unset. tgtd often requires this to be 1
iname
Initiator name. Currently unused.
file
The path on the server of the OS image the node should boot from.
userid
The userid of the iscsi server containing the boot device for this node.
passwd
The password for the iscsi server containing the boot device for this node.
kernel
The path of the linux kernel to boot from.
kcmdline
The kernel command line to use with iSCSI for this node.
initrd
The initial ramdisk to use when network booting this node.
comments
Any user-written notes.
disable
Set to ‘yes’ or ‘1’ to comment out this row.
554
Chapter 1. Table of Contents
xCAT3 Documentation, Release 2.12
SEE ALSO
nodels(1), chtab(8), tabdump(8), tabedit(8)
kit.5
NAME
kit - a table in the xCAT database.
SYNOPSIS
kit Attributes: kitname, basename, description, version, release, ostype, isinternal, kitdeployparams, kitdir, comments, disable
DESCRIPTION
This table stores all kits added to the xCAT cluster.
kit Attributes:
kitname
The unique generated kit name, when kit is added to the cluster.
basename
The kit base name
description
The Kit description.
version
The kit version
release
The kit release
ostype
The kit OS type. Linux or AIX.
isinternal
A flag to indicated if the K