Backup/Restore Solution for Lotus Domino

Best Practices
Prepared by Messaging and
Collaborative Applications
Business Unit Compaq Computer
Corporation
Contents
Introduction and Overview ........... 3
Backup/Restore Solution for Lotus
Domino ....................................... 4
EVM/SAN Backup &
Restore Solution ...................... 5
Hardware Configuration............ 8
Software Configuration............. 8
Performance Logging Tools...... 9
System Sizing .......................... 9
Configuration
Recommendations ..................10
Third-Party Backup Software.......14
SAN Setup .................................15
Storage Enclosures
(RA8000) ................................15
CLI Procedure.........................15
Configuring Storage Sets ........16
Connections............................17
Fibre Channel Switch ..............18
Fibre Channel Tape
Controller................................18
TL895 Tape Library.................19
Server Setup...........................19
Appendix A: Sample scripts ........20
Script process .........................21
EVM script examples...............21
Appendix C: Overview of SANworks
Enterprise Volume Manager........25
EVM Graphical User
Interface .................................25
EVM job interaction with
external applications ...............26
EVM Storage Unit
Replication Features ...............26
Appendix D: Product Overview –
Lotus Domino R5........................27
Appendix E: Test disclosure........28
Workload methodology............29
Test Procedure .......................30
Test Results............................30
Appendix F: Standard Terms .....30
Appendix G: Additional References
..................................................33
Backup/Restore Solution for
Lotus Domino
Abstract: This document addresses the needs of Lotus
Domino Administrators, faced with critically short database
backup windows, ever increasing database and mail file
backup responsibilities, and challenges with restoring
databases with a minimal business disruption.
The SAN technology presented here is managed by the
Compaq SANworks Enterprise Volume Manager (EVM).
EVM presents a clone of the Domino data volume to a
backup server on the SAN. All backup specific storage is
also presented to the environment on the SAN, including
disk arrays and tape drives in a library. To maximize
throughput, tape drives are “multiplexed” via third party
backup applications. This takes the backup offline to both
the Domino server and the public LAN.
This paper provides the hardware configurations for all the
machines used in testing, and also documents the specific
software versions. Performance data using Lotus Notes
Bench load simulator is included. In addition, this paper
describes how to configure Lotus Domino R5 and deploy
SANworks Enterprise Volume Manager as part of the
overall data protection plan.
Performance tuning is not discussed, nor are
recommendations made, aside from describing the particular
test environment and incidental setup practices. The results
documented here are a response to customer inquiries about
appropriate expectation levels for backup strategies to
protect corporate infrastructure data.
The performance points presented are necessarily a
reflection of the actual hardware used in testing. Relative
performance comparisons may be made regardless of the
hardware used in a given example.
The results documented herein are non-certified and are
accurate to the best of our knowledge at the time of
publication.
Backup/Restore Solution for Lotus Domino Server R5
Notice
The information in this publication is proprietary to Compaq and is protected by the terms of an
end-user license agreement. The information is subject to change without notice and is provided
“AS IS” WITHOUT WARRANTY OF ANY KIND. THE ENTIRE RISK ARISING OUT OF
THE USE OF THIS INFORMATION REMAINS WITH RECIPIENT. IN NO EVENT SHALL
COMPAQ BE LIABLE FOR ANY DIRECT, CONSEQUENTIAL, INCIDENTAL, SPECIAL,
PUNITIVE OR OTHER DAMAGES WHATSOEVER (INCLUDING WITHOUT
LIMITATION, DAMAGES FOR LOSS OF BUSINESS PROFITS, BUSINESS
INTERRUPTION OR LOSS OF BUSINESS INFORMATION), EVEN IF COMPAQ HAS
BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
The limited warranties for Compaq products are exclusively set forth in the documentation
accompanying such products. Nothing herein should be construed as constituting a further or
additional warranty.
This publication does not constitute an endorsement of the product or products that were tested.
The configuration or configurations tested or described may or may not be the only available
solution. This test is not a determination of product quality or correctness, nor does it ensure
compliance with any federal state or local requirements.
The software described in this guide is furnished under a license agreement or nondisclosure
agreement. The software may be used or copied only in accordance with the terms of the
agreement.
Compaq, Deskpro, Fastart, Compaq Insight Manager, Systempro, Systempro/LT, ProLiant,
ROMPaq, QVision, SmartStart, NetFlex, QuickFind, PaqFax, ProSignia, registered United States
Patent and Trademark Office.
Netelligent, Systempro/XL, SoftPaq, QuickBlank, QuickLock are trademarks and/or service marks
of Compaq Computer Corporation.
Microsoft, Windows, Windows NT, are trademarks and/or registered trademarks of Microsoft
Corporation.
Other product names mentioned herein may be trademarks and/or registered trademarks of their
respective companies.
©2000 Compaq Computer Corporation. All rights reserved. Printed in the U.S.A.
Lotus Domino Backup/Restore Solution prepared by Messaging and Collaborative
Applications/Industry Standard Server Division.
2
Backup/Restore Solution for Lotus Domino Server R5
Introduction and Overview
This informal test document presents the relative impact of a full backup to tape on a dedicated
Domino R5 server from a dedicated backup server configured in a fibre channel SAN to
common shared disk storage. Compaq SANworks Enterprise Volume Manager (EVM) is used
to set up and control SAN functionality, namely the creation of a copy of the Domino data
volume called a Business Continuance Volume (BCV) using cloning technology. BCV clones
can be backed up to tape over the SAN and offline to the Domino server, thereby eliminating
either hours of Domino server downtime or the need for a dedicated “replication server” to
facilitate an offline, daily backup window. If individual Domino databases become corrupt or
otherwise unusable, they can be restored promptly as desired, from the broken off clone
volume. Also, the complete Domino data volume can be restored in a matter of minutes rather
than hours, through the redeployment of the clone as the active Domino data volume.
The Domino server is shutdown for less than three minutes per backup cycle, as the
SANworks EVM clone is split off. Through employment of its graphical user interface and
script creation utility, SANworks EVM can control all these processes:
• Creating the clone
• Shutting down the Domino server
• Splitting off the clone
• Mounting the clone as a local volume on the backup server
• Restarting the Domino server
• Initiating the backup to tape or tape library
The ability to create a BCV clone on the backup server expands the available backup window
to virtually 24 hours per day. This represents a significant departure from the most common
backup scenario in the industry — that of an exclusively off-hours backup strategy,.
Compaq software engineers performed test design, engineering, and characterization of the
Compaq ProLiant Server running Lotus Domino Server. They selected backup software to
provide a realistic scenario to help Lotus Domino ProLiant customers evaluate and plan their
backup strategy under a variety of conditions. Unless otherwise noted, the default hardware
and software configuration described initially remains common throughout this testing and
characterization. Parameters defining the configuration of Windows NT and the backup
software are as set by default or modified per vendor recommendations and practical
experience of the Messaging and Collaborative Applications engineering team. An attempt has
been made to fully comply with the goals at hand.
Important factors involving the set up and configuration of a system to obtain the best Domino
performance will not be presented in this document. You can obtain the information to
configure a system for all of your unique needs from the documents specified in AppendixG.
Conclusion and Recommendation
Administrators can use this Backup/Restore Solution for Lotus Domino to enhance their
backup strategy and reduce Domino downtime for both backup and restore operations. This
solution gives administrators the opportunity to use Domino servers more efficiently in their
enterprise.
3
Backup/Restore Solution for Lotus Domino Server R5
This paper contains recommendations for configuring the Domino server disk subsystem and
procedures for using BCV clones to backup or restore Domino databases. It also contains
information regarding configuration of the SAN and installation of SANworks Enterprise
Volume Manager software. This includes comparative test results describing baseline testing
and performance characteristics of Computer Associates ARCserve 2000, VERITAS Backup
Exec and Legato NetWorker as they pertain to this EVM backup/restore solution.
IMPORTANT: The settings used in this document are not intended to be used as
guidelines for any customer specific installation. They are only incidental to the
compilation of experimental results. The included guidelines attempt to address specific
circumstances with generalized information, but should not override the realities of your
own production environment. Consult with your Compaq sales or support
representatives for help in setting up the best possible configuration to meet your needs.
Backup/Restore Solution for Lotus
Domino
Backup and recovery of Domino databases is a critical process to ensure availability of
Domino services. Through the use of BCV clones, the Backup/Restore Solution for Lotus
Domino helps ensure that critical Domino data is protected and readily available for recovery.
From our extensive experience in storage technology and Lotus Domino, Compaq has
developed a set of best practices, which include recommendations and procedures for
implementing the Backup/Restore Solution for Lotus Domino. These best practices address
the primary component of the solution — the SANworks Enterprise Volume Manager — and
include customizable scripts for Lotus Domino R5, which can be used to automate backup and
restore operations.
4
Backup/Restore Solution for Lotus Domino Server R5
EVM/SAN Backup & Restore Solution
Solution Components
Bill Of Materials
The table below provides a Bill of Materials that includes some of the products used in the
Lotus Domino Backup/Restore Solution. The table is an example only. You can choose the
products best suited to your needs, including products not listed here.
5
Backup/Restore Solution for Lotus Domino Server R5
6
Example: Components of Lotus Domino Backup/Restore Solution
Item
Item Descriptions
Base SKU
1
EVM Software for Window NT-Intel
160094-B21
2
ACS 8.5F Controller Software
128697-B21
3
Operating System Platform Software kit for NT (SWCC)
380551-001
4
RA8000 Rack with two HSG80 Controllers
380670-B21
5
Hard Disk Drives, 9 GB, 10K UWSE
380588-B21
6*
ProLiant NT Backup Server (PL1850 – typical)
153554-001
7
Fibre Channel HBA (switched fabric), KGPSA-BC
380574-001
8
Fibre Channel Storage Switch, 8 port
380591-B21
9
FC Optical Short Wave GBIC
158222-B21
10
Fibre Channel Optical Cable
234457-B22
11
Modular Data Router
163082-B21
12
TL895 Tape Library
349350-B24
13
Computer Associates ARCserve2000
3 party P/N
Legato NetWorker
3 party P/N
Veritas Backup Exec
3 party P/N
rd
rd
rd
Table Explanation
Item 1: EVM Software – EVM software is necessary on every client and server on the SAN
that needs to communicate or coordinate communications with hosts or storage devices. One
instance of EVM is designated as the EVM server. In our testing the application server was
used as the EVM server as well, with the backup server designated as client for a total of two
instances of EVM loaded.
Driver Version: EVM
Item 2: Controller Firmware – One instance of ACS is necessary for each HSG80
controller. ACS firmware is in the form of a PCMCIA card inserted into the front panel of the
controller. ACS 8.5F supports cloning and ACS 8.5S supports cloning and snapshots as well.
Driver Version: Hsxdisk.sys, v.5.4.18.0 or later for Windows2000
ACS8.5F: HSG80 zg92810079 Software V8.5s-0, Hardware e09
Item 3: OS Platform Kit CDROM, “ StorageWorks Solutions Software RA8000/MA8000
Fibre Channel Solution Software NT, Windows2000, Proliant, x86” includes drivers and
installation for the specific operating system and host involved.
Item 4: Disk Array – For our testing we used two StorageWorks RAID Array 8000’s. The
first unit in the SCSI chained configuration housed the two HSG80 controllers, which were
configured for Transparent Failover mode. Each of six ports on the rear of one RA8000 was
connected to the same numbered port connector on the second RA8000.
Item 5: Disks – The choice of disk type does not affect the functionality of this solution.
Two factors to consider for disk type are rpm and the number of spindles. For applications
like moil mail, the access pattern is random. Using 10,000-RPM disks can greatly improve
performance in this type of workload. Also, it is desirable to use as many spindles as possible
to increase parallel access to the volume as opposed to fewer disks. Presently, the largest
Backup/Restore Solution for Lotus Domino Server R5
number of total sets that EVM can address is 30. In our test we created 15, two disk mirrors,
and then striped them together to form one logical unit. This left room for the creation of 15
mirror sets of one disk, which would later comprise the clone.
Item 6: Backup Server – The backup server must be powerful enough to maintain
performance levels during a backup/restore operation. For our testing we used a single
processor 500 MHz PL1850 with 256 MB RAM. No server sizing information is presented in
this document.
Item 7: Fibre Channel Host Bus Adapter (HBA) – Use the KGPSA-BC for all switched
fabric solutions and loop solutions, which contain the HSG80 controller on the SAN.
Driver Version: KGPSA-xx PCI – Fibre Channel HBA, v.5.4.41.0 5/16/2000 Windows
2000
Item 8: Fibre Channel Interconnect – SAN interconnect switches allow for significantly
better performance, whereas loops (hubs) present significantly limited performance and
connectivity options. However, since they are newer, switches currently have limited
supported configurations where loops are supported by a variety of components. Storage
arrays and tape can reside on the same switched fabric, making a simpler, more extensible
installation possible. Disk and tape can not exist on the same loop; dual loops are required in
this case.
Item 9: FC Optical Short Wave GBIC – GigaBitInterConnect (GBIC) – GBICs don’t come
with the hubs or switches and must be ordered separately. SAN switches require the GBIC,
380561-B21.
Item 10: Fibre Channel Optical Cable – order the length needed. The basic part number is
234457 and the suffix B2x denotes the length in meters. Suffix B21 is 2, B22 is 5, B23 is 15,
B24 is 30, and B25 denotes a 50 meter cable. Each controller comes with one five meter
cable. All other cables will have to be ordered.
Item 11: Modular Data Router -- Compaq StorageWorks Modular Data Router (Fibre
Channel Tape Controller). SCSI attached tape libraries require a special Host Bus Adapters
(do not use standard server HBA adapters). Fibre Channel-attached tape libraries (being SCSI
devices) require a special Fibre Channel to SCSI bridge (FCTC). Requires VHDCI Cables for
SCSI connection. #341176.B21 for 6ft cable or #341177-B21 for 12ft cable.
Item 12: TL895 Tape Library – Compaq StorageWorks TL895 Tape Library.
(7) Compaq 35/70 GB DLT tape drives
Driver Version: Digital TL810 DLT Library, atlmc.sys v.5.00.2159.1 11/14/1999 Microsoft
DLT 35/70 v.5.0.2183.1 Microsoft 11/14/1999
rd
Item 13: 3 Party Backup Software –
Computer Associates ARCserve 2000 Advanced Server Edition v.7.0.Build.1018
(recommended)
VERITAS Backup Exec v.8.0 Rev.3314
Legato NetWorker for Microsoft Windows 2000, v.5.7.0.0.Build.11.2
7
Backup/Restore Solution for Lotus Domino Server R5
Hardware Configuration
Overview
Compaq ProLiant 1850 Server
• Application Server
• Backup Server
CPU
x86 Family 6 Model 7 Stepping 3 Genuine Intel ~ 500 MHz
RAM
1024 MB: Application Server
256 MB: Backup Server
Storage Sub systems
Symbios Logic 875XS|D,
2280X PCI SCSI Adapter
Compaq 9.1 GB 10,000 RPM Ultra2 SCSI (internal hard drives)
Compaq KGPSA-BC, Fibre
Channel Host Bus Adapter
Compaq StorageWorks SAN Switch 8
(2) Compaq StorageWorks
HSG80 Array Controller ACS
v.8.5F
(2) Compaq StorageWorks RAID Array 8000
Compaq StorageWorks
Modular Data Router (Fibre
Channel Tape Controller)
Fibre Channel to SCSI tape controller with (2) 2-port SCSI modules
Compaq StorageWorks
TL895 Tape Library
(7) Compaq 35/70 GB DLT tape drives
(45) Compaq DS-RZ1DD-VW 9.1 GB 10,000 RPM UltraSCSI 16-bit
Software Configuration
Overview
Operating Systems:
Microsoft Windows 2000 Server, version 5.0.2195 Build 2195
Software Placement Scheme
C:\WINNT
Microsoft Windows 2000 Server
D:\pagefile.sys
1x RAM
C:\Domino\Notes
Notes 5.03
E:\Arcserve
Computer Associates ARCserve 2000 Advanced Server Edition
v.7.0.Build.1018
8
Backup/Restore Solution for Lotus Domino Server R5
E:\Computer Associates
VERITAS Backup Exec v.8.0 Rev.3314
E:\Legato
Legato NetWorker for MS Windows 2000, v.5.7.0.0.Build.11.2
E:\Lotus Notes\data\mail or
multiple streaming to seven 35/70 DLT drives concurrently or seven tape
drive stripe set
Q:\Lotus Notes\data\mail
q:\ equals clone on backup server
Performance Logging Tools
Windows 2000 Performance Monitor
Performance Logs of the following:
% CPU Time
% Page File Utilization
% Disk Time
Disk Queue Length
% RAM Utilization
Lotus NotesBench 5.03
Lotus NotesBench v.5.03, the industry standard bench marking tool created by the Lotus
Development Corporation, was installed and run on three client machines in order to drive a
2000 user default load on the Domino application server.
System Sizing
Lotus Domino performance depends on the number of users who are actively working on the
system. Performance will also be affected by any of the following variables:
• Amount and size of mail sent
• Amount of mail sent local verses remote
• Number of accesses to the Domino databases
• Site or other type of domain connections
Sizing of the RAID subsystem was determined on the basis of what was available at the time
of testing. No implication of adequacy regarding the task of storage for the number of mail
users represented is implied. It is reasonable to believe that the average Notes mail user will
have a mail file of between 50 MB and 300 MB in size.
Please consult the following Compaq Active Answers Sizing Guides for assistance in sizing
your individual requirements:
Domino R5 Sizer:
http://vcmproapp02.compaq.com/ActiveAnswers/Global/en/solutions.1004/offline.9803/defau
lt.asp
9
Backup/Restore Solution for Lotus Domino Server R5
Sizer Tools Homepage:
http://vcmproapp02.compaq.com/ActiveAnswers/Global/en/tools.6017/default.asp
Enterprise Backup System Sizer:
http://www.compaq.com/products/storageworks/ebs/ebstoolsbackupsizing.html
SAN Sizer:
Contact your Compaq Storage Specialist to utilize the Compaq SAN sizer Tool V2.0 for
configuring your SAN.
Configuration Recommendations
The Lotus Domino Backup/Restore Solution has been validated in performance
characterization tests using specific test configurations. For those tested configurations, there
are a number of recommendations you should follow to ensure the best results from this
solution.
Using a BCV clone as part of a total backup and recovery solution requires planning for
additional capacity when you design your disk subsystem. For a Domino server, the disk
subsystem for the Domino databases is commonly designed using RAID fault-tolerant
settings. Best practices recommends RAID 0 + 1, striped mirror volume sets for performance
and reliability.
The SANworks Enterprise Volume Manager can be used to create BCV clones of hardwarebased, mirrored (RAID 1) or stripe-mirrored (RAID 0+1) volumes.
For best results, the Domino data directory and mail files should be located on a separate
volume from the program executable files. The data directory must be on its own separate
dedicated spindles, using RAID 1 or RAID 0 +1. The pagefile should also be separate from
the data directory to facilitate lowest impact on server response time during clone
resynchronization to the working mirror set, particularly in systems outfitted with marginal
amounts of RAM. In addition, when using BCV clones, you must have free space on the
RAID array equal to the size of the volume containing the Domino databases.
Creating a BCV Clone
The BCV clone should be created using EVM while Domino is running. Testing indicates that
the BCV clone will be built – actually synchronized – at about 120 GB/hr. This practice will
ensure that the clone can be split off for the backup in a fully synchronized state with no
waiting period required.
Once the clone is established, you are left with a full read-write copy of your data. After the
clone set has been backed up to tape, you have the choice of either:
1. Operating with one clone by rejoining the clone set members back to the source
volume, eliminating “hot standby” capability.
-OR2. Creating a second clone by selecting another set of drives to renormalize the mirror
sets back to 3 members and leaving the “hot standby” volume available and offline.
At the onset of each backup routine, you would alternate clone 1 (offline “hot
10
Backup/Restore Solution for Lotus Domino Server R5
standby”) and clone 2 (online 3rd mirror set). This practice provides the quickest
option for data recoverability in the event of catastrophe
The tested configuration for this solution utilized option 1 above. However, EVM
functionality exists to operate under a two clone condition which provides the ultimate level of
data availability. Dependent upon individual business needs, you have the option of selecting
the number of clones to operate with. See Appendix B for a sample procedure detailing how
to operate with two clones.
Clone Split Process
To bring the Domino data directory into a consistent state, the Domino server must be
shutdown. This Domino shutdown period is required for approximately 2.5 minutes while the
clone is split off. After this interval the Domino service should be immediately restarted. This
server shutdown and restart is facilitated by EVM as follows:
1. The execution of a batch file command, called by EVM, instructing the Domino server
service to stop.
2. The clone is split off immediately.
3. The clone is mounted on the backup server as a local volume, with a unique drive
letter and label (if desired) of the operator’s choice.
4. The command is issued to the application server start the Domino service. The actual
time a Domino server will be offline is unique to particular server configurations.
These steps can be easily automated using tested, documented, and customizable scripts
provided as part of the Backup/Restore Solution for Lotus Domino. Sample scripts are
included in this paper.
The user can modify this script for a particular application. See Appendix A.
Backing up a clone to tape
A key feature of SANworks Enterprise Volume Manager is the ability to automate backup of
the BCV clone to tape with customized scripts. Specific scripts are available for Computer
Associates ArcServe 2000. Specific commands to execute these scripts are provided in an
EVM white paper available at the following URL
http://www.compaq.com/
Recovering from a clone
Administrators can use BCV clones to recover either the full Domino database directories or
individual Domino databases or files. In addition, the BCV clone can later be used to recreate
a parent disk or become the parent disk.
While the decision to implement Domino recovery requires human intervention, the actual
recovery process can be scripted. Tested, documented, and customizable scripts enable
scheduled BATCH/COM processes to run the necessary commands and utilities within
SANworks Enterprise Volume Manager to restore the database files from a BCV clone. Now
let’s examine the steps to recover the full Domino data directory from a BCV clone in more
detail.
11
Backup/Restore Solution for Lotus Domino Server R5
Step 1: Check the integrity of the BCV clone
It is critical to integrity check the BCV clone prior to using it for a restore. If the BCV clone
fails the integrity check, than the restore must use the last-known-good tape backup.
Windows 2000 check disk utility will report any inconsistencies in the physical table structure
of the database directories.
Step 2: Stop the Domino Server service
After the recovery process has verified the BCV clone, stop the Domino Server service on the
application server if it is still running.
Note: Any users connected to the Domino Server when the server service is stopped will be
required to reconnect after the Domino Server service is restarted.
Step 3: Map the BCV clone to the parent disk drive letter
The BCV clone is currently mounted to a drive letter that is not correct for the database path.
This step involves re-mapping both the BCV clone and the parent disk from their respective
drive letters. The BCV clone can now be mapped to the drive letter that was previously used
by its parent. Verify that the drive letter is the correct path for the database files.
Follow the procedure outlined below. This procedure is to be executed at the HSG80 console.
1. Complete the following steps on the application server to power down both servers
and ensure clean initialization of the system:
a. Navigate in Windows 2000
i. Open Computer Management
ii. Open Device Manager
iii. Right mouse click on machine name on top right hand window for
drop down menu
iv. Select Scan for New Hardware
v. Check Disk Manager and Windows Explorer for presence of new
data volume.
2. The CLI commands listed below should be issued to the HSG controller through a
serial port from a dedicated terminal or HyperTerminal
NOTE: For this example D1= the original unit, mapped to the Domino server and D2= clone,
mapped to the backup server.
->show D1
• Note which machine has access to this unit
• Note the Unit Number, stripe set and disks that compose this unit. It is only
necessary to make a note to the stripe set level.
12
Backup/Restore Solution for Lotus Domino Server R5
->set D1 disable=all
• This disables access rights to the server users, allowing deletion of the unit.
->del D1
->sho D2
• Note which machine has access to this unit
• Note the unit number, stripe set and disks that compose this unit. It is only necessary
to make a note to the stripe set level.
• This stripe set will be in the naming format of S!S!nnn
->set D2 disable=all
->del D2
A command of show stripe will display two stripe sets not associated with any units.
->add unit D1 Snnn (stripe set from D2) dis=all ena=[sytstem ID from original unit D1]
• Stripe Set from D2 = The units that you noted from issuing command "show D2".
• Enable access for system ID from original unit D1 = The server that has access, as
noted from issuing the command "show D1".
Note: If a “disk is in use” message appears when attempting to unmap a drive letter, the server
has a process running that needs to be disabled. For example, a common situation is that the
server may have Performance Monitor disk counters enabled (diskperf –y). Using the
command line prompt, type diskperf –n. The server must then be rebooted.
Step 4: Start the Domino Server Service
Start the Domino Server service from either Computer Management – Services or the
command prompt.
Recovering an individual database or file
A reliable procedure for recovering individual Domino databases, or files that have been either
deleted or corrupted, is the use of conventional tape restore. With this procedure, the files to
be restored from backup can be restored to their original location or to another location of your
choice and file system security can be maintained. Depending on the size of the file and/or the
speed of your LAN, an administrator may choose to restore either over the LAN or over the
SAN by direct connection to the active data volume from the backup server.
13
Backup/Restore Solution for Lotus Domino Server R5
The solution simplifies and speeds the process of bringing a replacement or additional server
online in the event of either a catastrophic failure of the primary Domino server or a desire to
increase server availability quickly. In this scenario, a production Domino server running
SANworks Enterprise Volume Manager can make BCV clones available to other servers on the
network. For example, a Domino server with a recent BCV clone of the data directory would
serve the BCV clone to another server. The recovery or additional server would then mount
the networked BCV clone as its data directory. Large and complex environments can be built
or duplicated quickly in this manner. By making the BCV clone available to additional
servers, you can immediately serve up an individual database or a complete data directory.
This alleviates the need either to restore from tape or to wait for timely replication of large
amounts of data.
Third-Party Backup Software
Legato NetWorker for Microsoft Windows 2000, v.5.7.0.0.Build.11.2
Legato NetWorker multiplexes read and write operations to a multi-drive tape library. While
this strategy is totally satisfactory for general enterprise backups, either over the WAN, LAN
or SAN, the ideal strategy for a multi-drive tape library management is one of tape striping to
create one large, striped, seven tape save set.
Computer Associates ARCserve 2000 Advanced Server Edition
v.7.0.Build.1018
RAIDirector
RAIDirector is an add-in Computer Associates utility that makes it possible to create tape
RAID or RAIT sets of similar stand-alone tape drives or multi-drive components of a tape
library. RAIDirector installs as a software SCSI subsystem and creates a virtual tape drive that
is visible and useable to Windows, thereby making it visible to other backup applications as
well. This tape striping strategy is highly recommended for the administrative needs this test
environment addresses. It eliminates the need to create multiple save set backup jobs that
require specific administrative attention to the division and placement of all the Domino
databases on one or more data directories. Without this tape striping strategy, multiple save
sets would be needed to facilitate fast, logical backup and restore strategies.
Best practices recommends the use of Computer Associates ARCserve 2000 as the backup
application for Lotus Domino Backup/Restore Solution specifically because of its tape striping
capabilities.
Veritas Backup Exec v.8.0 Rev.3314
VERITAS Backup Exec is an efficient backup application that executes tape mulitiplexing
advantages quickly (79.212 GB/hr) at the highest through put level recorded in our testing. We
recommend VERITAS Backup Exec as an outstanding backup product for use with a SAN
environment as well as in LAN/WAN environments.
However, because this version of VERITAS Backup Exec does not provide tape drive RAID
capabilities to multi-drive tape libraries, for this solution we prefer to utilize the inherent
expedience afforded by tape RAID capabilities.
14
Backup/Restore Solution for Lotus Domino Server R5
SAN Setup
Storage Enclosures (RA8000)
Transparent Fail Over
See HSG80 Configuration Guide for Hardware for limitations of the controller.
1. Install cache memory, 128 MBs, not mirrored. Mirror 512 MB for best performance.
2. For (3) LUN mirror set installation, use (3) RA8000, (24)9.1 GB disks x 3.
3. Install (2) HSG80 controllers in first RA8000, (1) Maximum (3) cabinets, 72 drives
per HSG80 controller pair.
4. Set EMU addresses: EMU 0: id 1, EMU 2: id 2 , EMU 3: id 3
5. Connect EMUs with RJ45 connector cables. Inside jack (right) is outgoing , outside
jack (left) is incoming. See HSG80 Array Controller Maintenance & Service Guide
for instruction on physical connections.
6. Set EMU addresses: EMU 0: id 1, EMU 2: id 2 , EMU 3: id 3
Connect EMU’s with RJ45 connector cables. Inside jack (right) is outgoing,
outside jack (left) is incoming. See HSG80 Array Controller Maintenance &
Service Guide for instruction on physical connections.
7. Connect (2) HSG80 controllers via fibre cable to switch: Controller one, Port 1 to
switch. Controller 2, Port 2 to switch for Transparent Failover. See CLI Reference
Guide in RA8000/ESA12000 Kit for the command line interface commands for
establishment of Transparent Failover.
8. Extend SCSI bus with appropriate cables out lower rear of cabinet. Connect
RA8000’s channel-to -channel. Connect consecutive enclosures to each other in daisy
chain fashion via channel six using IN/OUT SCSI ports.
CLI Procedure
For creating maximum RAID0+1 LUN on RA8000 (Rev 0.1)
15 mirror set stripe across two cabinets with one pair of HSG80 controllers.
Initial Controller Configuration
Reset this
Reset other
Set this node_ID=5000-nnnn-nnnn-nnnn cs (where cs is checksum)
Set nofailover or set nomultibus_failover
Clear _errors cli
15
Backup/Restore Solution for Lotus Domino Server R5
Set failover copy=this (for transparent failover mode, multibus is set
multibus_failover)
Set this scsi_version=Scsi-2
Set this port_1_topology=fabric
Set this port_2_topology=fabric
Set other port_1_topology=fabric
Set other port_2_topology=fabric
Set this prompt=“HSG80-Top>”
Set other prompt=“HSG80->Bot”
Set this time=23-aug-2000:09:00:00
Set other time=23-aug-2000:09:00:00
Run FMU (run locally on this controller - select Y for battery replacement to reset
expiration)
Run FMU (run locally on other controller select Y for battery replacement to reset
expiration)
Run config (will detect all drives)
Configuring Storage Sets
Create Mirror Sets
Add mir m1 disk10000 disk20000
Add mir m2 disk30000 disk40000
Add mir m3 disk50000 disk60000
Add mir m4 disk10100 disk20100
Add mir m5 disk30100 disk40100
Add mir m6 disk50100 disk60100
Add mir m7 disk10200 disk20200
Add mir m8 disk30200 disk40200
Add mir m9 disk50200 disk60200
Add mir m10 disk10300 disk20300
Add mir m11 disk30300 disk40300
Add mir m12 disk50300 disk60300
16
Backup/Restore Solution for Lotus Domino Server R5
Add mir m13 disk10800 disk20800
Add mir m14 disk30800 disk40800
Add mir m15 disk50800 disk60800
Note: Numbers derived for disk row are 0-3(SCSI ID 0) for first cabinet, and 8-11(SCSI ID 2)
for second cabinet. See drive map in RA8000 configuration guide for cabinet port and channel
mapping.
Create Stripe Set
Add str s1 m1 m2 m3 m4 m5 m6 m7 m8 m9 m10 m11 m12 m13 m14 m15
Init Stripe Set
Init s1
Create Unit
Add uni d1 s1 dis=all
Set Maximum_Cached_Transfer_Size=256 (for best performance with Lotus Domino)
Connections
Install KGPSA driver on server
Restart server
Show connections
Sho conn
Rename connections to match OS and any unique identification methods (eg.
W2Kdom1,W2Kdom2) using “rename !newconXX”
Confirm connection OS type is correct. If necessary change with
“set W2Kdom1 Operating_system=WinNT”
NOTE: Bringing servers online one at a time facilitates the easy identification of connections
to the RA8000.
Add connections to appropriate units
Set d1 ena=W2Kdom1,W2Kdom2
(in transparent failover mode the LUN cannot move from port1 to port2 so only one
connection name matching the offset is required)
17
Backup/Restore Solution for Lotus Domino Server R5
To bring the KGPSA into an enabled state, one connection to any unit must be enabled for the
backup server. If no unit is enabled, Windows 2000 will disable the HBA and it will appear to
be non-functional as viewed by LPUTILNT.exe. However, when the clone is created this
anomaly will be resolved.
Notes
• No cloning or set changes can occur until normalization of the mirror sets is complete.
Normalization will not begin until the unit is created.
• A maximum of 15 mirror sets can be in any unit if EVM will be used for cloning. The
EVM procedure temporarily doubles the number of mirror sets and cannot exceed the
controller limit of 30.
• In order to improve cloning normalization time, an extra member can be added to the
mirror sets in advance using the “set m1 mem=3” command. The new member will draw
a disk from the spareset using the default mirror settings.
• A spareset can be used for replacement disks, and disks can be created by adding disks one
at a time using the “add spareset DiskX0X00” command
• Clones can occur using firmware 8.5F,8.5S,8.5P with 128 MB Cache-subject to
verification.
• Multi-bus mode requires use of host SecurePath software.
• Transparent failover mode only requires host SecurePath software if using more than one
HBA.
Fibre Channel Switch
Physical Connections
1. Install GBICS into switch ports.
2. Connect (2) fibre cables to each HSG80. Arrange in logical order for ease of trouble
shooting later.
3. Connect each fibre channel tape controller with (1) fibre cable each to switch.
4. Connect each server fibre channel adapter with one fibre cable each to switch.
Fibre Channel Tape Controller
1. Connect to FC SAN switch.
2. Connect to TL895 tape library with the following cable.
a. (4) DS-BN38B-01 1.0 Meter 68 VHDCI Male Straight to 68 HD Female
Straight
3. Turn on after completing SCSI connections to tape library and the tape library is on
and initialized.
18
Backup/Restore Solution for Lotus Domino Server R5
4. After tape controller initialization, set switch via serial port command console to
“default” settings.
TL895 Tape Library
1. Connect odd numbered SCSI ports 1 – 7 (68pin HD) to tape controller.
2. Set library to default configuration from front touch panel.
Server Setup
1. Install SNMP network services on Windows 2000 application server and backup
server. SNMP is needed by SANworks EVM.
−
Via Network Properties – Advanced – Optional Networking Components
– Management & Monitoring Tools - SNMP
2. Install KGPSA fibre channel adapter drivers from HSG80 kit CDROM (OS Platform
Kit) labeled: “StorageWorks Solutions Software RA8000/MA8000 Fibre Channel
Solution Software Windows NT, Windows 2000, Proliant, x86”
−
Via Control Panel – Admin Tools – Computer Management – Device
Manager – SCSI & Raid Controllers – Emulex Fibre channel HBA –
Driver Tab – Install or Update – search CD-ROM – next after driver
locate.
−
Found in \kgpsa\win2k\oemsetup.inf.
−
Restart computer.
−
Check status of KGPSA: run \winnt\system32\LPUTILNT.EXE, see
firmware settings of HBA.
5. Restart computer.
6. Install StorageWorks Fibre Channel Switch software from OS Platform Kit CDROM,
“StorageWorks Solutions Software RA8000/MA8000 Fibre Channel Solution Software
Windows NT, Windows 2000, Proliant, x86”
−
Select Solution Software for ACSv8.5 – select Fibre Channel Switch –
typical install – restart.
−
Update HSX driver if necessary.
−
Avoid any further driver updating from this CD.
7. Install HS agent. Agent means “server.”
8. Install SWCC, StorageWorks Command Console agents.
−
Install HS_Series latest version.
−
Click next, repeatedly to Typical installation.
−
Put <server name> in client, check TCPIP, choice of level, choose highest
configuration.
−
Scanning for subsystems, should show subsys1, OK. No params
selections needed.
19
Backup/Restore Solution for Lotus Domino Server R5
−
Restart to take effect. Actually stopping and starting the STEAM service.
SWCC Agent Service is called “STEAM” in the Services utility, and
called HS_Services Agent in add/remove programs. Compaq HS_Service
is a Compaq event logging service, updated to interpret HSG80 messages.
It is seen in the Programs menu as SW Command Console, which is the
client.
−
Verify that you can see disks and tapes after reboot.
9. If you don’t see disks, check the connections on HSG80 console; if there are new
connections, correct and rename. If new connections are not added to access list, the disks
are not seen.
−
Install EVM on application server. Due to the need to communicate with
storage, just install server.
10. Install Internet Explorer 5.0 on application server.
−
Objective is to have Java support on the browser.
−
Connect to EVM with the browser.
−
Use 127.0.0.1:2301 or server name.
−
EVM creates virtual HTTP server.
11. Install EVM client ONLY on backup server (read release notes – EVM installation).
12. BCV can be mounted on any server on the SAN that has EVM client installed.
Additional Information
You will find the hardware limitations for disk arrays in the HSG80 Configuration Guide.
Appendix A: Sample scripts
EVM scripts are made up of at least four files. Each EVM operation is divided into “Do” and
“Undo” sets of commands. Required with each of these is a “keyword” file. Example script
fragments are shown in the Best Practices for Domino Database Management using Compaq
StorageWorks Enterprise Volume Manager. The document will be available at
http://www.compaq.com/products/storageworks/Storage-ManagementSoftware/evmindex.html.
Due to the significant amount of individualization each EVM installation and operation
requires, this document does not include complete sample script listings.
EVM scripts are stored as ASCII files, but they must be created and modified using the EVM,
browser based GUI. Directly editing the ASCII scripts incurs a significant risk that they will
not be read by the EVM GUI. Not only are these scripts unique to each host, they are sensitive
to the sequence in which they are run.
Let us say the backup host is expecting to backup a BCV named e:. If another BCV named d:
has not yet been mounted to the backup host then the new BCV will be called d: instead of e:.
This will cause the backup to fail, because the host will be trying to backup e: which does not
exist.
20
Backup/Restore Solution for Lotus Domino Server R5
Similarly, if spare disks are needed in the controller to create a BCV, and another BCV is still
using them, then the job will fail.
Script process
Scripts must be authored and maintained using EVM. Although this process creates text files,
the text should not be edited directly, as the resulting changes may not work with EVM.
The process requires the following decisions.
• Pick a name for this EVM script.
• Decide if the user application uses a Clone
• Decide which of the existing LUNs on that controller are to be copied to the BCV.
• Create the application commands to pause and to restart.
• Indicate the remote backup host where the BCV will be mounted and the backup will
be done.
• Type in the backup host command to start the backup job of the newly mounted BCV.
EVM script examples
Sample Script for starting the Domino Server
REM ****************************************************************
REM Script to stop Lotus Domino
REM 10/20/00, Revision 1.0.0
REM Compaq Computer Corporation
REM
REM ****************************************************************
REM Start Lotus Domino
REM ****************************************************************
net start "Lotus Domino Server (LotusDominoData)"
echo Application Started... Exiting...
pause
REM ****************************************************************
REM End of Script
REM ****************************************************************
:end
Sample Script to Call an EVM Job
REM
****************************************************************************
***
REM Script to call EVM job
REM 10/17/00, Revision 0.1a-4
REM Compaq Computer Corporation
21
Backup/Restore Solution for Lotus Domino Server R5
REM
REM
*******************************************************************************
REM Set Script Variables
REM
*******************************************************************************
SET DOMINOserver=<servername>
SET EVMJOB=<fulljobpath>
REM EVM_HOME variable set by EVM installation
REM ****************************************************************
REM Stop Lotus Domino
REM ****************************************************************
net stop "Lotus Domino Server (LotusDominoData)"
echo Application Stopped...
echo Press any key to continue...
pause
REM
*******************************************************************************
REM Start logging for this Script
REM
REM
*******************************************************************************
md %windir%\rundomevm
net time \\%computername% >> %windir%\rundomevm\rundomevm.log
echo Beginning of job... >> %windir%\rundomevm\rundomevm.log
echo DOMNIOserver=%DOMINOserver% >> %windir%\rundomevm\rundomevm.log
echo EVMJOB=%EVMJOB% >> %windir%\rundomevm\rundomevm.log
echo EVM_HOME=%EVM_HOME% >> %windir%\rundomevm\rundomevm.log
REM
*******************************************************************************
REM Verify required variables
REM
REM
*******************************************************************************
IF NOT DEFINED DOMINOserver goto nodom
IF NOT DEFINED EVMJOB goto noevmjb
IF NOT DEFINED EVM_HOME goto noevmhm
goto runevm
:nodom
echo Dominoserver variable not set... Exiting...
echo Dominoserver variable not set... Exiting... >> %windir%\rundomevm\rundomevm.log
goto end
:noevmjb
echo EVMJOB variable not set... Exiting...
echo EVMJOB variable not set... Exiting... >> %windir%\rundomevm\rundomevm.log
22
Backup/Restore Solution for Lotus Domino Server R5
goto end
:noevmhm
echo EVM_HOME variable not set... Exiting...
echo EVM_HOME variable not set... Exiting... >> %windir%\rundomevm\rundomevm.log
goto end
:runevm
REM
****************************************************************************
***
REM RUN EVM JOB
REM
REM
****************************************************************************
***
CD %evm_home%\bin
### evmcl %DOMINOserver% %EVMJOB%
IF %ERRORLEVEL%==0 CPQLE 4 2000 "PROCESS:%0, EVM complete".
REM CPQLE 4 2010 "PROCESS:%0, Clone:%EVMJOBClone% DID NOT DELETE. "
REM
****************************************************************************
***
REM Close logging for this Script
REM
REM
****************************************************************************
***
echo End of job... >> %windir%\rundomevm\rundomevm.log
net time \\%computername% >> %windir%\rundomevm\rundomevm.log
REM
****************************************************************************
***
REM End of Script
REM
REM
****************************************************************************
***
:end
Further documentation on these command lines can be found in your backup application user
manual.
23
Backup/Restore Solution for Lotus Domino Server R5
24
Appendix B: Sample Procedure for Operating
with Two Clones
To establish the scenario of a two-clone rotation the user must:
1. Maintain two clones
2. Create a separate EVM job for each clone
3. Mount each clone as a different drive letter
4. Create modified backup batch files for each one of the clones
A simple process for creating 2 clones is to implement two sets of scripts. Scripts are
provided with this solution to help automate this backup process. However, if you want to use
the method of rotating two clones, you will discover that EVM will not allow you to create
one job to make two clones. When an EVM job is created, it only has the capability to make
one clone and undo that clone. The names of scripts and the mounted drive letters will always
remain the same for that one job.
In order to achieve two clones mounted and identified as different drive letters, the user will
have to create a separate EVM job for each clone. The two scripts will look almost identical,
differing only in the name of the job and the letter that you would mount the drive as.
The next step is to customize the batch files. You will need to make a batch file for each
separate clone. Within the batch file, you will define the name of the EVM job. Once the EMV
jobs and scripts are created, the process of clone rotation can be automated with the Compaq®
Batch Scheduler for Windows® NT™ Version 2.1. If there is a need to run a backup program
using a script, a separate batch file must be customized for each job.
Once jobs and scripts are created for each clone, schedule each of the two EVM jobs to rotate
at every scheduled backup window. The following is an example of how you may want to
implement the process:
Job 1
Job 2
-Clone
-Clone
-Mount as drive L:\
-Mount as drive M:\
-Call backup script for L:\
-Call backup script for M:\
-Set system variable for L:\
-Set system variable for M:\
Scheduler Script
Call EVM undo %Job (older job)
Call EVM %Job (new job)
Backup/Restore Solution for Lotus Domino Server R5
Appendix C: Overview of SANworks
Enterprise Volume Manager
Compaq SANworks Enterprise Volume Manager V1.1 is a key component of the Compaq
Enterprise Network Storage Architecture strategy (ENSA). SANworks Enterprise Volume
Manager provides advanced, centralized storage management capabilities in Windows 2000
computing environments. Using the BCV clone feature can decrease the amount of time
required to restore a Domino mail server and provides an additional tool that complements and
enhances regular online backup techniques. The BCV clone of the database file allows
administrators to perform integrity checks and do other work from offline copies of the clone,
without stopping the Domino Server service. Clones should not be viewed as a replacement
for tape backups, or as a justification for unchecked growth of the Domino data directory.
EVM is a Web-enabled application program that allows you to establish an EVM Network of
up to ten computers that can participate in automated storage unit replication jobs on Compaq
StorageWorks Enterprise Storage Array 12000 systems or StorageWorks RAID Array 8000
Fibre Channel systems.
Each EVM Network and its associated StorageWorks subsystem can be managed remotely
from a central location using a Web browser. Access control features are included in EVM's
HTTP server.
The figure below shows a typical EVM Network. The EVM Server and EVM Clients are
noted with red and orange dots, respectively.
EVM Graphical User Interface
Through an easy-to-use Web browser interface, EVM provides the means to create, manage
and run storage replication jobs on the StorageWorks subsystem. Completing just four simple
steps with the EVM Create Job wizard defines most jobs. The most complex jobs can be
created in six steps using the wizard. EVM job pages provide you with the ability to create,
run, validate, monitor, review and delete jobs. The EVM Network View and EVM Storage
View pages provide graphical displays of the available resources on the EVM Network.
25
Backup/Restore Solution for Lotus Domino Server R5
EVM job interaction with external applications
EVM provides a simple, yet powerful set of features that allow you to design storage unit
replication jobs that can be seamlessly combined with external applications. For example, it is
easy to create:
3. EVM-database jobs that create nearline backup units on the StorageWorks subsystem.
4. EVM-database-backup jobs that copy online databases and mount them on backup
servers for automated tape backups.
5. EVM-database jobs that copy databases and mount them on another server for data
mining, or testing.
Creating High-Availability Solutions
By using batch files and scheduling applications, it is possible to create job sequences that can
be run 24 hours a day, seven days a week.
EVM Storage Unit Replication Features
Replicating Storage Units Using Cloning
Defining cloning
Cloning is a method of making a physical, point-in-time copy of a storage unit. EVM creates
clones with a simple single-click operation. All of the underlying StorageWorks CLI
commands that are required for cloning have been automated and incorporated into EVM.
When you specify the clone method in a job, EVM will automatically make duplicates, or
BCV units, and mount them on a computer with the drive letters that you specify. The BCV
units are then available for a variety of purposes. You can also have the EVM job leave the
BCV units unmounted, for example, as nearline backup.
ACS versions supporting cloning
Versions 8.5F
Performance of clones
A clone can be built using one of two modes— Fast or Normal. Fast mode builds the clone at
20 MB/Sec, while Normal mode builds it at 7 MB/Sec. Normal mode is the default and
recommended setting as it runs at lower priority than host I/O.
When replicating the Domino data directory, the most important factors are the capacity and
number of disks being normalized. Assuming a moderate I/O load on the StorageWorks
HSG80 controller, and an adequate configuration of disks in the array, a replication rate of 120
GB per hour can be achieved.
Cloning a storage unit using EVM
For each unit to be cloned, EVM automatically checks the RAID Level, determines the correct
procedures, then searches the StorageWorks subsystem and selects free disks for the additional
(BCV) mirror. EVM then copies the data and normalizes the disks in the mirror set. When
26
Backup/Restore Solution for Lotus Domino Server R5
normalization is complete, all members of the mirror set have identical data and EVM splits
the additional (BCV) mirror from the mirror set.
EVM applies slightly different procedures, depending upon the type of source unit. You
should review the differences in procedures outlined below and consider them when planning
EVM jobs and storage configurations.
Source Unit is a Three-member Mirror Set (RAID 1, RAID 0+1) and is the recommended
mode in which to run a Domino server. To achieve near instantaneous cloning with EVM, a
source unit should be configured as a three-member mirror set and be fully normalized. In this
configuration, EVM instantly splits off the third member to create the BCV unit. This
approach to instant cloning also applies to mirror sets with four or more members.
Source Unit is a Two-member Mirror Set (RAID 1, RAID 0+1). EVM automatically adds a
third member to the mirror set and normalizes the members. When the members are
normalized, EVM splits the third member to create the new BCV unit. This method requires
the administrator to schedule a wait for the clone to synchronize (120 GB per hour).
Source Unit is Not Mirrored (RAID 0 and JBOD). EVM automatically creates a single
member mirror set and then adds a second member. When the members are normalized, EVM
splits the second member to create the new BCV unit.
Cloning RAID 3/5 units
RAID 3/5 units cannot be cloned. Due to the presence of parity data, units that are configured
as RAID 3/5 cannot be replicated by the clone method.
Cloning BCV units
BCV units cannot be cloned. EVM tracks the BCV units that it creates with each job and
will not allow you to select one as a source for another job.
Appendix D: Product Overview – Lotus
Domino R5
Lotus Domino is a highly effective application in the Mail and Messaging, and Knowledge
management space. Lotus Domino is designed to run at the application layer on the operating
system. Its basic disk architecture is similar to simple file storage. There are three basic types
of Domino files: Server Configuration, Mail files, and Application files. Each user is assigned
a separate mail file during the user's registration. These typically will be the user's last name
combined with the first character of the first name. The server creates the new file based on a
mail template file that includes the proper forms and views.
The Domino server is configured with a single main directory structure to maintain all the
corresponding databases, \lotus\domino\data as default. This data directory houses the mail
database files (usually in a separate folder called Mail), system database files, and any shared
application databases (these can exist within the root of the directory structure or in separate
folders). Each individual database contains within it the forms, views, and access control data
required by the particular database. This allows an individual database to be independent of
the Domino environment.
27
Backup/Restore Solution for Lotus Domino Server R5
Because of this design, Lotus installs Domino on a single logical disk. While you can allow
Domino Servers to span multiple logical disks by redirecting to separate directory structures
using file and directory pointers (text files consisting of the real directory path and ending in
.NSF), this is a manual process which is not supported during the Lotus installation. Because
all user mail boxes, and all database applications, appear to the Windows operating system as
individual files with a .NSF extension, they are handled the same way as Word documents or
text files, which allows copying or restoring of individual mail files as necessary.
What this means for backing up Lotus Domino, is that there are no special considerations
beyond those taken for a normal file server (I.E. open files). This also means that individual
mail files or application files of higher importance can be singled out for more frequent
backups, without having to backup the entire Domino Environment.
During the testing of EVM, the Lotus Tool NotesBench was used to create a constant load of
2000 users on the server delivering mail and browsing the Domino Directory, and mail files.
Due to the design of the NotesBench tool, 2000 user mail databases were created in the root of
the Domino directory. While this is not a typical location for these files, it was required to
have the test operate correctly. The location of the mail files on the logical disk (i.e. the
directory structure they are contained in) has been shown to have no effect on performance as
long as the files are still contained on the same logical disk.
By using the R5Mail test derived by Lotus, we created a typical Lotus environment of active
users sending and receiving mail, and doing calendar and scheduling events. An average user
will execute this script 4 times per hour. For each iteration of the script, there are:
• 5 documents read
• 2 documents updated
• 2 documents deleted
• 1 view scrolling operation
• 1 database opened and closed
• 1 view opened and closed
• 3 messages comprised of 1 memo to 3 recipients
• 3 lookups against the Domino Directory
• 1 calendar and scheduling appointment
• 1 calendar and scheduling invitation sent to the recipients approximately every 90 minutes.
Appendix E: Test disclosure
Performance characterization tests were conducted using the Lotus Domino Backup/Restore
Solution under typical load conditions. The results of these tests show that the use of a BCV
clone has no significant impact on the performance of Lotus Domino and that backup to tape
is almost three times faster than online backup.
Table D-1 below shows the results from tests using our typical configuration.
28
Backup/Restore Solution for Lotus Domino Server R5
29
Table D-1. NotesBench results and backup rates
Operation
Backup Type
NotesBench Client
Access Time Avg. (ms)
Backup Rate
Baseline
None
62
N/A
Backup
LAN
N/A
24.403 GB/Hr
Domino Downtime
4hr 54min 1sec
Backup
From Clone
72
69.612 GB/hr
Domino Downtime
2.5 minutes
Restore
Direct to Volume
N/A
81.244 GB/hr
Domino Downtime
1hr 28min 53sec
Restore
Clone Mapped to server
data volume
N/A
Domino Downtime:
11 seconds
* Compaq’s Best Practice for Domino recommends using BCV clone for Backup.
Workload methodology
One Domino server was on the network receiving a standard NotesBench 2000 client mail
load. A standard mail load includes opening of databases and calendaring. Each of the tests
was run on the SAN with and without load as applicable. A two-disk mirror configuration of
the hardware was tested to develop a baseline. The baseline did not use SANworks Enterprise
Volume Manager BCVs. Additional tests were run to compare the overhead of the copy-out
operation. The following configurations were tested:
• Baseline configuration, 2 disk mirror
• Baseline configuration, 3 disk mirror
• LAN Backup
• Clone
• Backup Clone using Legato NetWorker, Computer Associates ArcServe2000 and
Veritas Backup Exec
• Restore using Computer Associates ArcServe2000 over LAN
• Restore using Computer Associates ArcServe2000 over SAN
• Restore using clone remapped to data volume on Domino Server
These test configurations are further described in the section on “System Configurations.”
The main tool used in generating the workload was the Lotus NotesBench load simulator, the
industry standard tool for simulating a client user load on a Domino Server. Its purpose is to
enable a single Windows NT machine, called a “NotesBench client,” to simulate multiple
Lotus Domino client users.
Backup/Restore Solution for Lotus Domino Server R5
The Response Time score represents an average of the measured test run. The score is
expressed in milliseconds (ms). A Response Time score of 1000 ms or less is considered an
acceptable response time for e-mail users using Domino servers.
Two NotesBench clients were used to generate the default 2000 client load. The decision to
use 2000 clients was based upon the available capacity of the data volume of 120 GB. A total
of 1000 users were simulated on each client system. The baseline the tests were run for eight
hours to verify that the response times and all measured events and counters were within
acceptable limits.
For detailed information on the workload used for testing, refer to the Compaq ProLiant server
Benchmark testing documents. This workload-sizing document is available from the
Microsoft Web site at:
http://vcmproapp02.compaq.com/ActiveAnswers/Global/en/solutions.1004/Solutions_Plannin
g.1005/default.asp#tabbar
Test Procedure
Setting up the simulation environment is the first task in a NotesBench test. NotesBench
generates the user directory database and then initializes the test by populating the Domino
mail and calendar databases. A test typically lasts for eight hours. During a test, the simulated
users log onto the Domino server and create, send, delete, and otherwise process messages and
attachments, set up appointments, open and close databases. The performance data is written
to a log file and the user actions are logged to the file for later analysis.
Test Results
Performance results were measured using Microsoft NT Performance Monitor. Measurements
were obtained by measuring averages for the period of steady-state activity (after 2000 users
were successfully logged on). Tests measure the messaging throughput of a single server,
single-site topology.
IMPORTANT: There was no significant impact on the performance of Domino using a
BCV clone in this tested environment.
Performance Impact of Restoring from BCVs
It is important to verify the integrity of your BCV prior to using it for an offline recovery. NT
check disk utilities are sufficient, since Domino databases are seen as common flat files by the
operating system. With the Domino server shut down, the total time required to restore the
full data volume from clone over the SAN was 11 seconds using the clone re-mapping
method. Best practice would be to first re-establish full mirroring before using the restored
data. However, in an extreme situation, the Domino database can be available nearly instantly
Appendix F: Standard Terms
ACS (Array Controller Software) Controller Firmware
Array Controller Software (ACS) is the software component of the HSx80 array controller
subsystem. ACS software executes on the HSx80 controller and processes I/O requests from
the host, performing the device level operations required to satisfy the requests.
30
Backup/Restore Solution for Lotus Domino Server R5
ACS operating firmware is stored in a PCMCIA program card. Compaq ships the card along
with your controller. Each time ACS operating firmware is updated, new cards are
manufactured for each controller model.
Baseline Testing
Tests that are done using standard, present day configuration methods where statistics are
collected and used for comparison with tests conducted for solution.
BCV Clone = Business Continuous Volume Clone
**See the description for EVM Clone
Compaq SANworks Enterprise Volume Manager (EVM)
SANworks Enterprise Volume Manager (EVM) is a web-enabled application program that
allows users to establish an EVM Network of up to ten computers that can participate in
automated storage unit replication jobs on Compaq StorageWorks systems. SANworks EVM
addresses the need for business continuance and data protection by providing the ability to
create, run and manage automated storage replication jobs and to link them with external jobs
to achieve fully parallel processing operations.
EBS (Enterprise Backup Solution)
The Compaq StorageWorks Enterprise Backup Solution (EBS) is a Fibre Channel Backup and
Restore solution consisting of backup application software from leading vendors (see below)
and Compaq hardware to provide a complete enterprise-class solution. Compaq carries a
variety of solutions to solve customer data protection issues in the Workgroup, Departmental,
and Data Center segments.
EVM Clone = Enterprise Volume Manager Clone
EVM clone configurations help eliminate the contention for data between production
applications and backup or testing applications. EVM makes a physical copy of a production
volume when making a clone. The clone function is particularly suited to transactional
environments where sharing a volume between applications is difficult due to high numbers of
I/Os that can result in application contention for data.
HSG80 Controller
The heart of the Fibre Channel subsystem is its controller. The intelligent, StorageWorks
HSG80 controller supports all major RAID levels: 0, 1, 0+1, and Compaq's Adaptive 3/5. It
also supports non-RAID arrays. The controller is designed to take full advantage of Fibre
Channel hub and switch technology.
The StorageWorks Fibre Channel HSG80 controller supports dual-redundant configurations
with automatic failover, ensuring continuous controller operations and delivering dramatic
performance increases. Battery-protected, mirrored write-back cache minimizes
performance lags and ensures your valuable business data is not lost. Each controller provides
up to 512 MB of read-ahead cache and dual-port connection to the host.
31
Backup/Restore Solution for Lotus Domino Server R5
RAID Level 0+1
RAID 0+1 is a combination of striping and mirroring data. RAID 0+1 write data across the
drives in the striping fashion as RAID 0, and achieves redundancy by mirroring data similar to
RAID 1. In RAID 0+1, unlike RAID 1, the data disks are also the mirror disks. Consequently,
this kind of storageset combines the performance of striping with the reliability of mirroring.
Replication Server
A second Server holding an exact copy of the original Domino database, also known as a
replication client.
SAN
A Storage Area Network (SAN) is a network that connects storage devices, such as disk, tape,
and CD ROM drives to all types of computing devices. It is based on the Fibre Channel
interconnect. The SAN provides high-speed, fault-tolerant access to data for the various client,
server, and host computers. These computing devices can be as simple as a small workstation
or a large mainframe system.
Transparent Failover Mode
Controller managed failover mode, host side does not see failover and storage is still available
to hosts. At any moment in time only one controller has an active port one and only one
controller has an active port two, and the other two ports are in standby mode.
Compaq Active Answers Sizing Guide
ActiveAnswers Solution Sizers are automated tools that assist you with managing the size and
scope of a solution environment. The sizing information and algorithms have been developed
using testing and performance data on Compaq servers running Solutions from Compaq
partners such as Microsoft, Lotus and Oracle. The online sizers are directly linked to the
Compaq ActiveAnswers System Configurator. The downloadable sizers consist of executables
that run under several different platforms.
The following definitions are referenced from the Microsoft Windows 2000 Performance
Monitor.
%Committed Bytes In Use
The percent of virtual memory in use by process.
% Committed Bytes In Use is the ratio of Memory: Committed Bytes to Memory: Commit
Limit. (Committed memory is physical memory in use for which space has been reserved in
the paging file should it need to be written to disk. The commit limit is determined by the size
of the paging file. If the paging file is enlarged, the commit limit increases, and the ratio is
reduced). This counter displays the current percentage value only; it is not an average.
%CPU Time = % Processor Time
% Processor Time is the percentage of time that the processor is executing a non-Idle thread.
This counter was designed as a primary indicator of processor activity. It is calculated by
32
Backup/Restore Solution for Lotus Domino Server R5
measuring the time that the processor spends executing the thread of the idle process in each
sample interval, and subtracting that value from 100%. (Each processor has an idle thread
which consumes cycles when no other threads are ready to run). It can be viewed as the
percentage of the sample interval spent doing useful work. This counter displays the average
percentage of busy time observed during the sample interval. It is calculated by monitoring
the time the service was inactive, and then subtracting that value from 100%.
%Disk Time
This counter monitors the percentage of time that the physical disk is doing work with
read/write activity.% Disk Time is the percentage of elapsed time that the selected disk drive is
busy servicing read or write requests.
Disk Queue Length
Current Disk Queue Length is the number of requests outstanding on the disk at the time the
performance data is collected. It includes requests in service at the time of the snapshot. This
is an instantaneous length, not an average over the time interval. Multi-spindle disk devices
can have multiple requests active at one time, but other concurrent requests are awaiting
service. This counter might reflect a transitory high or low queue length, but if there is a
sustained load on the disk drive, it is likely that this will be consistently high. Requests are
experiencing delays proportional to the length of this queue minus the number of spindles on
the disks. This difference should average less than 2 for good performance.
%Page File Utilization
The amount of the Page File instance in use in percent. See also Process: Page File Bytes.
Page File Bytes is the current number of bytes this process has used in the paging file(s).
Paging files are used to store pages of memory used by the process that are not contained in
other files. Paging files are shared by all processes, and lack of space in paging files can
prevent other processes from allocating memory.
Percent of reserved memory from operating system used to back up physical memory
Appendix G: Additional References
The following key documents and locations provide a wealth of information
regarding successful deployment of Lotus Domino Server on Compaq
platforms.
Compaq ActiveAnswers
www.compaq.com/activeanswers
Messaging and Collaboration Solutions
Compaq White Paper Index
www.compaq.com/support/techpubs/whitepapers
33
Backup/Restore Solution for Lotus Domino Server R5
Compaq TechNote Index
www.compaq.com/support/techpubs
RAID Technology for Database Servers
Lotus Development Corporation Web Site
http://www.lotus.com/home.nsf/welcome/lotusnotes
34