Sun Storage 7000 for VMware ESX

Quick Start Guide

Application Integration Engineering (AIE)

Sun Microsystems, Inc.

Ryan Arneson

April 2009, Rev 1.0

© 2007 Sun Microsystems, Inc., 4150 Network Circle, Santa Clara, CA 95054 USA

All rights reserved.

This product or document is protected by copyright and distributed under licenses restricting its use, copying, distribution, and decompilation. No part of this product or document may be reproduced in any form by any means without prior written authorization of Sun and its licensors, if any. Third-party software, including font technology, is copyrighted and licensed from Sun suppliers.

Parts of the product may be derived from Berkeley BSD appliances, licensed from the University of California.

Sun, Sun Microsystems, Sun StorageTek, the Sun logo, are trademarks, registered trademarks, or service marks of Sun Microsystems, Inc. in the U.S. and other countries.

UNIX is a registered trademark in the United States and other countries, exclusively licensed through X/Open Company, Ltd.

Oracle and Orion are registered trademark of Oracle corporation in the United States and other countries.

All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. in the U.S. and other countries. Products bearing SPARC trademarks are based upon an architecture developed by Sun Microsystems, Inc.

The OPEN LOOK and Sun's Graphical User Interface was developed by Sun Microsystems, Inc. for its users and licensees. Sun acknowledges the pioneering efforts of Xerox in researching and developing the concept of visual or graphical user interfaces for the computer industry. Sun holds a non-exclusive license from Xerox to the Xerox Graphical User Interface, which license also covers Sun's licensees who implement OPEN LOOK GUIs and otherwise comply with Sun's written license agreements.

RESTRICTED RIGHTS: Use, duplication, or disclosure by the U.S. Government is subject to restrictions of FAR 52.227-14(g)(2)(6/87) and

FAR 52.227-1987), or DFAR 252.227-7015(b)(6/95) and DFAR 227.7202-3(a). DOCUMENTATION IS PROVIDED ÒAS ISÓ AND ALL

EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS AND WARRANTIES, INCLUDING ANY IMPLIED WARRANTY OF

MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR NON-INFRINGEMENT, ARE DISCLAIMED, EXCEPT TO THE

EXTENT THAT SUCH DISCLAIMERS HELD TO BE LEGALLY

Table of Contents

1 Objective................................................................................................................................................. 4

1.1 NFS (NAS) Access..........................................................................................................................4

1.1 iSCSI Access................................................................................................................................... 5

2 Configuring Sun 7000 for VMware ESX................................................................................................6

2.1 Security............................................................................................................................................6

2.1.1 Configure public SSH for password-less login............................................................................ 8

2.2 Network........................................................................................................................................... 9

2.2.1 Datalink Configuration Example..................................................................................................9

2.2.2 Interface Configuration Example............................................................................................... 10

2.3 Pool................................................................................................................................................10

2.4 Projects and Shares........................................................................................................................11

2.4.1 Creating a New Project...............................................................................................................12

2.4.2 Creating a New Filesystem (NFS)..............................................................................................12

2.4.3 Creating a New Lun (iSCSI)...................................................................................................... 12

2.5 Analytics........................................................................................................................................14

3 Configuring VMware ESX with Sun 7000 .......................................................................................... 16

3.1 Changing NFS Settings in ESX.....................................................................................................16

3.2 Network......................................................................................................................................... 16

3.3 iSCSI Access................................................................................................................................. 17

3.4 Enable NFSv3 access.................................................................................................................... 19

4 Advanced Data Services........................................................................................................................20

4.1 Snapshots.......................................................................................................................................20

4.2 Cloning a Snapshot........................................................................................................................20

4.3 Replication.....................................................................................................................................21

4.4 Accessing Replicated Data............................................................................................................ 21

5 Reference Material................................................................................................................................ 23

6 Appendix A........................................................................................................................................... 24

Illustration Index

Illustration 1: NFS Access..........................................................................................................................4

Illustration 2: iSCSI Access....................................................................................................................... 5

Illustration 3: Roles.................................................................................................................................... 6

Illustration 4: Adding Local User...............................................................................................................7

Illustration 5: SSH Public Keys................................................................................................................. 8

Illustration 6: Datalink Example................................................................................................................ 9

Illustration 7: Interface Example..............................................................................................................10

Illustration 8: File System Properties....................................................................................................... 11

Illustration 9: iSCSI Access Control........................................................................................................ 14

Illustration 10: VMware ESX Network Configuration............................................................................ 16

Illustration 11: iSCSI Initiator..................................................................................................................17

Illustration 12: Add NFS Datastore..........................................................................................................19

Revision 1.1

page 3

1 Objective

This Quick Start Guide is intended to give an Administrator a quick overview on how to quickly setup a Sun Storage 7000 to use with VMware ESX. Both NFS (NAS) and iSCSI protocols are addressed.

This document assumes the administrator is familiar with both Sun Storage 7000 concepts as well as

VMware ESX concepts.

1.1 NFS (NAS) Access

The below diagram gives an overview of accessing the Sun 7000 via the NFS (NFSv3) protocol to host

Virtual Machine Virtual Disks (vmdk). A NFS file system is exported from the Sun Storage 7000 and mounted via the vmkernel in the ESX host.

Illustration 1: NFS Access

Revision 1.1

page 4

1.1 iSCSI Access

The below diagram gives an overview of accessing the Sun Storage 7000 via the iSCSI protocol to host

Virtual Machine Virtual Disks (vmdk). An iSCSI lun is exported from the Sun Storage 7000 and attached via the vmkernel in the ESX host. A VMFS3 file system can be created on the iSCSI lun to host vmdks, or the iSCSI lun can be attached directly to a Virtual Machine using Raw Device Mapping

(RDM).

Illustration 2: iSCSI Access

Revision 1.1

page 5

2 Configuring Sun 7000 for VMware ESX

2.1 Security

It is recommended to create a local user on the Sun 7000 that is associated with a role that allows limited configuration and control for shares within a specific project dedicated to VMware ESX use.

That project is first created by the Super-User role.

For that purpose, a role called “VMUser_Role” is created in “Configuration -> Users -> Roles”. This role is given access to two projects, “vm-nfs” and “vm-iscsi”. The role will have the ability to perform operations in those projects only. The role has also been given authorizations for using Analytics and

Worksheets as well as Replication. Specific policies around this role may vary from the example.

Illustration 3: Roles

Revision 1.1

page 6

Once the role is created, the next step is to create a user that can be associated with this role. In the example shown, the user is created as a local user on the Sun 7000. This user could also be maintained in a Directory Service. Note that this user configuration is solely to allow administration rights on the

Sun Storage 7000, it is not for file system ownership or file creation privileges. The user should also be associated with the built-in “basic” role as well as the “VMUser_Role” configured above.

Illustration 4: Adding Local User

Revision 1.1

page 7

2.1.1 Configure public SSH for password-less login

The final (optional) step is to upload a SSH Public Key for the user to allow administration and scripting without having to enter a password for every login.

1. On a UNIX/Windows client or VMware Remote CLI Virtual Appliance, login as the “vmuser” and generate keys

2. $ ssh-keygen -t rsa

3. $ ssh-keygen -t dsa

4. The above steps will create public key files in ~/.ssh/id_rsa.pub and ~/.ssh/id_dsa.pub

5. Login to the Sun 7000 as the “vmuser”

6. Under Configuration -> Preferences, click (+) to add an entry

7. Choose either RSA or DSA

8. Copy the contents of the .pub file (only the key itself, not the type field or the comment field) and paste in the entry box. Next copy and paste (or type) in an optional comment to clearly indicate who the key is for.

Illustration 5: SSH Public Keys

9. Click “Apply” from the main screen to complete the procedure.

10. Test by running a simple command to check that you are not prompted for a password. You may be prompted to accept the Sun 7000 key the first time the command is run

$ ssh -l vmuser <Sun 7000 IP> show

Revision 1.1

page 8

2.2 Network

There are no special network requirements mandated for VMware ESX interoperability. All methods of networking, Link Aggregation, IPMP, 10Gbe, are supported. It is recommended that a private network be configured to handle the storage traffic between the ESX servers and the Sun

7000. It is also recommended that some level of redundancy be configured either via Link

Aggregation or IPMP. The example below shows an Interface (Private-Net) created from a Link

Aggregation of two physical 1Gb Ethernet Ports (PrivNet-Aggregation). Those ports are also configured for LACP. Consult your switch vendor documentation for details on enabling LACP and

Link Aggregation at the switch level.

2.2.1 Datalink Configuration Example

Revision 1.1

Illustration 6: Datalink Example

page 9

2.2.2 Interface Configuration Example

Illustration 7: Interface Example

2.3 Pool

Configuring the pool storage RAID layout is a balance between availability, capacity and performance.

As can be seen during pool configuration, the Sun 7000 BUI gives a description plus the relative tradeoffs around availability, capacity and performance. The administrator must weigh all three factors when deciding on a Data Profile. For VMware ESX, and it's highly random I/O nature, the best performance is usually found with the Mirror profile.

Also, depending on the Sun Storage 7000 model, you may also have to decide on data profile for any

Log SSDs that are present in the configuration. Striping the Log devices should give the best performance. Mirroring the Log Devices will give an extra degree of Log availability, however, while losing a Log Device may impact write performance, it will not impact overall availability of the pool.

Revision 1.1

page 10

The writes will go directly to disk, but possibly at lower performance levels.

2.4 Projects and Shares

Projects and shares should be configured to present NFSv3 mount points or iSCSI luns to the ESX servers. The number of shares will be dependent on ESX storage sizing needs and other business factors such as snapshot and cloning granularity. Also, project layouts should be considered if

Replication is going to be implemented. The Sun 7000 replicates at the project level. If you have differing replication schedules for different types of Virtual Machines (or even subsets of Virtual Disk types, OS/Boot versus Production Data) you should configure multiple projects, each to replicate on their own schedule.

For VMware ESX, it is recommended to turn off “Update access time on read” for NFS filesystems.

For the rest of the options, the defaults are sufficient for most general deployments. If you are going to create multiple shares under the same project you can change the properties at the project level and those properties will be inherited by all shares in the project. Individual properties can be overridden at the share level as needed.

Illustration 8: File System Properties

Revision 1.1

page 11

2.4.1 Creating a New Project

To create a new project

1. Configuration -> Shares -> Projects

2. Click (+) to add new Project

3. Give it a name and click “Apply”

4. Click the Edit icon next to Project name to enter Project Properties.

2.4.2 Creating a New Filesystem (NFS)

To create a new share

1. Configuration -> Shares

2. Click (+) to add a new Filesystem

3. Use the Drop-down menu to select which project the share should go into

4. Give the share a name. Other fields can be left at the defaults.

5. Click “Apply”

6. Click the Edit icon next to Share to change any of the defaults.

By default, all Sun 7000 filesystems are created under “/export/sharename” mount points. This can be overridden at the project or share level if needed.

2.4.3 Creating a New Lun (iSCSI)

Before creating a new iSCSI lun, initiator entries should be configured to allow fine grained access control.

Revision 1.1

page 12

To add an Initiator

1. Configuration -> Services -> iSCSI -> Initiators

2. Click (+) to add a new initiator.

3. Enter an alias and the IQN of the initiator. The IQN is found in the Storage Adapter section of the ESX VI Client application. See Section 3 for more details.

4. Optionally, configure the CHAP settings for Uni-Directional CHAP. These credentials must match those set on the initiator in ESX.

5. Click Add to finish the procedure.

To add a new iSCSI lun

1. Configuration -> Shares -> Luns.

2. Click (+) to add a new Lun.

3. Use the Drop-down menu to select which project the lun should go into.

4. Give the lun a name. Enter in volume size, whether the lun should be sparse and the volume block size.

5. Click Apply.

6. Click the Edit icon to enter the lun properties. Click the Protocols tab

7. Uncheck “Inherit from project” and “Allow any initiator access”

8. Check the desired initiator that was configured above.

Revision 1.1

page 13

Illustration 9: iSCSI Access Control

The initiator access can also be configured at the project level if all luns in a project are going to havethe same initiator access.

2.5 Analytics

The following analytics are useful to observe the workloads on the Sun 7000

●

NFSv3 operations broken down by file name (NFS only)

○

Allow the administrator to view operations at the virtual disk level.

●

NFSv3 operations broken down by client (NFS only)

○

Allow the administrator to view operations on a per ESX server level

●

Interface bytes as a raw statistic

○

Allow the administrator to view network performance

●

I/O bytes broken down by disk

○

Allow the administrator to view disk I/O bandwidth usage

●

ARC access broken down by Hit/Miss

Revision 1.1

page 14

○

Allow the administrator to view Adaptive Replacement Cache activity

At the end of this guide is a sample Workflow script that will build a Worksheet with these Analytics.

Revision 1.1

page 15

3 Configuring VMware ESX with Sun 7000

3.1 Changing NFS Settings in ESX

By default, VMware ESX has a NFS Datastore limit of 8 file systems. To change that limit, use the following steps.

1. Login To Virtual Center

2. Choose an ESX host

3. Select Configuration -> Advanced Settings -> NFS

4. Change NFS.MaxVolumes to 32.

5. Repeat for each ESX server

3.2 Network

In order to use NFSv3 or iSCSI to store virtual machines, VMware ESX needs to have an additional network interface configured to allow the vmkernel to access the Sun 7000.

1. Login to Virtual Center

2. Choose an ESX host

3. Select Configuration-> Networking

4. Click “Add Networking”, choose Vmkernel, click Next

5. Select which Virtual Switch you want to create the new vmkernel interface on. This could be an existing vSwitch or you could create a new vSwitch on a different network adapter. Click Next

6. Change the label of the interface if needed or Click Next

7. Enter a IP address and subnet mask. Click Next.

8. Click Finish. You may be prompted to enter a gateway for the vmkernel interface.

Revision 1.1

page 16

9. Repeat for each ESX server.

In the example below. Two physical NICs (vmnic2 and vmnic3) are configured in vSwitch1 for vmkernel access. This provides connectivity to the Sun 7000 via a private vLAN.

Illustration 10: VMware ESX Network Configuration

3.3 iSCSI Access

To enable the built-in VMware ESX iSCSI Software Initiator

1. Login to Virtual Center

2. Select Configuration -> Storage Adapters

3. Select the iSCSI Software Adapter. Click Properties

4. Click Configure. Check the “Enabled” box. Click OK

Revision 1.1

page 17

Illustration 11: iSCSI Initiator

5. Select the Dynamic Discovery Tab.

6. Click Add. Enter in the IP address of the Sun 7000. Click OK

7. Click Close

8. Click the Rescan link to have VMware ESX discover the Sun 7000 iSCSI luns.

9. It is recommended to enable any CHAP configuration after all iSCSI luns have been discovered.

10. After the Rescan completes. Select Configuration -> Storage to add the iSCSI lun as a DiskDevice and create a

VMFS3 file system on the lun. Optionally the iSCSI lun can be used as a Raw Device Mapping device.

Revision 1.1

page 18

3.4 Enable NFSv3 access

1. Login to Virtual Center

2. Select Configuration -> Storage - > Add Storage

3. Choose Network File System

4. Enter the Sun 7000 name or IP, file system mount point and a Datastore label.

Illustration 12: Add NFS Datastore

5. Click Next. Click Finish

Revision 1.1

page 19

4 Advanced Data Services

The following section will give a high level overview on how to enable snapshots and replication on the Sun 7000 to use with VMware ESX. It is outside the scope of this particular document to delve into the lower level implementation details for snapshot and replication. Please see the “NEED_A_TITLE” for further information on using snapshots and replication with VMware ESX. Also refer to the online

Sun Storage 7000 Help Wiki for details on using both features.

4.1 Snapshots

Snapshots can be taken at the project or the share (file system or iSCSI lun) level.

To take a snapshot use the following procedure.

1. Select Shares

2. Select the Share or Project that you want to create a snapshot on

3. Select the Snapshots tab

4. Click (+) to add a new manual snapshot or Select the Schedule tab and Click (+) to add a new scheduled snapshot

5. Click Apply

4.2 Cloning a Snapshot

In order to access the snapshot as a readable/writable file system or lun, the snapshot must be cloned and then mounted on the VMware ESX Server

To clone a snaphost use the following procedure

1. Select Shares

2. Select the Share or Project that contains the snapshot you wish to clone

3. Select the Snapshots tab

4. Select the snapshot you wish to clone

Revision 1.1

page 20

5. Click (+) to “Clone snapshot as a new share”

6. Give the clone a unique name to identify it as separate from the snapshot. (example: fs1-clone)

7. Click Apply

You can now use the procedures for adding a new NFS file system or a new iSCSI lun to VMware ESX to access the clone.

4.3 Replication

Replication is configured at the project level only. All file systems or iSCSI luns contained within a project will be asynchronously replicated on the same schedule to the target system. As the replication is at the project level, this allows the administrator to intelligently group virtual machines or virtual disk layouts based on replication needs. All objects within a specific project will be replicated with the same policy.

To enable replication to another Sun Storage 7000.

1. Select Shares -> Projects

2. Select the Project you wish to replicate and Click the “Edit Entry” icon

3. Select the Replication tab

4. Click (+) to add a new replication target

5. Fill in the target address and root user password. Select Continuous our Scheduled mode.

6. Click Add

4.4 Accessing Replicated Data

To access the replicated data on the target system, the system must complete at least one successful replication. After the data has finished replicating, the project can be access on the target system.

Revision 1.1

page 21

To access the replicated project, use the following procedures

1. Login to the Target system

2. Select Shares -> Replication

3. Select the Replicated project you wish to access

4. Click (+) to “Clone (export) this replica”

5. Give the replicated project a new clone project name

6. Optionally change the mount points of any file systems in the project by using “Override

Mountpoint” Click Continue

The project, including all file systems and luns, is now available on the target system. Follow the procedures on adding NFS file systems or iSCSI luns in ESX to attach these shares to your ESX server.

Revision 1.1

page 22

5 Reference Material

Sun Storage 7000 Product Page : http://www.sun.com/storage/disk_systems/unified_storage/

Fishworks Blogs: http://blogs.sun.com/fishworks

Sun BigAdmin Site: http://www.sun.com/bigadmin/home/index.jsp

Sun Storage Forums: http://forums.sun.com/category.jspa?categoryID=66

VMware on Sun Page: http://www.sun.com/software/vmware/index.jsp

VMware ESX Server Documentation: http://www.vmware.com/support/pubs/

Vmware Communities: http://communities.vmware.com/home.jspa

Revision 1.1

page 23

6 Appendix A

Sample workflow script to build an Analytics worksheet for monitoring NFSv3 and iSCSI operations.

This workflow should be copied to a text file and then uploaded to the Sun Storage 7000 by the following steps.

1. Select Maintenance -> Workflows

2. Click (+) to Add a Workflow

3. Browse to the text file location

4. Activate Workflow by double-clicking on the name of the workflow.

5. The resulting worksheet can be accessed by selecting Analytics -> Saved Worksheets

Workflow Example:

/*

* This is an example workflow for building a Analytics worksheet

* The worksheet is intended to monitor key metrics for NFSv3 and

* iSCSI operations in a VMware ESX environment

*

*/ var steps = [ { step: 'Checking for existing worksheet', execute: function (params) {

/*

* In this step, we're going to see if the worksheet that

* we're going to create already exists. If the worksheet

* already exists, we blow it away if the user has indicated

* that they desire this behavior. Note that we store our

* derived worksheet name with the parameters, even though

* it is not a parameter per se; this is explicitly allowed,

* and it allows us to build state in one step that is

* processed in another without requiring additional global

* variables.

*/ params.worksheet = 'VMware ESX Worksheet'; try { run('analytics worksheets select name="' +

params.worksheet + '"'); if (params.overwrite) { run('confirm destroy');

Revision 1.1

page 24

}, { return;

} throw('Worksheet called "' + params.worksheet +

'" already exists!');

} catch (err) { if (err.code != EAKSH_ENTITY_BADSELECT) throw (err);

}

} step: 'Creating worksheet', execute: function (params) {

/*

* In this step, we're ready to actually create the worksheet

* itself. Finally, note that we

* aren't saving the datasets -- we'll let the user do that

* from the created worksheet if they so desire. (It would

* be straightforward to add a boolean parameter to this

* workflow that allows that last behavior to be optionally

* changed.)

*/ run('analytics worksheets'); run('create "' + params.worksheet + '"'); run('select name="' + params.worksheet + '"');

if (params.proto == 'iscsi') {

ds = [ 'iscsi.ops[client]', 'net.kilobytes', 'io.bytes[disk]',

'arc.accesses[hit/miss]'];

} else if (params.proto == 'nfs') {

ds = ['nfs3.ops[file]','nfs3.ops[client]','net.kilobytes',

'io.bytes[disk]','arc.accesses[hit/miss]'];

}

else {

ds = ['nfs3.ops[file]','nfs3.ops[client]', 'iscsi.ops[client]',

'net.kilobytes', 'io.bytes[disk]','arc.accesses[hit/miss]'];

} for (i = 0; i < ds.length; i++) { run('dataset'); run('set name=' + ds[i]); run('commit');

}

}

} ]; var workflow = { name: 'VMware ESX Worksheet', description: 'VMware ESX Worksheet', parameters: { overwrite: { label: 'Overwrite the worksheet if it exists', type: 'Boolean'

},

proto: {

label: 'Protocol',

type: 'ChooseOne',

options: [ 'nfs', 'iscsi', 'both' ],

optionlabels: [ 'NFSv3', 'iSCSI', 'NFSv3+iSCSI' ],

}

Revision 1.1

page 25

};

}, execute: function (params) { return (steps); }

Revision 1.1

page 26