HPE 3PAR Thin Technologies
Technical white paper
Technical white paper
Contents
Introduction ................................................................................................................................................................................................................................................................................................................................................... 5
Overview of HPE 3PAR Thin Technologies for data compaction ............................................................................................................................................................................................................... 5
Product highlights ............................................................................................................................................................................................................................................................................................................................ 5
HPE 3PAR Thin Provisioning software .........................................................................................................................................................................................................................................................................6
HPE 3PAR Thin Deduplication software......................................................................................................................................................................................................................................................................6
HPE 3PAR Thin Clones software ........................................................................................................................................................................................................................................................................................6
HPE 3PAR Thin Persistence software............................................................................................................................................................................................................................................................................6
HPE 3PAR Thin Conversion software ............................................................................................................................................................................................................................................................................6
HPE 3PAR Thin Copy Reclamation software........................................................................................................................................................................................................................................................... 7
HPE 3PAR ASIC with Thin Built In ........................................................................................................................................................................................................................................................................................... 7
The benefits of Thin Provisioning ............................................................................................................................................................................................................................................................................................ 7
Avoid frequent storage capacity addition to servers........................................................................................................................................................................................................................................ 7
Accelerate time to market .........................................................................................................................................................................................................................................................................................................8
Chargeback model in utility storage ................................................................................................................................................................................................................................................................................8
Thin Deduplication .................................................................................................................................................................................................................................................................................................................................8
Using fully provisioned volumes ................................................................................................................................................................................................................................................................................................9
System requirements............................................................................................................................................................................................................................................................................................................................9
HPE 3PAR Volume Manager.......................................................................................................................................................................................................................................................................................................10
Physical disks ......................................................................................................................................................................................................................................................................................................................................10
Logical disks .........................................................................................................................................................................................................................................................................................................................................10
Common provisioning groups..............................................................................................................................................................................................................................................................................................10
Virtual volumes..................................................................................................................................................................................................................................................................................................................................10
The anatomy of a thin volume .............................................................................................................................................................................................................................................................................................11
Thin volume metadata ...............................................................................................................................................................................................................................................................................................................12
Thin Deduplication implementation...............................................................................................................................................................................................................................................................................12
Express Indexing..............................................................................................................................................................................................................................................................................................................................13
Common provisioning groups ....................................................................................................................................................................................................................................................................................................14
CPGs and workloads .....................................................................................................................................................................................................................................................................................................................14
Availability level ................................................................................................................................................................................................................................................................................................................................14
Reasons to create multiple CPGs ......................................................................................................................................................................................................................................................................................15
CPG automatic growth ...............................................................................................................................................................................................................................................................................................................15
Changing LD characteristics within a CPG...............................................................................................................................................................................................................................................................16
Snapshot space within CPGs ................................................................................................................................................................................................................................................................................................16
Allocating CPG space to thin volumes .........................................................................................................................................................................................................................................................................17
Sizing guidelines .....................................................................................................................................................................................................................................................................................................................................17
Technical white paper
Thin volumes administration.......................................................................................................................................................................................................................................................................................................17
Allocation warnings and alerts ...................................................................................................................................................................................................................................................................................................18
Allocation warnings .......................................................................................................................................................................................................................................................................................................................18
Allocation limits .................................................................................................................................................................................................................................................................................................................................19
Remaining physical capacity alerts ................................................................................................................................................................................................................................................................................20
Remaining CPG free space alerts.....................................................................................................................................................................................................................................................................................20
View logged warning and limit alerts ...........................................................................................................................................................................................................................................................................20
User recommendations .............................................................................................................................................................................................................................................................................................................20
Capacity efficiency ................................................................................................................................................................................................................................................................................................................................21
Manually updating the capacity efficiency statistics ......................................................................................................................................................................................................................................26
Understanding capacity efficiency ratios.................................................................................................................................................................................................................................................................. 27
Tracking volume space usage...................................................................................................................................................................................................................................................................................................28
TPVVs .......................................................................................................................................................................................................................................................................................................................................................28
CPGs ...........................................................................................................................................................................................................................................................................................................................................................28
System space......................................................................................................................................................................................................................................................................................................................................29
Total used space .............................................................................................................................................................................................................................................................................................................................29
Usage reporting and trend analysis .....................................................................................................................................................................................................................................................................................30
Running out of space .........................................................................................................................................................................................................................................................................................................................31
Running out of space in a TV...............................................................................................................................................................................................................................................................................................31
Running out of space in a CPG ...........................................................................................................................................................................................................................................................................................31
Running out of space on the system .............................................................................................................................................................................................................................................................................31
Reclaiming unused space ...............................................................................................................................................................................................................................................................................................................31
Reclaiming unmapped logical disk space from CPGs ................................................................................................................................................................................................................................... 32
Automatically reclaiming unused space from volumes ............................................................................................................................................................................................................................... 32
Deleted volume snapshot space....................................................................................................................................................................................................................................................................................... 32
Logical disks and chunklet initialization .................................................................................................................................................................................................................................................................... 32
Free page consolidation ........................................................................................................................................................................................................................................................................................................... 32
HPE 3PAR Thin Persistence software ............................................................................................................................................................................................................................................................................... 33
Thin Persistence reclamation .............................................................................................................................................................................................................................................................................................. 33
Thin Persistence methods .....................................................................................................................................................................................................................................................................................................34
Thin Deduplication with snapshots and clones ........................................................................................................................................................................................................................................................34
Thin Deduplication with Remote Copy ............................................................................................................................................................................................................................................................................. 35
Dynamic Optimization of virtual volumes....................................................................................................................................................................................................................................................................... 35
Migrate between full and thin volumes on the same array...................................................................................................................................................................................................................... 36
Estimating Thin Deduplication space savings .....................................................................................................................................................................................................................................................36
Online migration to HPE 3PAR StoreServ...................................................................................................................................................................................................................................................................... 37
Preparing data for migration................................................................................................................................................................................................................................................................................................38
Conclusion ...................................................................................................................................................................................................................................................................................................................................................38
Technical white paper
Appendix A—Thin volumes and applications ............................................................................................................................................................................................................................................................38
Oracle ........................................................................................................................................................................................................................................................................................................................................................38
Microsoft SQL Server ..................................................................................................................................................................................................................................................................................................................39
Microsoft Hyper-V .........................................................................................................................................................................................................................................................................................................................39
SAP ..............................................................................................................................................................................................................................................................................................................................................................39
VMware vSphere ............................................................................................................................................................................................................................................................................................................................40
Appendix B—Thin Persistence methods ....................................................................................................................................................................................................................................................................... 40
Thin Reclamation for Microsoft Windows Server 2012 ............................................................................................................................................................................................................................... 40
Thin Reclamation for Microsoft Windows Server 2003 and 2008 .....................................................................................................................................................................................................41
Thin Reclamation for VMware vSphere ......................................................................................................................................................................................................................................................................41
Thin Reclamation for Linux ...................................................................................................................................................................................................................................................................................................42
Thin Reclamation for HP-UX ...............................................................................................................................................................................................................................................................................................43
Thin Reclamation for UNIX....................................................................................................................................................................................................................................................................................................43
Thin Reclamation for HPE OpenVMS..........................................................................................................................................................................................................................................................................43
Thin Reclamation for Symantec Storage Foundation ...................................................................................................................................................................................................................................44
Thin Reclamation for Oracle databases .....................................................................................................................................................................................................................................................................44
Appendix C—File Systems and Thin Deduplication .............................................................................................................................................................................................................................................46
Microsoft Windows ........................................................................................................................................................................................................................................................................................................................46
Microsoft Hyper-V .........................................................................................................................................................................................................................................................................................................................48
VMware vSphere .............................................................................................................................................................................................................................................................................................................................49
Oracle Solaris ......................................................................................................................................................................................................................................................................................................................................49
Linux...........................................................................................................................................................................................................................................................................................................................................................50
Symantec Storage Foundation............................................................................................................................................................................................................................................................................................51
HP-UX........................................................................................................................................................................................................................................................................................................................................................51
IBM AIX .................................................................................................................................................................................................................................................................................................................................................... 52
HPE OpenVMS ................................................................................................................................................................................................................................................................................................................................. 52
Technical white paper
Page 5
Introduction
Compaction technologies such as thin provisioning, Thin Deduplication, and thin reclamation offer efficiency benefits for primary storage that can
significantly reduce both capital and operational costs. Thin provisioning has achieved widespread adoption as it dramatically increases capacity
efficiencies. It has become a data center “must have” for its ability to break the connection between logical and physical capacity. Deduplication is
also an essential consideration when looking into deploying workloads onto a flash tier or an all flash array. Thin technologies can vary widely in
how they are implemented; some are complex to deploy, while others use coarse allocation units and cannot deliver the required space savings.
Not only is HPE 3PAR StoreServ Storage viewed as the industry’s thin technology leader, but third-party testing and competitive analysis confirm
that HPE 3PAR StoreServ offers the most comprehensive and efficient thin technologies among the major enterprise storage platforms. 1
HPE 3PAR Thin Technologies including HPE 3PAR Thin Deduplication, Thin Provisioning, Thin Conversion, Thin Persistence, and Thin Copy
Reclamation achieve advanced data compaction through leveraging built-in hardware capabilities and Express Indexing technology.
Thin provisioning allows a volume to be created and made available as a logical unit number (LUN) to a host without the need to dedicate
physical storage until it is actually needed. HPE 3PAR Thin Provisioning software has long been considered the gold standard in thin
provisioning for its simplicity and efficiency. Unlike other “bolt-on” implementations, HPE 3PAR Thin Provisioning software is simple and efficient,
helps your organization start new projects more quickly and on demand, and saves millions of dollars. HPE 3PAR Thin Provisioning leverages the
dedicate-on-write approach of HPE 3PAR StoreServ Storage, allowing enterprises like yours to purchase only the disk capacity you actually need.
HPE 3PAR Thin Provisioning integrates seamlessly with VMware® vSphere, Windows® Server 2012, Red Hat® Enterprise Linux® (RHEL), and
Symantec Storage Foundation—greatly enhancing the operative and administrative efficiency of these platforms.
While HPE 3PAR Thin Technologies software is extremely simple to deploy and use, a certain amount of planning is advantageous to help
maximize its benefits. This paper documents best practices on thin provisioning on HPE 3PAR StoreServ Storage and is intended for
administrators looking to get the most out of their HPE 3PAR StoreServ deployment. In addition, it describes other HPE 3PAR Thin Technologies
that you can use in conjunction with HPE 3PAR Thin Provisioning software to help maximize its effectiveness. Unique to HPE 3PAR StoreServ,
HPE 3PAR Thin Conversion software enables you to reduce capacity requirements by 50 percent or more by deploying HPE 3PAR StoreServ in
place of legacy storage. 2
HPE 3PAR Thin Persistence software and other thin-reclamation solutions enable thin-provisioned storage on HPE 3PAR StoreServ arrays to
stay thin over time by helping ensure that unused capacity is reclaimed for use by the array on an ongoing basis.
Now, HPE 3PAR Thin Deduplication and related HPE 3PAR Thin Clones software take thin efficiency to the next level. In addition, HPE 3PAR
Thin Technologies protect SSD performance and extend flash-based media lifespan while ensuring resiliency.
Overview of HPE 3PAR Thin Technologies for data compaction
Product highlights
• HPE 3PAR Thin Technologies are completely automated.
• HPE 3PAR Operating System (OS) software uses a reservationless, dedicate-on-write approach to thin technologies and virtual copies that
draws and configures capacity in fine-grained increments from a single free space reservoir without pre-dedication of any kind.
• HPE 3PAR Thin Technologies uses an allocation unit size of just 16 KiB, so you don’t have to worry about small writes consuming megabytes
or even gigabytes of capacity.
• HPE 3PAR StoreServ is a storage platform built from the ground up to support thin technologies by reducing the diminished performance and
functional limitations that plague bolt-on thin solutions.
1
2
HPE Thin Technologies: A Competitive Comparison, Edison Group 2012. theedison.com/pdf/Samples_HP_3PAR_Thin_Provisioning_WP.pdf
Based on documented client results that are subject to unique business conditions, client IT environment, Hewlett Packard Enterprise products deployed, and other factors. These
results may not be typical; your results may vary.
Technical white paper
Page 6
HPE 3PAR Thin Provisioning software
HPE 3PAR Thin Provisioning is the most comprehensive thin provisioning software solution available. Since its introduction in 2002, HPE 3PAR
Thin Provisioning has been widely considered the gold standard in thin provisioning. It leverages dedicate-on-write capabilities of HPE 3PAR
StoreServ to make storage more efficient and greener, allowing customers to purchase only the disk capacity they actually need and only when
they actually need it.
With Thin Provisioning software, there is no more up-front capacity allocation. No more dedicating resources for each application. No more
paying to house, power, and cool drives that might not be needed for months or years to come, or may seldom be needed at all.
HPE 3PAR Thin Deduplication software
HPE 3PAR StoreServ Storage systems employ purpose-built HPE 3PAR ASICs at the heart of each controller node that feature efficient, silicon-based
mechanisms to drive inline deduplication. This unique implementation relies on built-in hardware capability to assign a unique hash to any
incoming write request and leverages the HPE 3PAR Thin Provisioning metadata lookup table for fast hash comparisons. When new I/O requests
come in, the signatures of the incoming data are compared to the signatures of the data already stored in the array. When a match is found, the
software marks the data as duplicated. The software also leverages the controller node ASICs to perform a bit-to-bit comparison that reduces the
possibility of hash collision. The CPU-intensive job of calculating signatures of incoming data and read verify is offloaded to the hardware assist
engines, freeing up processor cycles to deliver advanced data services and service I/O requests.
This inline deduplication process carries multiple benefits, including increasing capacity efficiency, protecting flash performance, and extending flash
media lifespan. Other storage architectures lack the processing power to simultaneously drive inline deduplication and the high-performance levels
demanded by flash-based media while also offering advanced data services (replication, quality of service [QoS], and federation).
HPE 3PAR Thin Clones software
HPE 3PAR Thin Clones software is an extension of Thin Deduplication software for server virtualization environments, providing host-initiated
deduplication that produces non-duplicative virtual machine clones for Microsoft® Hyper-V and VMware ESXi. These VM clones are created
instantly by leveraging copy offload for VMware vStorage APIs for Array Integration (VAAI) and Microsoft Offloaded Data Transfer (ODX)
technology without increasing capacity consumption on the HPE 3PAR StoreServ Storage system. HPE 3PAR Thin Clones software leverages
Thin Deduplication to update the metadata table without copying data, using the inline deduplication technology to reduce capacity footprint as
new write requests come in.
A thin clone is a replica of a VM that is created through copying only the metadata that associates a virtual volume with the physical data on disk.
At initial creation, thin clones point to the same blocks of data as the cloned VM, however, as volumes are updated and the content of data
changes, new writes will map to different deduplicated blocks (or create new blocks), so no direct overwrite process occurs. Thin clones continue
to “stay thin” if updated data continues to map to existing deduplicated data on the array.
HPE 3PAR Thin Persistence software
HPE 3PAR Thin Persistence software is an optional feature that keeps thin volumes (TVs) and read/write snapshots of TVs small by detecting
pages of zeros during data transfers and not allocating space for the zeros. This feature works in real time and analyzes the data before it is
written to the source TV or read/write snapshot of the TV. Freed blocks of 16 KiB of contiguous space are returned to the source volume, freed
blocks of 128 MiB of contiguous space are returned to the common provisioning group (CPG) for use by other volumes. With HPE 3PAR Thin
Persistence, HPE 3PAR StoreServ Storage customers can now leverage next-generation space reclamation technology to help minimize storage
total cost of ownership (TCO).
HPE 3PAR Thin Conversion software
With HPE 3PAR Thin Conversion, a technology refresh no longer requires a terabyte-for-terabyte replacement, but instead offers the opportunity
to reduce up to 50 percent of your legacy capacity, simply and rapidly. This savings alone can save you up to 60 percent on the cost of a
technology refresh.
With HPE 3PAR Thin Conversion, you can quickly shrink your storage footprint, reduce storage TCO, and meet your green IT targets. HPE 3PAR
Thin Conversion software makes this possible by leveraging the zero-detection and inline deduplication capabilities within the HPE 3PAR ASIC
and HPE 3PAR Thin Engine (a unique virtualization mapping engine for space reclamation) to power the simple and rapid conversion of
inefficient, “fat” volumes on legacy arrays to more efficient, higher-utilization “thin” volumes. Getting thin has never been so easy.
Technical white paper
Page 7
HPE 3PAR Thin Conversion software is an optional feature that converts a fully provisioned volume to a thin provisioned or Thin Deduplication
volume during data migration. Virtual volumes (VVs) with large amounts of allocated but unused space are converted to thin volumes that are
much smaller than the original volume. During the conversion process, allocated but unused space is discarded and the result is a volume that
uses less space than the original volume.
HPE 3PAR Thin Copy Reclamation software
An industry first, Thin Copy Reclamation keeps your storage as lean and efficient as possible by reclaiming unused space resulting from deleted
virtual copy snapshots and remote copy volumes. Thin Copy Reclamation is built on a virtualization mapping engine for space reclamation called
HPE 3PAR Thin Engine, which is included as part of the HPE 3PAR OS. HPE 3PAR Thin Copy Reclamation software is an optional feature that
reclaims space when snapshots are deleted from a system.
As snapshots are deleted, the snapshot space is reclaimed from a TV or fully provisioned virtual volume and returned to the CPG for reuse by
other volumes. Deleted snapshot space can be reclaimed from virtual copies, physical copies, or remote copies.
HPE 3PAR ASIC with Thin Built In
At the heart of every HPE 3PAR StoreServ node there is an HPE 3PAR ASIC with Thin Built In which features an efficient, silicon-based zero-detection
and hashing mechanism. This unique hardware capability gives HPE 3PAR StoreServ Storage the power to perform inline deduplication and
remove allocated but unused space inline and non-disruptively.
The zero-detect capability can recognize an incoming write request of 16 KiB of zeros and either not allocate space for the zero block or free the
space if it was already allocated for that block. All this happens in cache, and therefore no zeroes are written to the backend. When a read request
comes in for a block that is unallocated, the HPE 3PAR StoreServ will immediately return zeros back to the host.
Many other storage arrays do not detect blocks of zeroes on write. Instead, the zeros are written to disk and a scrubbing process later detects
these zeroed blocks and discards them. With this approach, the zeroed blocks consume space until they’re scrubbed, and therefore they may not
be available for use by other volumes when needed. Also, there is increased load placed on the storage as the scrubbing process examines the
block contents on the physical storage.
The HPE 3PAR StoreServ built-in zero-detection capability can be controlled per TV, and it is enabled by default.
The HPE 3PAR ASIC is also the only solution in the industry with a built-in, silicon-based hash calculation engine. With HPE 3PAR Thin
Deduplication software, the process of calculating the hash signatures for incoming data and verifying reads are offloaded to the HPE 3PAR
ASICs, freeing up processor cycles to deliver advanced data services and service I/O requests. This hardware-assisted approach together with a
unique Express Indexing feature enables extremely efficient, extremely granular block-level inline deduplication that carries multiple benefits,
including increased capacity efficiency, flash performance protection, and flash media lifespan extension. Unlike other approaches, HPE 3PAR
Thin Deduplication software performs a full check on all data before marking it as duplicated, which is essential to ensuring data integrity for
mission-critical environments.
The benefits of Thin Provisioning
Thin Provisioning can be used with nearly all applications available in the market to improve storage utilization dramatically. The following use
cases illustrate the superior value from Thin Provisioning on HPE 3PAR StoreServ.
Avoid frequent storage capacity addition to servers
Storage administrators often allocate and present a large amount of storage to a server at the start of a project to accommodate its long-term
growth requirements. This practice reduces the number of times that LUNs mapped to the server have to be expanded, an operation that can be
complex, time consuming, and cause server downtime. With the HPE 3PAR Thin Provisioning software, it is possible to allocate large amounts of
storage to a particular server but only consume physical space on the array as used. This fits environments where:
• The addition or expansion of storage provisioned to servers is not desired, for example, in mission-critical environments.
• Relatively slow growth rates or unpredictable growth over time occurs. This can happen on large file systems used for group shares, mailbox
databases, or general database space.
Technical white paper
Page 8
Accelerate time to market
In an application’s development stage, there is sometimes a requirement for the application storage to be in place and ready before the
application goes live. With the HPE 3PAR Thin Provisioning software, it is possible to present the full amount of storage immediately so that it is
ready for the developers to work on without the requirement for the full physical capacity being in place. Because thin provisioning gives storage
administrators the ability to place limits on TPVVs and CPGs, the administrator can make sure that the developer’s work does not affect other
production systems by using the free space within the HPE 3PAR StoreServ. After the application is ready to go live, these limits can be removed.
Chargeback model in utility storage
HPE 3PAR Thin Provisioning is ideal for enterprise customers and service providers wishing to deploy a storage offering where usage
chargeback is an important component of the service. Thin Provisioning offers these customers the following benefits:
• Deploy quickly
• Decouple the charges for storage from the limitations of actual presented storage
• Remove the exposure to disruption of service during future capacity expansions
• Collect detailed charging data at the individual VV level or at the CPG level
When planning to collect charging data, it is recommended that the names chosen for objects like CPGs, VVs, snapshots, and domains contain a
meaningful prefix or suffix referring to the project, application, line of business (LOB), or department the objects belong to. This enables the
grouping of objects in the chargeback report. The HPE 3PAR OS allows up to 31 characters for the name of an object.
Thin Deduplication
Deduplication has become standard with disk-based backup due to a high degree of data redundancy and less emphasis on high performance,
backup, and archive workloads have been an ideal target for deduplication technologies. Traditional primary storage workloads such as OLTP
have lower data redundancy and hence lower deduplication ratios and therefore deduplication of primary storage has not been seen as
beneficial. However, the landscape around primary storage deduplication is changing. The high data redundancy of server virtual machine (VM)
images and client virtualization environments with hosted virtual desktops have tremendous potential for benefiting from deduplication. Home
and file directory consolidation is another area where primary storage deduplication can offer significant space savings.
With the increasing use of SSD storage, deduplication for primary storage arrays has become critical. The cost differential between SSDs and hard
disk drives (HDDs) requires compaction technologies like thin provisioning and deduplication to make flash-based media more cost-efficient. The
widespread deployment of server virtualization is also driving the demand for primary storage deduplication.
The following should be taken into account before implementing Thin Deduplication:
• It is only applicable to virtual volumes residing solely on SSD storage. Any system with an SSD tier can take advantage of Thin Deduplication.
Since a Thin Deduplication can only be on SSD storage, they are not compatible with the sub-LUN tiering of Adaptive Optimization (AO). If a
thinly deduped volume exists within a common provisioning group (CPG) then the CPG is not available for use in an AO configuration.
Conversely, if a CPG is already in an AO configuration then it is not possible to create a thinly deduped volume in the CPG.
• The granularity of deduplication is 16 KiB and therefore the efficiency is greatest when I/Os are aligned to this granularity. For hosts that use
file systems with tunable allocation units consider setting the allocation unit to 16 KiB or a multiple of 16 KiB. For more information on Thin
Deduplication and file systems, see Appendix C. For applications that have tunable block sizes consider to setting the block size to 16 KiB or a
multiple of 16 KiB.
• Deduplication is performed not only on the data contained within the virtual volumes but also between virtual volumes in the same CPG. For
maximum deduplication store data with duplicate affinity on virtual volumes within the same CPG.
• Thin Deduplication is ideal for data that has a high level of redundancy. Data with a low level of redundancy such as databases or data that has
been previously been deduplicated, compressed, or encrypted are not good candidates for deduplication and should be stored on thin
provisioned volumes. Use the Dedup Estimate tool to check the dedup ratio of existing volumes before conversion to thinly deduped volumes.
Technical white paper
Page 9
Using fully provisioned volumes
The use of thin provisioning has minimal performance impact and has the significant operational benefit of reducing storage consumption.
However, there are certain workloads and applications thin provisioning may not be of benefit such as:
• Systems with a high file system utilization—File systems on TPVVs that are nearly full offer reduced benefits of thin provisioning. This relates
to the fact that, for thin volumes, the thin-provisioning license charge is incurred in addition to the cost for physical disk capacity used. In the
case of file system utilization rates of 80 percent or higher, it may be more cost-efficient to use fat-provisioned volumes to hold the data.
• Applications that write continuously to new space—An example of this is Oracle redo log file.
• Databases not in “autoextend” mode (or equivalent) —Some databases initialize their assigned storage at creation time by writing markers at
regular intervals over the entire volume. This has the same effect as provisioning file systems with a high utilization on thin volumes.
• Small capacity requirements—Thin Provisioning is ideal for large-scale volumes. For small size VVs (256 MB up to a few tens of GB), even the
minimum growth increment of the CPG may mean that minimal benefit is realized. Use care in the selection of the CPG growth increment in
this case.
• Environments that require host encrypted volumes—Writing blocks of zeros to a host encrypted volume on a newly created HPE 3PAR
StoreServ thin-provisioned volume will cause space to be allocated on the TPVV because the encryption alters the content of the blocks.
Applying encryption to thin-provisioned volumes that already contain data or rekeying them also inflates the zero blocks, making the volume
consume space as if it was fully provisioned. Attempting to re-thin the volume by writing zeros to allocated but unused space is not possible as
well. As a result, host encryption and thin provisioning do not cooperate well.
• Environments that require SAN encrypted volumes—Like host-based encryption, encryption by a device in the data path (e.g., SAN switch) will
also alter the data stream so that blocks of zeros written by the host are not passed onto the storage. A notable exception is Brocade SAN
switches. With the introduction of Fabric OS 7.1.0, the Fabric OS encryption switch can automatically detect if a disk LUN is a thin-provisioned
LUN. If a LUN is detected as being thin-provisioned, then first-time encryption and rekey are done on the allocated blocks only. This thin
provision LUN support requires no action by the user.
• Copy-on-write file systems—File systems that write to new blocks rather than overwrite existing data are not suitable for thin provisioning, as
every write will allocate new storage until the volume is fully allocated. An example of a copy-on-write (CoW) file system is Oracle Solaris ZFS.
However, if a significant part of the array will be utilized for thin volumes, it is advised to use thin provisioning for all volumes, to help minimize
management overhead and help maximize space efficiency.
System requirements
HPE 3PAR Thin Technologies are included as part of the HPE 3PAR OS Software Suite and are available on all HPE 3PAR StoreServ models. The
functionality offered by each system license is summarized in table 1.
Table 1. Summary of the thin storage technology features
LICENSE
FUNCTIONALITY
Thin Provisioning
The creation of TPVVs
Thin Deduplication
Creation of TDVVs for inline deduplication on SSD Tier and Thin Clones for host-initiated deduplication on TDVVs
Thin Persistence
Zero-detection to free previously allocated space
Thin Conversion
Zero-detection to prevent allocation of space and enables T10 UNMAP support
Thin Copy Reclamation
Reclaiming unused space resulting from deleted virtual copy snapshots and remote copy volumes
Technical white paper
Page 10
HPE 3PAR Volume Manager
The HPE 3PAR OS has a volume manager that provides the virtual volume abstraction. It is comprised of several layers with each layer being
created from elements of the layer below.
Physical disks
Every physical disk (PD) that is admitted into the system is divided into 1 GB chunklets. A chunklet is the most basic element of data storage of
the HPE 3PAR StoreServ. These chunklets form the basis of the RAID sets; depending on the sparing algorithm and system configuration, some
chunklets are allocated as spares.
Logical disks
The logical disk (LD) layer is where the RAID functionality occurs. Multiple chunklet RAID sets, typically from different PDs, are striped together
to form a LD. All chunklets belonging to a given LD will be from the same drive type. LDs can consist of all Nearline (NL), Fibre Channel (FC), or
solid-state drive (SSD) type chunklets.
There are three types of logical disk:
1. User (Usr) logical disks provide user storage space to fully provisioned virtual volumes.
2. Shared data (SD) logical disks provide the storage space for snapshots, virtual copies, thin provisioned virtual volumes (TPVVs) or thin
duplicated virtual volumes (TDVVs).
3. Shared administration (SA) logical disks provide the storage space for snapshot and TPVV/TDVV administration.
They contain the bitmaps pointing to which pages of which SD LD are in use. The LDs are divided into “regions,” which are contiguous 128 MiB
blocks. The space for the virtual volumes is allocated across these regions.
Common provisioning groups
The next layer is the CPG that defines the LD creation characteristics, such as RAID type, set size, disk type for chunklet selection, plus total space
warning, and limit points. A CPG is a virtual pool of LDs that allows volumes to share resources and to allocate space on demand. A thin-provisioned
volume created from a CPG will automatically allocate space on demand by mapping new regions from the LDs associated with the CPG. CPGs and
their importance to thin provisioning will be discussed in more detail in the next section.
Virtual volumes
The top layer is the virtual volume (VV). VVs draw their resources from CPGs, and are the only data layer visible to hosts when they are exported
as virtual logical unit numbers (VLUNs).
A VV is classified by its type provisioning which can be one of the following:
• Full—Fully Provisioned VV, with space for the base volume allocated from the associated user CPG and no snapshot space.
• CPVV—Commonly Provisioned VV. A fully provisioned VV with space for the base volume allocated from the associated user CPG and
snapshot space allocated from the associated snapshot CPG.
• TPVV—Thin Provisioned VV, with space for the base volume allocated from the associated user CPG and snapshot space allocated from the
associated snapshot CPG (if any).
• TDVV—Thin Deduplicated VV, with space for the base volume allocated from the associated user CPG and snapshot space allocated from the
associated snapshot CPG (if any).
On creation of a thin volume (TPVV or TDVV), the size of the VV is specified, but no storage is allocated. Storage is allocated on demand in the
shared data area as required by the host operation being performed. The shared admin area contains the metadata indexes that point to the
user data in the SD area. Since the SA metadata needs to be accessed to locate the user data, the indexes are cached in policy memory to help
minimize the performance impact of the lookups.
Technical white paper
Page 11
Thin volumes associated with the same CPG share the same LDs and draw space from that pool as needed, allocating space on demand in small
increments for each controller node. As the volumes that draw space from the CPG require additional storage, the HPE 3PAR OS automatically
creates increase in the logical disk storage until either all available storage is consumed or if specified, the CPG reaches the user-defined growth
limit, which restricts the CPG’s maximum size. The size limit for an individual virtual volume is 16 TiB.
The relationship between the HPE 3PAR OS abstraction layers is illustrated in figure 1.
Figure 1. Overview of thin virtual volumes
The anatomy of a thin volume
The sparse nature of thin volume requires a mechanism to map the virtual volume address space to physical storage pages, which are 16 KiB in
size. The HPE 3PAR OS does this by using page tables, which are similar in concept to the virtual memory mapping page tables of OSs. The
logical block address (LBA) of a SCSI read or write command is then used as an index into three different page tables to find a particular 16 KiB
block belonging to a TV. New writes will result in a free 16 KiB page being allocated (or multiple 16 KiB pages for larger writes); rewrites of data
will simply reuse any previously allocated space and writes of 16 KiB or larger of zeros will result in space reclaim.
The address space of a TV is also assigned across all the nodes in the system in 32 MiB increments in a round-robin manner. The LBA for a new
write I/O therefore determines which node the I/O will be directed to and this aids the load balancing of I/Os across all nodes in the system.
The system also needs a mechanism to track TV SD page usage since space reclaim creates unused “holes” in the 128 MiB regions allocated to
particular TVs. To do this, there are two levels of bitmaps used to track the TV used space. The SD logical disks are made up of 128 MiB regions
that contain 8,192 16 KiB pages. The first level (L1) bitmap indicates whether a given 128 MiB region in SD space is in use or not. If a region is in
use then a second level (L2) bitmap indicates whether a particular 16 KiB page in that region is in use or not. The relationship between regions,
pages, and bitmaps is shown in figure 2.
As individual 16 KiB pages are freed, the space in the SD LDs can become fragmented and a defragment thread periodically examines the
bitmaps to determine whether defragmentation should be initiated. The data from partially allocated regions is then reorganized to create larger
contiguous blocks of free space that can be reclaimed.
Technical white paper
Page 12
Figure 2. Overview of the SD space tracking bitmaps
Thin volume metadata
The user data from thin volumes are stored in SD LDs that use the RAID type and set size defined by the CPG they have a naming convention of
tp-x-sd-y.z. The bitmap and page table metadata indexes that point to the user data in the SD LDs are stored in shared administration (SA)
logical disks. Since the SA LDs are critical to the system, they are three-way RAID 1 protected with high availability (HA) cage availability, and
they have a naming convention of tp-x-sa-y.z. In addition, the SA metadata indexes are cached in policy memory to help minimize the
performance impact of the lookups.
Although there is SA space in addition to the SD space, the SA size is not a simple percentage of the SD size. The first TPVV/TDVV created in a
CPG will have 8 GB per node-pair of SA LD space created (due to the three-way RAID 1 protection this will be 24 GB of raw space) for the
metadata indexes. As the volume is populated, or more thin volumes added, the SD space usage will increase (more LDs may be added or the
existing LDs expanded) but the amount of SA space will not expand until the initial 8 GB allocation is used. The CPG will then grow the SA space
based on the growth parameters that have been set.
On a small system with a few thin volumes, the SA space may be as much as 10 percent of the SD space, but on a medium-large system, the SA
space used is typically only 1 percent of the SD space.
Thin Deduplication implementation
Thin Deduplication is an extension of industry leading HPE 3PAR Thin Provisioning. With primary storage, it is very important to keep to I/O
latencies as low as possible. Therefore, the process of transferring host data to the array is the same for both normal thin provisioned volumes
and Thin Deduplication volumes. The deduplication is performed inline as part of the process of flushing the acknowledged I/Os from cache.
A new volume type called TDVV is used for thinly deduplicated virtual volumes. TDVVs in the same CPG will share common pages of data. A thin
provisioned volume called the Dedup Store (DDS) is automatically created when the first TDVV is created from a CPG and it is used to hold the data
(both unique and duplicate) written to the TDVVs. The calculation of the hash, which is normally a CPU intensive task, is offloaded to the ASIC. If a
hash match is found, the ASIC is used to do a bit level comparison of the new data with the existing data to ensure no hash collision has occurred.
Technical white paper
Page 13
Figure 3. ASIC-based signature generation for inline deduplication
Express Indexing
HPE 3PAR Thin Deduplication uses an advanced metadata lookup mechanism called Express Indexing which is unique as it combines the built-in
hash generation capability of the HPE 3PAR ASIC with HPE 3PAR Thin Provisioning metadata lookup table for extremely fast hash comparisons.
Once the hash signature of the incoming data has been generated, there must be a check to see if data with the same signature already exists.
This is typically a CPU and memory intensive operation involving search mechanisms on large pools of reserved memory containing the
signatures of existing data. The HPE 3PAR OS instead uses a technique called Express Indexing to detect duplicate page data. This process
takes advantage of the highly optimized and robust address space to physical storage page indexing mechanism of thin provisioned volumes.
When a new write I/O request comes in, the Logical Block Address (LBA) is used as an index into page tables as per a regular TPVV but instead
of allocating a new page the hash signature of the incoming data page is computed by the HPE 3PAR ASIC and compared to the signatures of
the data already stored in the CPG. If a match is found then the existing block is compared at a bit level with the new block by the ASIC. A
successful comparison is a “dedup hit,” in which case the virtual volume pointers are updated to reference the existing data. In the unlikely event
a hash collision is detected, then the data is stored in the virtual volume and not treated as duplicate. If the hash of the new data was not found
by the lookup, a new page is allocated in the DDS. The value of the hash is used as the offset of the page in the DDS therefore a simple a page
translation will turn the hash into a physical page location.
When a read is performed, the page translation of the TDVV LBA can point either to the local store or to the DDS volume.
Technical white paper
Page 14
Common provisioning groups
CPGs are policies for how free chunklets within the HPE 3PAR StoreServ should be used when creating volumes. A CPG policy contains
parameters such as disk type, disk speed, RAID type, growth increment, chunklet radial placement, and availability level. CPGs automatically grow
the underlying LD storage, according to the stated growth parameters, on demand to store data in a TPVV or TDVV. No administrator
intervention is needed with this provisioning operation.
CPGs and workloads
HPE 3PAR StoreServ performs more efficiently for any type of workload, and different workloads can be mixed on the same array. These different
workloads may need different types of service levels to store their data. For example, for high-performance mission-critical workloads, it may be
best to create volumes with RAID 5 protection on SSDs or RAID 1 protection on Fast Class (FC or SAS performance hard disk drives [HDDs]). For
less-demanding projects, RAID 5 on FC drives or RAID 6 on NL drives may suffice. For each of these workloads, you can create a CPG to serve as
the template for creating VVs for the workload. It is most efficient to keep the number of CPGs low as each CPG reserves its amount of growth
space. All virtual volume types can be moved between CPGs with the HPE 3PAR Dynamic Optimization (DO) software command tunevv,
thereby changing their underlying physical disk layout and hence their service level. The same Dynamic Optimization technology can be used to
convert volume types; this is described in greater detail later in the paper.
The following sections discuss what to consider when planning CPGs for thin provisioning and virtual copy snapshots and recommend a set of
best practices.
Availability level
To provide HA, chunklets from the same RAID set should be distributed across multiple components. 3
There are three levels of availability that can be selected with HPE 3PAR StoreServ.
• HA CAGE means that no two members of the same RAID set can be in the same drive enclosure. For example, to support RAID 5 3+1 (set size
four), four drive chassis connected to the same node pair are required. This helps ensure that data is still available in the event that access to
an entire drive cage is lost. This applies to drive chassis that are point-to-point connected to the nodes (no daisy chain).
• HA MAG means that no two members of the same RAID set are in the same drive magazine. This allows a wider stripe with fewer drive chassis;
for example, a RAID 5 stripe size of 7+1 (set size eight) would be possible with only four drive chassis, provided each chassis had at least two
drive magazines.
• HA PORT applies only to daisy-chained drive chassis. When this level of availability is selected, no two members of the same RAID set can be
in drive chassis that are dependent on one another for node connectivity. For example, in a system in which there are eight drive chassis with
four of the drive chassis connected to another drive chassis for node access, HA PORT would only allow RAID 5 3+1 (set size four) in order to
prevent the loss of one drive chassis from causing a loss of data access. On systems that do not have daisy-chained cages, such as the
HPE 3PAR StoreServ 10000, setting HA PORT is the same as setting HA CAGE.
When creating CPGs, it is strongly recommended to select HA CAGE availability always.
3
It is important to understand that drive magazines consist of four drives for HPE 3PAR StoreServ 10000. Drive magazines consist of only a single drive in the HPE 3PAR StoreServ
7000 and 7450 Storage systems.
Technical white paper
Page 15
Reasons to create multiple CPGs
All TPVVs and TDVVs associated with a CPG allocate space from a shared pool of LDs. This means that VVs associated with a particular CPG
have identical LD characteristics. VVs that require different characteristics must use a different CPG.
Reasons to create multiple CPGs include:
• To define different service levels, e.g., RAID 1, RAID 5, or RAID 6 with FC, NL, or SSDs.
• To map VVs belonging to different lines of business, departments, or customers onto particular CPGs for reporting and management
purposes; creating CPGs with exactly the same characteristics but a different name is possible. This allows a logical separation of resources
and may help with chargeback models, as chargeback could be based on CPG space usage rather than usage at an individual VV level.
• There can be multiple deduplication CPGs in the system, each deduplication CPG providing storage for different sets of volumes. This allows
volumes with similar datasets to be grouped together and facilitates deduplication at the virtual domain level in a multi-tenancy environment.
• When virtual domains are used, because a CPG can only belong to one virtual domain.
• When HPE 3PAR Adaptive Optimization (AO) software is used, because a CPG can only belong to one Adaptive Optimization policy.
• When using Thin Deduplication there is a limit of 2048 TDVVs per CPG on HPE 3PAR StoreServ 20000 systems and 1024 TDVVs per CPG on
all other HPE 3PAR StoreServ systems. 4
While there are several reasons to create multiple CPGs, it is recommended that the number of CPGs be kept to a minimum as each CPG will
reserve its own growth space. For deduplication, the recommended maximum number of CPGs is shown in table 2.
Table 2. Recommended maximum dedup CPGs
SYSTEM TYPE
MAXIMUM DEDUP CPGs PER NODE PAIR
HPE 3PAR StoreServ 7200/7200c/7400/7400c
2
HPE 3PAR StoreServ 7450/7440c/7450c
4
HPE 3PAR StoreServ 8200/8400
2
HPE 3PAR StoreServ 8440/8450
8
HPE 3PAR StoreServ 20000 series
12
CPG automatic growth
By default, CPGs dynamically allocate storage in increments specified by the CPG’s growth increment. This on-demand allocation unit determines
the automated response to the growth demands of TPVVs and TDVVs. The growth increment should be large enough to ensure wide striping
across most or all physical disks that are part of the CPG. To grow the TPVV/TDVV, the HPE 3PAR OS may expand existing LDs according to the
CPGs growth increment or create additional ones. Growth is triggered when the CPG’s available space falls below 85 percent of the growth
increment value. The CPG growth increment can be changed at any time, which also changes the threshold for the next growth increment to
happen. A mechanism with warnings and limits can be configured on the array to control the growth of a CPG. When the growth increment is set
to zero, the CPG does not grow automatically. If the CPG cannot grow then when the free space in the CPG is exhausted I/Os that require space
to be allocated will fail.
4
Prior to 3.2.2 MU2 the limit was 256 TDVVs per CPG for all HPE 3PAR StoreServ systems.
Technical white paper
Page 16
The default and the minimum growth increment for a CPG depend on the number of nodes in the system. Table 3 lists the default growth
increment and its limits for different numbers of node pairs in a system. The maximum growth increment for a CPG is 2,047.750 GB being 2 TB
minus 256 MB.
Table 3. Default and limits for the growth increment per node pair
NUMBER OF NODES
DEFAULT (GB)
MINIMUM (GB)
MAXIMUM (GB)
2
32
8
2,047.75
4
64
16
2,047.75
6
96
24
2,047.75
8
128
32
2,047.75
Considerations when selecting a CPG growth increment for a CPG include:
• The CPG growth increment can be changed at any time.
• The grow size of a TDVV is 25 percent of the CPG grow size.
• To determine the optimal growth increment for your environment, review the following factors influencing the value for it:
– With the default growth increment of 32 GB per node pair and 1 GB chunklets, the new CPG growth will be spread across 16 disks on each node.
– If there are less than 16 drives of the device type associated with the CPG connected to each node type, then consideration should be given
to reducing the CPG growth increment to help minimize the amount of reserved.
– Growth space—This can be important for CPGs created on SSDs as they often account for a minority of the space within the array.
– If there are more than 16 drives of the device type associated with the CPG connected to each node type, then it may seem tempting to
increase the growth increment to spread the data across more disks. This is not necessary, as the next growth will use chunks from the
remaining drives. It is therefore recommended to keep the growth at the default value.
– However, if the environment is write-intensive, the rate of consumption of CPG space might be significant. In this case, it is recommended
that the growth increment be set to a value above the default value listed in table 3.
Changing LD characteristics within a CPG
LD space within a CPG is automatically allocated to thin volumes as needed. Storage administrators have no direct control over which LDs are
used for a particular allocation. Although the capability exists to modify the LD characteristics for new LDs within a CPG, its use is advisable only
in rare instances, for example, when different LD characteristics would enable the CPG to fully utilize the remaining available physical capacity.
The changes to CPGs characteristics will only be reflected on newly created LDs, the layout of existing LD will not change. For this to occur the
administrator must use the tunevv or tunesys commands.
Snapshot space within CPGs
A thin volume can be associated at creation time or at a later stage with snapshot space, which is used to store data changed in the original volume.
The space for one or more snapshots or virtual copies can originate from the same CPG as for the user space in the TV or from a different one. The
CPG for the snapshot space of a TV can be changed anytime later using the tunevv command. The snapshot space grows on-demand with growth
increments as defined by the CPG in which it was created. This association mitigates the planning required to estimate and dedicate capacity upfront
for copy-on-write snapshots of a volume.
Technical white paper
Page 17
Allocating CPG space to thin volumes
Each TPVV or TDVV associated with a CPG allocates space from the following sources in the listed order:
• Unallocated LD space in CPGs—LD space is shared by all the thin volumes associated with the CPG. This space can grow to a considerable
amount as the SA and SD space of any VV associated with the CPG is returned to the CPG as unallocated LD space upon VV removal or when
the free space command is issued for a volume. The returned, unallocated LDs are re-mapped to the SD space of associated TPVVs or their
snapshots as needed over time. Unallocated LD space for a CPG can be displayed using the command showld -cpg <CPG>. Newly created
LDs to accommodate thin volume growth will show as unused initially.
• Free chunklets—Free chunklets available to a CPG for creating new LDs may be limited by the LD creation parameters for the CPG. It is
important to understand that if different CPGs can draw from the same pool of chunklets, the chunklets allocated to one CPG will reduce the
pool of storage available to the other CPGs. It is recommended that storage administrators implement the following strategies to stay abreast
of available free space:
– Set the CPG allocation warnings (and limits, if necessary)
– Monitor the rate of free space reduction using the command showspace -cpg <CPG> -hist
– Set the free space capacity warning with the setsys RawSpaceAlertXX command (as detailed in “allocation warnings” section later in
this paper)
– Monitor available free space alerts set by the showalert command
Sizing guidelines
What is the ideal size for a thin volume? There is no definite answer to this, but you should consider the following when deciding on the size for
thin volumes:
• The minimum size for a thin volume is 256 MB; the maximum size is 16 TB for all HPE 3PAR OS versions on all types of HPE 3PAR StoreServ systems.
• Thin Provisioning demonstrates the highest value in situations involving large-scale consolidation. For small virtual volumes (256 MB up to a
few tens of GB), the growth increment of the CPG of the TPVV may be many times higher than the TPVV size which means that minimal
benefit is realized after one growth increment is applied.
• It is possible to increase the size of a thin volume. This provides a significant amount of flexibility. However, the impact of growing volumes for
host-based OS needs to be considered. It is not possible to shrink a TPVV.
• When faced with a choice, it is preferable to make volumes larger than needed over making them too small. If the volumes that are presented
are too small for the ongoing requirements, then TPVV growth or additional volumes will be required in the future, which is something that
needs to be managed.
Thin volumes administration
Over-provisioning capacity to hosts is one of the prime reasons why people choose to create thin volumes. It is therefore essential to be in
control of the space use to ensure that the allocated capacity does not reach the physical capacity of the system. Natural growth or a process
writing excessive amounts of data to VVs can cause a CPG or the entire array to run out of space potentially causing an interruption of service
for applications consuming space from that CPG or array.
While HPE 3PAR Thin Technologies are simple to use, a certain level of management is required to help maximize its benefits. HPE 3PAR Thin
Technologies offers a comprehensive set of tools that not only allows storage to start thin, but also to stay thin.
A key component of successful over-provisioning is the implementation of a space reclaim regime to leverage HPE 3PAR Thin Persistence capability
so that there is a constant ebb and flow of space use and reclaim. See the HPE 3PAR Thin Persistence software section for more details.
The extensive reporting capabilities enable storage administrators to monitor space usage trends and predict future requirements while the
alerting functions allow for timely interventions to remedy unexpected capacity shortages.
Technical white paper
Page 18
Allocation warnings and alerts
The HPE 3PAR OS provides multiple categories of alerts that notify storage administrators of important events to help ensure the smooth
running of an HPE 3PAR StoreServ. Figure 4 illustrates the categories for which warnings can be set.
Figure 4. HPE 3PAR thin volume alert options
These include:
• Allocation warnings and limits for TPVVs
• Growth warnings and limits for CPGs
• Used physical capacity alerts
• Free physical capacity alerts
Allocation warnings
Allocation warnings provide a mechanism for informing storage administrators when a specific capacity threshold is reached. An allocation
warning can be specified independently for each TV and each CPG. It is recommended that allocation warnings be used, at least on the CPG
level, and acted upon when they are triggered.
The relevant CLI commands for setting allocation and growth warnings are:
setvv
-usr_aw
<percent>
<TV>:
sets the allocation warning for the user space of the TV as a percentage of the TV size
setvv
-snp_aw
<percent>
<TV>:
sets the allocation warning for the snapshot space of the TV as a percentage of the TV size
setcpg -sdgw
<num>
<CPG>:
sets the growth warning for the CPG in MB (append to the value num “g” or “G” for GB or “t” or
“T” for TB)
These warnings can be changed at any time and are effective immediately. The CLI commands showvv -alert and showcpg -alert lists
the allocation warnings that were set per TV and CPG.
Technical white paper
Page 19
Allocation limits
Applications sometimes get into an abnormal state writing data continuously to the storage device. Allocation limits provide a mechanism to
prevent such “runaway” applications from consuming disk capacity beyond a specified threshold. Allocation limits can be specified independently
for each TV and each CPG. For a TV, after the allocation limit is reached, the capacity allocated to the TV stops growing and new writes by the
application fail. Similarly, for a CPG, after the allocation limit is reached, the automatic creation of new LDs, if configured, is disabled.
The relevant CLI commands related to setting allocation and growth limits are:
setvv
-usr_al
<percent>
<TV>:
sets the allocation limit for the user space of the TV as a percentage of the TV size
setvv
-snp_al
<percent>
<TV>:
sets the allocation limit for the snapshot space of the TV as a percentage of the TV size
setcpg
-sdgl
<num>
<CPG>:
sets the growth limit of the CPG in MB (append to the value num “g” or “G” for GB, or “t” or
“T” for TB)
These alerts can be changed at any time and are effective immediately. The CLI commands showvv -alert and showcpg -alert list the
allocation limits that were set per TV and CPG.
The VV allocation limits and warnings can be set with the HPE 3PAR Management Console by selecting Show advanced options checkbox
when creating or editing a VV as shown in figure 5.
Figure 5. VV allocation limit and warning options
The CPG growth limits and warnings can be set with the HPE 3PAR Management Console by selecting Show advanced options checkbox when
creating or editing a CPG as shown in figure 6.
Figure 6. CPG allocation limit and warning options
It is important to note that the growth limit for a CPG is a hard limit and the CPG will not grow beyond it. Once the CPG hard limit has been
reached any VVs that require more space will not be able to grow. This will result in write errors to host systems until the CPG allocation limit is
Technical white paper
Page 20
raised. Therefore, it is recommended that TV, CPG, and free space warnings and limits are set to sensible levels and managed when they are
triggered. As an example, the CPG warning limit should be set sufficiently below the CPG allocation limit so that it alerts the storage administrator
with ample time to react before the CPG allocation limit is reached.
Remaining physical capacity alerts
As available, physical capacity across the HPE 3PAR StoreServ Storage gets consumed by VVs, preconfigured alerts are generated at
50 percent, 75 percent, 85 percent, and 95 percent of physical capacity in use per drive type (FC, NL, or SSD). Furthermore, the storage
administrator can use the CLI command setsys as follows to set another warning level when the available space within the system falls
below a custom-defined capacity point:
setsys
RawSpaceAlertFC
<value>:
where value is the remaining capacity on FC disks in GB
setsys
RawSpaceAlertNL
<value>:
where value is the remaining capacity on NL disks in GB
setsys
RawSpaceAlertSSD
<value>: where value is the remaining capacity on SSD disks in GB
These serve as array-wide, advance warnings to the storage administrator to plan for and add necessary physical capacity. The alerts generated
should be monitored and promptly acted upon to prevent all free space of a particular drive type be consumed.
Remaining CPG free space alerts
HPE 3PAR OS samples the space available to CPGs once per day. The history of used and free CPG space is stored in an internal table and can
be displayed using the -hist option in the showspace and showcpg commands. An alert is automatically generated if the available free
space for a CPG falls below the CPG warning limit or the CPG allocation limit.
View logged warning and limit alerts
All warning and limit alerts mentioned above can be viewed in several ways:
• The CLI commands showalert and showeventlog list the alerts in various formats and with various options.
• The HPE 3PAR Management Console shows the alerts in the “Events” section.
• Storage Management Initiative software (SMI-S) integrated in HPE 3PAR OS provides asynchronous notification of events for changes in the
elements managed by the Common Information Model (CIM) server. A CIM client can subscribe to selected CIM indications to receive event
notifications from the CIM server.
• The SNMP agent within HPE 3PAR OS allows for retrieving the alerts by remote SNMP clients.
Alerts can be forwarded (setsys RemoteSyslogHost) to a log host for viewing them in an enterprise management application.
User recommendations
The monitoring of alerts for available capacity by storage administrators and internal business processes are a critical component of a successful
HPE 3PAR Thin Provisioning management and administration strategy. You should nominate a primary and if possible a backup storage
administrator for each site with HPE 3PAR StoreServ equipment. The storage administrator’s roles include:
• Proactively monitor free space availability per TV and CPG.
• Proactively monitor consumption rates for TVs and CPGs.
• Proactively monitor consumed TV capacity and compare to licensed thin provisioning capacity.
• Proactively monitor physical capacity thresholds for each disk type and for the entire array.
• Ensure adequate purchasing and installation of additional physical disk capacity buffer and thin-provisioning license upgrades in a timely manner.
• Nominate an escalation contact who has proper authority to drive the customer responsibilities outlined in this document if the nominated
storage administrators fail to carry out their responsibilities.
Technical white paper
Page 21
If you have a network connection with HPE 3PAR Central via the Service Processor, the health of the HPE 3PAR StoreServ can be proactively
monitored for CPG growth problems, and you can request to receive thin provisioning and other alerts by mail or via phone. You retain
responsibility for managing the thin-provisioning capacity and CPGs; Hewlett Packard Enterprise is not responsible for any failure when
thresholds are met or exceeded.
Capacity efficiency
HPE 3PAR OS 3.2.1 MU1 introduces two metrics to measure the capacity efficiency of the HPE 3PAR Thin Technologies, compaction ratio, and
dedup ratio. The compaction ratio is how much physical storage space a volume consumes compared to its virtual size and applies to both thin
provisioned and thinly deduped volumes. The dedup ratio is how much physical storage space would have been used without deduplication,
compared to the actual storage space used by a thinly deduped volume. The ratios are shown as decimals and have an implied: 1 i.e., 4.00 is
actually 4:1 (4 to 1). The dedup ratio does not include savings from inline zero-detection.
These capacity efficiencies can be shown per volume, volume family, CPG, virtual domain or system and are in terms of usable storage (i.e., not
including RAID overhead). The efficiencies displayed will depend on the scope of the request. For example, it is likely that the dedup ratio of a
CPG will be higher than that of the individual TDVVs because the data in the Dedup Store can be shared by multiple TDVVs. Note that the
capacity efficiency ratios are not dynamically updated and therefore changes to the data may not be immediately reflected in the capacity
efficiency ratios displayed.
Base volumes
The capacity efficiencies of a base volume are shown by the showvv -space command and are calculated as follows:
• The compaction ratio of a TPVV is the virtual size of the volume divided by its used data space.
• The compaction ratio of a TDVV is the virtual size of the volume divided by the sum of its used data space and Hewlett Packard Enterprise
used Dedup Store space.
• The dedup ratio of a TDVV is the size of the data written to the TDVV divided by the sum of the data stored in the TDVV and the data
associated with the TDVV in the Dedup Store.
In this following example, vv1 is a TPVV that has virtual size of 100 GB and has 10 GB of data written to it, vv2 is a TDVV that has virtual size of
100 GB and has 10 GB of non-deduplicable data written to it, and vv3 is a TDVV that has virtual size of 100 GB and has two copies of the vv2
data written to it.
cli% showvv -space vv1 vv2 vv3
---Adm---- --------Snp---------- ----------Usr-------------(MB)--- --(MB)-- -(% VSize)-- ---(MB)---- -(% VSize)-- ---------(MB)----------- -Capacity EfficiencyId
Name Prov
Type
Rsvd Used Rsvd Used
483
vv1 tpvv
base
9
0
0
0.0
485
vv2 tdvv
base
5184 4428
0
0
0.0
486
vv3 tdvv
base
5184 4433
0
0
0.0
256
Used Wrn
Lim
Rsvd
Used Used
Wrn
--
-- 12800 10240 10.0
0
--
-- 13056
5129
5.0
0
--
-- 13056
5129
5.0
0
Lim Tot_Rsvd
VSize
Compaction
Dedup
0
13056 102400
10.0
--
0
18240 102400
10.7
1.0
0
18240 102400
10.7
2.0
-----------------------------------------------------------------------------------------------------------------------------3 total
10624 8870
0
0
38912 20498
49536 307200
10.5
1.5
Note that the capacity efficiencies in the total row are averages of the volumes displayed.
The Adm Used and Usr Used values displayed for each TDVV are an estimation of contribution of the volume to the Dedup Store. Both vv2 and
vv3 share the same data and therefore the Usr Used space is divided between them (10 GB/2 = 5 GB).
The base volume savings can also be seen in the Management Console by selecting the particular vv. Figure 7 shows the space savings displayed
by the HPE 3PAR Management Console (IMC) for vv3. Note that the virtual volume Allocation shows the reserved space usage (i.e., the number
regions allocated to the VV), whereas the Used total in Capacity Efficiency shows the in use space (i.e., number of pages in use).
Technical white paper
Page 22
Figure 7. Virtual volume space savings in IMC
Figure 8 shows the space savings displayed by the HPE 3PAR StoreServ Management Console (SSMC) for vv3.
Figure 8. Virtual volume space savings in SSMC
Volume families
If a volume has snapshots then the showvv command will display the individual snapshots below their parent volumes but the capacity
efficiency displayed is that of the entire volume family because all the snaps of a base volume share the same snapshot area. If a block changes
on a base volume, the old data is copied to the snapshot area for that volume and all snaps of the volume point to that single block. This allows a
volume to have hundreds of snaps without requiring additional space or incurring additional performance impact.
The capacity efficiencies of a volume family are shown by the showvv -space command and are calculated as follows:
• The compaction ratio of a TPVV volume family is the sum of the virtual size of the volume and the virtual volume sizes of all its snapshots
divided by the used data space.
• The compaction ratio of a TDVV volume family is the sum of the virtual size of the volume and the virtual volume sizes of all its snapshots
divided by the sum of the used data space and the used Dedup Store space of the volume and its snapshots.
• The dedup ratio a TDVV volume family is the sum of the data written to the TDVV and all its snapshots divided by the sum of the data stored
in the TDVV and the Dedup Store for the volume and its snapshots.
Technical white paper
Page 23
In this example, snapshots were created from the previous VVs and 1 GB of data was changed on vv1 and 100 MB of data was changed on vv3.
The result is the compaction ratio for the vv2 volume family is twice that of the vv2 base volume and the compaction ratios of the other volume
families have increased but not by as much due to the change in the data.
cli% showvv -space vv*
---Adm---- ---------Snp---------Id
Name Prov Type
483
489
vv1 tpvv base
vv1_snp
485
491
493
Rsvd
256
--(MB)---
Used Rsvd
10 2560
snp vcopy
--
*0
--
vv2 tdvv base
5184
4465
512
snp vcopy
--
*0
--
vv3 tdvv base
5208
4492
512
--
*0
--
vv2_snp
486
---(MB)---
vv3_snp
snp vcopy
----------Usr-----------
-(% VSize)--
Used Used
1000
Wrn Lim
1.0
*0 *0.0
0
0
---(MB)---Rsvd
-(% VSize)--
Used Used
12800 10240 10.0
-----(MB)------ -Capacity Efficiency-
Wrn Lim Tot_Rsvd
0
0
VSize
15616 102400
Compaction Dedup
18.2
--
0
0
--
--
--
--
--
-- 102400
--
--
0.0
0
0
13056
5170
5.0
0
0
18752 102400
21.3
1.0
*0 *0.0
0
0
--
--
--
--
--
-- 102400
--
--
0.0
0
0
13087
5196
5.1
0
0
18807 102400
21.1
2.0
*0 *0.0
0
0
--
--
--
--
--
-- 102400
--
--
0
0
-------------------------------------------------------------------------------------------------------------------------------6 total
10648
8967 3584
1000
38943 20606
53175 614400
20.1
1.5
Note that the space usage columns for the snapshots contain “—” as the space usage of the snapshots is maintained in the base volume.
Common provisioning groups
A CPG with multiple volumes may have a higher dedup ratio than that of each TDVV in the CPG because the Dedup Store pages may be shared
by more than one volume (in addition to sharing within a volume). It is therefore a more accurate reflection of the space savings than shown by
the virtual volume dedup ratios.
The capacity efficiencies of a CPG are shown by the showcpg -space command and are calculated as follows:
• The compaction ratio is the sum of the virtual sizes of all the volumes and snapshots in the CPG (CPVV, TPVV, and TDVV) divided by the sum
of their in use data space, snapshot space, and used Dedup Store space.
• The dedup ratio is the sum of all the data written to the TDVVs and TDVV snapshots of a CPG divided by the size of the data stored in the
Dedup Store of the CPG and the sum of the data stored in the TDVVs of the CPG.
In this example, the TPVV_CPG CPG contains only TPVVs so it only has a compaction ratio, whereas the TDVV_CPG CPG contains TDVVs so it
also has a dedup ratio.
cli% showcpg -space
---------------(MB)--------------------------- Usr --Id
Name Warn%
-- Snp ---
--- Adm ---
- Capacity Efficiency –
Total
Used
Total
Used
Total
Used
Compaction
Dedup
0
TDVV_CPG
-
26112
26112
10752
0
24576
10368
10.7
3.0
1
TPVV_CPG
-
12800
12800
11776
0
8192
256
10.0
-
----------------------------------------------------------------------------------------2 total
38912
38912
22528
0
32768
10624
10.5
3.0
The base volume savings can also be seen in the Management Console by selecting the particular CPG. Figure 9 shows the space savings
displayed by the HPE 3PAR Management Console (IMC) for TDVV_CPG.
Technical white paper
Page 24
Figure 9. CPG space savings in IMC
Figure 10 shows the space savings displayed by the HPE 3PAR StoreServ Management Console (SSMC) for TDVV_CPG.
Figure 10. CPG space savings in SSMC
Virtual domains
The capacity efficiencies of a virtual domain are shown by the showsys -domainspace command and are calculated as follows:
• The compaction ratio is the sum of the virtual sizes of all the volumes in the virtual domain (CPVV, TPVV, and TDVV) divided by the sum of
the used data space and the used snapshot space of all CPGs in the virtual domain.
• The dedup ratio is the sum of all the data written to the TDVVs and TDVV snapshots in the virtual domain divided by the sum of the space
used by the TDVVs, the TDVV snapshots and the Dedup Stores in the virtual domain.
In this example, there is an additional dom1 virtual domain which contains just TPVVs.
Technical white paper
cli%
showsys
Page 25
-domainspace
--------------CPG(MB)---------------
-Non-CPG(MB)Domain
-----Usr-----
Usr
Snp
Adm
-
0
0
0
dom0
0
0
0
dom1
0
0
0
----Snp----
Total
Used Unmapped
-Capacity Efficiency-
0
0
0
0
0
0
0
0
0.0
-
102400 102400
94208
1024
98304
32256
0
294912
10.7
3.0
31744
0
24576
768
0
90112
10.0
-
33792
Used
-----(MB)------
Used
33792
Total
----Adm-----
Total
Total
Compaction
Dedup
--------------------------------------------------------------------------------------------------------------0
0
0
136192 136192 125952
1024 122880
33024
0
385024
10.3
3.0
System
The capacity efficiencies of the system are shown by the showsys -space command and are calculated as follows:
• The compaction ratio is the sum of the virtual sizes of all the volumes in the system (CPVV, TPVV, and TDVV) divided by the sum of the used
data space and the used snapshot space of all CPGs.
• The dedup ratio is the sum of all the data written to the TDVVs and TDVV snapshots in the system divided by the sum of the space used by
the all TDVVs, the TDVV snapshots and the Dedup Stores.
This example shows how the system wide capacity efficiencies are displayed.
cli% showsys -space
------------- System Capacity (MB) ------------Total Capacity
:
20054016
Allocated
:
1542144
Volumes
:
385024
:
0
User
:
0
Snapshot
:
0
Admin
:
0
:
385024
Non-CPGs
CPGs (TPVVs & TDVVs & CPVVs)
User
:
136192
Used
:
136192
Unused
:
0
Snapshot
:
125952
Used
:
1024
Unused
:
124928
:
122880
Used
:
33024
Unused
:
89856
Admin
Unmapped
:
0
:
1157120
Internal
:
321536
Spare
:
835584
System
Free
Used
:
0
Unused
:
835584
:
18511872
Technical white paper
Page 26
Initialized
:
18511872
Uninitialized
:
0
Unavailable
:
0
Failed
:
0
------------- Capacity Efficiency -------------Compaction
:
10.3
Dedup
:
3.0
Figure 11 shows the system wide space savings displayed by the HPE 3PAR StoreServ Management Console (SSMC).
Figure 11. System space savings
Manually updating the capacity efficiency statistics
Since the capacity efficiency ratios are not dynamically updated recent changes to the data may not be immediately reflected in the capacity
efficiency ratios displayed. It is possible to manually initiate an update of the statistics using the updatesnapspace -dedup_stats
command with the argument of a TDVV name. Even though one TDVV name is specified the capacity efficiency ratios all the volumes in the CPG
will be updated as well as capacity efficiency ratios of the CPG.
The following example shows how to initiate the manual statistics update.
cli% showvv -space tdvv*
---Adm---- ---------Snp------------(MB)--Id
--(MB)---
Used Rsvd
----------Usr-----------
-(% VSize)--
Used Used
---(MB)----
Wrn Lim
Rsvd
-(% VSize)--
Used Used
-----(MB)------ -Capacity Efficiency-
Name Prov Type
Rsvd
Wrn Lim Tot_Rsvd
VSize
Compaction Dedup
130
tdvv_1 tdvv base
256
52
0
0
0.0
--
--
2560
0
0.0
0
0
2816 102400
1947.6
0.0
136
tdvv_2 tdvv base
256
52
0
0
0.0
--
--
2560
0
0.0
0
0
2816 102400
1934.4
0.0
-------------------------------------------------------------------------------------------------------------------------------2 total
512
104
0
0
5120
0
5632 204800
1940.9
0.0
cli% updatesnapspace -dedup_stats tdvv_1
Task “dedup_stats_update TDVV_CPG” has been started.
cli% showvv -space tdvv*
---Adm---- ---------Snp------------(MB)--Id
Name Prov Type
Rsvd
--(MB)---
Used Rsvd
----------Usr-----------
-(% VSize)--
Used Used
Wrn Lim
---(MB)---Rsvd
-(% VSize)--
Used Used
-----(MB)------ -Capacity Efficiency-
Wrn Lim Tot_Rsvd
VSize
Compaction Dedup
130
tdvv_1 tdvv base 17125 16650
0
0
0.0
--
--
15105
5277
5.2
0
0
32230 102400
4.7
3.0
136
tdvv_2 tdvv base
0
0
0.0
--
--
8959
2691
2.6
0
0
17818 102400
9.1
1.2
8859
8518
-------------------------------------------------------------------------------------------------------------------------------2 total
25984 25168
0
0
24064
7968
50048 204800
6.2
2.4
Technical white paper
Page 27
The capacity efficiency ratios can also be updated from the HPE 3PAR StoreServ Management Console (SSMC) by selecting the Refresh
capacity efficiency action for a CPG as shown in figure 12.
Figure 12. Refreshing capacity efficiency
Understanding capacity efficiency ratios
It may not be immediately apparent that even low capacity efficiency ratios indicate significant space savings. As capacity efficiency ratios
increase, there are diminishing returns in terms of space savings. The percentage of space reduction obtained is 100 percent less the inverse of
the capacity efficiency ratio. Space savings for selected capacity efficiency ratios are shown in table 4.
Table 4. Space savings by capacity efficiency ratio
CAPACITY EFFICIENCY RATIO
SPACE REDUCTION PERCENTAGE
1:1
0%
1.5:1
33%
2:1
50%
4:1
75%
10:1
90%
Technical white paper
Page 28
Tracking volume space usage
Thin volumes consume user space, admin space, and possibly snapshot space on the disk array. The following sections provide the CLI
commands needed to determine how much space of every type is being consumed within the array. The output of these CLI commands shows
the reserved and the raw reserved space. The reserved space is what is offered by the array as usable space to the host. This value is also shown
in the HPE 3PAR Management Console in the Reserved User Size column for a TPVV and in the pie chart for the Logical option in the
Summary tab for the virtual volume details screen. The raw reserved space is calculated from the reserved space by multiplying the latter by its
RAID overhead factor. For example, this factor has a value of 2 for RAID 1 and 8/7 for RAID 5 with a set size equal to 8. The HPE 3PAR
Management Console shows the Raw Reserved space in the pie chart for the Raw option in the Summary tab for the virtual volume details. In
chargeback models, most IT departments bill their customers on the amount of raw reserved space consumed.
TPVVs
Use the showvv -s -p -prov tpvv command to see how much admin, user, and snapshot space is used by each TPVV. The reserved
totals show how much space has been allocated, whereas the used totals show how much of the space is currently in use by the VV. A significant
difference between the space in use and the reserved space would indicate that space reclaim has been initiated on the VV, and the reserved
space will decrease over time as the space is reclaimed in the background. This is an example of the showvv -s output:
cli% showvv -s -p -prov tpvv
---Adm---- ---------Snp---------Id
Name Prov Type
371
vv2 tpvv base
372
vv3 tpvv base
373
vv4 tpvv base
370
vv1 tpvv base
---(MB)--Rsvd
256
--(MB)---
Used Rsvd
16
0
256
40
256
47
256
28
----------Usr-----------
-(% VSize)--
Used Used
0
0.0
0
0
0
0
0
0
Wrn Lim
---(MB)---Rsvd
-(% VSize)--
Used Used
--
--
25088 21582 21.1
0.0
--
--
0.0
--
--
0.0
--
--
-----(MB)------ -Capacity Efficiency-
Wrn Lim Tot_Rsvd
VSize
0
0
25344 102400
66048 61642 60.2
0
0
74240 73378 71.7
0
0
47616 41062 40.1
0
0
Compaction Dedup
4.7
--
66304 102400
1.7
--
74496 102400
1.4
--
47872 102400
2.5
--
-------------------------------------------------------------------------------------------------------------------------------4 total
1024
131
0
0
212992 197664
214016 409600
2.1
The same information is displayed in the HPE 3PAR Management Console when viewing the VV provisioning as shown in figure 13.
Figure 13. VV allocation sizes
CPGs
Space in use on the array can be tracked per CPG. The showcpg -r command shows the user, snapshot, and admin space in Used and Raw
Used amounts. You can work out the unallocated space within the CPGs by subtracting the used space from the Totals listed.
In addition to showing the CPG usage, the showspace -cpg command will also show how much LD space may still be created, given the
amount of free chunklets in the system and the CPG parameters (e.g., RAID level, HA level, device types, etc.).
--
Technical white paper
Page 29
cli% showspace -cpg *
------------------------------(MB)---------------------------CPG
-----EstFree-------
--------Usr------
-----Snp----
----Adm----
Name
RawFree
LDFree
Total
Used
Total
Used
TPVV_CPG
18499584
9249792
16896
16896
15872
512
TDVV_CPG
18499584
9249792
34304
34304
31232
0
FC_r5
176078848
154068992
21951616 21951616
104320
3072
NL_r6
54263808
40697856
27969280 27969280
96512
69632
Total
-Capacity Efficiency-
Used
Compaction
Dedup
8192
256
10.0
-
24576
10496
10.7
1.6
32768
8480
13.1
-
23552
21248
8.3
-
System space
Not all space on the physical disks is used for storing your data. A small portion of the space on the array is dedicated to volumes with an
administrative function.
There is a fully provisioned volume named admin that is used to store system administrative data such as the System Event Log. The logging
LDs, starting with the name log, are used to store data temporarily during physical disk failures and disk replacement procedures. There are also
preserved data space logical disks (PDSLDs) which start with the name pdsld. Preserved data is the data moved from the system’s cache
memory to the PDSLD space in the eventuality of multiple disk or cage failures. On HPE 3PAR StoreServ systems running HPE 3PAR OS 3.1.2, the
HPE 3PAR System Reporter software is integrated into the OS and executed on the controller nodes, and the database files are stored in a fully
provisioned volume called .srdata.
The amount of raw disk space consumed by these system functions is summarized in table 5.
Table 5. System space usage 5
NUMBER OF NODES
LOGGING LDS
ADMIN LDS
PRESERVED DATA 6
SRDATA
2
240 GB
20 GB
128 GB
60 GB
4
480 GB
20 GB
256 GB
80 GB
6
720 GB
20 GB
384 GB
100 GB
8
960 GB
20 GB
512 GB
100 GB
Total used space
The showsys -space command gives the total raw space in use on the system with a breakdown of how the space is allocated. This is an
example of showsys output:
cli% showsys -space
------------- System Capacity (MB) ------------Total Capacity
:
20054016
Allocated
:
1542144
Volumes
:
385024
:
0
Non-CPGs
5
6
These are maximum values. The exact values will vary depending on the HPE 3PAR StoreServ model and disk configuration.
The total capacity of the PDSLDs is equal to the sum of all data cache memory located in the controller nodes of the system.
Technical white paper
Page 30
User
:
0
Snapshot
:
0
Admin
:
0
:
385024
:
136192
Used
:
136192
Unused
:
0
Snapshot
:
125952
Used
:
1024
Unused
:
124928
:
122880
Used
:
33024
Unused
:
89856
CPGs (TPVVs & TDVVs & CPVVs)
User
Admin
Unmapped
:
0
:
1157120
Internal
:
321536
Spare
System
:
835584
Used
:
0
Unused
:
835584
:
18511872
Initialized
:
18511872
Uninitialized
:
0
Unavailable
:
0
Failed
:
0
Free
------------- Capacity Efficiency -------------Compaction
:
10.3
Dedup
:
1.6
The value in the System section of the output is the raw space in use by the admin and the .srdata volume, the PDSLDs, the logging LDs, and
the spare chunklets. The internal entry lists the space associated with the local boot disks of the nodes so this should be treated separately from
the space allocated from the regular drives.
Usage reporting and trend analysis
The CLI command showcpg -hist <CPG> gives a daily account of CPG usage split into user, snapshot, and admin space. The command
showspace -cpg <CPG> -hist also shows this information.
You can also use the srcpgspace and srvvspace commands to query the internal System Reporter database. 7 Additionally, the optional
HPE 3PAR System Reporter software has the ability to track the CPG and TV usage for comprehensive usage and trend analysis.
7
The System Reporter functionality has been included in HPE 3PAR OS 3.1.2. Prior versions of HPE 3PAR OS will require the optional HPE 3PAR System Reporter software on an
external host.
Technical white paper
Page 31
The following example shows the output of a srvvspace command for a CPG.
cli% srvvspace -hourly -btsecs -1h -usr_cpg TDVV_CPG
------RawRsvd(MB)-----Time
----User(MB)-----
Secs
User
Snap
Admin
Total
Used
Free
2014-10-16 23:00:00 CEST 1413493200
68608
0
31488
100096
10244
24060
------Admin(MB)------Used
Free
8896
1600
34304
---------------Total(MB)---------------
Rsvd Vcopy
10496
------Snap(MB)-----Rsvd Used Free Rsvd Vcopy
0
Vcopy
Used
Rsvd
0
19140
44800
0
0
0
-Capacity Efficiency-
VirtualSize Compaction
67313664
0
10.0
Dedup
1.5
Running out of space
The following sections describe the severity of the situation when space limitations are met, from the least critical (for example, running out of
space on a single TV) to the more severe (for example, running out of space on a CPG or on the entire system).
Running out of space in a TV
If an allocation limit was set for a TV, the array stops all writes to that TV after the limit is reached-returning SCSI error codes if more writes come
in. Depending on the OS and application, the effect of this can range from the host application stalling to the host server crashing. Reaching a TV
limit purely affects that volume and its resident application(s).
TV limits can be changed at any time with the setvv command. After an appropriate change, writes to the TV continue to be processed.
Running out of space in a CPG
Reaching a CPG limit has the same effect as reaching a TV limit; however, many more TVs and other types of VVs may be impacted. CPG limits
can be changed at any time with the setcpg -sdgl command. After an appropriate change, writes to the volumes in the affected CPG
continue to be processed.
It is also possible for a CPG to run out of space if there are not enough free chunklets of the selected device type to complete a grow. In such a
circumstance the CPG will attempt to complete the grow by using a lower availability level (i.e., HA MAG instead of HA CAGE) and if that is not
possible it will use chunklets from another tier. The system will report that the CPG was grown with degraded parameters and once the space
shortage has been corrected it is important to run the tunesys command to correct the degraded grow.
Running out of space on the system
Running out of space on the system will affect TVs and all other types of volumes being written to in all CPGs. In the unlikely scenario that all
physical capacity becomes used, the system prevents new writes from occurring until more capacity is added.
Reclaiming unused space
The HPE 3PAR OS space consolidation features allow you to change the way that virtual volumes are mapped to logical disks in a CPG. Deleting
TVs or moving virtual volume regions from one logical disk to another enables you to compact logical disks, and free up disk space so that it can
be reclaimed for use by the system.
Mapping is the correspondence of logical disk regions to the virtual volume. Virtual volumes are made up of multiple logical disks, and each
logical disk contains regions that are mapped to the virtual volume. All types of volumes are created by mapping data from one or more logical
disks to the virtual volume.
Logical disks can be shared by multiple virtual volumes or snapshots. As volumes are deleted or as volume copy space grow and then shrink,
logical disks can use space less efficiently. When logical disks do not efficiently use space, the unused space consumes regions on the LD that are
not available for use by the system when creating new logical disks. The space management features enable you to consolidate used space onto
fewer fully used logical disks so that unused regions are forced onto one or more logical disks that are then deleted. Deleting these logical disks
frees the unused space for general use by the system. You can also truncate LDs to free up space. The LD’s used regions are compacted by
moving them to the beginning of the LD, and then the LD is truncated so that unused space can be returned to the system’s free chunklet pool.
Technical white paper
Page 32
Reclaiming unmapped logical disk space from CPGs
CPGs provide a shared pool of logical disk capacity for use by all virtual volumes that draw space from that pool if volumes that draw from a CPG
are deleted, or if copy space for these volumes grows and then shrinks, the underlying logical disks in the CPG pool can become less efficient in
space usage. One or more logical disks in the CPG pool may have only a small portion of their regions mapped to existing virtual volumes.
However, the logical disk’s unused regions are only available for use by the volumes mapped to the CPG.
It is possible to compact the logical disk regions mapped to these volumes to recover and free logical disk space that can be returned to the free
space pool for reuse by other CPGs. Compacting a CPG allows you to reclaim space from a CPG that has become less efficient in space usage
from creating, deleting, and relocating volumes. Compacting consolidates logical disk space in CPGs into as few logical disks as possible.
Compacting CPGs can be performed with both the HPE 3PAR CLI and the HPE 3PAR Management Console.
By default, the compactcpg command will defragment the LDs owned by the CPG and then return entire unused LDs back into the free space
pool. There is also a -trimonly option that causes compactcpg to simply check for unused LDs and return them to the free space pool
without running a defragmentation.
When using thin provisioning, it is recommended to schedule regular CPG compactions during periods of low activity. On larger systems, it is
advisable to stagger the compaction of the CPGs to reduce the load on the system.
Note
If using Adaptive Optimization on all the CPGs, automatic CPG compaction is a configurable option of Adaptive Optimization. It is recommended to
run the startao command with the -compact trimonly option to defer the LD defragmentation until the scheduled CPG compaction runs.
Automatically reclaiming unused space from volumes
The HPE 3PAR OS automatically reclaims unused user space from thin volumes and returns the space to the LDs. The system examines the
shared space for large areas of unused space. The identified areas are unmapped from the corresponding LD regions, and the space is returned
to the LDs.
Deleted volume snapshot space
HPE 3PAR Thin Copy Reclamation is an optional feature that reclaims space when a snapshot of a TPVV, virtual copy, full copy, or remote
copy volume is deleted from the system. The deleted snapshot space is reclaimed from either a standard or a thin-provisioned volume and is
automatically returned to its CPG for reuse by other volumes. The HPE 3PAR Thin Copy Reclamation feature works on all HPE 3PAR StoreServ
and requires that an HPE 3PAR Thin Provisioning software license be present on the array.
Logical disks and chunklet initialization
After logical disks are deleted or space is reclaimed by the Thin Persistence software, the underlying chunklets are reinitialized before their space
is available to be used by other LDs. The initialization process for chunklets happens in the background and at a low priority so as not to impact
system performance. To see chunklets that are currently in the process of being initialized, issue the showpd -c command. Chunklets that are
uninitialized are listed in the Uninit column.
Free page consolidation
As individual 16 KiB pages are freed the space in the 128 MiB regions of the LDs can become fragmented, and therefore the HPE 3PAR OS
implements a defragment process similar to what an OS may run on a file system. There is a thread that periodically examines the VVs to
determine whether defragmentation should be initiated. The data from partially allocated regions is then reorganized to create larger contiguous
blocks of free space that can be reclaimed.
Technical white paper
Page 33
HPE 3PAR Thin Persistence software
Traditionally when data is deleted on a host, the OS will report that space has been freed, but the storage is not informed that the data is no
longer in use. With fully provisioned volumes, this is not an issue but with thin-provisioned volumes, the unused space will remain allocated on
the array causing the volumes to grow over time. This creates a hidden utilization penalty that can significantly reduce the space savings of thin
provisioning.
HPE 3PAR Thin Persistence software is able to maintain the benefits of HPE 3PAR Thin Provisioning software and HPE 3PAR Thin Conversion
software by enabling “thin reclamation” of allocated but unused capacity so the thin volumes are able to stay as lean and efficient as possible. It
leverages the HPE 3PAR OS support for the WRITE_SAME and UNMAP commands of the T10 SCSI Primary Commands SPC-3 standard and the
unique zero-detection capabilities of the HPE 3PAR ASIC to give HPE 3PAR StoreServ Storage the power to reclaim unused space associated
with deleted data simply, quickly, and non-disruptively.
UNMAP is a SCSI command that a host can issue to tell the storage that blocks are no longer need to be allocated. This is particularly important
in thinly provisioned environments as it allows the storage array to recognize that these blocks are not used and to return them to the free
capacity pool for reuse by other volumes.
The HPE 3PAR ASIC features an efficient, silicon-based zero-detection mechanism. This unique hardware capability gives HPE 3PAR StoreServ
Storage the ability to remove allocated but unused space as small as 16 KiB on the fly without impacting performance.
In addition, the benefits of HPE 3PAR Thin Persistence are available to read/write snapshots of TPVVs. The mechanism for initiating reclamation
is the same as for the parent TPVV: writing zeros to the allocated but unused space in a read/write snapshot will trigger the ASIC to initiate
reclamation of the deleted space. To benefit from thin reclamation, the zero-detect policy needs to be enabled on each read/write snapshot.
Thin Persistence reclamation
Thin Persistence reclamation occurs at several levels. Initially all freed 16 KiB pages are returned to the TPVV. This means that on a file system
that supports automatic reclaim the spaced freed by an UNMAP after a file deletion is immediately available for reuse by a file creation or file
extension operation on the same file system.
To make space available for reuse by other volumes, there is a reclaim thread that returns freed 128 MiB regions allocated to a TPVV back to the
CPG. This thread scans volumes every five minutes for SD space that potentially can be reclaimed. If a TPVV has free 128 MiB regions or there is
enough free space to warrant a defragmentation of the volume then space will be reclaimed back to the CPG. Defragmentation occurs if there is
more than 1 GB of available space to reclaim and results in free 128 MiB regions. Up to 16 volumes at a time can be queued for reclaim processing.
How quickly the space is reclaimed depends on a number of factors. If there is a large amount of freed space on a volume, then this may not be
processed within a single reclaim period and once the reclaim process runs on a TPVV, the reclaim process will not run again on that TPVV again
for at least 90 minutes. Therefore, large space reclaims can take several hours to complete.
In addition, the reclamation on a TPVV can be deferred for various reasons. For example, if the SD space of a TPVV is grown, then reclaims on
the volume will be deferred for 60 minutes. Also, if reclaim is defragmenting a TPVV and the defragmentation does not complete during the
reclaim interval further reclaim will be deferred for 8 hours.
Thin Persistence reclamation may not reclaim all the free space on a volume. There is a 4 GB per node threshold below which the TPVV will not
be inspected for available 128 MiB regions that can be reclaimed back to the CPG. The free space will still be available for reuse by the TPVV.
Those new to Thin Provisioning often like to verify Thin Persistence reclamation by creating a test scenario of filling a file system then deleting
the files and running a space reclamation tool. It is important to understand that the space will not be returned to the CPG immediately. The
showvv -s command will show how much space has been allocated to the TPVV and the difference between the space in use and the
reserved space shows the amount of space reclaimed for use within the TPVV. The amount of reserved space will decrease over time as the
space is reclaimed back to the CPG in the background by the reclaim thread.
Technical white paper
Page 34
Thin Persistence and deduplication
From a host perspective, the methods used to initiate space reclaim on TDVVs are the same as those on TPVVs (i.e., zerofiles or UNMAP) but
internally there are differences in the way space reclaim operates. With TDVVs, it is not necessary to have a zero-detection mechanism to scan all
incoming I/Os as all zero blocks will be reduced to a single zero block by the deduplication engine. However, the original pages in the Dedup
Store cannot be removed as they may be in use by other TDVVs. This is also the case for space reclaimed by the UNMAP command.
HPE 3PAR Thin Deduplication uses an online garbage collection process that periodically checks for data in a Dedup Store that is not referenced
by any TDVVs in that CPG.
When reclaim is initiated on a TDVV the metadata for the pages being freed are pointed to the deduped zero block. The garbage collection
process then scans all the TDVVs belonging to the same CPG and builds a list of hashes being referenced. This is then compared with the
hashes in the Dedup Store and if any pages are no longer referenced then they are marked as free.
Once the garbage collection has completed the SD space associated with the TDVVs and Dedup Store is processed by the normal Thin
Persistence reclaim thread and the space is returned to the CPG.
Thin Persistence methods
The most optimal Thin Persistence method is for the host OS to issue an UNMAP for the unwanted blocks when the data is deleted. The most
optimal Thin Persistence method is for the host operating system to issue SCSI UNMAP commands for unwanted data blocks. Typically, this
would be done by the file system when files are deleted. However, if the operating systems offers a suitable UNMAP application programming
interface (API), an application can directly communicate to the HPE 3PAR StoreServ system that it no longer needed some data. This method
provides continual reclaiming of space to allow the storage volumes to stay thin. The disadvantage is it requires significant changes to the
storage stack and only the most modern OSs have implemented native UNMAP support.
Another method is to have an OS utility that will examine the blocks of a volume and issue UNMAPs for those that are not being used. This type
of utility also requires host OS UNMAP support but to a lesser extent the continual method, and they are specific to a file system or volume
manager type. Most of these utilities can be run when the data is online but as they generate the UNMAP requests for all the unused blocks in
one go they are generally run manually during an outage window or scheduled to run during a quiet period so the reclaims do not adversely
impact other workloads on the storage.
The final method is to reclaim space using a “zerofile” utility that writes zeros to all allocated but unused space on a file system. On HPE 3PAR
StoreServ systems, the zero-detection capability of the HPE 3PAR ASIC intercepts the blocks of zeros being written and automatically triggers
the reclamation of the space. The advantage of this method is that it does not require any special OS support, and the utilities to generate
zerofiles are often supplied with the base OS distribution. To achieve good reclaim rates, the utilities need to fill the majority of the available free
space so they are generally run manually during an outage window or scheduled to run during a quiet period to avoid applications failing due to
a lack of space.
For the zerofile utilities to work the zero-detect policy needs to be set for each TPVV. Blocks of 16 KiB of contiguous zeros are freed and returned
for reuse by the VV; if 128 MiB of space is freed, it is returned to the CPG for use by other volumes.
For more information on Thin Persistence methods for various operating systems, see Appendix B.
Thin Deduplication with snapshots and clones
Volumes in HPE 3PAR OS can have read-only or read-write snapshots also known as virtual copies. The storage for the base volume and the
storage for the snapshot volume can potentially come from different CPGs. When the user creates a TDVV, if a snapshot CPG is specified which
is of dedup CPG type, then any snapshot data stored in the Dedup Store associated with that CPG will be deduplicated.
If the snapshots can have a different CPG to the TDVV then the deduplicated data will still reside in the CPG associated with the TDVV. Only
collision data will reside in the snapshot CPG.
A clone, also known as a Physical Copy, from a source volume of any type can also be created as a TDVV.
Technical white paper
Page 35
Thin Deduplication with Remote Copy
HPE 3PAR Thin Deduplication is fully supported as either a source or destination with Remote Copy, however, data transmission is not
dedup aware:
• In synchronous replication mode data is immediately sent to the secondary site and since deduplication happens post acknowledging to the
host and before flushing data to the backend all data is transmitted over the wire. The synchronous replication process is zero-detect aware,
as 16 KiB pages of zeroes are detected before being transmitted and only 64 bytes of metadata are transmitted over the wire. This behavior is
also applicable to HPE 3PAR Peer Persistence as it uses synchronous replication.
• In asynchronous replication mode data is stored in snapshots and then read from the secondary volume when the period is due. Snapshots
of the primary volume are dedup aware, however, data will be rehydrated before being sent over the wire.
The replication can be between volumes of different types on different tiers. For example, a thinly deduplicated volume on SSD drives on a
primary array can be in a remote copy relationship with a thin provisioned volume on NL drives on a secondary array.
While deduplicate data is rehydrated HPE 3PAR Remote Copy offers advanced data transmission techniques to optimize bandwidth utilization,
for more information please refer to the HPE 3PAR Remote Copy white paper. 8
Dynamic Optimization of virtual volumes
Traditionally, data migrations associated with a storage technology refresh have carried forward the poor utilization rates from one system to the
other. Administrators had to procure at least the same amount of capacity on a new array as on the legacy systems from which data was being
migrated, even if the migration involved a large amount of allocated but unused capacity.
HPE 3PAR StoreServ storage offers several products that can be used for service-level optimization.
HPE 3PAR Dynamic Optimization is a powerful software feature that enables storage administrators to perform several online volumes
optimizations.
• Perform online data movement of existing volumes to different RAID levels and different tiers of storage. For example, application that requires
high performance only during certain windows can be tuned to RAID 1 on SSDs and during lower activity periods moved to more cost-effective
RAID 6 storage on Nearline disks.
• Convert existing volumes to a different volume type. For example, thinly provisioned volumes on a HDD CPG to a thinly deduped volume on
an SSD tier. Or the conversion of a full volume to a thinly provisioned volume. This conversion happens within the HPE 3PAR StoreServ
Storage system transparently and non-disruptively.
This “thinning” of volumes enables data centers to meet green IT targets, reduce wasted space, and increase utilization rates. Because the fat-to-thin
conversion mechanism is built into the array hardware, volume conversions take place inline and at wire speeds, while preserving service levels,
and without causing disruption to production workloads.
The conversion requires the HPE 3PAR Dynamic Optimization Software license.
8
h20195.www2.hp.com/V2/GetPDF.aspx%2F4AA3-8318ENW.pdf
Technical white paper
Page 36
Migrate between full and thin volumes on the same array
All HPE 3PAR StoreServ systems running HPE 3PAR OS 3.1.2 and above have the ability to make a convert from a thin-provisioned volume to a
fully provisioned volume (or vice versa) without requiring an offline transition. 9 HPE 3PAR OS 3.2.1 MU1 adds the ability to convert to a Thin
Deduplication volume.
The tunevv command is used to convert between full provisioned, thin provisioned and Thin Deduplication virtual volumes without requiring
an offline transition. The convert virtual volume operation from the HPE 3PAR Management Console or the tunevv command can used to
perform the conversion. In the following example, the virtual volume vol01 is moved to the FC_r5 CPG and converted to a TPVV:
cli% tunevv usr_cpg FC_r5 -tpvv vol01
To convert the volume to a Thin Deduplication volume in the SSD_r5 CPG the following command would be used:
cli% tunevv usr_cpg SSD_r5 -tdvv vol01
When the -tpvv, -tdvv or -full options for the usr_cpg subcommand are specified, the tune will automatically rollback on a failure. These
options do not support virtual volumes with remote copy. These options will only convert virtual volumes using snapshots if the -keepvv option
is used, but the snapshots will reside in the virtual volume specified by the -keepvv option.
During the thin conversion, the HPE 3PAR ASIC will assist in reducing the amount copied by using its zero-detect capability to remove the need
to copy blocks of zeros and deduplicate data if converting to a TDVV. To make optimal use of this feature, it is advantageous to write zeros to
the allocated but unused space on the fully provisioned volume prior to the conversion.
Estimating Thin Deduplication space savings
A Dedup Estimate tool is available to show the space saving benefits of Thin Deduplication on existing data without the need to convert the
volumes. The tool attempts to estimate the amount of space savings that can potentially be achieved by finding common data across specified
TPVVs, and provides the dedup ratio based on the calculated data.
To launch Dedup Estimate from the CLI use the checkvv command with the -dedup_dryrun option on a group of VVs or a VV set to
perform a dry run conversion which will report the space savings Thin Deduplication would achieve if the VVs specified were in the same
deduplication CPG. The specified VVs must be of type TPVV but they can reside on any type of drive not just SSDs.
The following example launches a dedup estimation task on VVs vv1 and vv2:
cli% checkvv -dedup_dryrun vv1 vv2
The Dedup Estimate can also be launched from the HPE 3PAR Management Console by selecting a VV set or multiple TPVVs and then Right
clicking and choosing the Dedup Estimate menu item as shown in figure 14.
9
For earlier versions of HPE 3PAR OS, this “thinning” operation does not complete online: a brief disruption of service is required to change the host mapping from the full to the
thin-provisioned volume. An HPE 3PAR Full Copy software (or Physical Copy) operation of the source volume is required for HPE 3PAR OS 3.1.1 and prior, the license for which is
included in HPE 3PAR OS.
Technical white paper
Page 37
Figure 14. Dedup Estimate
The estimated dedup ratio will be shown under the task’s detailed information, which can be accessed via showtask -d.
cli% showtask -d 11705
Id
11705
Type
Name
Status
dedup_dryrun
checkvv done
Phase
Step -------StartTime--------
-------FinishTime-------
-Priority-
-User—
---
---
2014-10-12 22:26:15 CEST
n/a
3parsvc
2014-10-12 22:25:45 CEST
-----(MB)------ (DedupRatio)
Id
Name
Usr
Estimated
395
vv1
10240
--
vv2
10240
--
431
Estimated
--
-------------------------------------2 total
20480
10239
1.82
In this example, checkvv is estimating that if vv1 and vv2 were converted to TDVVs and put in the same CPG there would be a dedup ratio of
1.82:1 for the CPG. The tool does not calculate the amount of unique data within each TPVV so the estimated values for each TPVV are marked
with “—.”
Online migration to HPE 3PAR StoreServ
There are several approaches to consider when migrating data from a source array to thin provisioned or Thin Deduplication volumes on an
HPE 3PAR StoreServ. If the source array is also an HPE 3PAR StoreServ then Peer Motion can be used to import a full or thin provisioned volume
from the source system to a thin provisioned or thinly deduped volume on the destination system.
The HPE 3PAR Online Import Software enables an effortless migration of HPE EVA; EMC VMAX, VNX, CX4; HDS NSC, USP, VSP or IBM XIV Storage
to an HPE 3PAR StoreServ Storage array, without the need of host-based resources. The source data can be migrated inline to thin provisioned
or thinly deduped volumes on HPE 3PAR StoreServ Storage.
A block-based migration tools such as a host-based volume manager or a SAN-based virtualization appliance can also be used copy a source
volume to the thin volume destination.
It is also possible to use file-based migration tools such as rsync in UNIX®/Linux, Quest in Windows, or backup and restore. With these tools the
target thin volume will grow only to the capacity of data contained within the file system, therefore making efficient use of thin provisioning on
the target HPE 3PAR StoreServ.
Technical white paper
Page 38
Preparing data for migration
HPE 3PAR Peer Motion, HPE 3PAR Online Import Software and block-based migration tools will copy any non-zero blocks from the source
volume to the TPVV or TDVV, whether they contain valid data or not. This means that the customer may not see all benefits from thin
provisioning after using the migration tools.
It is therefore recommended to defragment the file systems (where applicable) and run a zerofile tool prior to the migration so that the allocated but
unused space in the volume on the source array is filled with zeros. When the data is migrated the HPE 3PAR ASIC accelerated zero-detection will then
allow customers to reap the benefits of thin volumes immediately.
For virtual machines the same process should be followed for all VM guests whether the VM files are copied directly or online migration
technique such as Hyper-V Live Migration or VMware vMotion are used.
Conclusion
HPE 3PAR StoreServ Storage is the only platform that offers a comprehensive thin strategy that not only allows storage to start thin, but to get
thin and stay thin. Compaction technologies such as Thin Deduplication, thin provisioning and thin reclamation offer efficiency benefits for
primary storage that can significantly reduce both capital and operational costs with spinning media and SSDs. Not only is HPE 3PAR StoreServ
Storage viewed as the industry’s thin technology leader, but third-party testing and competitive analysis confirm that HPE 3PAR StoreServ offers
the most comprehensive and efficient thin technologies among the major enterprise storage platforms. In addition, HPE 3PAR Thin Technologies
protect SSD performance and extend flash-based media lifespan while ensuring resiliency.
HPE 3PAR Thin Technologies enables simple, yet powerful tools for improving the efficiency of storage. Following the best practices outlined in
this paper will allow IT staff to help maximize the benefit of HPE 3PAR Thin Technologies and do more with less. To supplement the dramatic
savings of thin provisioning, HPE 3PAR StoreServ features a unique HPE 3PAR ASIC with thin capabilities built in and a range of software
offerings that can save enterprises 50 percent or more on the cost of a storage technology refresh.
Appendix A—Thin volumes and applications
All applications including mission-critical ones can use thin-provisioned volumes. The use of thin volumes may need some consideration upfront
to enable them to utilize the volume space thin-efficiently. The following sections include recommendations for installing a few well-known
applications on thin-provisioned volumes.
Oracle
In an Oracle environment, HPE 3PAR Thin Provisioning provides the better storage space utilization when used in the following cases:
• Autoextend-enabled tablespaces—Oracle Automatic Storage Management (ASM)-managed datafiles with the autoextend feature grow as the
need for tablespace grows. Using such autoextendable datafiles on thin-provisioned volumes allows the system to allocate disk space to a
database as it grows.
However, Oracle’s process of extending a tablespace is I/O intensive and can affect the performance of the database during the file extension.
Lessen the frequency of the performance impact by increasing the increment of space that Oracle adds (the AUTOEXTEND ON NEXT
parameter of the CREATE TABLESPACE command). Set autoextend to a larger value for rapidly growing datafiles, and a smaller value if the
data is growing at a slower rate.
• ASM disk groups as archive log destinations—The combination of Oracle ASM and HPE 3PAR Thin Provisioning enables an expanding ASM
archive log destination. Oracle databases become inaccessible when the archive log destinations fill up. Put Oracle archive logs on thin-provisioned
ASM disk groups, which allows the underlying storage to self-tune, accommodating unexpected increases in log switch activity. After a level 0
backup, remove the archive logs and use the Oracle ASM Storage Reclamation Utility (ASRU, described later) to free up the storage that was
allocated to the old logs.
HPE 3PAR Thin Provisioning may not be the best option for the following:
• Datafiles residing on file systems—When an Oracle datafile is first created, Oracle initializes the data blocks with a block headers and other
metadata. Due to small block size of most databases, there are usually no contiguous ranges of zero blocks for space reclamation. This causes
a TPVV to provision all of its space, nullifying the value of thin provisioning.
Technical white paper
Page 39
• Systems with high file system utilization—If the file systems or ASM disk groups are full, then the benefits of thin provisioning are reduced.
Consider any ASM disk group or file systems with utilization rates above 80 percent as inappropriate for use with thin provisioning. In this
case, it may be more efficient to use fully provisioned virtual volumes to hold this data.
• Oracle datafiles that are not in “autoextend” mode—Oracle databases write format information to datafiles during tablespace creation. This
has the same effect as provisioning file systems with high utilization and may be inefficient depending upon the ratio of provisioned storage to
database size.
Microsoft SQL Server
Microsoft SQL Server transaction log files are formatted when a database is created. The pages written mostly contain contiguous zeros and
therefore the HPE 3PAR Thin Persistence software will keep the allocation of the TPVV to a minimum. Over time, as transactions are written to
the log files, the log file TPVV allocation will grow, so the CPG associated with log files should be appropriately sized.
As of Microsoft SQL Server 2005, the database creation phase no longer initializes datafiles if the account under which the SQL Server service is
running has the “perform volume maintenance tasks” user rights assignment under the local security policy. The instant file initialization process
creates a sparse file, which is thin friendly and does not allocate storage space in advance.
Microsoft SQL Server 2012 is able to take advantage of the SCSI UNMAP support in Windows Server® 2012 to automatically reclaim space after
shrink/drop datafile operations. In addition, when datafiles and databases are deleted on Windows Server 2012, the system automatically reclaims
the space without administrator intervention. Prior to Windows Server 2012 space reclamation required a zerofile tool to write zeros to the free
space in the file system.
Microsoft Hyper-V
The virtual hard disk (VHD) and the new virtual hard disk (VHDX) formats of Microsoft Hyper-V are fully compatible with HPE 3PAR Thin
Provisioning. Both formats offer a choice of fixed or dynamic sizing. A fixed VHD has an allocated size that does not change whereas a dynamic
VHD will expand as data is written to it.
Although it may seem natural to use a fully provisioned VV with a fixed VHD, contiguous zeros are written to the space allocated so they are
ideal candidates for thin provisioning, as the zeros will be reclaimed by the HPE 3PAR Thin Persistence software.
In the past, fixed VHDs were recommended for production instead of dynamic VHDs, as their I/O performance was higher. With the new VHDX
format introduced in Windows Server 2012, performance of dynamic VHDs has been significantly improved and there is now little difference
between the two VHD types.
In addition, VHDX disks report themselves to the guest OSs as being “thin-provision capable.” This means that if the guest OS is UNMAP-capable
it will be able to send UNMAPs to the VHDX file, which will then be used to ensure that block allocations within the VHDX file that are freed up
for subsequent allocations as well as forwarding the UNMAP requests to the physical storage.
SAP
By migrating data from traditional arrays to HPE 3PAR StoreServ via Thin Conversion, legacy SAP® systems can reduce up to 80 percent of the
capacity in the storage environment. In an SAP system, data gets moved around or deleted within system storage volumes. HPE 3PAR Thin
Persistence enables that the thin volumes used by SAP systems stay efficient as possible by reclaiming unused space associated with deleted data.
Typically, SAP databases store data in the form of a matrix (tables and indexes) consisting of rows and columns. Most of the columns in the
tables are of fixed length, and there are often leading or trailing zeroes in these columns. The HPE 3PAR ASIC is able to detect these zeros if they
form a contiguous 16 KB block and prevent storage from being allocated. In testing with SAP ERP 6 IDES, a LUN with HPE 3PAR Thin
Persistence software enabled consumed 10 percent less storage space than a traditional thin LUNs after the initial database creation. See the
HPE 3PAR StoreServ Storage for SAP Systems—technical white paper for further details.
Technical white paper
Page 40
VMware vSphere
VMware vSphere also has its own thin provisioning capabilities therefore a decision has to be made about which level to implement thin provisioning.
When implementing HPE 3PAR StoreServ TPVVs, administrators often ask whether implementing vSphere Thin Provisioning for Virtual Machine
Disk (VMDK) files makes any sense. In general, Thin Provisioning with HPE 3PAR StoreServ and vSphere accomplish the same end-result, albeit
at different logical layers. With VMware vSphere Thin Provisioning, administrators realize greater VM density at the Virtual Machine File System
(VMFS) layer, at the cost of some CPU and disk I/O overhead as the volume is incrementally grown on the ESXi hosts. By implementing
HPE 3PAR StoreServ TPVVs, the same VM density levels are achieved; however, the thin provisioning CPU work is offloaded to the HPE 3PAR
StoreServ ASIC. If the goal is to reduce storage costs, help maximize storage utilization, and maintain performance, then use HPE 3PAR Thin
Provisioning software to provision Virtual Machine File System (VMFS) volumes. If performance is not a concern but overprovisioning VMs at the
VMFS layer is important, then administrators can consider implementing both Thin Provisioning solutions. However, administrators should realize
that there is no additional storage savings realized by using vSphere Thin Provisioning on top of HPE 3PAR TPVVs; and in fact, implementing
both solutions adds more management complexity to the environment.
When creating VMs, there are several options for the file system layout of the VMDK files. By default, VMDK files are created with the “Thick
Provision Lazy Zeroed” option, which means the VMDK is sparsely populated so not all the blocks are immediately allocated. When a guest VM
reads from the unallocated areas of the VMDK, the vSphere server detects this and returns zeros rather than reading from disk. This VMware
thin provisioning capability enables the oversubscription of the sizes of the VMDK files within the datastore.
For performance-intensive environments, VMware recommends using “Thick Provision Eager Zeroed” (EZT) virtual disks. These EZT disks have
lower runtime overhead but require zeros to be written across all of the capacity of the VMDK at the time of creation. On traditional arrays, this
VMDK format would negate all the benefits of thinly provisioned LUNs as all of the physical storage is allocated when the volume is zero-filled
during creation. However, HPE 3PAR Thin Persistence software allows clients to retain thin provisioning benefits when using EZT VMDKs
without sacrificing any of the performance benefits offered by this VMDK option. In this case, Thin Persistence helps ensure that, when a new
EZT volume is created, the entire volume is not allocated from physical storage since all zeros that have been written to the VMDK were
intercepted and discarded by the HPE 3PAR ASIC.
Appendix B—Thin Persistence methods
Thin Reclamation for Microsoft Windows Server 2012
Windows Server 2012 integrates very well with thin-provisioned volumes on HPE 3PAR StoreServ. It identifies thinly provisioned volumes on
HPE 3PAR StoreServ systems, writes entries in the Windows event log file when storage thresholds are reached on the CPG, and the TPVV level
and supports active reclaim by issuing UNMAPs upon file deletion or file shrinking on thin-provisioned volumes on NT File System (NTFS)
formatted volumes. The standard defragmentation scheduled task also automatically reclaims storage.
In addition, Windows Server 2012 extends UNMAP support to the virtual layer. The Hyper-V VHDX disks report themselves to the guest OSs as
being “thin-provision capable.” This means that if the guest OS is UNMAP-capable, it can send UNMAPs to the VHDX file, which will then be used
to help ensure that block allocations within the VHDX file are freed up for subsequent allocations as well as forwarding the UNMAP requests to
the physical storage.
There is also a File TRIM API which is mapped to the TRIM command for ATA devices and the UNMAP command for SCSI devices. TRIM hints
allow the application to notify the storage that blocks that, previously were allocated, are no longer needed and can be reclaimed.
In summary, the following operations trigger storage space reclamation in Windows Server 2012:
• Deletion of a file from a file system on a thin-provisioned volume
• Running storage optimization, a new feature of Windows Server 2012 disk management
– You can use manual or automatic scheduling of the Optimize operation utility
– The standard Defrag scheduled task automatically runs Optimize
Technical white paper
Page 41
• UNMAP requests from a Hyper-V guest OS
– Deleting a file from the file system of an UNMAP capable guest OS sends UNMAP requests to the driver stack of the Hyper-V host
• UNMAP requests from applications using the TRIM API
The default behavior of issuing UNMAPs on file deletion can be disabled on a Windows 2012 server by setting the “disabledeletenotify”
parameter of the fsutil command. This will prevent reclaim operations from being issued against all volumes on the server.
To disable reclaim operations, run the following PowerShell command:
fsutil behavior set DisableDeleteNotify 1
Thin Reclamation for Microsoft Windows Server 2003 and 2008
Windows Server versions prior to 2012 do not implement UNMAP support and therefore to reclaim thinly provisioned storage you must leverage
the zero-detection capabilities of HPE 3PAR StoreServ.
Microsoft provide a Sysinternals advanced system utilities suite that includes the Secure Delete (SDelete) application that can used to overwrite
a deleted files on-disk data to make disk data unrecoverable.
As well as overwriting a single file data space, SDelete can indirectly overwrite free space by allocating the largest file it can and then performing
a secure overwrite to ensuring that all the disk space that was previously free becomes securely cleansed. You can utilize this feature of SDelete
to perform thin reclamation on zero-detect enabled HPE 3PAR StoreServ volumes by specifying the -z flag when running SDelete to writes zeros
to the free space.
One disadvantage of SDelete is that it does not support mount points so a volume must have a drive letter associated with it before the utility
can be run. Using the subst command one can temporarily attach a drive letter to a mount point before running SDelete.
It is recommended that applications are shut down before running SDelete as it can cause a file_system_full condition due to consuming all “free”
space. An alternative solution is to create a PowerShell script that uses fsutil to create a balloon file that is limited to a certain percentage of the
“free” space.
This is an example of using fsutil to zero 100 MiB of space:
fsutil file createnew
fsutil
file
zerotemp.txt
setvaliddata
104857600
zerotemp.txt
104857600
fsutil file setzerodata offset=0 length=104857600 zerotemp.txt del zerotemp.txt
Thin Reclamation for VMware vSphere
vSphere 5.0 introduced automatic space reclamation upon deletion of a VMDK. However, due to poor performance from some storage devices
when the UNMAP command was used, VMware recommended this feature was to be disabled on vSphere 5.0 hosts. As an alternative vSphere
5.1 Update 1 included an update to the vmkfstools command that provided a new option (-y) to issue UNMAPs to reclaim unused
space on a thin-provisioned volume. This new implementation is not an automated process and will need to be run manually. On vSphere
5.0 Update 1 and 5.2 the command should be run after changing to the root directory of the desired VMFS volume.
#
cd
/vmfs/volumes/<volume-name>
# vmkfstools -y <percentage of free space to reclaim>
The vmkfstools -y command creates a temporary “balloon” file in the datastore that can be up to the size of the free space available in the
datastore, and then it issues UNMAPs for the blocks of the balloon file. However, it is recommended that the reclaim percentage is not more than
60 percent as the resulting balloon file temporarily uses space on the datastore that could cause the deployment of new virtual disks to fail while
the vmkfstools command is running. This is an important consideration as there is no way of knowing how long a vmkfstools -y
operation will take to complete. It can be anywhere from few minutes to several hours depending on the size of the datastore and the amount of
space that needs to be reclaimed.
Technical white paper
Page 42
There is an alternative method that can be faster and takes advantage of the ASIC-based zero-detect capability of HPE 3PAR StoreServ. Run the
vmkfstools command with the -d eagerzeroedthick option to create a zerofile that can then be removed.
#
cd
/vmfs/volumes/<volume-name>
# vmkfstools -c <size of space to reclaim> -d eagerzeroedthick zerofile
#
rm
zerofile-flat
These methods can be run when the file system is online but if reclaims of very large amounts of data (100’s of GB) are required consideration
should be given to running it during a quiet period so the reclaims do not adversely impact other workloads on the storage.
In vSphere 5.5, the vmkfstools -y command is deprecated and in its place a new esxcli command UNMAP namespace has been added
that allows deleted blocks to be reclaimed on thin-provisioned LUNs that support the UNMAP primitive. The reclaim mechanism has been
enhanced so that the reclaim size can be specified in blocks instead of a percentage value to make it more intuitive to calculate, and the unused
space is reclaimed in increments instead of all at once. However, reclaim operation is still manual.
To reclaim unused storage blocks on a vSphere 5.5 VMFS datastore for a thin-provisioned device, run the command:
# esxcli storage vmfs unmap -l volume_label | -u volume_uuid [-n number]
The datastore to operate on is determined by either using the -l flag to specify the volume label or -u to specify the universal unique identifier
(UUID). The optional -n flag sets the number of VMFS blocks to UNMAP per iteration. If it is not specified, the command uses a default value of 200.
The VMware hypervisor does not report disks as being thin provisioning capable to the guest OS, when using VMDKs therefore to reclaim thinly
provisioned storage you must leverage the zero-detection capabilities of HPE 3PAR StoreServ. This means using standard file system tools (such as
SDelete in Microsoft Windows, dd in UNIX/Linux) to write zeros across deleted and unused space in the file system of a VM. The zeros will be
autonomically detected by the HPE 3PAR ASIC and the disk space they were consuming will be freed up and returned to the thin-provisioned volume.
If running VMware ESXi 4.1, it is strongly recommended to install the HPE 3PAR VAAI plug-in before creating EZT VMDKs as it will speed up
their creation by 10X to 20X.
Thin Reclamation for Linux
Linux uses the term discard for the act of informing a storage device that blocks are no longer in use. This is because it uses the same mechanism to
support both TRIM commands on ATA SSDs and UNMAP commands on SCSI devices, so discard was chosen as a protocol neutral name. RHEL 6
was the first major distribution to support discards and offers both real-time and batched support.
Real-time support is offered by an ext4 or XFS file system when mounted with the -o discard option; the default is not to issue discards. When
data or files are deleted on a discard enabled ext4 file system, UNMAPs are generated to free up space on the thinly provisioned virtual volume
on the storage. The LVM and the device-mapper (DM) targets also support discards so space reclaim will also work on file systems created on
LVM and/or DM volumes.
For example, to mount the DM device tpvv_lun on /mnt with discards enabled, run:
# mount -t ext4 -o discard /dev/mapper/tpvv_lun /mnt
This will cause the RHEL 6 to issue the UNMAP command, which in turn causes space to be released back to the array from the TPVV volumes
for any deletions in that ext4 file system. This is not applicable for fully provisioned virtual volumes.
In addition, the mke2fs, e2fsck, and resize2fs utilities also support discards to help ensure the TPVV volumes are enhanced when
administration tasks are performed.
There is also batched discard support available using the fstrim command. This can be used on a mounted file system to discard blocks, which
are not in use by the file system. It supports ext3 and ext4 file systems and can also re-thin a file system not mounted with the discard option.
Technical white paper
Page 43
For example, to initiate storage reclaim on the /mnt file system, run:
# fstrim -v /mnt
/mnt: 21567070208 bytes were trimmed
The fstrim can be run when the file system is online but as it generates UNMAP requests for all the unused blocks in one go consideration
should be given to running it during a quiet period so the reclaims do not adversely impact other workloads on the storage.
It is also possible to configure the Logical Volume Manager (LVM) to automatically issue discards to a logical volumes’s underlying physical
volume(s) when the logical volume is no longer using the physical volumes’ space (e.g., by using the lvremove or lvreduce commands). This
can be done by enabling the option issue_discards in the LVM configuration file:
$ cat /etc/lvm/lvm.conf
devices {
}
issue_discards = 1
Then when removing a volume, you would see the following message:
# lvremove vg1/lvol0
Do you really want to remove and DISCARD logical volume lvol0? [y/n]:
The Linux swap code will also automatically issue discard commands for unused blocks on discard-enabled devices and there is no option to
control this behavior.
Thin Reclamation for HP-UX
HP-UX systems using Veritas Storage Foundation 5.1 or higher can reclaim space associated with file deletions or file shrinking (see the Thin
Reclamation for Symantec Storage Foundation section for more details). However, non-VxFS file systems or VxFS file systems on LVM volumes
will need to reclaim space using a “zerofile” script that writes zeros to all allocated but unused space on a file system. The zero-detection
capability of the HPE 3PAR ASIC will intercept the blocks of zeros being written and automatically trigger the reclamation of the space.
Thin Reclamation for UNIX
On UNIX systems or Linux distributions that do not support discard, you will need to reclaim space using a “zerofile” script that writes zeros to all
allocated but unused space on a file system. The zero-detection capability of the HPE 3PAR ASIC will intercept the blocks of zeros being written
and automatically trigger the reclamation of the space.
The script would use the dd command to copy zero data blocks from the /dev/zero device to a file in the file system. However, it is recommended
that the size of the space to zero is not more than 70 percent of the free space as a very large zerofile could cause the creation of new files to fail
while the zerofile script is running.
Thin Reclamation for HPE OpenVMS
Automatic space reclamation upon deletion is possible in OpenVMS by leveraging the erase-on-delete capability to write erasure patterns of
zeros. The zero-detection capability of the HPE 3PAR ASIC will intercept the blocks of zeros being written and automatically trigger the
reclamation of the space.
The inclusion of the /ERASE qualifier with the DELETE or the PURGE command causes the system to write an erasure pattern of zeros over the
entire file location when you delete or purge that file. Users can use this qualifier voluntarily or this can be made automatic by including the
following command definitions in the system login command procedure (usually SYS$MANAGER:SYLOGIN.COM):
DEL*ETE :== “DELETE/ERASE”
PUR*GE :== “PURGE/ERASE”
Technical white paper
Page 44
However, any user can bypass these definitions by adding the /NOERASE qualifier to the DELETE or PURGE commands. To guarantee
erase-on-delete, it is possible to turn on the feature for the entire volume by using the DCL command SET VOLUME/ERASE_ON_DELETE.
Once this is set, when files are deleted they are overwritten with an erasure pattern of zeros.
To completely erase the volume and enable erase-on-delete for the volume at volume initialization, use the DCL command INITIALIZE/ERASE.
Thin Reclamation for Symantec Storage Foundation
Since the introduction of Symantec Storage Foundation 5.1 hosts with VxFS file systems managed by the Veritas Volume Manager (VxVM)
software can also reclaim space associated with file deletions or file shrinking on HPE 3PAR StoreServ Storage.
The space reclamation is not automatic, the VxFS file system informs the VxVM about deleted blocks and a VxVM command has to be manually
run which send WRITE SAME SCSI commands with the UNMAP bit turned on to the HPE 3PAR StoreServ. No tool to write zeros to the deleted
space in the file systems is required for reclaiming the space.
The list of disks whose allocated but unused space can be reclaimed is given by the command executed on the host:
#
vxdisk -o thin,fssize -u m
list
This will display the VV allocated space and the file system usage. The space reclamation is initiated by the VxVM vxdisk command:
#
vxdisk
reclaim
[<disk>|<dg>|<encl>]
By default, the reclamation does not affect unmarked space, which is the unused space between subdisks. If a LUN has a lot of physical space
that was previously allocated, the space between the subdisks might be substantial. Use the -o full option to reclaim the unmarked space.
# /opt/VRTS/bin/fsadm -V vxfs -R /<VxFS_mount_point>
To monitor the reclamation status, run the following command:
# vxtask list
TASKID PTID TYPE/STATE
171
RECLAIM/R
PCT
PROGRESS
00.00% 0/41875931136/0 RECLAIM vol100 dg100
The vxrelocd daemon tracks the disks that require reclamation. The schedule for reclamation can be controlled using the vxdefault command. The
reclaim_on_delete_wait_period parameter specifies the number of days after a volume or plex is deleted when VxVM reclaims the
storage space. The default value is 1, which means the volume is deleted the next day. A value of -1 indicates that the storage is reclaimed
immediately and a value of 367 indicates that the storage space can only be reclaimed manually using the vxdisk reclaim command. The
reclaim_on_delete_start_time parameter specifies the time of day when VxVM starts the reclamation for deleted volumes and this
defaults to 22:10.
To completely disable thin-reclaim operations, add the parameter reclaim=off to the /etc/vxdefault/vxdisk file.
Thin Reclamation for Oracle databases
During an Oracle Automatic Storage Management (ASM) database lifecycle, the utilization of allocated storage capacity in a thin-provisioned
volume can decrease as changes are made to the database through common operations such as:
• Dropping of a tablespace or database upon deletion of transient data
• Resizing of an Oracle datafile upon shrinking a tablespace
• Addition of new disks to an ASM disk group to accommodate growth or load balance performance
These changes result in the creation of unused ASM disk space that can built up over time and although this space is available for reuse within
ASM, it remains allocated on the storage array. The net result is that the storage utilization on the array eventually falls below desirable levels.
Technical white paper
Page 45
To solve this problem, HPE and Oracle have partnered to improve storage efficiency for Oracle Database 10g and 11g environments by
reclaiming unused (but allocated) ASM disk space in thin-provisioned environments. The Oracle ASRU is a stand-alone utility that works with
HPE 3PAR Thin Persistence software to reclaim storage in an ASM disk group that was previously allocated but is no longer in use. Oracle ASRU
compacts the ASM disks, writes blocks of zeroes to the free space, and resizes the ASM disks to original size with a single command, online and
non-disruptively. The HPE 3PAR StoreServ, using the zero-detect capability of the HPE 3PAR ASIC, will detect these zero blocks and reclaim any
corresponding physical storage.
You can issue a SQL query to verify that ASM has free space available that can be reclaimed:
SQL>
select
NAME
name,
state,
STATE
type,
total_mb,
free_mb
from
v$asm_diskgroup
where
name
=
‘LDATA’;
TYPE TOTAL_MB FREE_MB
-----
------- ------
------- -------
LDATA
MOUNTED EXTERN
1023984
610948
Run the Oracle ASRU utility as the Oracle user with the name of the disk group for which space should be reclaimed:
# bash ASRU LDATA
Checking the system ...done
Calculating
Writing
Resizing
the
the
new
data
the
to
sizes
a
of
the
disks
file ...done
...done
disks...done
/u03/app/oracle/product/11.2.0/grid/perl/bin/perl -I /u03/app/oracle/product/
11.2.0/grid/perl/lib/5.10.0 /home/ora/zerofill 5 /dev/oracleasm/disks/LDATA2
129081 255996 /dev/oracleasm/disks/LDATA3 129070 255996 /dev/oracleasm/disks/
LDATA4 129081 255996 /dev/oracleasm/disks/LDATA1 129068 255996
126928+0 records in
126928+0 records out
133093654528
bytes
GB)
copied,
2436.45
seconds,
54.6
MB/s
(133
GB)
copied,
2511.25
seconds,
53.0
MB/s
(133
GB)
copied,
2514.57
seconds,
52.9
MB/s
126915+0 records in
(133
126915+0 records out
133080023040
bytes
126926+0 records in
126926+0 records out
133091557376
bytes
126915+0 records in
126915+0 records out
133080023040 bytes (133 GB) copied, 2524.14 seconds, 52.7 MB/s
Calculating
the
new
sizes
of
the
disks
...done
Resizing the disks...done Dropping the file ...done
Technical white paper
Page 46
Appendix C—File Systems and Thin Deduplication
File systems with lots of common files such as home directory shares are ideal candidates for deduplication. However, the space savings achieved
can vary greatly depending on the file system in use. Thin Deduplication uses a granularity, also called chunk size, of 16 KiB and therefore
optimum deduplication is achieved when the file system aligns the files with this granularity.
They are many different file systems choices available across the supported operating systems and they work in very different ways with
deduplication. Some file systems work perfectly, some need online tuning, some need reformatting with different options and some cannot
be changed. In general, older file system technologies designed to work with discrete drives do not perform as well as modern log structured
file systems.
It is often thought that deduplication ratio is dependent on the file system block size alone but it actually depends on block size, extent size and
alignment, all of which vary from file system to file system. Block size is the basic unit of storage for the file system and it is usually, but not
always, page size of the processor architecture. Extent size is the number of contiguous blocks that can be written. Alignment is the starting
offset of a file, most file systems align to block size boundaries.
For normal volumes changing the block size or alignment can increase the space usage for very small files (although it may improve performance
for larger files). However, by aligning the files the increase in deduplication should more than offset any space increase for small files so the
overall space usage will reduce.
Note that with virtualized systems the file systems inside the virtual machine, which may be under the control of a tenant, should be aligned well
as the file system of the hypervisor.
Microsoft Windows
New Technology File System (NTFS)
NTFS is the default file system for Microsoft Windows and by default has an allocation unit of 4 KiB. For optimal deduplication consider setting
the allocation unit to 16 KiB or a multiple of 16 KiB when the file system is created. Do this by selecting the desired the allocation unit in the
format dialog box when creating the file system. 10
Using a 16 KB allocation unit size not only ensures that all files start on a 16 KiB boundary, relative to the offset of the partition, but also any
UNMAP operations will also be aligned to 16 KiB boundaries.
To determine the allocation unit size of an existing NTFS file system run the fsutil fsinfo ntfsinfo command and check the Bytes per
Cluster value. The following example shows a file system with a 16 KiB allocation unit.
C:\>fsutil fsinfo ntfsinfo f:
NTFS Volume Serial Number: 0x004eab4c4eab38f4
Version:
3.1
Number Sectors: 0x0000000006f997ff
Total
Clusters:
0x00000000001be65f
Free Clusters: 0x00000000000317a5
Total Reserved: 0x0000000000000000
Bytes
Per
Sector:
512
Bytes Per Physical Sector: 512
10
For Microsoft Windows PowerShell Format-Volume cmdlet syntax see technet.microsoft.com/en-us/library/hh848665.aspx.
Technical white paper
Page 47
Bytes Per Cluster: 16384
Bytes Per FileRecord Segment: 1024
Clusters
Per
FileRecord
Segment:
0
Mft Valid Data Length: 0x0000000000040000
Mft Start Lcn: 0x0000000000018000
Mft2 Start Lcn: 0x0000000000000001
Mft Zone Start: 0x0000000000018000
Mft Zone End: 0x0000000000019920
RM Identifier: D252D40C-592D-11E4-803D-A0E201E4931F
The following example shows the effect of allocation unit on dedup ratios. Five TDVVs were created and then formatted with NTFS file
systems using allocation units of 4 KiB–64 KiB. Four copies of the Linux kernel source tree, which contains many small files, were unpacked
into each file system.
cli% showvv -space ntfs*
---Adm---- ---------Snp---------Id
Name Prov Type
149
ntfs_4k tdvv base
---(MB)---
--(MB)---
Rsvd
Used Rsvd
6233
0
6781
----------Usr-----------
-(% VSize)--
Used Used
0
0.0
Wrn Lim
---(MB)---Rsvd
--
--
11831
-(% VSize)--
Used Used
8660
1.7
-----(MB)------ -Capacity Efficiency-
Wrn Lim Tot_Rsvd
VSize
0
0
18612 512000
Compaction Dedup
34.4
1.1
151
ntfs_8k tdvv base
6200
5679
0
0
0.0
--
--
11006
7889
1.5
0
0
17206 512000
37.7
1.4
152
ntfs_16k tdvv base
3908
3498
0
0
0.0
--
--
7748
4853
0.9
0
0
11656 512000
61.3
3.1
153
ntfs_32k tdvv base
3907
3491
0
0
0.0
--
--
7748
4846
0.9
0
0
11655 512000
61.4
3.1
154
ntfs_64k tdvv base
3907
3494
0
0
0.0
--
--
7748
4847
0.9
0
0
11655 512000
61.4
3.1
It is clear that the file system with allocation units of 16 KiB and above have significantly higher dedup ratios and from the compaction ratios you
can see that they also save more space even though the file systems contained only small files.
Setting boot disk allocation units
While changing the NTFS allocation unit on a file system requires a reformat, it is relatively easy to do for non-system disks. The Windows
operating system unfortunately does not give an option to set the allocation unit during installation so the system disk will have 4 KiB by default.
Although there is little dedupable data within a single Windows OS instance, having a 4 KiB allocation unit reduces the deduplication achievable
between multiple Windows Servers or virtual machines. Therefore, it is still desirable to set 16 KiB allocation units for boot disks.
The Windows installation does not give an option to change the allocation unit on Windows boot disks and Microsoft does not have a
documented procedure. The challenge is that Windows boots from a hidden System Partition which must have a 4 KiB allocation unit. A solution
is at the start of the installation process is to create a small (at least 500 MB) System Partition formatted with a 4 KiB allocation unit and then the
main Windows partition formatted with a 16 KiB allocation unit.
The following is an example of the procedure:
1. Start Windows installation
2. Press Shift + F10 to open CMD
a. disk part
b. select disk 0
c. create partition primary size=1000
d. format quick
e. create partition primary
f.
format quick unit=16K
Technical white paper
Page 48
3. Close CMD
4. Install Windows
This method has been tested on Windows Server 2008 and Windows Server 2012.
If the Windows System Partition is not formatted with a 4 KiB allocation unit the message shown in figure 15 will be seen during installation.
Figure 15. System partition error
Resilient File System (ReFS)
ReFS is a new file system introduced with Windows Server 2012. It was designed to have higher data integrity and scalability than NTFS.
Internally ReFS allocates data on disk in clusters of 64 KiB and metadata in clusters of 16 KiB. Therefore, no tuning is necessary for use with Thin
Deduplication.
Formatting a volume with ReFS can be done from the UI by choosing ReFS as the file system in the drop down of the Format dialog box or from
the command line using the following syntax:
Command-Line
Format /fs:ReFS /q J:
PowerShell Equivalent
Format-Volume -DriveLetter J -FileSystem ReFS
Microsoft Hyper-V
Windows Server 2012 brought many new virtualization improvements, including the introduction of the VHDX file format. In Windows Server
2008, Hyper-V used a virtual hard disk (VHD) file format that had a 2 TiB limit. The new VHDX file format has a maximum capacity 64 TiB. The
advantages of VHDX aren’t limited to improved capacity, it has a 4 KiB logical sector size that improves performance and can increase
deduplication compared with VHD files. It is therefore recommended to convert VHD files to VHDX format when using thin volumes.
Technical white paper
Page 49
Use one of the following methods to convert between formats:
1. Use the Hyper-V UI in Windows Server 2012, select edit the VHD file and choose to convert to VHDX.
2. Use the new Convert-VHD PowerShell cmdlet. 11
Note that the VHD conversion must be done with the VM shut down.
Furthermore, the virtual machines using the VHDs may have file systems that have alignments that are different from the host file system. It is
therefore recommended to follow the operating system guidelines from this section for each virtual machine.
VMware vSphere
The VMFS-5 file system of VMware vSphere 5.x uses a 1 MiB block size, the VMFS versions used in prior releases used block sizes that depended
on the file system size but were always a multiple of 1 MiB. Due to these large block sizes there is no tuning necessary for deduplication with
VMware.
However, the virtual machines using the VMDK files within the VMFS may have file systems that have alignments that are not optimal for
deduplication. It is therefore recommended to follow the operating system guidelines from this section for each virtual machine.
Oracle Solaris
ZFS
ZFS is a combined file system and logical volume manager for Oracle Solaris systems. It uses variable-sized blocks, with 128 KB as the default size.
While this is good for deduplication within a file, it aligns files on physical block boundaries which are 512 bytes by default and this significantly
reduces deduplication between files containing like data. The inter-file deduplication rate can be increased by configuring Solaris to use 16 KiB
physical block sizes for HPE 3PAR StoreServ volumes.
On SPARC based Solaris platforms the SAN SCSI Disk driver configuration file /kernel/drv/ssd.conf should be edited to include the
following line:
ssd-config-list=“3PARdataVV”,“physical-block-size:16384”;
On Intel® based Solaris platforms the SCSI Disk driver configuration file /kernel/drv/sd.conf should be edited to include the following line:
sd-config-list=“3PARdataVV”,“physical-block-size:16384”;
After the system is rebooted any new zpools created should have a 16 KiB block size. You can verify this by running the command zdb -L. The
zpools created with 16 KiB block sizes will have an ashift value of 14 and the zpools created with the default block size will have an ashift value of
9 as shown in this example.
# zdb -L | grep ashift
ashift: 14
ashift:
11
9
For Microsoft Windows PowerShell Convert-VHD cmdlet syntax see: technet.microsoft.com/en-us/library/hh848454.aspx.
Technical white paper
Page 50
The following example shows the difference in dedup ratio between a zpool created with 16 KiB block size and the default block size. Ten copies
of the same 1 GB file were copied into to each file system.
cli% showvv -space zfs*
---Adm---- ---------Snp---------Id
Name Prov Type
12
zfs_def tdvv base
13
zfs_16K tdvv base
---(MB)--Rsvd
256
--(MB)---
Used Rsvd
256
9
0
9
0
----------Usr-----------
-(% VSize)--
Used Used
0
0.0
0
0.0
Wrn Lim
--
--
--
--
---(MB)---Rsvd
5632
-(% VSize)--
Used Used
10
0.0
0
0.0
5632
-----(MB)------ -Capacity Efficiency-
Wrn Lim Tot_Rsvd
0
0
5888
0
0
5888
VSize
Compaction Dedup
4.8
1.2
51200
39.4
9.9
51200
It is clear that with the default block size there is very little deduplication due to the 512 byte alignment but with the 16 KiB block size there is
almost perfect deduplication.
UNIX file system (UFS)
UFS is the original Solaris file system and is based on the BSD Fast File System. The maximum file system size is 1 TB on Intel Solaris (x86) and
16 TB on SPARC Solaris. The default logical block size which files are aligned to is 8 KiB and this is also the maximum block size, therefore to
obtain the best deduplication rate it may be necessary to migrate to ZFS with the aforementioned block size tuning.
Linux
There are many different file systems available on Linux but the most widely used are ext3, ext4, and XFS.
Ext3 is an extension of the ext2 file system with journal support. It does not support extents and therefore files will be fragmented into block
sized chucks which have a default, and maximum, size of 4 KiB. Ext3 was introduced to the Linux kernel in 2001 and for a long time was
considered to be the standard Linux file system.
Ext4 superseded ext3 and replaced the traditional block mapping scheme by extents. An extent is a range of contiguous physical blocks,
improving large file performance and reducing fragmentation. Ext4 uses a delayed allocation mechanism that buffers data and allows it to
allocate blocks in groups. Ext3 file systems can be migrated offline to ext4 by using the tune2fs command, however, exiting files will retain
their ext3 layout therefore backup and restore operations are required to take full advantage of the ext4 improvements. Ext4 was made the
default file system in Red Hat Enterprise Linux 6.
Extend file system (XFS) is a highly scalable, high-performance file system with metadata journaling and online defragmentation. Compared to
ext4 it excels at multi-threaded, parallel I/O workloads and scales higher with support for file systems up to 16 EB and files of 8 EB. There is no
direct conversion available from ext3/ext4 to XFS so migration involves backup and restore operations. XFS was the default file system in SUSE
Linux Enterprise Server from SLES8-SLES11 and is the default file system in Red Hat Enterprise Linux 7. With SLES12, the B-tree file system
(Btrfs) is the default file system for the operating system but XFS is the default for all other use cases.
The following example shows three TDVVs with ext3, ext4, and XFS file systems created on them. Four copies of a Linux system backup were
restored to each file system.
cli% showvv -space ext3 ext4 xfs
---Adm---- ---------Snp------------(MB)---
--(MB)---
Id
Name Prov Type
ext3 tdvv base 13306 12037
0
20
21
19
Rsvd
Used Rsvd
ext4 tdvv base
5552
4887
xfs tdvv base
2989
2526
----------Usr-----------
-(% VSize)--
Used Used
0
0.0
0
0
0
0
Wrn Lim
---(MB)----
-(% VSize)--
Rsvd
Used Used
--
--
17401 12831 12.5
0.0
--
--
8583
5208
0.0
--
--
5668
2688
-----(MB)------ -Capacity Efficiency-
Wrn Lim Tot_Rsvd
VSize
Compaction Dedup
0
0
30706 102400
4.1
1.6
5.1
0
0
14136 102400
10.1
2.3
2.6
0
0
8658 102400
19.6
3.5
-------------------------------------------------------------------------------------------------------------------------------3 total
21847 19450
0
0
31652 20727
53500 307200
7.6
2.2
Technical white paper
Page 51
It is clear from this example how the use extents allows ext4 to give more than double the space savings of ext3. However, the more
sophisticated XFS provides the greatest space savings and therefore consideration should be given to migrating existing ext3 or ext4 file
systems to XFS when using thinly deduped volumes.
Symantec Storage Foundation
The Veritas File System (VxFS) of Symantec Storage Foundation is available on many platforms including AIX, HP-UX, Linux, Solaris, and
Windows. It is an extent based file system that allocates storage in groups of extents rather than a block at a time.
VxFS file systems are formatted with a fixed block size ranging from 1 KiB to 8 KiB which represents the smallest amount of disk space allocated
to a file. The block size largely used to define the maximum size of the file system, a block size of 1 KiB allows a maximum file system size of up to
32 TB and a block size of 8 KiB allows for a file system up to 256 TB.
The extent based nature of VxFS means that no tuning is necessary for use with Thin Deduplication as the alignment is not based on the block
size. This is demonstrated by the following example which shows two TDVVs, one with a 1 KiB block size file system and the other with an 8 KiB
file system. Four copies of a backup were restored to each file system.
cli% showvv -space vxfs*
---Adm---- ---------Snp---------Id
Name Prov Type
156
vxfs_1k tdvv base
157
vxfs_8k tdvv base
---(MB)---
--(MB)---
Rsvd
Used Rsvd
7889
0
9271
8743
0
8393
----------Usr-----------
-(% VSize)--
Used Used
0
0.0
0
0.0
Wrn Lim
--
--
--
--
---(MB)----
-(% VSize)--
Rsvd
Used Used
4657
2.3
9292
5167
2.5
8627
-----(MB)------ -Capacity Efficiency-
Wrn Lim Tot_Rsvd
VSize
0
0
17021 204800
0
0
18563 204800
Compaction Dedup
16.3
3.9
14.7
3.6
-------------------------------------------------------------------------------------------------------------------------------2 total
17664 16632
0
0
17919
9824
35584 409600
15.5
3.8
Thin Deduplication achieves good results with both file systems with the 1 KiB block size having slightly a lower space usage (~10 percent) and a
higher dedup ratio than the one with the 8 KiB block size.
HP-UX
Journaled File System (JFS)
JFS is an implementation of the Veritas File System (VxFS) and due to the extent based nature of VxFS means that no tuning is necessary for
use with Thin Deduplication.
Hierarchical File System (HFS)
HFS is based on the BSD Fast File System. HFS file systems are not commonly used as their maximum size is limited to 128 GB. However, HFS
does support a wider range of block sizes (4 KiB-64 KiB) than other FFS implementations and therefore is suitable for use with Thin
Deduplication.
The default primary block size of HFS is 8192 bytes and can be checked by looking at the bsize field of the dumpfs command output.
$
dumpfs
/dev/rdisk/disk63
**/dev/rdisk/disk63:
magic
5231994 time
Fri Mar 20 09:30:54 2015
sblkno
16
24
iblkno
sbsize
2048
cgsize
2048
cgoffset
size
104857600
ncg
12800
cblkno
32
32
blocks
dblkno
192
cgmask
0xfffffff0
102604584
bsize
8192
shift
13
mask
0xffffe000
fsize
1024
shift
10
mask
0xfffffc00
frag
8
shift
3
fsbtodb 0
maxbpg
256
minfree 10%
Technical white paper
maxcontig
1
Page 52
rotdelay 0ms
rps
60
csaddr
192
cssize
204800
shift
9
mask
0xfffffe00
ntrak
16
nsect
32
spc
512
ncyl
204800
ipg
1280
cpg
16
bpg
1024
fpg
8192
nindir
2048
inopb
64
nspf
1
To create a new HFS file system suitable for use with Thin Deduplication use the -b flag of the newfs command to set a 16 KiB block size.
$ newfs -F hfs -b 16384 -f 4096 /dev/rdisk/disk63
mkfs (hfs): Warning - 608 sector(s) in the last cylinder are not allocated.
mkfs (hfs): /dev/rdisk/disk63 - 104857600 sectors in 168042 cylinders of 16 tracks, 39 sectors 107374.2Mb in 10503 cyl groups
(16 c/g, 10.22Mb/g, 1536 i/g)
IBM AIX
Journaled File System (JFS) is a 64-bit extent based file system. The initial version of JFS (also called JFS1) can support a file system of up to
1 TB and a file size of 64 GB. JFS2 (the Enhanced Journaled File System) is similar to JFS, but supports a file system of up to 32 TB and a file size
of 16 TB.
Both JFS and JFS2 use a block size of 4 KiB and use extents of up to 64 GiB to allocate blocks to files. JFS2 also divides the space into allocation
groups which are ranging from 8 MiB to 64 MiB which allow the resource allocation policies to cluster disk blocks for related data. Due to the
extent based nature of JFS no tuning is necessary for use with Thin Deduplication.
HPE OpenVMS
OpenVMS uses the Files-11 file system with On-Disk Structure Levels 2 (ODS-2) or 5 (ODS-5). The minimum unit of storage allocation is known
as a cluster and is sized in terms of 512 byte blocks.
For ODS-5 disks, the default cluster size is 16 (8 KiB) and the minimum value allowed is defined by the following formula:
For ODS-2 disks, the default cluster size depends on the disk capacity; disks with less than 50,000 blocks have a default of 1. Disks that are larger
than 50,000 blocks have a default of either 16 (8 KiB) or the result of the following formula, whichever is greater:
The cluster size is set when the file systems are created using INITIALIZE command to make the disk volume structures. The syntax of the
INITIALIZE command is:
INITIALIZE /CLUSTER_SIZE=number-of-blocks device-name[:] volume-label
For optimal deduplication initialize the volumes with a cluster size which is a multiple of 32.
Technical white paper
Learn more at
hp.com/go/3PARStoreServ
Sign up for updates
Rate this document
© Copyright 2012–2016 Hewlett Packard Enterprise Development LP. The information contained herein is subject to change without
notice. The only warranties for Hewlett Packard Enterprise products and services are set forth in the express warranty statements
accompanying such products and services. Nothing herein should be construed as constituting an additional warranty.
Hewlett Packard Enterprise shall not be liable for technical or editorial errors or omissions contained herein.
Intel is a trademark of Intel Corporation in the U.S. and other countries. Linux is the registered trademark of Linus Torvalds in the U.S. and
other countries. Microsoft, Windows, and Windows Server are either registered trademarks or trademarks of Microsoft Corporation in the
United States and/or other countries. Oracle is a registered trademark of Oracle and/or its affiliates. Red Hat is a registered trademark of
Red Hat, Inc. in the United States and other countries. SAP is the trademark or registered trademark of SAP SE in Germany and in several
other countries. UNIX is a registered trademark of The Open Group. VMware is a registered trademark or trademark of VMware, Inc. in the
United States and/or other jurisdictions.
4AA3-8987ENW, March 2016, Rev. 9
Download PDF
Similar pages