Read a White Paper

Read a White Paper
Datadobi White Paper
NAS Storage Platform Migration
March 2016. Copyright © Datadobi, all rights reserved.
Datadobi believes the information in this publication is accurate as of its publication date.
The information is subject to change without notice.
Use, copying, and distribution of any Datadobi software described in this publication requires an
applicable software license.
DobiMiner® is a trademark registered in the US Patent and Trademark Office. All other trademarks
used herein are the property of their respective owners.
Table of contents
Introduction .......................................................................................................... 5
The NAS Platform Migration Process ................................................................... 7
Discovery and Analysis ................................................................................. 7
Planning and Design ..................................................................................... 8
Migration Execution ..................................................................................... 10
Verification and Reporting ........................................................................... 11
Tools and Support ....................................................................................... 12
NAS Platform Migrations with DobiMiner ............................................................ 13
DobiMiner Deployment ................................................................................ 14
DobiMiner Proxies ................................................................................ 15
DobiMiner Core..................................................................................... 16
DobiMiner System Requirements.......................................................... 16
Supported Migration Scenarios ............................................................. 17
Audience for this White Paper ....................................................................... 6
Migration Configuration ............................................................................... 18
Configuration of Integrated File Servers ................................................ 18
Configuration of SMB Shares or NFS Exports....................................... 18
Configuration of Migration Pairs ............................................................ 19
Configuration of Migration Paths ........................................................... 19
Validation of Migration Configuration .................................................... 21
Import Bulk Configurations .................................................................... 22
DobiMiner Migration Workflow ............................................................................ 23
DobiMiner Migration Phases ....................................................................... 23
Discovery and Analysis Phase .............................................................. 23
Planning and Design Phase .................................................................. 23
Migration Execution Phase ................................................................... 24
Verification and Reporting Phase .......................................................... 26
Functionality in Detail .................................................................................. 27
Discovery and Analysis ......................................................................... 27
Planning and Design ............................................................................. 29
Bandwidth Throttling and Scheduling .................................................... 34
Monitoring, Troubleshooting, and Reporting ......................................... 36
Validation .............................................................................................. 38
Conclusion ......................................................................................................... 40
1 Introduction
Continuous growth of data and fast-paced technology changes are the typical drivers
for storage platform transitions. Additionally, companies want to consolidate some or
all of their existing systems to save on management overhead, infrastructure,
footprint, and energy costs.
Historically, storage platform transitions have caused companies a great deal of
expense, pain, and disruption due to a lack of proper tools. And this is far more
common on unstructured file or object data than on block data. Most storage platform
migrations are still carried out using tools that were created when a 10 Terabyte
migration was considered the norm and when only one protocol was required (SMB
or NFS).
In this era of rapid technology change, tools are needed that can help companies
transition from one storage platform to another no matter what size or complexity,
without the cost, pain, or disruption.
This white paper first describes the NAS Platform Migration Process in general and
the various challenges of platform migrations. It then illustrates how Datadobi
provides a comprehensive end-to-end solution to transition from one NAS storage
platform to another in a controllable, reliable, and cost effective way.
Chapter 2 explains that a platform migration is more than simply moving large chunks
of indistinct data from one system to another. Any platform migration follows the
same overall process that can be described in a repeatable number of steps. The
better you understand this process, the more capable you are in defining and running
a successful migration project. After reading this chapter you will understand what is
required for each platform migration and also what the challenges are of every
migration project.
Chapter 3 provides a detailed description of Datadobi's migration solution that is
based on our in-depth knowledge of data storage and vast experience with storage
platform migrations. As you will discover, our unique DobiMiner software assists you
throughout the entire migration from scoping to finalization. With DobiMiner your
migrations are not only easy to set up and transparent to monitor and control, but the
migration results will also meet your expectations and avoid unpleasant surprises or
many of the risks associated with manual script-based migration approaches.
1.1 Audience for this White Paper
The intended audience for this white paper includes all stakeholders in a platform
migration such as:
migration experts/consultants
project managers
solution architects
technical architects
system administrators
application owners
business unit managers
It is assumed that you have a basic understanding of different data storage platforms
and related technologies.
2 The NAS Platform Migration Process
When considering a platform transition, it is crucial to fully understand the
requirements and impacts of a migration on your business to ensure the best
outcome of the move to the new platform.
Migrating from one storage platform to another should be simple, fast, cost effective,
and above all reliable. However, due to the complexity of platform migrations, without
proper tools and methodology, you will likely encounter numerous challenges. This
chapter describes the platform migration process and the intricacies of each step
involved in a NAS storage platform migration. Once you understand the overall
migration process and its challenges, you are better equipped to select the right tools
and therefore perform a successful migration.
The process for each migration is basically the same regardless of the scope and
type of source and target infrastructure. Each migration should cover the following
phases and requirements:
Discovery and Analysis
Planning and Design
Migration Execution
Verification and Reporting
Tools and Support
The following sections describe each of these phases and requirements in detail.
2.1 Discovery and Analysis
Understanding exactly how the current platform is set up and how that fits with the
new platform can be a daunting challenge for a migration project team. Although a
person may be expert in one platform, very rarely are the persons responsible for a
migration project cognizant of all aspects of their source and target platforms in every
Many migration projects therefore begin without a proper and complete
understanding of the exact nature of the current platform or how this should be
translated to the new platform. In other words, the very foundation of their project is
already at risk.
To properly plan a platform migration and to make informed decisions, you need to
know as much as possible about both the source and target platforms before you
begin, including:
The data: amount, capacity, age, file size, ownership, type, frequency of
changes, etc.
The environment: number of servers, versions of hardware and software,
types of servers, compression settings, amount and listing of files systems,
volumes, shares or exports, security models and permissions, access
methods, quotas, disaster recovery and network interfaces, etc.
Once this information has been gathered, the arduous task of analysis begins. But
the discovery information can become so large and complex that the task of
organizing and analyzing it can significantly increase the risk and duration of the
However, without this analysis it is nearly impossible to define the scope, predictably
plan the migration, validate the migration results, and declare the project a success.
Traditional migration tools provide very little support for the discovery and analysis
phase of a project, leaving the project team to piece together what information can be
gleaned from often scarce documentation and putting this together in spreadsheets
and other inappropriate tools.
2.2 Planning and Design
Having discovered and analyzed all relevant information about the source and target
platforms, you can then begin the task of planning and designing your migration
project. Any migration planning should involve all stakeholders including:
data center personnel
business unit managers
senior management
application owners
home directory users
any relevant external parties
A key element of the planning and design phase is the design of the new platform.
Key decisions to be made during this phase are:
The data: Is all data to be migrated or only subsets of it; and if subsets what
criteria will you use to select the data? For example, do you want to exclude
certain data such as MP3 or TMP files; or other patterns in directories or files?
Will all data be migrated to exactly the same place on the target system or will
the organizational structure change and will characteristics such as retention
dates, ownership etc. remain the same?
The users and applications: Are all users and applications to be migrated;
are the permissions to remain the same? Does application data need to be
merged into combined locations; how will you handle orphaned users; do the
users or applications need to be consolidated to specific platforms, etc.?
The environment: Which servers must be migrated; what deadlines are
imperative; what requirements are on the Must Do- versus Like To Do-list;
which industry, business, infrastructure, or network restrictions apply; are
there priorities or sequences to be respected; is the new platform provisioned
correctly, etc.?
In addition to the target platform design, plans for the following also need to be put in
platform downtime windows
troubleshooting activities
reporting requirements
replication and back-up requirements
roll-back scenarios
issue escalation path
roles and responsibilities
resources required for data movement (servers, network, etc.)
Key during this phase is to consolidate all planning information from all stakeholders
and provide them with relevant information on time so they can make or adjust their
plans accordingly.
Traditional migration tools provide very little in the way of support for planning and
design leaving the project team to couple together spreadsheets, project planning
software, and other inappropriate tools increasing duration and risk, and potentially
reducing trust in the project.
2.3 Migration Execution
The actual migration phase can only begin when planning and design are complete
and all stakeholders are in agreement. In this phase, the data, metadata, user
permissions, shares, exports, etc. are all moved or recreated on the new platform.
The ultimate result of this phase should be a fast and smooth transition from one
platform to another with one hundred percent accuracy and full transparency to the
end users.
During this phase, the project team should be able to easily:
Predict migration durations.
Keep track of what should and what should not be migrated, what has started
and what has finished.
Control bandwidth or network consumption and load (connections) to the
Identify issues and retry after fixes.
Prepare switchover events such as creating read-only shares and exports on
the target server.
Predict the number and duration of switchover events.
Roll back the migration in case of issues endangering the business.
Switchover is a major activity in this phase and typically involves:
Enabling secure (read-only) access to the new platform for User Acceptance
Testing (UAT) before the actual switchover.
Removing the access of users and applications from the old platform.
Migrating any last-minute changes so that the target and source platform are
in sync (including the resolution of any last-minute issues that might occur at
this stage).
Enabling user and application access to the new storage platform.
Redirecting access to the new storage platform.
In addition, data-integrity issues need to be detected, aggregated, and root-caused
and solutions have to be found quickly throughout this phase to avoid disruption to
the migration schedule. Issues could include:
Lack of or incorrect permissions.
Remapping ownership of the user data.
Missing or incorrect shares and exports.
Data-format incompatibilities between the target and source platform.
Data unavailability and data loss.
Data corruption during the transfer.
Infrastructure issues.
Traditional migration tools focus only on data pumping leaving the project team to
guess at migration and switchover durations and painstakingly investigate and rootcause any issues or disruptions that occur by analyzing tens or even hundreds of log
2.4 Verification and Reporting
During all phases of the migration project, staying in control at all times is paramount
to success. Migration projects can take months or even years during which the
project, its stakeholders, and infrastructure are subject to unplanned changes such
Change in priorities impacting the sequence and planning of the project.
Infrastructure modifications impacting the scope, progress and planning.
Modified or new restrictions due to change in business operations.
New resource constraints.
Project stakeholders must be able to access reliable reports quickly and easily to
verify all aspects of the migration project. They must be able to:
Have real-time insights into the progress and health of the project.
Act on changes immediately with minimal effort and minimal impact on both
ongoing business operations and project goals.
Get immediate feedback on actions performed and on their impact on the
project milestones.
It is not only vital that all reporting can be provided quickly and easily, but that it also
provides auto-updated discovery of the source storage systems to watch out for
environmental changes outside the control of the project team.
For communication to all stakeholders involved, the monitoring and reporting levels
should include:
High-level traffic light reporting for management.
Mid-level progress reporting for project managers.
In-depth reporting down to the file level for the technical team.
Final reporting and sign-off should include the creation of:
A full inventory of what was migrated for safe keeping in case of future audits
(including a full list of every migrated file).
An extensive human readable final report for communication and sign-off by
all stakeholders.
Only when all stakeholders understand how the actual migration compares against
the original scope, the explanation for any exceptions, and the plan of action for
additional activities or projects, can the migration be signed off and considered to be
successfully finalized.
Traditional migration tools provide simple means for verification and support. What
can be gathered by the project from these tools or log files provides only rudimentary
information which can hardly be relied on as a source of good quality data and
therefore high quality decision making.
2.5 Tools and Support
Integral to the migration project are the tools used to carry out the migration.
Traditional migration tools include data pumping tools such as Robocopy, data
comparison tools such as Beyond Compare, and other supporting tools such as
spreadsheets and project planning applications.
However, these tools lack significant functionality, provide no end-to-end support,
provide limited troubleshooting facilities, and generally are ill equipped for today's
platform migration projects.
What is needed is a dedicated tool, purpose built to successfully carry out a migration
project with the least possible disruption and at the lowest cost.
Datadobi's unique migration software DobiMiner is such a tool and is described in
detail in the following sections.
3 NAS Platform Migrations with DobiMiner
As can be seen from the previous chapter, the single biggest issue confronting the
project team that will carry out a NAS platform migration is the many and varied tasks
that need to be performed. Without the aid of dedicated migration software, a NAS
platform migration can potentially cause:
High cost of both internal and external personnel.
Increased risk across all aspects of the project from data integrity to reputation
of the migration team.
Extended durations and increased switchover events.
Skilled staff distracted by migrations instead of more important work.
Disruption to the business.
Lack of proper reporting and governance.
Datadobi has taken a fresh look at NAS platform migrations and has re-written the
playbook. Our DobiMiner software approaches migrations holistically, encapsulates
the many disparate migration processes, and combines them with best practices to
create a single, integrated, end-to-end, automated migration solution.
DobiMiner is not a patchwork of existing tools or applications. DobiMiner is purposebuilt for executing complex NAS migrations; it has been engineered with new and
exciting technology and only has reused the best of the best existing software when
DobiMiner is built on big-data technology with the capability to analyze billions of
metadata files. Based on this analysis, DobiMiner is able to execute a controlled and
automated migration without the need for constant user interaction. DobiMiner's
workflow engine and automation features provide users the ultimate experience in
NAS platform migrations.
With DobiMiner, Datadobi has re-imagined the entire migration process and
produced a software suite that releases the project team from costly, risky, and timeconsuming tasks allowing them to focus on more important business issues.
3.1 DobiMiner Deployment
DobiMiner's flexible architecture allows it to fit in any NAS infrastructure environment.
The heart of the DobiMiner architecture is the DobiMiner Core. DobiMiner Core is
used to command the DobiMiner proxies which are distributed as needed across the
network to increase bandwidth. DobiMiner Core also acts as an aggregation point for
all statistics and metadata reported by the proxies. This allows the user to monitor
and report on the migration from a single management point.
Figure 1: DobiMiner Architecture
3.1.1 DobiMiner Proxies
DobiMiner proxies are the workers of DobiMiner. They take commands from the
DobiMiner Core, carry out the tasks, and report back to the DobiMiner Core.
Regardless of project size, multiple DobiMiner proxies can be deployed and assigned
to any source/target migration pair to increase the migration bandwidth. If needed,
proxies can be added and used temporarily, for example to rapidly migrate legacy
DobiMiner proxies can be assigned to either SMB or NFS file servers. DobiMiner
proxies can be throttled at bandwidth and scheduled independently to manage the
network load.
Proxies are typically deployed on the network close to where the migration activities
will occur. Proxies can be flexibly assigned to the source and target file servers
requiring migration. For example:
With one proxy the following setups are possible:
o One proxy can be assigned to one source and one target file server
o One proxy can be assigned to one source and many target file servers
(1-N). This approach enables a minimal deployment of proxies for
either small or larger projects.
o One proxy can be assigned to many source files servers and one target
file server (N-1).
o One proxy can be assigned to many source and many target file
servers (N-N).
With multiple proxies the following setups are possible:
o Many proxies can be assigned to one source and one target file server
to increase bandwidth for migration purposes. This model is particularly
useful when large numbers of large files need to be consolidated into
one target platform (1-N-1).
o Many proxies can be assigned to many file servers. This model is
useful when many systems need to be migrated in parallel. DobiMiner
can help manage the priorities of each system and between the
systems (N-N-N).
o Many proxies can be assigned to many source file servers and one
target file server (N-N-1).
o Many proxies can be assigned to one source file server and many
target file servers (1-N-N).
Proxies can be reassigned easily at any time to other servers to help tune the
progress of the migration project. Performance and scheduling are monitored for
each individual proxy.
3.1.2 DobiMiner Core
The DobiMiner Core is the single pane of glass for the entire migration project.
Through an easy to use GUI, the user plans and starts the migration project, views
progress in real time, and gets detailed reports at numerous vantage points
regardless of the size of the migration project.
DobiMiner takes a unique approach with the user interface providing the project team
a bird's-eye view on the status of the project while simultaneously allowing immediate
drill down to every possible detail if needed.
DobiMiner Core supports role-based access. This allows for distribution of
responsibilities to all members of the project team at the appropriate level. Each team
member has access to the GUI over HTTP simultaneously and at any time. Access
can be either read only or with full control. Access control can be integrated with
3.1.3 DobiMiner System Requirements
The DobiMiner Core is available as either a virtual appliance, delivered as an Open
Virtualization Archive (OVA file) with complete specifications and operating system of
the virtual machine, or as an RPM package that can be installed on Red Hat® or
CentOS® Linux®. DobiMiner is a lightweight package requiring:
4 CPU cores
1 GigE or 10 GigE network connection
approximately 1 GB of storage per million files that need to be discovered and
DobiMiner Core is available in different packages with predefined virtual disks of
50 GB, 100 GB, 250 GB or 500 GB, and a 16 GB disk for installation of the software
only. Depending on the size of the migration, extra disks can be allocated easily
during installation.
DobiMiner proxies run as a service:
SMB: via an MSI installer on a Microsoft® Windows Server® 2008 and above.
NFS: via an OVA file as a virtual appliance or via an RPM file for installation
on Red Hat v6.6 or CentOS v6.6 Linux.
The minimum setup of DobiMiner is:
For NFS projects only: DobiMiner Core which also includes one NFS proxy.
For SMB projects only or for SMB and NFS projects: DobiMiner Core and one
SMB proxy.
The required sizing for each proxy is:
4 CPU cores
1 GigE or 10 GigE network connection
16 GB disk space
Note: Because of the amount of data that DobiMiner can migrate, it is recommended
to use 10 GigE VMXNET 3 network adapters on the DobiMiner proxies if both source
and target storage systems have 10 GigE network connections available to them.
Setting up DobiMiner Core and a number of proxies usually takes less than an hour.
3.1.4 Supported Migration Scenarios
DobiMiner supports migrations from any NAS device to any other NAS device with
the following protocols:
SMB v1.0, v2.x, v3.x
NFS v3.0
Mixed mode
DobiMiner has integrated the APIs of the following systems to enhance the migration
end-to-end flow even further:
NetApp® as a source (version v7.3.6 or higher)
Isilon® as a target (version v7.1.1 or higher)
VNX® as a source (v1 and v2)
3.2 Migration Configuration
To start a migration project, the project team follows the documentation to set up the
DobiMiner Core and one or more DobiMiner proxies. The next step is to configure the
individual source and target file servers in DobiMiner.
3.2.1 Configuration of Integrated File Servers
Note: This only applies to file servers with an integrated API.
First the IP address and credentials of the file server are requested for access to
the file server's management API. Once connected, DobiMiner will retrieve and
display file server information such as the software version and the unique
identifier. For each file server the project team can configure the maximum load
for scanning and migration operations.
Then DobiMiner displays all accessible volumes, shares and exports with their IP
addresses for SMB and/or NFS for use by the proxies. For SMB the credentials
are requested to access the shares. Each proxy can connect to one or to multiple
IP addresses of the file server.
3.2.2 Configuration of SMB Shares or NFS Exports
Note: This applies to file servers with a non-integrated API.
For SMB shares, the name, the IP address and share credentials of the file server
are requested. For each file server the project team can configure the maximum
load for scanning and migration operations. Once connected, DobiMiner will
retrieve and display a list of visible shares to select from.
For NFS exports, the name and the IP address of the file server are requested.
The proxies used for these exports need to be added as a root and read-write
client of the exports. For each file server the project team can configure the
maximum load for scanning and migration operations. Once connected,
DobiMiner will retrieve and display a list of visible exports to select from.
Once all source and target file servers have been configured, the project team will
define the server migration pairs and paths.
3.2.3 Configuration of Migration Pairs
A migration pair is the relationship between source and target file server involved in
the migration. When configuring a migration pair, the project team can define:
A tracking schedule: This allows the project team to track the progress of a
migration pair individually or in combination with other migration pairs.
SID maps: DobiMiner provides an easy way of (re)assigning Security IDentifier
(SID) maps to user accounts and/or groups. This can be done by uploading an
SID mapping file during the setup of a migration pair. During the upload,
DobiMiner validates the file's content for syntax and the ability to resolve the
imported user accounts. Conflicts are shown in the GUI immediately and can be
resolved by correcting the SID mapping file.
Paths to scan: After selecting the source and target server pair, DobiMiner will
collect and show the list of relevant paths on the source file server which can be
selected for further data analysis. By selecting one of the paths, DobiMiner will
collect the metadata of all files and folders from this path. The results can be
exported for further review.
Once the migration pair has been configured, the project team selects the paths to
include in the migration and sets their security style.
3.2.4 Configuration of Migration Paths
The migration can be done at any level: the root or any level subdirectory. A
migration path is the collection of the individual volumes, Qtrees, or directory paths
representing the actual data to be migrated in the confines of a migration pair.
When selecting the source path, the protocol used for the migration is typically based
on the security style of the source data. DobiMiner supports the following migration
SMB only: The source file system has stored its data with NTFS security. The
Access Control Lists (ACLs) are copied from the source to the target system
using the SMB protocol.
NFS only: The source file system has stored its data with UNIX permissions.
The permission bits are copied from the source to the target system using the
NFS protocol.
Mixed mode (SMB and NFS): A combination of the SMB and NFS protocol is
used to copy NTFS and UNIX permissions from the source to the target
system. In this scenario it is also possible to use only the SMB or NFS
protocol in case the target system is not a mixed-mode system.
After configuring the source path, the project team can define the destination path.
This can be an existing path on the target server or a path that will be created by
The following global options can be configured at path level:
Iteration delay: this setting defines how long DobiMiner has to wait between
iterations. Options are:
o Default: DobiMiner will wait 1 hour before starting the next iteration.
o Manual: DobiMiner will wait until the next iteration is manually started.
o None: DobiMiner will start the next iteration immediately.
o Duration: DobiMiner will wait for the set duration before starting the next
No deletes at root level: this setting is only applicable when setting up multiple
migration paths that target the same root directory. For example in the case that
multiple directories containing home directories of users on the source file server
are merged to the same root directory on the target file server.
Copy root folder security settings: this option is set by default and will transfer
the security settings from the source root folder to the target root folder.
Read-back verification: when this option is enabled, all data that is migrated
during an iteration will be validated after it has been written on the target file
server. The verification process calculates an MD5 hash on both the source and
target file content and compares these hashes to check if the content has been
copied accurately.
Overwrite existing metadata: this option can be enabled to overwrite already
existing metadata of the migrated data.
Interleave directory operations: this option can be enabled to instruct the
proxies to operate on multiple directories at once resulting in a performance
improvement (typically on Isilon file servers).
Ignore case: this option can be enabled to ignore changes in file name case.
DobiMiner will ignore changes made to the characters of an existing file name
(between small and capitalized).
Exclude pattern: after entering a pattern, DobiMiner will not migrate files and
directories matching the given pattern.
In addition to the global migration options which apply to the selected migration path,
the following protocol-specific options can be configured at path level:
For SMB:
o Follow junctions: By default, DobiMiner will ignore junctions when these
are discovered. It is possible to override this default behavior so that
DobiMiner will not ignore the junction and migrate all the content it refers to
as if it was a directory with that content.
o Copy hidden attributes
o Preserve the access time of the source data when migrating the data to
the target file server.
o Copy security descriptor
o Copy system access control list (SACL) containing the access control
entries (ACEs) that specify the types of access attempts that generate
audit reports.
o Keep or overwrite owner and group SIDs of source: if the owner and/or
group SIDs need to be overwritten, new SID values can be configured.
For NFS, the only option is to Preserve the access time of the source data
when migrating the data to the target file server.
3.2.5 Validation of Migration Configuration
Based on the configuration of the migration paths, a number of pre-checks are done
to check whether:
The right permissions have been given to DobiMiner to perform the migrations
from the source to the target location.
The target locations are empty or not.
Enough free capacity is available to perform this migration.
The required ACL Policy settings are OK.
The required SMB/NFS filename encoding is available.
The SMB proxy configurations are set correctly.
The results of the configuration checks are displayed in the GUI and the project team
can take action immediately to resolve issues that might appear and to follow-up on
3.2.6 Import Bulk Configurations
For large-scale migrations, existing migration configurations can be re-used when
defining a new migration. DobiMiner supports the import of configured:
file servers
migration pairs
shares & exports
Before and after the import process, DobiMiner will:
Validate the syntax of the configuration definitions.
Validate whether specific criteria have been met for the import (for example, is the
file server to be added to a migration pair already defined).
Cross check the definitions against what has already been defined in the
DobiMiner environment.
Reconcile the definitions with information collected by DobiMiner (for example,
validate whether an access zone definition exists on the targeted Isilon file
Provide feedback on all of the above to help the project team with importing the
correct definitions for smooth execution during the migration project.
To support the import of different configurations, a set of templates is available which
can be exported into a .zip file from DobiMiner. The configuration files created with
the templates can be imported into the DobiMiner environment whenever needed.
4 DobiMiner Migration Workflow
This chapter starts with a sample DobiMiner migration workflow which offers a highlevel view of the process phases described in more detail in the remainder of this
4.1 DobiMiner Migration Phases
As shown in the following sections, the DobiMiner migration workflow includes all
migration phases from beginning to end.
4.1.1 Discovery and Analysis Phase
Once the project team has set up the migration pairs, DobiMiner starts with the
discovery of the source paths. The results of the discovery phase are shown in the
GUI and can be viewed in detail.
Figure 2: Discovery Information
4.1.2 Planning and Design Phase
The project team uses the discovered information to select the data to
migrate at any level of the system structure. The selected data to migrate is
collectively referred to as the migration path.
Note: At any time during the migration, paths can be added or removed; not only
during the planning and design phase.
Figure 3: Selection of Migration Paths
4.1.3 Migration Execution Phase
In this phase, DobiMiner begins the task of migrating the data while the project team
monitors progress in real time in the Schedule Window.
DobiMiner carries out the following migration tasks in parallel:
First Scan: DobiMiner queries the metadata of the selected migration path.
First Copy: As soon as each migration source path completes its First Scan,
DobiMiner begins a First Copy of the data to the target system.
Steady State: After DobiMiner copies a particular migration path, the path is
passed to a Steady State mode. During this mode DobiMiner periodically
recopies any files or metadata of files that have changed, until the migration
path is switched over to the target system.
Note: This recopying phase is called an iteration in DobiMiner and
the delay between iterations can be configured per migration path.
Figure 4: Schedule Window – First Scan, First Copy & Steady State
As soon as migration paths enter the Steady State mode, the project team plans the
switchover of the migration paths into switchover windows which are a series of times
on specific dates (e.g. noon to 2 p.m. on the first Saturday of the month; but this is
completely adaptable to any customer's preferred schedule).
Figure 5: Schedule Window - Windows
Multiple switchover windows can be planned and DobiMiner indicates how many
migration paths can be processed in each window (Zone) given the allotted time and
number of files.
Figure 6: Switchover Zones
When the switchover time arrives, the Current Window opens and the switchover
phase begins. During the switchover window, users are redirected from the source to
the target platform.
Figure 7: Current Window
When the switchover phase is finished, users will be able to read from and write to
the switched-over shares and exports on the target system.
At any time, the project team is able to lengthen a particular switchover window or
remove, add, or reprioritize specific migration paths ensuring the best use of time.
Any issues that are encountered are clearly displayed in real time allowing the project
team to take remedial action and change plans on the fly if necessary. A report of all
issues is available through DobiMiner's reporting module.
4.1.4 Verification and Reporting Phase
With DobiMiner, verification and reporting occurs automatically and continuously. For
example, the project team can run reports at multiple levels of detail including:
 weekly management and steering committee reports that summarize the progress
and overall status of the project
 technical reports that provide detailed information at the file level
 audit and record keeping reports containing a complete list of exactly which files
were migrated and a detailed status of the migration
 ad hoc (on the fly) reports to answer what-if questions
4.2 Functionality in Detail
This section explores DobiMiner functionality in detail and explains how DobiMiner
Additionally while DobiMiner automatically performs many tasks, this section
provides more detail to the project team members who: 1) require a deeper level of
information about the source and target systems; 2) manage the project at a more
granular level; 3) require greater flexibility with the migration.
4.2.1 Discovery and Analysis
The single biggest criterion for a successful technology transition is that users and
applications are transferred to the new technology in a transparent and nondisruptive manner.
To this end, it is vital that a detailed map of the environment is made before
embarking on a migration project. This map is used to establish a baseline before the
project begins; to plan properly for the new environment and to establish criteria by
which to measure the project for success.
DobiMiner's extensive discovery functionality helps the project team to create this
map. Apart from basic information including number of files and folders that exist,
DobiMiner collects other metadata including:
Who is using the system.
What information is stored (file size, type, etc.).
When the users have created or modified their data.
At what rate the user assets grow.
To minimize disruption to the users, analysis is also done to provide the project team
an understanding of the technical context of the current working environments: how
is the environment configured to serve the users and applications? For example:
which access methods are used (SMB, NFS, mixed mode); details about the shares
and exports; access information about physical and virtual filers and more.
Figure 8: Detailed System Discovery
Figure 9: Detailed Data Discovery
Discovery information feeds into the planning of the migration project.
4.2.2 Planning and Design
Using the information collected from the source environment, the project team is
better equipped to:
Size the target environment to support the end users and applications.
Understand the dependencies that exist on the current environment and that
will have to be taken into account for the new environment.
Make decisions on the migration approach, priorities and plan.
Understand limitations that may be introduced when moving to the new
environment such as number of shares that fit into a switchover window.
Once all stakeholders agree to the project plan, DobiMiner becomes the pivotal
single pane of glass for executing the plan. Planning in DobiMiner consists of
scheduling the:
copy activities
switchover windows
After the project begins, the project team can use DobiMiner's extensive reporting
functionality throughout the execution phase to report progress, activity status and
milestones to the stakeholders.
Scheduling the Copy Activities
A migration schedule in DobiMiner is a logical collection of migration pairs that the
project team wants to bundle together for ease of management.
Any number of migration pairs can be added to a single schedule and any number of
schedules can be created. Setting up a migration pair and adding it to a schedule
takes just a few seconds. Once the pairs are added to the schedule and DobiMiner
has discovered the volumes, directories, shares, exports, etc., the project team can
easily select what needs to be migrated (the migration paths).
Figure 10: Migration Schedule Overview
For each individual migration path, the team can apply a number of options such as
attributes that need to be removed; SID mapping that needs to occur to change the
owners of the data (necessary to fix issues with historical SIDs of users and groups
on the source); or cadence of the migration path iterations.
Figure 11: Detailed Schedule Information
When setting up a migration pair for integrated file servers, DobiMiner will
automatically create the migration shares and/or exports that are needed by the
proxies to copy data from the source file server to the target file server. These shares
and exports are created at the volume level or below.
Shares and exports can be set up on the target exactly as they are on the source or
remapped with new share names on the target system if required.
Once the migration paths are defined, DobiMiner begins a query of the source and
(normally empty) target systems to collect basis information such as file names and
last access time (First Scan).
DobiMiner then compares the results from the queries and copies the delta from the
source to the target (First Copy).
Once the First Copy is complete, the migration path enters into Steady Sate. During
this phase, DobiMiner monitors the changes in data on the source systems,
compares this with the data on the target system, calculates what has changed, and
then copies the differences to the target system.
Changes can include an addition, a change, or a deletion of a file, directory, or
symlink or changes to the metadata. In case only the metadata has changed,
DobiMiner will only re-copy the metadata.
In Steady State mode, DobiMiner migrates source data based on its last modification
or change time: i.e. any file with a modification time (the last time content has
changed) or change time (the last time metadata has changed) older than the
timestamps on the target file server. To further improve the efficiency of the migration
process, a minimum-age setting can be defined by the project team to avoid the
constant re-copying of rapidly changing data until the dry runs and switchover
activities are started (see below).
DobiMiner continues to run in this iterative mode and provides estimated timings for
completion of each iteration and each migration path. This information is vital for both
DobiMiner and the project team to assess the amount of ongoing changes on the
system and therefore make better projections utilizing the switchover windows.
Scheduling the Switchover Windows
A switchover window is a scheduled down-time slot (an engineering window) typically
on a weekend or late at night when users are inactive. Multiple switchover windows
can be scheduled over the period of the migration project. This allows the project
team to plan all migration paths and provide an overall estimated completion time for
the entire project. Additionally, each migration schedule can have its own
independent switchover window. DobiMiner provides the project team the flexibility to
create as many switchover windows as are required to properly plan and complete
the project.
Figure 12: Switchover Window Schedule Detail
During the switchover window, the following occurs:
1. User access on the source system is set to read only. For migrations with an
integrated API this can be done via DobiMiner, for other migrations this needs
to be done outside DobiMiner.
2. The project team schedules the final iteration for the migration path and
DobiMiner automatically copies any last changes to the new system.
3. The project team instructs DobiMiner to create the shares and/or exports on
the target file server in read-only mode.
4. The project team checks if users can read their data on the new system.
5. Using DobiMiner, the project team then transfers all the security settings for
the shares and exports from the source file server to the target system.
6. The project team changes the DNS/DFS to point to the new system.
The steps above are repeated until all data, shares, or exports are transferred to the
new system.
Note: Because a number of these activities need to be done outside of
DobiMiner, it is important that the defined Switchover Window (schedule) is
smaller than the total change window offered by the business.
Figure 13: Current Switchover Window
Before or during the switchover window, DobiMiner can easily adapt to last-minute
changes such as removing, adding, or reprioritizing the migration paths to be
switched over. This provides the project team the flexibility to deal with last-minute
business priority changes while at the same time maximizing the provided timeslot to
switchover as many shares as possible so that no time is wasted.
Figure 14: Migration Paths not fitting in Switchover Window
Switchover windows are expensive and disruptive and the key to reducing the
number of switchover windows required during any migration is to utilize each
window to the fullest extent.
Through its automated dry run support, DobiMiner can accurately access the
dynamics of the environment and use this information to make a best-effort estimate
of how long it will take to switchover a migration path.
Dry runs should be carried out on a similar schedule and under the same
circumstances as the planned real switchover window to get the most accurate
estimate. Multiple dry runs can be scheduled allowing DobiMiner to gather
information for assessing switchover durations.
This approach provides the project team both a reasonably accurate estimate of the
time required for switching over the selected migration paths and allows the project
team to make decisions on adding or removing migration paths from switchover
Note: Because user behavior is unpredictable, DobiMiner's dry run support is
only able to give a best-effort estimate based on the activity measured during
the dry runs. Actual activity immediately prior to the real switchover window
can vary.
In order to switch over the most migration paths during a particular window,
DobiMiner shuts down all activity that does not contribute to the switchover (such as
scanning and copying of migration paths that are not part of the switchover window).
These processes automatically resume once the switchover window is closed by the
4.2.3 Bandwidth Throttling and Scheduling
DobiMiner supports two different types of scheduled throttling:
Bandwidth throttling of the proxies.
Load throttling of the source and/or target file server.
Proxy-level Throttling
Proxies are assigned to different migration pairs and can be throttled and scheduled
individually within their own time zone. In this way, bandwidth can be tuned towards
network capabilities and operational requirements. For example, the migration
process may need to be reduced or even stopped entirely during certain hours of the
day where the network needs to be fully dedicated to a particular business activity.
But after that activity, the migration process should resume again to get the best
possible migration times. In DobiMiner, a dedicated schedule can be created for each
proxy to better control network throttling activities.
Figure 15: Network Throttling Schedule
The detailed network bandwidth used by DobiMiner can also be monitored.
Figure 16: Monitoring Network Activity
Server-level Throttling
DobiMiner can also throttle server loads by defining how many concurrent threads
can be used against a particular file server. Similar to proxy scheduling, server
scheduling can be defined on an hourly basis for each day of the week. The
migration process threads can be reduced or increased to control or even stop
4.2.4 Monitoring, Troubleshooting, and Reporting
DobiMiner supports true Management by Exception because it is both iterative and
workflow based. This approach allows the user to 1) track the migration project
quickly, 2) only interact with real exceptions within the migration process, and 3)
easily report on the project at various levels of detail to provide stakeholders
appropriate information about the project.
Monitoring a project with DobiMiner starts with the advanced GUI. For easy
monitoring of the migration, DobiMiner presents each process individually in a tile.
Each tile shows exactly what is happening in each step of the migration.
The tiles are color-coded so that the user can see at a glance whether everything is
on track (green), action is required to progress the migration process (orange), or a
persistent issue has been encountered and needs human intervention (red).
Figure 17: Migration Schedule Overview
A simple click on a tile provides the user with the next level of detail of each process.
Figure 18: Migration Detail Window
This visualization method provides both a bird's-eye view on the project and an easy
way to access more detailed information if required.
DobiMiner automatically handles temporary errors (e.g. network glitches) by
periodically retrying files that have failed to copy, allowing the project team to focus
on persistent issues such as:
Missing shares and exports on the target platform.
Open files locked by the end user.
Permission issues for accessing and copying data.
Out of capacity issues.
Permanently lost network connectivity.
These issues are reported in every phase and are categorized with specific
information to make problem resolution fast and straight forward.
Figure 19: Detailed Troubleshooting Information
Communication is the essence of any project and proper reporting to project
stakeholders is crucial to maintain trust in the project objectives and the project team.
DobiMiner provides extensive reporting that can be created on the fly for all levels of
stakeholders from the technical team to senior management.
Figure 20: Detailed Progress Reporting
DobiMiner also provides Email Home functionality so project stakeholders (including
the Datadobi Support Desk if permitted) can be informed of project progress
automatically by email.
Most information in the DobiMiner GUI can also be exported into CSV files for further
analysis and reporting offline if required.
4.2.5 Validation
DobiMiner tracks changes on the source and target file servers to understand exactly
what must be processed in a migration iteration. These iterations either run
continuously (one iteration after the other), or by following an iteration delay setting
that is customized per migration path.
Whenever DobiMiner encounters an error during these migration iterations, it flags
this as an issue which can be inspected and analyzed per individual file or per file
The list of issues is refreshed automatically with each iteration. Temporary issues
such as lack of capacity, loss in connectivity to either source or target file servers, or
inaccessible data disappear from the issues list when they have been resolved
during subsequent iterations.
This approach promotes a very short feedback cycle during issue resolution
activities. Because DobiMiner's scanning engine operates independently of the
migration engine, extra checks and balances exist to validate that all data was copied
correctly to the target system.
Issues that arise during each step of the migration process are flagged in the user
interface (e.g. the color of a tile changes to red) so that the project team can quickly
identify process steps, migration paths, and individual items that require root-cause
analysis and resolution.
With the pre-switchover support in DobiMiner, the project team can allow end users
to validate access to their data before giving them the ability to actually change the
data. This is done by mapping and creating shares and exports with read-only rights
on the target file server.
Shares and exports are discovered on the source and target file servers through the
storage platform management APIs; detected changes are displayed via the GUI and
mismatches are flagged for resolution by the project team.
5 Conclusion
NAS storage platform migrations historically have been painful, manual, timeconsuming, and expensive endeavors. They have been carried out by teams of
people using outdated tools and complex manual processes.
Substantial resources were needed to ensure a successful technology transition
which was both disruptive and risky.
The tools used were created to migrate only small amounts of data and are
substantially limited in functionality: no end-to-end approach, little troubleshooting
support, covering only one protocol (SMB or NFS), etc.
The result of this approach is that companies need more time to recognize their
return on investment (ROI) or even completely fail to take advantage of superior
storage platform technology.
Datadobi has completely rewritten the playbook allowing companies to fully grasp
NAS technology transitions without the fear of a long and expensive migration
DobiMiner, Datadobi's unique software suite, delivers an end-to-end, automated,
multi-protocol migration solution specifically designed to make today's migration
projects quick and cost effective.
Contact Datadobi today to learn more about how we can help your company
transition to new technology quicker increasing your ROI:
Was this manual useful for you? yes no
Thank you for your participation!

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Download PDF