Dell DataIQ software Installation Guide
إعلان
إعلان
DataIQ 2.2.x.x
Installation Guide
2.2.x.x
July 2022
Notes, cautions, and warnings
NOTE: A NOTE indicates important information that helps you make better use of your product.
CAUTION: A CAUTION indicates either potential damage to hardware or loss of data and tells you how to avoid the problem.
WARNING: A WARNING indicates a potential for property damage, personal injury, or death.
© 20202022 Dell Inc. or its subsidiaries. All rights reserved. Dell Technologies, Dell, and other trademarks are trademarks of Dell Inc. or its subsidiaries. Other trademarks may be trademarks of their respective owners.
Contents
Contents 3
4 Contents
1
About this guide
This guide provides requirements and instructions on how to install, upgrade, or uninstall DataIQ and its plugins and services.
Where to go for support
This topic contains resources for getting answers to questions about DataIQ.
Support and downloads
Telephone support
Community support
Manuals, videos, and blogs
Demos
Licensing
DataIQ Product Support
United States: 1-800-782-4362 (1-800-SVC-4EMC)
Canada: 1-800-543-4782
Worldwide: +1-508-497-7901
Find phone numbers for more countries and regions at Dell
EMC Customer Support Centers .
Dell Community
DataIQ Info Hub
DataIQ Developer Info Hub
Dell Technologies Demo Center . Log in and search for DataIQ.
Software Licensing Central, at https://licensing.emc.com/ .
Learn more about licensing at https://www.dell.com/support/contents/article/ product-support/self-support-knowledgebase/software-anddownloads/software-licensing-central-documentation .
Feedback
We value your feedback. Please let us know how we can improve this document.
● Take the survey at http://bit.ly/isilon-docfeedback .
● Send comments or suggestions to [email protected]
.
About this guide 5
2
Requirements
This chapter describes DataIQ installation requirements.
Host requirements
DataIQ can be installed to a Linux server or virtual machine (VM).
Operating system
Option 1: Install on an existing server or VM.
The server or VM requires one of the following versions of Linux:
● Red Hat Enterprise Linux Server: 64-bit version 7.6, 7.7, 7.8, or 7.9. (Version 8.x is not supported)
● CentOS: 64-bit version 7.6, 7.7, 7.8, or 7.9. (Version 8.x is not supported.)
● Oracle Linux 7.6, 7.7, 7.8, or 7.9. (Version 8.x is not supported ) Requires Oracle Red Hat Compatible Kernel (RHCK).
(Unbreakable Enterprise Kernel / UEK is not supported).
Option 2: Create a VM using an Open Virtualization Application (OVA) bundle.
Supported virtual environments include:
● VMware ESXi 6.7
When you create a VM using the OVA bundle, you get:
● A minimal CentOS 7.6 installation with some settings preconfigured.
● Preinstalled security patches that are up-to-date as of the package release date.
● The DataIQ installer RPM package, ready to be initiated.
NOTE: For sites without a network connection: it is recommended to install DataIQ using the OVA bundle so that DataIQ can manage future application updates using standard upgrade files.
Memory (RAM)
DataIQ requires at least 32 GB of memory, 64 GB is recommended.
64 GB is required when using data management on systems with more than 1 billion files and folders, or storage monitoring for systems with more than 8 clusters.
Processors (CPU)
DataIQ requires at least 4 processor cores. 8 cores are recommended.
12 cores are recommended when using data management features on systems with more than 1 billion files and folders.
DataIQ can take advantage of multi-core and multithreaded processor features. DataIQ requires "server-class" CPUs. Higher performance CPUs can improve DataIQ performance when generating complex reports.
Other software
During setup, DataIQ installs Docker and Kubernetes.
If Docker or Kubernetes are installed on the host system, uninstall them before installing DataIQ.
6 Requirements
If you are using Option 1: Install on an existing server or VM, then you also will need:
● OpenSSL version 1.1.1n. (To upgrade, see
.)
Drives: Planning for future growth
To balance cost savings and performance, it is recommended to use a combination of both solid-state drives (SSD) and traditional hard disk drives (HDD). It is recommended to set up drives as logical drives to support future expansion.
DataIQ can be installed on either logical volume-managed partitions or static primary partitions. To plan for future expansion, it is recommended to use logical volumes. Logical volumes can be expanded later onto new drives when more storage is needed.
● For OVA-based installations, drives are configured as logical volumes by default.
● For non-OVA installations, see
Create and mount logical volumes
for steps on preparing logical volumes for future expansion.
To balance drive cost and speed, we recommend the following logical volumes:
● A logical volume for the OS, Docker, DataIQ, Keycloak, and the storage monitoring database (TimescaleDB). These features can be run on hard disk drives, solid state drives, or a combination of the two.
● A separate logical volume for the data management database. For performance reasons, this database must be hosted on solid-state drives.
To add capacity later, expand the volume groups and logical volumes onto new drives. To learn more, see
Add a new drive to a volume (LVM)
.
Drive requirements
Drive requirements may vary greatly depending on whether the data management features are used, the number of clusters managed and monitored, and the length of time that storage monitoring data is retained.
Directories, storage types, and size requirements
Directory
Operating system
Description and requirements
May be hosted on a traditional hard disk drive (HDD) or a solid-state drive (SSD).
Requirements 7
Directory
/opt/dataiq
Description and requirements
Do not host the data management database on an NFS storage location, because it requires extra network layer protocol overhead, and it may cause stability issues or data loss when there are network issues.
The /opt/dataiq directory stores the DataIQ application (including the Keycloak database and other configuration files), and the storage monitoring database (TimescaleDB).
During installation, if this directory does not exist, the installer creates it. The directory may be hosted on a traditional hard disk drive (HDD) or a solid-state drive (SSD).
To calculate the amount of space needed in the /opt/dataiq directory:
● Add 20 GB for the DataIQ application.
● Add 10 GB if the /var/lib/docker is on the same partition as /opt/dataiq .
● Add space for the storage monitoring database. For a conservative estimate of a few nodes, plan on adding 200 MB per week per monitored node. For a better estimate, see the Storage monitoring database section below.
● Plan to keep at least 15% of the total partition space + 40 GB as free working space.
If free space is less than (15% total drive space+40 GB), DataIQ sends an alert in the notification area.
If free space is less than (15% total drive space+20 GB), DataIQ sends an alert and stops collecting storage monitoring data. When free space is more than (15% total drive space + 60 GB), DataIQ resumes collecting storage monitoring data.
/var/lib/ docker
/mnt/ssd
The /var/lib/docker directory stores the Docker application and DataIQ pods.
During installation, if this directory does not exist, the installer creates it. The directory may be hosted on a traditional hard disk drive (HDD) or a solid-state drive (SSD).
The /var/lib/docker requires at least 55 GB, which includes some room for free working space.
To upgrade from DataIQ v1, this partition must have at least 25 GB free working space.
Plan to keep at least 5 GB of free working space in this partition. If this directory is on the same partition as /opt/dataiq , see the free space calculations above.
By default, the data management database is stored at /mnt/ssd . The directory location can be changed during installation.
For performance reasons, the data management database should be hosted on a solid-state drive (SSD).
The directory requires a minimum of 250 GB, and a minimum of 250 GB for each 1 billion files and folders to be scanned.
The drive must be local to the server. Do not host the data management database on an NFS storage location, because it requires extra network layer protocol overhead, and it may cause stability issues or data loss when there are network issues.
Storage monitoring database
The storage monitoring database grows based on factors including the number of clusters that are monitored, the number of nodes monitored, the number of entries displayed in charts (Top N value), and the load rate on the clusters.
Table 1. Estimated drive space needed for different scenarios
Scenario Number of clusters Number of nodes
Scenario 1
Scenario 2
Scenario 3
Scenario 4
1
1
10
70
6
30
300
2000
1
3
3
Years of data retention
1
Estimated drive space needed
0.12 TB
0.36 TB
10.93 TB
76.49 TB
8 Requirements
Administrators can set the maximum amount of data to keep by setting a data retention policy. For example, if you are monitoring 1 cluster with 30 nodes and are retaining a maximum of 3 years of data, use the value 1.08 TB (0.36 TB * 3). To learn more, see Data retention policy in the Dell EMC DataIQ Admin Guide.
Supported browsers
You can use the following browsers with DataIQ:
● Google Chrome
● Microsoft Edge, Chromium edition (released January 15, 2020)
● Apple Safari
● Mozilla Firefox (standard edition). Not supported on Extended Support Release (ESR) versions.
DataIQ does not work with:
● Internet Explorer 11
Monitored cluster requirements
DataIQ can manage storage across multiple types of unstructured storage products, and can monitor storage on Dell EMC
PowerScale and Isilon systems.
Requirements for data management
DataIQ manages, scans, indexes, and reports on network-attached file systems including NFS and SMB. DataIQ requires root access to clusters that it is directly managing.
DataIQ can connect to other platforms that use standard Amazon S3-compatible, VFS, NFS, or SMB/CIFS file and access protocols. While connected, users with permissions can browse external volumes. With the Data Mover plugin, you can move data between your managed volumes and other connected volumes. Examples of volumes you can connect:
● Dell EMC PowerScale and Isilon systems
● Amazon Web Services (AWS) S3
● Google Cloud Platform (GCP)
● Dell EMC Elastic Cloud Storage (ECS)
● Dell EMC Unity
DataIQ works well on systems that have up to 20 billion files and folders. If your system has more files or folders, you may need to deploy multiple DataIQ instances, with each instance hosted on a separate node. Note: There is not a way to combine data from multiple DataIQ instances.
Requirements for storage monitoring
DataIQ can manage and monitor storage on Dell EMC PowerScale and Isilon systems using OneFS 8.0.0.0 and later, including
Gen 4, Gen 5, and Gen 6 product models.
A single DataIQ instance (virtual or physical) can monitor storage on up to 70 Dell EMC PowerScale and Isilon systems with a total of up to 2,000 nodes.
DataIQ requires access to a user account with read permission to the clusters that it is monitoring.
Requirements 9
Network requirements
This topic describes the network requirements of DataIQ.
General requirements
DataIQ does not require an Internet connection. However, some features such as connection to Amazon S3-compatible services and proactive support do require an Internet connection.
The DataIQ host and each worker node require a static IPv4 address. DataIQ does not support IPv6.
A 1 Gbps connection is usually sufficient. Fast and stable networking between nodes improves performance and scalability.
Plugins like Data Mover may require additional network throughput.
● Appropriate default gateway.
● Appropriate DNS entries.
● No IP address conflicts.
Ports
DataIQ requires access to commonly-used network ports for sending and receiving data. Access to additional ports may be required to use optional features.
Table 2. DataIQ host
Port Used for Effect if closed
80, 443
Protocol Connection type
(outbound/inbound)
HTTP/
HTTPS
Inbound
30003
22
NFS/SMB ports
S3 HTTPS ports
389 or 636
8080
465
21
25
HTTPS
SSH
TCP/IP
TCP/IP
HTTPS
HTTPS
TCP/IP
Passive
FTP
SMTP
Inbound
Inbound
Inbound
Inbound
Outbound
Outbound
Inbound and outbound
Used to access DataIQ user interface (API gateway)
Login screen
Installation and configuration
Connection to NFS/SMB volumes
Connection to S3 volumes
Connection to LDAP/
LDAPS (Active
Directory)
Verify OneFS SSL certificates
Email (SMTP)
Secure Remote Services:
Sends "connect home" messages
Secure Remote Services:
Processes "Connect home" files, sends email notifications
No access to DataIQ user interface.
No ability to log in.
No ability to install or configure DataIQ.
Not able to use data management features for
NFS/SMB volumes.
Not able to use data management features for
S3 volumes.
Not able to use LDAP/
LDAPS/Active Directory for user management and authentication.
Not able to use data management features for
OneFS clients.
Not able to send email alerts.
Not able to use
Secure Remote Services to send proactive communications.
Not able to use
Secure Remote Services to send proactive
10 Requirements
Table 2. DataIQ host (continued)
Port Protocol Connection type
(outbound/inbound)
Used for Effect if closed
443
443 and 8443
9443
52100
Varies
Note: Dell recommends blocking external connections to ports 6443 and 10250 of the Kubernetes service.
Table 3. Ports: SRS host (if used)
Port
21
Protocol Connection type
(outbound/inbound)
Passive
FTP
Outbound
Used for Effect if closed
25
443
443 and 8443
9443
SMTP
HTTPS
HTTP
HTTP
Inbound and outbound
Inbound and outbound
Inbound and outbound
SRS: Sends "connect home" messages
SRS: Processes
"Connect home" files, sends email notifications
SRS: Initial setup, service notifications
Not able to use SRS
SRS: Remote support
SRS: Managed file transfer
Not able to use
SRS to send proactive communications
Not able to use
SRS to send proactive communications or alerts in email
Not able to use SRS
Dell Support will not be able to send or receive files from the local system to troubleshoot and solve issues
Table 4. Ports: Data Mover workers (if used)
Port
30080
Protocol Connection type
(outbound/inbound)
Inbound and outbound
NFS/SMB ports TCP/IP Inbound
S3 HTTPS ports
HTTPS
HTTP
HTTP
TCP/IP
TCP/IP
Inbound and outbound
Inbound and outbound
Inbound and outbound
Varies
Inbound
Secure Remote Services:
Initial setup, service notifications
Secure Remote Services:
Remote support
Secure Remote Services:
Managed file transfer
Data Mover (optional plugin)
May be used by optional
Custom Action Items communications or alerts in email.
Not able to use Secure
Remote Services.
Not able to use Secure
Remote Services.
Dell Support will not be able to send or receive files from the local system to troubleshoot and solve issues.
Not able to use Data
Mover external worker nodes to assist with moving data.
Custom Action Items may not work.
Used for Effect if closed
Moving data
Connection to NFS/SMB volumes
Connection to S3 volumes
Not able to move data
Not able to use data management features for
NFS/SMB volumes.
Not able to use data management features for
S3 volumes.
Requirements 11
Virtual machine / OVA installation
On some networks, you may need to change the Network Adapter from Bridged to network address translation (NAT).
Hostname requirements
DataIQ-specific hostname requirements:
● After DataIQ is set up, changes to the hostname are not supported.
● The hostname must not start with localhost . For example, localhost-dataiq is invalid.
● The hostname cannot contain upper-case letters (A-Z).
The hostname must also meet general DNS hostname requirements:
● The hostname may contain letters (a-z) and digits (0-9).
● The hostname may contain hyphens (-) if they are in the middle of a label. Example: dataiq-host .
● While the initial name section of a DNS A-record cannot have an underscore, underscores ( _ ) are permitted as part of subdomains. Examples:
OK: dataiq._subdomain.myorganization.org
Invalid: _dataiq.myorganization.org
Invalid: _dataiq.subdomain.myorganization.org
To learn more, see http://domainkeys.sourceforge.net/underscore.html
.
● Other characters, including blank or space characters are not permitted.
● Single-character names are not permitted.
Recommendations:
● A matching entry should be created as an A-level record in the DNS domain for the network.
Tool: dnsmasq
If the dnsmasq tool is installed, it must be stopped and disabled. The dnsmasq tool is often configured and enabled on standard
Linux installations.
1. Stop the utility: systemctl stop dnsmasq
2. Disable the utility: systemctl disable dnsmasq
Authentication requirements
DataIQ supports tools like Active Directory with LDAP or LDAPS to authenticate users.
You can create users through DataIQ, or import existing lists of Active Directory users or groups, and manage access on an individual or group level.
Other tools such as OpenLDAP are unsupported at this time.
12 Requirements
3
Installation
This chapter discusses installing, upgrading, and uninstalling DataIQ.
Method 1: Create a virtual machine (VM) using an
OVA
To make installation simpler, you can install DataIQ onto a new VM using the open virtual appliance (OVA/OVF) installer. This method requires a virtualization platform such as VMware vSphere. The installer prepares a new VM with CentOS, Docker,
Kubernetes installed, prepares many settings for DataIQ, and prepares DataIQ to be initialized after other setup steps are complete.
Download installation and upgrade packages
Download the installer and optionally, a DataIQ license file. A DataIQ license can be added either during installation or after installation through the UI.
Steps
1. Download the DataIQ Virtual Machine OVA , for example, dataiq-2.2.2.0-67.el7-vmware.ova
. from Dell Support, at https://www.dell.com/support/home/product-support/product/data-iq .
2. Optional: Download a license file ( license.xml
) from Software Licensing Central, at https://licensing.emc.com/ .
Check the authenticity of a DataIQ OVA file
Verify DataIQ OVA with the Open Virtualization Format Tool (ovftool) utility provided by VMware.
Prerequisites
Requires a system that has the OpenSSL and Open Virtualization Format Tool (ovftool) utilities installed.
Steps
Examine the certificate:
$ ovftool --machineOutput dataiq-*-vmware.ova | \
awk '/-----BEGIN.*CERT/,/-----END.*CERT/ { gsub(/^+ /, ""); print> "ova.crt"}'
$ openssl x509 -in ova.crt -text -noout
Prepare a virtual machine to host DataIQ
This topic describes how to set up and log onto the new VM created from the OVA.
About this task
You can host DataIQ and its features entirely on a solid-state drive (SSD), or you can split it across drives so that the operating system and other features that don't require the performance of an SSD are stored on a less expensive hard disk drive (HDD).
Installation 13
Steps
1. Save the OVA file to the server that will be hosting the DataIQ application. The OVA file is only used for creating the VM and can be stored on any drive type.
2. Create a new VM based on the OVA using a virtualization platform such as VMware vSphere, using options to deploy an
OVA or an OVF. For example, in VMWare vSphere, Deploy an OVF template starts the process of creating the VM immediately using the OVA.
When prompted, select the storage path for the operating system for the new virtual machine. If you are using both SSD and HDD drives, you should host this location on an HDD. The OVA configures the datastore as a logical volume that can be resized later by adding new drives or partitions.
3. In the virtualization platform, modify the VM to meet your system needs, meeting at least these system requirements:
● Memory: 32 GB required; 64 GB recommended. To learn more, see
● Processors: 4 required; 8 recommended. To learn more, see
.
● Network configured for the VLAN. On some networks, you may need to change the Network Adapter from Bridged to network address translation (NAT).
4. Modify the VM datastore(s) to meet the
.
● To host the data management database on a separate SSD (recommended): Create a new empty datastore with at least
500 GB, saved on an SSD drive location.
● To host the data management database on the same drive: Expand the primary drive from 80 GB to at least 580 GB.
●
5. Power on the new VM.
6. Log in using the username: dataiqadmin and password dataiqadmin .
7. Change to root: sudo su -
8. Set the new root password to a secure password.
passwd root
Keep this password safe using your organization's recommended security practices. This password is used for backups and restores.
Create and mount logical volumes
Prepare logical volumes using logical volume manager (LVM) for your storage locations before installing DataIQ. This allows you to maintain storage on different types of drives to control storage costs, and allows you to more easily expand the available storage by adding new drives later.
About this task
Repeat these instructions to prepare each of these logical volumes:
14 Installation
● The data management database. By default, DataIQ stores this database at /mnt/ssd . Prepare a new logical volume and map it to /mnt/ssd . For performance reasons, we recommend that the physical drives hosting this logical volume are solid-state drives (SSDs).
● The main DataIQ folder and storage monitoring database (TimescaleDB), which is stored at /opt/dataiq . If the existing /opt folder is not already a logical volume, create a new logical volume and map it to /opt/dataiq . For cost reasons, we recommend that the physical drives hosting this logical volume are traditional hard disk drives (HDDs). Make sure the /var directory still has 10 GB free.
Later, if you need to expand a volume, see
Expand an LVM volume by adding a drive
.
Steps
1. Create the database folder, for example: mkdir /mnt/ssd
If the operating system, application, and databases will all be stored on the same drive, skip the rest of this procedure.
2. Identify the SSD datastore: lsscsi
3. Get additional details to identify the drive: lvdisplay pvdisplay
4. Create a physical volume that is based on the drive you identified as the SSD, for example, /dev/sdb : pvcreate /dev/sdb
5. Create a volume group vg0 based on the same drive: vgcreate vg0 /dev/sdb
6. Verify that the group has been created: vgdisplay
Installation 15
7. Create a logical volume ssd in the new volume group.
lvcreate -l +100%FREE -n ssd vg0
8. Confirm the logical volume exists: lvdisplay
9. If the drive is a new virtual drive, erase and configure the volume and a file system: mkfs -t xfs /dev/vg0/ssd
16 Installation
10. Mount the file system to the mount point: mount /dev/vg0/ssd /mnt/ssd
11. Confirm that the disk is mounted: df -h
You should see several mount points, including the /mnt/ssd mount point.
12. Edit the file system table (fstab) to mount the drive on bootup using a text editor, for example, vim (on CentOS or Oracle
Linux) or nano (on Red Hat Enterprise Linux): vi /etc/fstab
Add an entry for the file system based on the file system location:
/dev/mapper/vg0-ssd /mnt/ssd xfs defaults 0 0
NOTE: Using VIM: Press i to enter insert mode, press Esc to escape. Press :wq to write changes and quit. To learn more, see A Great VIM Cheat Sheet .
13.
Installation 17
Set up networking
Set up a static IP address, subnet mask, and hostname. Perform these steps even if you are installing DataIQ on a host that is not connected to the internet.
Steps
1. Request a static IP address from your network administrator or IT organization.
2. Open the NetworkManager text user interface (NMTUI). Use keyboard commands (Tab key, arrows) to navigate NMTUI menus.
nmtui
3. Select Edit a connection .
4. Select the ethernet connection, for example, enp0s3 .
5. Edit IP v4 CONFIGURATION : in Address , enter the networking information that was provided by your organization: a. At the top level, change the dropdown from Automatic to Manual .
b. Select Show to see more options.
c. Select OK , then Back .
6. Select Activate a connection .
7. Deactivate and reactivate the ethernet connection: select the connection, for example, enp0s3 , and select Deactivate , then select Activate .
8. Select Back .
9. Select Set hostname .
10. Type a hostname that meets the following requirements, for example: dataiq-server . Select OK .
Hostname requirements
DataIQ-specific hostname requirements:
● After DataIQ is set up, changes to the hostname are not supported.
● The hostname must not start with localhost . For example, localhost-dataiq is invalid.
● The hostname cannot contain upper-case letters (A-Z).
The hostname must also meet general DNS hostname requirements:
● The hostname may contain letters (a-z) and digits (0-9).
● The hostname may contain hyphens (-) if they are in the middle of a label. Example: dataiq-host .
● While the initial name section of a DNS A-record cannot have an underscore, underscores ( _ ) are permitted as part of subdomains. Examples:
OK: dataiq._subdomain.myorganization.org
Invalid: _dataiq.myorganization.org
Invalid: _dataiq.subdomain.myorganization.org
To learn more, see http://domainkeys.sourceforge.net/underscore.html
.
● Other characters, including blank or space characters are not permitted.
● Single-character names are not permitted.
Recommendations:
● A matching entry should be created as an A-level record in the DNS domain for the network.
11. Select Quit .
12. Restart the network configuration using the new settings: systemctl restart network
18 Installation
13. Verify the network information is correct, for example: ip addr show enp0s3
14. Optional: set up a fully qualified domain name (FQDN), such as https://dataiq.myorganization.com.
a. Edit the /etc/hosts file: vi /etc/hosts
If an entry for localregistry.emc.com
exists, remove it.
Add an entry for the file system based on the file system location. In this example, the sample IP address 192.168.0.34 is mapped to the sample FQDN: dataiq.myorganization.com.
192.168.0.34 dataiq.myorganization.com
NOTE: Using VIM: Press i to enter insert mode, press Esc to escape. Press :wq to write changes and quit. To learn more, see A Great VIM Cheat Sheet .
b. Set up the FQDN from the command line: hostnamectl set-hostname {fqdn}
Results
At this point, you should be able to ping the DataIQ server using the FQDN.
Initialize DataIQ
Run the DataIQ installer, receive initial passwords, and optionally set up Secure Remote Services.
Steps
1. Initialize DataIQ: dataiq init
Or, initialize DataIQ with a license file: dataiq init --license-path /mydata/license.xml
.
Follow the instructions to initialize DataIQ:
● Accept the installation EULA. (Type yes .)
Wait up to 20 minutes for DataIQ to initialize.
NOTE:
● If DataIQ fails to initialize: Remove (destroy) any installed DataIQ components and remove the installation package.
To perform an offline installation, extract the dataiq-dependencies files. Extract the DataIQ installer, and initialize
DataIQ again. Example: dataiq destroy yum remove -y dataiq yum install dataiq-dependencies-2.2.2.0-5.el7.noarch.rpm #(Offline install) yum install dataiq-2.2.2.0-3783.el7.noarch.rpm dataiq init
Installation 19
● To troubleshoot installation or initialization, review the logs at /opt/dataiq/install/dataiq_install.log
and /opt/dataiq/install/playbook.log
.
Figure 1. End of DataIQ 2.2.0.0 installation
2. When prompted: "Do you have an installed SRS gateway ... [y/N]" , you can optionally type y to set up
Secure Remote Services using your existing Secure Remote Services credentials. You can also skip setup by typing N and then set up Secure Remote Services later through the DataIQ UI.
3. Copy the temporary password for the administrator account. In DataIQ, the administrator credentials are sometimes called Keycloak credentials.
You can also retrieve this temporary password using the command: cat /opt/dataiq/install/keycloak_admin.passwd
4. In a browser window, go to the IP address used for DataIQ to open the DataIQ UI. Enter the administrator username and paste or enter the temporary password, then change the password when prompted.
Keep the new password safe using recommended security practices for your organization.
5. Optional: If you use Secure Remote Services, copy down the remote service password. You can give this password to a Dell
Support agent to allow them to gather DataIQ troubleshooting logs when needed.
cat /opt/dataiq/install/keycloak_remoteservice.passwd
Keep the password safe using recommended security practices for your organization. To generate a new password, see
Update service account password .
thumbnail-sized preview images for graphics and videos.
20 Installation
Method 2: Install on a physical machine or existing virtual machine
One option to install DataIQ is to install on a physical machine or a virtual machine that already has Linux installed.
Install and register the operating system
You can install DataIQ on supported Linux systems.
Steps
1. Install a supported version of Linux.
2. For Red Hat Enterprise Linux or Oracle Linux, register your Linux version. To learn more, see: How to register and subscribe to Red Hat Enterprise Linux or Registering Client Systems with Oracle Linux Manager .
Download installation and upgrade packages
Get the installer, and optionally, a license file. Licenses can be added either during initialization or after installation through the
DataIQ UI.
Steps
1. Download the DataIQ Install and Upgrade RPM Package , for example, dataiq-2.2.x.x-xxxx.el7.noarch.rpm
.
from Dell Support, at https://www.dell.com/support/home/product-support/product/data-iq .
2. If your existing server or virtual machine (VM) that is not connected to the Internet, also download the DataIQ
Dependencies RPM Package for offline sites : dataiq-dependencies-2.2.x.x-xxxx.noarch.rpm
.
NOTE: For sites without a network connection: it is recommended to install DataIQ using the OVA bundle so that
DataIQ can manage future application updates using standard upgrade files.
3. Optional: Download a license file ( license.xml
) from the Software Licensing Central, at https://licensing.emc.com/ .
Check the authenticity of a DataIQ RPM file
Help verify the authenticity of DataIQ RPM Package Manager download files with the DataIQ RPM signing key .
About this task
To learn more about verifying RPM file signatures, see https://access.redhat.com/documentation/en-us/ red_hat_enterprise_linux/5/html/deployment_guide/s1-check-rpm-sig .
Steps
1. Download the DataIQ RPM signing key from Dell Support, at https://www.dell.com/support/home/product-support/ product/data-iq . The file is named RPM-GPG-KEY-DATAIQ.gpg
.
2. Import the signature file into your trusted signatures: rpm --import RPM-GPG-KEY-DATAIQ.gpg
3. Verify the RPM file, for example: rpm -K dataiq-2.2.2.0-67.el7.noarch.rpm
Installation 21
Create and mount logical volumes
Prepare logical volumes using logical volume manager (LVM) for your storage locations before installing DataIQ. This allows you to maintain storage on different types of drives to control storage costs, and allows you to more easily expand the available storage by adding new drives later.
About this task
Repeat these instructions to prepare each of these logical volumes:
● The data management database. By default, DataIQ stores this database at /mnt/ssd . Prepare a new logical volume and map it to /mnt/ssd . For performance reasons, we recommend that the physical drives hosting this logical volume are solid-state drives (SSDs).
● The main DataIQ folder and storage monitoring database (TimescaleDB), which is stored at /opt/dataiq . If the existing /opt folder is not already a logical volume, create a new logical volume and map it to /opt/dataiq . For cost reasons, we recommend that the physical drives hosting this logical volume are traditional hard disk drives (HDDs). Make sure the /var directory still has 10 GB free.
Later, if you need to expand a volume, see
Expand an LVM volume by adding a drive
.
Steps
1. Create the database folder, for example: mkdir /mnt/ssd
If the operating system, application, and databases will all be stored on the same drive, skip the rest of this procedure.
2. Identify the SSD datastore: lsscsi
3. Get additional details to identify the drive: lvdisplay pvdisplay
4. Create a physical volume that is based on the drive you identified as the SSD, for example, /dev/sdb : pvcreate /dev/sdb
5. Create a volume group vg0 based on the same drive: vgcreate vg0 /dev/sdb
6. Verify that the group has been created: vgdisplay
22 Installation
7. Create a logical volume ssd in the new volume group.
lvcreate -l +100%FREE -n ssd vg0
8. Confirm the logical volume exists: lvdisplay
9. If the drive is a new virtual drive, erase and configure the volume and a file system: mkfs -t xfs /dev/vg0/ssd
Installation 23
10. Mount the file system to the mount point: mount /dev/vg0/ssd /mnt/ssd
11. Confirm that the disk is mounted: df -h
You should see several mount points, including the /mnt/ssd mount point.
12. Edit the file system table (fstab) to mount the drive on bootup using a text editor, for example, vim (on CentOS or Oracle
Linux) or nano (on Red Hat Enterprise Linux): vi /etc/fstab
Add an entry for the file system based on the file system location:
/dev/mapper/vg0-ssd /mnt/ssd xfs defaults 0 0
NOTE: Using VIM: Press i to enter insert mode, press Esc to escape. Press :wq to write changes and quit. To learn more, see A Great VIM Cheat Sheet .
13.
24 Installation
Start NetworkManager
Enable and start Linux NetworkManager tools, if they are not running already. Perform these steps even if you are installing
DataIQ on a host that is not connected to the internet.
Steps
Enable and start Network Manager: systemctl enable NetworkManager systemctl start NetworkManager
Set up networking
Set up a static IP address, subnet mask, and hostname. Perform these steps even if you are installing DataIQ on a host that is not connected to the internet.
Steps
1. Request a static IP address from your network administrator or IT organization.
2. Open the NetworkManager text user interface (NMTUI). Use keyboard commands (Tab key, arrows) to navigate NMTUI menus.
nmtui
3. Select Edit a connection .
4. Select the ethernet connection, for example, enp0s3 .
5. Edit IP v4 CONFIGURATION : in Address , enter the networking information that was provided by your organization: a. At the top level, change the dropdown from Automatic to Manual .
b. Select Show to see more options.
c. Select OK , then Back .
6. Select Activate a connection .
7. Deactivate and reactivate the ethernet connection: select the connection, for example, enp0s3 , and select Deactivate , then select Activate .
8. Select Back .
9. Select Set hostname .
10. Type a hostname that meets the following requirements, for example: dataiq-server . Select OK .
Hostname requirements
DataIQ-specific hostname requirements:
● After DataIQ is set up, changes to the hostname are not supported.
● The hostname must not start with localhost . For example, localhost-dataiq is invalid.
● The hostname cannot contain upper-case letters (A-Z).
The hostname must also meet general DNS hostname requirements:
● The hostname may contain letters (a-z) and digits (0-9).
● The hostname may contain hyphens (-) if they are in the middle of a label. Example: dataiq-host .
● While the initial name section of a DNS A-record cannot have an underscore, underscores ( _ ) are permitted as part of subdomains. Examples:
OK: dataiq._subdomain.myorganization.org
Invalid: _dataiq.myorganization.org
Invalid: _dataiq.subdomain.myorganization.org
To learn more, see http://domainkeys.sourceforge.net/underscore.html
.
Installation 25
Hostname requirements
● Other characters, including blank or space characters are not permitted.
● Single-character names are not permitted.
Recommendations:
● A matching entry should be created as an A-level record in the DNS domain for the network.
11. Select Quit .
12. Restart the network configuration using the new settings: systemctl restart network
13. Verify the network information is correct, for example: ip addr show enp0s3
14. Optional: set up a fully qualified domain name (FQDN), such as https://dataiq.myorganization.com.
a. Edit the /etc/hosts file: vi /etc/hosts
If an entry for localregistry.emc.com
exists, remove it.
Add an entry for the file system based on the file system location. In this example, the sample IP address 192.168.0.34 is mapped to the sample FQDN: dataiq.myorganization.com.
192.168.0.34 dataiq.myorganization.com
NOTE: Using VIM: Press i to enter insert mode, press Esc to escape. Press :wq to write changes and quit. To learn more, see A Great VIM Cheat Sheet .
b. Set up the FQDN from the command line: hostnamectl set-hostname {fqdn}
Results
At this point, you should be able to ping the DataIQ server using the FQDN.
Prepare the DataIQ installer
Copy the DataIQ installer packages to the DataIQ server and extract the installer files.
Steps
1. Copy the DataIQ RPM package to the DataIQ server to a folder, for example, /home/administrator .
If you are setting up DataIQ on a system that is not connected to the Internet, also copy over the DataIQ Dependencies
RPM package .
2. Open the superuser prompt, if it is not already open: sudo -i
3. Go to the folder with the installer, and note the name of the installer file: cd /home/administrator ls
26 Installation
4. If you are setting up DataIQ on a system that is not connected to the Internet, extract files from the DataIQ Dependencies
RPM package , for example: yum install dataiq-dependencies-2.2.2.0-5.el7.noarch.rpm
5. Extract files from the DataIQ Install and Upgrade RPM package , for example: yum install dataiq-2.2.2.0-67.el7.noarch.rpm
6. Recommended: If your system is connected to the Internet, add the latest security updates for your version of Linux.
yum -y update --security
NOTE: Do not run a full yum update, as a full update may update Linux to an unsupported version.
For Oracle Linux: Install add-on packages
On Oracle Linux, to prepare for an offline installation of DataIQ, you must install add-on packages.
For Oracle Linux: Install add-on packages for sites with an internet connection
On Oracle Linux, to prepare for an installation of DataIQ on a site that has an internet connection, install both the Extra
Packages for Enterprise Linux and Oracle Linux add-ons.
About this task
If your site has internal package mirrors for Oracle Linux 7, then you can use these to install the packages.
Steps
1. From a command line, install the Extra Packages for Enterprise Linux: yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
2. Install the Oracle Linux 7 Addons: yum install --enablerepo=ol7_addons container-selinux containerd.io-1.2.6-3.3
For Oracle Linux: Install dependency packages for sites without an internet connection
On Oracle Linux, to prepare for an offline installation of DataIQ, in addition to the DataIQ Dependencies RPM Package for offline sites , DataIQ requires plus two additional packages, iptables-services and policycoreutils-python , that are part of the Oracle Linux distribution.
About this task
If your site has internal package mirrors for Oracle Linux 7, then you can use these to install the packages.
Otherwise, you can use these instructions to set up an Oracle Linux 7 repository to resolve the dependencies for these two additional packages.
NOTE: These steps only cover installing the two additional packages. In later steps, you will also install the DataIQ
Dependencies RPM Package for offline sites .
Installation 27
Steps
1. Connect the Oracle Linux 7 media. For example, on a virtualization platform, add the Oracle Linux 7 .iso as a drive on the system.
2. Identify the drive. From a command line: lsscsi
3. Mount the Oracle Linux 7 media. For example, if the drive is /dev/sr0 : mkdir /media/OL7-iso mount /dev/sr0 /media/OL7-iso
4. Edit the file system table to add an entry for the Oracle 7 Linux media.
vi /etc/fstab
5. Create a YUM repository definition.
# cat /etc/yum.repos.d/ol7-media.repo [ol7-media] name=Oracle Linux 7 baseurl=file:/// media/OL7-iso gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY enabled=1 gpgcheck=1
6. Install the iptables-services and policycoreutils-python dependency files: yum --disablerepo=* --enablerepo=ol7-media install iptables-services policycoreutilspython
7. Fix the bootloader to load the supported kernel: grub2-set-default 'Oracle Linux Server 7.9, with Linux 3.10.0-1160.el7.x86_64'
8. Reboot the server.
Initialize DataIQ
Run the DataIQ installer, receive initial passwords, and optionally set up Secure Remote Services.
Steps
1. Initialize DataIQ: dataiq init
Or, initialize DataIQ with a license file: dataiq init --license-path /mydata/license.xml
.
Follow the instructions to initialize DataIQ:
● Accept the installation EULA. (Type yes .)
Wait up to 20 minutes for DataIQ to initialize.
NOTE:
● If DataIQ fails to initialize: Remove (destroy) any installed DataIQ components and remove the installation package.
To perform an offline installation, extract the dataiq-dependencies files. Extract the DataIQ installer, and initialize
DataIQ again. Example: dataiq destroy
28 Installation
yum remove -y dataiq yum install dataiq-dependencies-2.2.2.0-5.el7.noarch.rpm #(Offline install) yum install dataiq-2.2.2.0-3783.el7.noarch.rpm dataiq init
● To troubleshoot installation or initialization, review the logs at /opt/dataiq/install/dataiq_install.log
and /opt/dataiq/install/playbook.log
.
Figure 2. End of DataIQ 2.2.0.0 installation
2. When prompted: "Do you have an installed SRS gateway ... [y/N]" , you can optionally type y to set up
Secure Remote Services using your existing Secure Remote Services credentials. You can also skip setup by typing N and then set up Secure Remote Services later through the DataIQ UI.
3. Copy the temporary password for the administrator account. In DataIQ, the administrator credentials are sometimes called Keycloak credentials.
You can also retrieve this temporary password using the command: cat /opt/dataiq/install/keycloak_admin.passwd
4. In a browser window, go to the IP address used for DataIQ to open the DataIQ UI. Enter the administrator username and paste or enter the temporary password, then change the password when prompted.
Keep the new password safe using recommended security practices for your organization.
5. Optional: If you use Secure Remote Services, copy down the remote service password. You can give this password to a Dell
Support agent to allow them to gather DataIQ troubleshooting logs when needed.
cat /opt/dataiq/install/keycloak_remoteservice.passwd
Keep the password safe using recommended security practices for your organization. To generate a new password, see
Update service account password .
thumbnail-sized preview images for graphics and videos.
Licenses
Some features require a fully licensed version of DataIQ. Contact your Dell EMC representative for info on purchasing a license.
When you first install DataIQ, you have access to the trial functions for 1 year, including the ability to connect to third-party storage.
Installation 29
After you purchase and activate a license, you get access to the features you have purchased. When your DataIQ license expires, you will only have access to basic DataIQ features.
Activate DataIQ
Get your license file and import it into DataIQ.
Steps
1. Download your license file ( license.xml
) from Software Licensing Central, at https://licensing.emc.com/ .
2. If DataIQ is not connected to the Internet, transfer this file to a workstation that has web access to the DataIQ host.
3. From DataIQ UI, from the Settings page, select Licensing > Upload , then browse to the licensing file.
Results
The License information is updated, including License type , License file version , Customer , and Expires . The ID appears as an eSWID . (Trial versions appear as an iSWID .)
Update license
Get updated license files and upload them to DataIQ through the web UI.
Steps
1. Download your license file (license.xml) from Software Licensing Central, at https://licensing.emc.com/ . (Learn more about licensing at https://www.dell.com/support/contents/article/product-support/self-support-knowledgebase/softwareand-downloads/software-licensing-central-documentation .)
2. If your cluster is not connected to the Internet, transfer this file to a workstation that has web access to the DataIQ host.
3. From DataIQ UI, from the Settings page, select Licensing > Update licensing file , then browse to the licensing file.
Check capacity
Licenses are usually limited by the amount of data stored and monitored on Dell EMC PowerScale and Isilon systems, and on external storage, including Amazon Web Services (S3), Google Cloud Platform (GCP), Dell EMC PowerScale, and Dell Elastic
Cloud Service (ECS) storage.
About this task
You can check the amount of data stored on these sources from the web UI:
Steps
1. From the DataIQ web interface, from the Settings page, select Licensing .
2. Review the current scanning usage to find your current capacity. Compare this with Maximum mounted bytes .
dataiq command-line options
This topic describes the command-line options for the dataiq installer package.
Installation
Usage: dataiq [OPTIONS] init [ARGS]...
--config_file <path>
If you have existing configuration files, you can import them during
30 Installation
--force
--license-path <path>
--ssd_path <path>
--srs_ip <srs_ip_address>
--srs_user <srs_username>
--srs_passwd <srs_password>
--help installation using this option. The values from this option have lower priority than those from other
CLI options. The default paths for configuration files are as follows:
● /opt/dataiq/ maunakea/aws/config
● /opt/dataiq/maunakea/ aws/credentials
● /opt/dataiq/maunakea/ claritynow/etc/ autotag.cfg
● /opt/dataiq/maunakea/ claritynow/etc/ clarity.cfg
● /opt/dataiq/maunakea/ claritynow/etc/ service.cfg
● /opt/dataiq/maunakea/ claritynow/etc/ viewfilter.cfg
Example: dataiq init -config_file /mydrive/ configfiles
Installs using default values, skipping prompts.
Installs using a license file. Example: dataiq init --licensepath /mydata/license.xml
Use this option to change where
DataIQ stores the data management database. By default, DataIQ stores the data management database at /mnt/ssd . Example: dataiq init --ssd_path / mydata
IP address of the SRS server
Username of the SRS server
Password for the SRS server.
Example: dataiq init -srs_ip 10.20.30.40 -srs_user [email protected] -srs_passwd P@ssw0rd
Provides help for using the init command.
Installation 31
More commands
Usage: dataiq [OPTIONS] COMMAND [ARGS]...
certificates
Renews certificates destroy
Uninstalls DataIQ, Docker, and Kubernetes components.
After running this command, you can start DataIQ again by using the init command.
Starts all DataIQ pods.
start status stop
Shows the status of DataIQ pods.
Stops all DataIQ pods.
Use this before using the destroy command.
Options
Use these options with the previously listed commands.
-v, --verbosity [CRITICAL|
ERROR|WARNING|INFO|DEBUG]
Shows how much information to show during the installation process. Example: dataiq --verbosity DEBUG
--version
Show the version of DataIQ that will be installed.
Show help for this command.
--help
32 Installation
4
Upgrade
This section describes upgrading from previous versions of DataIQ, and migrating from ClarityNow.
Upgrade from a previous version of DataIQ
Upgrade requirements
Before upgrading, check that your system meets requirements.
Prerequisites
● A previous version of DataIQ must already be installed.
● The host system must meet the
. DataIQ version 2.x requires more CPUs, drive space, and memory
.
● During an upgrade from DataIQ 2.0 or 2.1, DataIQ requires that the storage management datastore disk has at least 33% of its space free. The free space is used to convert the database (TimescaleDB) into a new format that uses between 33-50% of the size of the original.
About this task
Usually, the upgrade initialization process completes within 20 minutes. However, on some systems with large amounts of data, the upgrade process may take several hours. During this time, users are logged out and storage is not monitored.
After the upgrade initialization process is complete, users can log into DataIQ.
When converting from DataIQ 2.0 or 2.1, DataIQ then begins converting the storage management database. During the time, storage management reports do not show historical storage management data. The amount of time required depends on how much data is being migrated. For example, on a system monitoring one cluster with 20 nodes and half a year of history data, this process takes 2 hours.
Steps
1. Recommended: Back up your data to a separate location
. If the upgrade fails, these backup files may be required to roll back the installation.
2. Connect to the DataIQ server and open the superuser prompt: sudo -i
3. When upgrading from DataIQ version 2.0 or 2.1, check to see if there is enough space to complete the upgrade: a. Select Settings > Storage monitoring database settings .
b. Check the datastore disk usage. If the disk is less than 60% used, then there is enough space.
Upgrade 33
If the disk is more than 60% used, then note the unused space and perform an additional check: c. From the DataIQ command line, check the size of the existing data store.
du -sh /opt/dataiq/maunakea/data/timescale
33.9G /opt/dataiq/maunakea/data/timescale
Compare this size to the amount of free space on the datastore disk from the previous step. If the free space is more than half of the size of the storage management database, then there is enough space. Continue with the upgrade.
4. Optional: Confirm that the Kubernetes pods have the status of Running , Ready , or Completed .
● For DataIQ v1, these pods include claritynow , clew , imanager , and ixui .
● For DataIQ v2, these pods include claritynow , clew , collector (sometimes, and may appear more than once), grafana , imanager , ixui , keycloak , keycloakdb , metadata , and timescale .
● For DataIQ v2.1.x, the pods are: claritynow , claritynow-plug-in-init , clew , collector (sometimes, and may appear more than once), grafana , imanager , ixui , keycloak , keycloakdb , metadata , and timescale .
NOTE: During the upgrade, the status of all pods is Stopped . After the upgrade, the status is Running .
[root@kmaster ~]# dataiq status
DataIQ Status
PLAY [ansible_localhost] *******************************************************
TASK [Gathering Facts] ********************************************************* ok: [localhost]
TASK [Show DataIQ PODs statuses] ***********************************************
NAME READY STATUS RESTARTS AGE claritynow-f68fccb4d-7llr4 1/1 Running 0 19h claritynow-plug-in-init-tzx8g 0/1 Completed 0 19h clew-7874f9595f-5dw2x 1/1 Running 0 19h collector-10-111-225-115-7c9666fd49-m8bzg 1/1 Running 0 19h collector-10-111-225-118-7549cf8c7b-97rjb 1/1 Running 19 19h dataiq-realm-init-qrp6l 0/1 Completed 0 19h grafana-6944844797-ggs7g 1/1 Running 0 12h imanager-794bbddd9-r4prm 1/1 Running 0 19h ixui-68fd7d4b8f-wxd7v 1/1 Running 0 19h keycloak-0 1/1 Running 0 19h keycloakdb-0 1/1 Running 0 19h metadata-0 1/1 Running 0 19h timescale-5c95795d79-pxmv2 1/1 Running 0 19h ok: [localhost]
PLAY RECAP ********************************************************************* localhost : ok=<#> changed=<#> unreachable=0 failed=0 skipped=<#> rescued=0 ignored=0
Playbook log was saved in here: /opt/dataiq/install/playbook.log
34 Upgrade
Download installation and upgrade packages
Get the installer, and optionally, a license file. Licenses can be added either during initialization or after installation through the
DataIQ UI.
Steps
1. Download the DataIQ Install and Upgrade RPM Package , for example, dataiq-2.2.x.x-xxxx.el7.noarch.rpm
.
from Dell Support, at https://www.dell.com/support/home/product-support/product/data-iq .
2. If your existing server or virtual machine (VM) that is not connected to the Internet, also download the DataIQ
Dependencies RPM Package for offline sites : dataiq-dependencies-2.2.x.x-xxxx.noarch.rpm
.
NOTE: For sites without a network connection: it is recommended to install DataIQ using the OVA bundle so that
DataIQ can manage future application updates using standard upgrade files.
3. Optional: Download a license file ( license.xml
) from the Software Licensing Central, at https://licensing.emc.com/ .
Check the authenticity of a DataIQ RPM file
Help verify the authenticity of DataIQ RPM Package Manager download files with the DataIQ RPM signing key .
About this task
To learn more about verifying RPM file signatures, see https://access.redhat.com/documentation/en-us/ red_hat_enterprise_linux/5/html/deployment_guide/s1-check-rpm-sig .
Steps
1. Download the DataIQ RPM signing key from Dell Support, at https://www.dell.com/support/home/product-support/ product/data-iq . The file is named RPM-GPG-KEY-DATAIQ.gpg
.
2. Import the signature file into your trusted signatures: rpm --import RPM-GPG-KEY-DATAIQ.gpg
3. Verify the RPM file, for example: rpm -K dataiq-2.2.2.0-67.el7.noarch.rpm
Upgrade DataIQ
Keep DataIQ upgraded with new features and fixes.
Prerequisites
Back up your data to a separate location
. Check the
.
Download and verify the RPM installer file.
Steps
1. Connect to the DataIQ server and open the superuser prompt: sudo -i
2. Recommended: Upgrade to Red Hat Enterprise Linux or CentOS version 7.9. (Note, do not run the yum update command by itself, because this could update Kubernetes or Docker, which could cause Kubernetes Pods to fail.) yum update --disablerepo=docker\*,kubernetes\*
3. Copy the RPM package or packages to the DataIQ server, to the /home/administrator folder.
Upgrade 35
● To transfer RPMs from another system, use a file-transfer tool such as WinSCP.
● To transfer RPMs from a portable drive, use the command line, for example: cp /yourdrive/dataiq-2.2.2.0-67.el7.noarch.rpm /home/administrator cp /yourdrive/dataiq-dependencies-2.2.2.0-5.el7.noarch.rpm /home/administrator
4. If you are setting up DataIQ on a server or VM that is not connected to the Internet, extract files from the DataIQ
Dependencies RPM Package for Offline Sites : cd /home/administrator yum install dataiq-dependencies-2.2.2.0-5.el7.noarch.rpm
5. Extract files from the DataIQ Install and Upgrade RPM Package : cd /home/administrator yum install ./dataiq-2.2.2.0-67.el7.noarch.rpm
6. Initialize DataIQ. The installer detects the previous installation of DataIQ and upgrades automatically: dataiq init
Follow the instructions to initialize DataIQ:
● Accept the installation EULA. (Type yes .)
Wait for DataIQ to initialize. Usually the upgrade process completes within 20 minutes. However, on some systems with large amounts of data, the upgrade process may take several hours.
When the upgrade is finished, the script reports "Upgrade successfully finished." For example:
TASK [Upgrade: Upgrading from dataiq-<old version> to <new version> (typically under
10 minutes)] *** changed: [localhost]
TASK [Upgrade: Upgrade dataiq-<old version> to <new version> successfully.] *** ok; [localhost] => [
"msg": "Upgrade script completed without error"
]
PLAY RECAP localhost : ok=<#> changed=<#> unreachable=0 failed=0 skipped=<#> rescued=<#> ignored=0
Playbook log wan saved in here: /opt/dataiq/install/playbook.log
The upgrade process has been completed successfully.
NOTE: To troubleshoot upgrading, review /opt/dataiq/install/playbook.log
. Also check to see if /opt/ dataiq/install/dataiq_upgrade.log
exists, and if so, review it too. These log files may provide help or instructions. After the problem is resolved, run dataiq init to try the upgrade again.
If DataIQ fails to initialize,
restore DataIQ back to the version you were running before the upgrade, and try again.
7. Optional: Confirm that the new pods are Running . For DataIQ v2.2.x, the pods are: claritynow , claritynowplug-in-init , clew , collector (sometimes, and may appear more than once), dataiq-realm-init , grafana , imanager , ixui , keycloak , keycloakdb , metadata , timescale , tsdb-down-sampling , and tsdb-schemaupgrade . Future versions may have new pods.
[root@dataiq-48 ~]# dataiq status
DataIQ Status
PLAY [ansible_localhost] *******************************************************
TASK [Gathering Facts] ********************************************************* ok: [localhost]
TASK [Show DataIQ PODs statuses] ***********************************************
NAME READY STATUS RESTARTS AGE claritynow-5cc9f4d459-vgmrn 1/1 Running 0 17d
36 Upgrade
claritynow-plugin-init-5z82r 0/1 Completed 0 17d clew-7bd775c965-mdd7h 1/1 Running 0 10d dataiq-realm-init-lg7xz 0/1 Completed 0 17d grafana-69bcd4fc4c-nc8m5 1/1 Running 0 17d imanager-99c7bd5bd-nhd5d 1/1 Running 0 17d ixui-79587d7d8d-x499b 1/1 Running 0 17d keycloak-0 1/1 Running 0 17d keycloakdb-0 1/1 Running 0 17d metadata-0 1/1 Running 0 17d timescale-5f9449bc99-6nnjm 1/1 Running 0 17d tsdb-down-sampling-6f4598b798-ks846 1/1 Running 0 17d tsdb-schema-upgrade-vggh4 0/1 Completed 0 17d ok: [localhost]
PLAY RECAP ********************************************************************* localhost : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
Playbook log was saved in here: /opt/dataiq/install/playbook.log
NOTE: If there is an issue with any of the pods, contact Dell Support.
8. Optional: To confirm the version number in the DataIQ UI, select Settings > About .
9. Perform a scan on all volumes to refresh the data management database, or wait for the first automatic scan to complete.
Until this first volume scan is completed, in Data Management :
● If Settings > Local settings > Show size: Physical is selected, folder and file size will show as 0 .
● When upgrading from DataIQ version 1.0 or 2.0,
○ Searches do not show results.
○ In Browse and Analyze views, data categories and bins are not shown ( Hot , Warm , Cold , Frozen ).
10. If you use DataIQ plug-ins, download the upgraded versions of the plug-ins, and then upgrade each plug-in. To learn more, see
.
11. Recommended: After storage management reports begin displaying historical storage data again, perform a DataIQ system backup.
Immediately after upgrading, DataIQ automatically begins converting the storage management database (TimescaleDB).
During this conversion process, storage management reports do not show historical storage management data.
The time of the conversion process depends on how much data is being migrated. For example, on a system monitoring one cluster with 20 nodes and half a year of history data, this process takes 2 hours.
Scan a volume or path
DataIQ performs regular scans on volumes. However, if data changes on a volume, the changes do not appear in DataIQ until the next scan. Perform a manual scan on a volume or path to see the latest information.
Steps
1. In the Data Management pane, in the Browse page, navigate to a volume or folder.
2. In the far-right pane, select the Actions tab.
3. From the Actions list, select Scan volume , Scan all volumes , Scan group , or Scan selected path , depending on what's selected.
4. Optional: Check the status of the scan. In the Jobs page, review the status and wait for the job to complete. Or, check the
Logs page to see details about jobs that have already completed.
Upgrade 37
Upgrade OpenSSL
To address a security vulnerability (CVE-2021-3712) , update OpenSSL to version 1.1.1n or later.
About this task
To learn about the security vulnerabilities addressed in the latest updates, see https://www.openssl.org/news/ vulnerabilities.html
.
Steps
1. On the DataIQ host, from a command line, check the version of OpenSSL on your system.
openssl version
The version is shown. For example:
OpenSSL 1.0.2k-fips 26 Jan 2017
If the version is 1.1.1n or later, you can skip the rest of this topic. Example:
OpenSSL 1.1.1n 24 Aug 2021
2. On a computer connected to the internet, download the OpenSSL installation files from https://www.openssl.org/source/ , for example, openssl-1.1.1n.tar.gz
, and then transfer them to the DataIQ host.
3. From the DataIQ host, remove the previous version of OpenSSL: yum remove -y openssl
4. Install Openssl 1.1.1n using tarball: yum install -y make gcc perl pcre-devel zlib-devel perl-core
5. Verify the installation file by comparing the MD5 checksum value. Example: md5sum openssl-1.1.1n.tar.gz
The MD5 checksum for OpenSSL version 1.1.1n should be MD5_HASH=2aad5635f9bb338bc2c6b7d19cbc9676 .
6. Extract the installer. Example: tar -xvf openssl-1.1.1n.tar.gz
7. Compile and install OpenSSL. Example: cd openssl-1.1.1n
./config make make install export LD_LIBRARY_PATH="/usr/local/lib:/usr/local/lib64" echo "export LD_LIBRARY_PATH=/usr/local/lib:/usr/local/lib64" >> ~/.bashrc
8. Check the version of OpenSSL on your system. The version should now be 1.1.1n or later. Example:
[root@...]# openssl version
OpenSSL 1.1.1n 24 Aug 2021
38 Upgrade
NOTE: If you receive the message: "No such file or directory," you can try either of the following steps:
● Close and reopen the connection to DataIQ, and check the version again.
● Or, add the directories: /usr/local/sbin and /usr/local/bin to the system path and try again: export PATH=$PATH:/usr/local/sbin:/usr/local/bin
9. Optional: Cleanup the installation by removing the installation files. Example: cd ..
rm -rf openssl-1.1.1n
rm -rf openssl-1.1.1n.tar.gz
yum remove -y pcre-devel zlib-devel perl-core
Upgrade OS packages
Keep your Linux OS updated on your DataIQ host.
yum check-update
See which packages are available for updating.
yum check-update
yum update [package-name]
To update specific packages, where [package-name] is the name of the package you are updating: yum update [package-name]
man yum
To see all update options, see the Yum Linux manual page.
man yum
Migrate from ClarityNow to DataIQ
DataIQ can perform all the functions of ClarityNow to manage the unstructured data on your storage. This document provides steps to make the switch to DataIQ.
Advantages
You can use DataIQ to find and analyze your data, and generate reports using meaningful, business-oriented categorization. In addition, DataIQ storage monitoring features deliver a single-pane-of-glass view of all file and object data across the enterprise as well as investigative and historical reporting on the performance of PowerScale clusters. DataIQ offers other advantages for managing data over ClarityNow:
● Can track tens of billions of files and folders (ClarityNow can track up to 500 million individual files. Over this number of files, it must be reconfigured to only track folders, up to 500 million folders).
Upgrade 39
● Requires less memory to manage large file and folder counts.
● Searching for files is faster.
● Tracks more statistics about files and folders.
● Reports show time-based categories (age binning) on all files, not just on tagged files.
● Browser UI can be used on more modern platforms, using HTML5 instead of Java Client.
● Supports OneFS for both Isilon and PowerScale customers.
● Much more powerful plugin capabilities. Plugins no longer have Java HTML3 limits.
To learn more about DataIQ, see the DataIQ homepage . and the DataIQ Info Hub .
Support and licensing
Contact your Dell EMC Sales team to learn about licensing and support service options for DataIQ.
Transition to DataIQ
When you transition from ClarityNow to DataIQ, you may be able to reuse some configurations, including autotag rules, viewfilter configurations, and some custom context menu scripts. Note the following changes when transitioning from
ClarityNow versions 2.9.x, 2.10.x, or 2.11.x. To convert from previous versions, more changes may apply. For support, see the Dell Support for DataIQ .
Installation
● DataIQ requirements : A minimal DataIQ install requires more CPU cores, memory (RAM), and drive space than a minimal
ClarityNow install. If you do plan to reuse the ClarityNow server for DataIQ, copy the configuration files mentioned in this guide to a location outside of the server before uninstalling ClarityNow. After ClarityNow is uninstalled, install DataIQ.
● Run DataIQ and ClarityNow simultaneously : It is possible to run both DataIQ and ClarityNow on two different systems, connected to the same volumes. This can help reduce interruptions associated with setting up and transitioning the organization to the new tools.
NOTE: Running both DataIQ and ClarityNow on the same system in separate VMs is not supported because this action might create network port conflicts. To learn more, see Requirements in the DataIQ Install Guide .
File tags
DataIQ uses the same tagging conventions as ClarityNow:
Automatic tags : You can copy and paste Autotag rules from ClarityNow into DataIQ and begin working immediately:
1. From ClarityNow: Copy the autotag rules from the command-line file: /usr/local/claritynow/etc/autotag.cfg
.
2. From the DataIQ UI: Select Settings > Data management configuration > Configure autotagging , and paste in the rules.
In DataIQ, unlike ClarityNow, each tag rule applies either to files or folders, but not to both. Rules pasted in from ClarityNow will by default apply to folders only. To apply these rules to individual files instead, add in the attribute: applies_to_files. To learn more, see the DataIQ Admin Guide . In addition, DataIQ supports a new autotag feature that can categorize individual files: applies_to_files. To learn more, see Autotag file format the DataIQ Admin Guide .
Manual tags : DataIQ continues to support ClarityNow .cntag
files to automatically categorize and tag folders.
Viewfilters
DataIQ uses the same conventions for setting up viewfilters as ClarityNow.
You can copy and paste viewfilter rules from ClarityNow into DataIQ and begin working immediately:
1. Recommended: Set up the group names in DataIQ the same way as they are set up in ClarityNow.
2. Recommended: Set up virtual paths in DataIQ the same way as they are set up in ClarityNow.
3. From ClarityNow: Copy the viewfilter rules from the command-line file: /usr/local/claritynow/etc/ viewfilter.cfg
.
40 Upgrade
4. In DataIQ: From the UI, go to Settings >Data management configuration > Configure data management , and paste in the rules.
5. If you used different group names or virtual paths in steps 1 and 2, adjust the rules as necessary.
To learn more, see Viewfilters in the DataIQ Admin Guide .
Data management configuration settings
Both ClarityNow and DataIQ use similar settings to manage volume scans. However, their effects on the systems may be different. In most cases, in DataIQ, the default data management settings should be left empty and not changed. This is true even if you used these settings in ClarityNow.
Supported ClarityNow data management settings that are still in use in DataIQ:
● concurrentIndexedScans
● maxScanErrors
● maxObjectsInFolderEnumeration
● enableS3Trace
Deprecated settings:
● patternStartLimitMode . This setting is no longer needed.
Email notifications
The procedure to set up email notifications is different in DataIQ. To learn more, see the DataIQ Admin Guide .
Authentication
The procedure to set up Active Directory authentication through LDAP is different in DataIQ. Note, the process for adding permissions to an LDAP group is simplified: You can now manage the permissions for individual LDAP groups directly. It is not necessary to add the LDAP group into a local group to set permissions. To learn more, see the DataIQ Admin Guide .
Volume mapping
To mount an NFS-based volume, use the NFS type. To learn more, see the DataIQ Admin Guide .
Custom context menu (CCM) scripts
See the DataIQ Developer Hub to learn how to convert ClarityNow custom context menus (CCM) scripts into plugins that are compatible with DataIQ.
In many cases, the CCMs can be converted with minimal changes. However, some variable substitution strings do not have direct equivalents and may require additional work to convert.
Some ClarityNow CCMs, including Graphite and Client Transfer, do not yet have direct equivalents with DataIQ.
If you need help, Dell EMC provides conversion assistance through Professional Services.
Upgrade 41
5
Uninstall
This chapter discusses uninstalling DataIQ.
Uninstall DataIQ
Use this procedure to uninstall DataIQ.
About this task
The uninstall process removes:
● All files and configurations in /opt/dataiq , except for some data backup files.
● Kubernetes and Docker packages.
● RPM packages installed by DataIQ, for example, nfs-utils .
Steps
1. From the command line, remove the DataIQ environment: dataiq destroy
(For more info, use dataiq destroy --help )
2. Uninstall DataIQ: yum remove -y dataiq
3. If the DataIQ dependency RPM is installed, uninstall it: yum remove dataiq-dependencies
4. Recommended: Remove the entry: localregistry.emc.com
in the /etc/hosts file.
Results
The uninstall process can take up to 5 minutes. Log files for the uninstall are at /opt/dataiq/install/playbook.log
.
After DataIQ is uninstalled, it can be reinstalled using the dataiq init command.
42 Uninstall
6
Backup and restore
You can back up and restore DataIQ configuration data, file and folder metadata, storage monitoring data, and plug-in data.
Table 5. Backup data locations
Type of data
Configuration data
Data management data
Storage monitoring data
Path
A path that you specify, for example, / mydrive/backup-dataiq .
The default backup path is /mnt/ssd/ data/claritynow/ssd/backup .
Size
Several MB.
This file starts as several hundred KB, and may grow over time to be several hundred MB.
Do not remove this folder, even after a system backup or recovery.
The default backup path for a backup is
/opt/dataiq/backup/timescale .
A backup is roughly 10% of the size of the total database.
Backup
When a backup is running, data management features are offline. The backup process may take up to a few minutes.
Steps
1. Create a folder to store the backup, for example: mkdir /mydrive/backup-dataiq/
2. From a command line, start a backup to the new volume, using the format: sh /opt/dataiq/bin/ svc_dataiq_backup_all.sh {backup_path} {username} [{password}] .
DataIQ only supports using the administrator account for backups. For username , use administrator . For password , use the administrator account password.
If the password is not included, the script prompts for the password. Example: cd /opt/dataiq/bin/ sh svc_dataiq_backup_all.sh /mydrive/backup-dataiq/ administrator
You need to input password of DataIQ:
Login DataIQ succeed.
Backup started, id 2
3. Optional: Review the backup status logs to ensure that the backup was successful: sudo /opt/dataiq/bin/svc_dataiq_get_backup_status.sh administrator password123
4. Optional: Copy the data to a separate drive, for example: mkdir /mydrive/datamanagement cp -rp /mnt/ssd/data/claritynow/ssd/backup /mydrive/datamanagement mkdir /mydrive/storagemonitoring cp -rp /opt/dataiq/backup/timescale /mydrive/storagemonitoring
Backup and restore 43
5. Optional: Copy plugin data to a separate drive, for example: cp /opt/dataiq/maunakea/data/plugins /mydrive/backup-dataiq/plugins
Restore to the same host
You can restore DataIQ configuration data from the most recent backup, including user authentication data and volume configuration.
Prerequisites
About this task
DataIQ only supports restoring to the most recent backup.
Steps
1. Remove the DataIQ environment: dataiq destroy
2. Optional: Uninstall and reinstall DataIQ: yum remove -y dataiq yum install dataiq-2.x.x.xxxxxxx.rpm
3. Reinitialize DataIQ: dataiq init
4. Optional: Restore the data management data:
● To restore from a manual backup location, for example, from /mydrive : cp -r /mydrive/datamanagement/ /mnt/ssd/data/claritynow/ssd
● To restore from the default location: cp -r /mnt/ssd/data/claritynow/ssd/backup /mnt/ssd/data/claritynow/ssd/
NOTE: If you skip this step, file and folder information will not be available until after the first rescan of the database.
5. Delete the /opt/dataiq/backup/timescale folder: rmdir -rf /opt/dataiq/backup/timescale
6. Optional: Restore the storage monitoring data:
● To restore from a manual backup location, for example, from /mydrive : cp -rp /mydrive/storagemonitoring /opt/dataiq/backup/timescale
● To restore from the default location, move back the files from /opt/dataiq/backup/timescale-<datestamp> .
Example: cp -rp /opt/dataiq/backup/timescale-xxxxx /opt/dataiq/backup/timescale
NOTE: If you skip this step, Storage Monitoring screens show no previous cluster history.
44 Backup and restore
7. Restore the configuration data, using the format: sh /opt/dataiq/bin/svc_dataiq_restore_all.sh
{backup_path} {dataiq_username} [{dataiq_password}] . Example: cd /opt/dataiq/bin/ sh svc_dataiq_restore_all.sh /mydrive/backup-dataiq administrator password123
If the password is not included, the script prompts for the password.
After the restore is complete, you should receive a success message:
% Total % Received
100 828 100 828
% Total % Received
100 850 100 21 ims_result:{ "status":"success" }
DatalQ restore accomplished.
Quit overall restore at Sun Oct 11 03:01:29 EDT 2020 Total time: 7:45
If the restore is not successful, uninstall and reinstall DataIQ, and then restore the DataIQ data again.
8. Recommended: Check the status of the restore: sudo /opt/dataiq/bin/svc_dataiq_get_restore_status.sh
9. Recommended: Perform a manual scan of all volumes. In the Data Management window, in Volumes , select the root volume. Select the Actions tab. Select Scan all volumes . (Alternately, wait for automatic file scans.)
10.
.
11. Optional: Restore plugin data. Example: cp /mydrive/backup-dataiq/plugins /opt/dataiq/maunakea/data/plugins
NOTE: If you skip this step, plugins will use the default configurations.
12. Start each plugin:
● For Previewer, Audited Delete, and Duplicate Finder, start the plugin:
/opt/dataiq/plugin_manager/bin/plugin_manager start previewer
/opt/dataiq/plugin_manager/bin/plugin_manager start audited-delete
/opt/dataiq/plugin_manager/bin/plugin_manager start duplicates
● For Data Mover (and other external plugins that include more than one Kubernetes pod), stop and restart the plugin: python datamover.py stop python datamover.py start
13. If you connect DataIQ to an S3 endpoint, re-create the credentials file
to add back login information.
Restore to a different host
You can restore DataIQ configuration data from the most recent backup, including user authentication data and volume configuration.
Prerequisites
●
● Set up the new host with the same IP address and hostname as the original host. We recommend also using the same operating system and system specs as the original host.
About this task
DataIQ only supports restoring to the most recent backup.
Backup and restore 45
Steps
1. Install DataIQ.
2. Optional: Restore the data management data: cp -r /mydrive/datamanagement/ /mnt/ssd/data/claritynow/ssd/
NOTE: If you skip this step, file and folder information will not be available until after the first rescan of the database.
3. Delete the /opt/dataiq/backup/timescale folder: rmdir -r /opt/dataiq/backup/timescale
4. Optional: Restore the storage monitoring data: cp -rp /mydrive/storagemonitoring /opt/dataiq/backup/timescale
NOTE: If you skip this step, Storage Monitoring screens show no previous cluster history.
5. Restore the configuration data using the format: sh /opt/dataiq/bin/svc_dataiq_restore_all.sh
{backup_path} {dataiq_username} [{dataiq_password}] , example: cd /opt/dataiq/bin/ sh svc_dataiq_restore_all.sh /mydrive/backup-dataiq/ administrator password123
If the password is not included, the script prompts for the password.
After the restore is complete, you should receive a success message:
% Total % Received
100 828 100 828
% Total % Received
100 850 100 21 ims_result:{ "status":"success" }
DatalQ restore accomplished.
Quit overall restore at Sun Oct 11 03:01:29 EDT 2020 Total time: 7:45
If the restore is not successful, uninstall and reinstall DataIQ, then restore the DataIQ configuration data again.
6. Recommended: Check the status of the restore: sudo /opt/dataiq/bin/svc_dataiq_get_restore_status.sh
7. Recommended: Perform a manual scan of all volumes. In the Data Management window, in Volumes , select the root volume. Select the Actions tab. Select Scan all volumes . (Alternately, wait for automatic file scans.)
8.
.
9. Optional: Restore plugin data. Example: cp /mydrive/backup-dataiq/plugins /opt/dataiq/maunakea/data/plugins
NOTE: If you skip this step, plugins will use the default configurations.
10. Start each plugin:
● For Previewer, Audited Delete, and Duplicate Finder, start the plugin:
/opt/dataiq/plugin_manager/bin/plugin_manager start previewer
/opt/dataiq/plugin_manager/bin/plugin_manager start audited-delete
/opt/dataiq/plugin_manager/bin/plugin_manager start duplicates
● For Data Mover (and other external plugins that include more than one Kubernetes pod), stop and restart the plugin: python datamover.py stop python datamover.py start
46 Backup and restore
11. If you connect DataIQ to an S3 endpoint, re-create the credentials file
to add back login information.
Create a credentials file
At the DataIQ command-line, add a credentials file that can access Amazon S3-compatible storage.
Steps
1. Create a credentials file using a text editor and save it to the DataIQ host at /opt/dataiq/maunakea/aws/ credentials . For example: vi /opt/dataiq/maunakea/aws/credentials
You can add credentials for multiple S3 providers into the same file. For each provider, add the following:
For Amazon S3
For Dell EMC ECS
For Dell EMC PowerScale
For Google Cloud Platform
Unique identifier, in brackets
Must use [default] . This name is required.
Use any unique value, for example: [ecs]
Use any unique value, for example: [powerscale]
Use any unique value, for example: [gcp] .
aws_access_key_id
Use the
Use the
Use the
Use the access key .
object user name access key ID .
private_key_id .
.
aws_secret_access_key
Use the
Use the
Use the
Use the secret access key secret key secret key .
.
private_key .
Write down the unique identifier that you use. For S3-compatible systems other than Amazon, you will use this identifier to add an endpoint.
Sample file:
[default] aws_access_key_id = 123234345456567678789890 aws_secret_access_key= ABCDE12345
[ecs] aws_access_key_id = dataiquser aws_secret_access_key= ABCl23def456G7h8I9j0=
[powerscale] aws_access_key_id = 12345abcde7890a1b2c3d4e5 aws_secret_access_key= PqRsT67890uVwXyZ/A1fB2gC3hD4iE5j\abc
[gcp] aws_access_key_id = XYZAB12345abcde7890a1b2c aws_secret_access_key= abc\PqRsT67890uVwXyZ/A1fB2gC3hD4iE5j
NOTE: Using VIM: Press i to enter insert mode, press Esc to escape. Press :wq to write changes and quit. To learn more, see A Great VIM Cheat Sheet .
2. Recommended: Hide the credentials file: chmod 600 credentials
.
Backup and restore 47
7
Plugin installation
This section discusses installing, upgrading, and uninstalling DataIQ plugins.
Plugins available from Dell Support
Plugins are disabled by default and are optional to use.
Free plugins:
● Previewer shows thumbnail-sized preview images for graphics and videos, and lists attributes such as resolution. It is extensible to other file types.
Paid plugins (may require additional license agreements):
● Audited Delete allows you to delete folders and files from the Data Management tab.
● Data Mover transfers files and directories between different volumes and file systems.
● Duplicate Finder finds duplicate files and directories across different volumes.
DataIQ Audited Delete plugin
Audited Delete is a paid plugin for DataIQ that allows you to delete folders and files from the Data Management : Browse window, in the Actions tab. The plugin asks for confirmation before deleting anything, and logs details of the delete operation.
Limitations
Does not work with object stores (S3, GCP, or ECS).
Enable the plugin
1. From the DataIQ browser: Select Settings > Data management configuration .
2. In the Plugins section, select the ⋮ (vertical ellipses) icon next to Audited Delete , and select Enable .
3. In the Plugins section, select the ⋮ (vertical ellipses) icon next to Audited Delete , and select Edit configuration . Check the settings to ensure they work for your organization.
View the audit trail
To see the Audited Delete audit trail, you can do any of the following:
● Use the Log button in the plugin window, or ...
● View the logs at /opt/dataiq/maunakea/data/plugins/audited-delete/outputs/full-deletehistory.log
, or...
● Use the command: sudo kubectl logs -ndataiq plugin-audited-delete-0 .
DataIQ Data Mover plugin
Data Mover is a paid plug-in for DataIQ that transfers files and directories between different volumes and file systems.
To learn more about installing, setting up, and using Data Mover, see the Dell EMC Data Mover Admin Guide .
48 Plugin installation
Features and limitations
Data Mover helps you move data between your DataIQ-connected volumes.
Features
● Moves data between different file systems, including NFS, VFS, and SMB volumes, and S3 endpoints such as Amazon Simple
Storage Service (S3), Google Cloud Platform (GCP), and Dell Elastic Cloud Storage (ECS).
● Logs file transfers to assist with audits.
● Supports transfer policies to allow or restrict use of Data Mover based on user group and by source and destination visibility.
● Keeps DataIQ updated by reindexing files after transfers.
● Verifies file transfers by using checksum algorithms, including MD5 and SHA256.
● Supports the use of additional servers (Data Mover worker nodes) to speed up file transfers and to reduce the burden on the DataIQ primary host.
● Can move multiple TBs of data per job.
● Captures and stores file permissions and attributes such as user id , group id , atimes , and mtimes on POSIXcompatible systems. When files are restored, Data Mover reapplies the attributes.
Administrators can decide whether non-admin users can perform the following tasks:
● See and edit transfer jobs started by other users.
● Delete the source file or directory after the transfer is completed and verified.
● Overwrite existing files at the destination.
● Transfer files in sequence to potentially improve video playback performance.
● Throttle active transfer jobs.
Limitations
● Cannot move data directly between two external object storage (S3) buckets.
● Cannot move files based on policies. (Example: Data not accessed in the last 30 days.)
● Does not capture file permissions or file attributes on Windows-based systems.
● Does not support integration with AD for SMB ownership.
● Does not support automatic archiving, and does not have incremental archiving capability. To learn more about archiving
OneFS data, see the OneFS CloudPool Administration Guide.
● Does not support stateful folder transfer or on-the-fly updates. To catch updates, you must restart the data transfer.
● Does not support multi-source tree moves.
DataIQ Duplicate Finder plugin
Duplicate Finder is a paid plugin for DataIQ that finds duplicate files and directories across different volumes.
Identify and remove duplicate files with Duplicate Finder
When Duplicate Finder is installed, DataIQ can find duplicate files across your managed clusters.
About this task
Limitation: Duplicate Finder does not work with object stores such as Amazon Web Services (S3), Google Cloud Platform
(GCP), Dell EMC PowerScale, and Dell Elastic Cloud Service (ECS).
Steps
1. From DataIQ, select Data Management , and then use either the Search or Browse features to find a file, folder, or volume that you want to scan.
2. Select the Actions tab > Show duplicates .
3. In the Duplicates Finder window, review the list of possible duplicates, and select those you'd like to get more detail on.
For the initial scan, Duplicates Finder compares:
● File name
● Size
● Last modified time (Mtime) of the file.
Plugin installation 49
4. For the second comparison, to check whether files or folders seen in the list are truly duplicates of each other, select checkboxes to the left of listed paths and then select MD5 Checksum . Identical files are highlighted in bright green.
5. Select any files you no longer need, and select Delete .
Automatically scan for duplicate files
Schedule automatic searches for duplicate files across your managed clusters.
Steps
1. From the DataIQ UI: Select Settings > Data management configuration .
2. In the Plugins section, select the ⋮ (vertical ellipses) icon next to Duplicates , and select Edit configuration .
3. Uncomment the Cron Jobs section. Keep all the lines for when you want the cron job to execute, and either comment out or remove the rest.
Optional: To run the cron job immediately after saving the config file, in the Execute On section, change the value of
Start to True .
4. Review the cron job output at /opt/dataiq/maunakea/data/plugins/duplicates/outputs/cron-<user name>/ . Or, run the command: kubectl logs -ndataiq plugin-duplicates-0 .
DataIQ Previewer plugin
Previewer is a free, optional plugin that shows thumbnail-sized preview images for graphics and videos.
Configure Previewer
Edit the Previewer configuration file from the DataIQ GUI.
From the DataIQ browser: Select Settings > Data management configuration .
In the Plugins section, select the ⋮ (vertical ellipses) icon next to Previewer , and select Edit configuration .
Configuration file conventions:
● Use spaces for indentations, not tabs.
● Ensure indentation matches exactly.
● When using Boolean values and numbers, do not use quote marks. For all other values, use quote marks.
NOTE: Do not edit the configuration file directly from the command-line, otherwise your changes may not be taken.
watermark (required)
The watermark value is a path to an image that is superimposed on graphics in the DataIQ previewer.
To add a custom watermark:
● Create a transparent .png image. The recommended size is 160x100 pixels.
● Add the image to the DataIQ host in the /opt/dataiq/maunakea/data/plugins/previewer/ directory.
● Refer to the plugin in the configuration file as /hoststorage/<filename> , for example:
"watermark":
value: "/hoststorage/MyCustomWatermark.png"
default: "watermark.png"
exclude_s3 (required)
The exclude_s3 value determines whether to refuse to preview image objects on S3 volumes.
True : Image objects on s3 volumes are not shown in the preview window.
50 Plugin installation
False : Image objects on s3 volumes are shown in the preview window.
"exclude_s3":
value: False
s3_max_bytes (required)
Use exclude_s3 to prevent situations from attempting to preview very large files. When a large file is selected for preview,
DataIQ immediately displays a message that a preview is not available.
The value is the maximum size in bytes of S3 image objects to try to preview. The default value is 1073741824 (= 1 GiB).
"s3_max_bytes":
value: 1073741824
S3 buckets (optional)
Format:
"S3 buckets:"
<On the DataIQ host, add "value:". Do not add "value:" on worker nodes>
"<bucket_name>": # Repeat this section for each bucket
"dataiq_volume": "< DataIQ Volume Name >"
"access_key": "< bucket access key >"
"secret_key": "< bucket secret key >"
"endpoint": "< endpoint url >"
< bucket_name >
(required) dataiq_volume
(optional)
The name of the S3 bucket. The name must match the bucket name that is used on the S3 service.
Buckets with the same name are not supported.
The name of the volume within DataIQ. Only use this setting when the S3 bucket is already represented as a volume within DataIQ. The name must match the DataIQ volume name, and it must be prepended with a forward slash (/). If this value is not present, the bucket may have files that are transferred to it, but no transfer will be allowed from the bucket to the file system.
The access key for the S3 bucket.
access_key
(required) secret_key
(required) endpoint
(required)
The secret key for the S3 bucket.
The endpoint URL of the S3 bucket.
Example for Data Mover Host:
"S3 buckets:"
value:
"amazon-s3-bucket":
"dataiq_volume": "/amazon_s3"
"access_key": "123234345456567678789890"
"secret_key": "ABCDE12345"
"endpoint": "s3.ap-northeast-1.amazonaws.com"
"ecs-s3-bucket":
"dataiq_volume": "/ecs"
"access_key": "dataiquser"
"secret_key": "FGHIJ67890"
"endpoint": "http://10.123.45.67:9090"
"google-s3-bucket-3":
"dataiq_volume": "/gcp_on_us_west_4"
"access_key": "12345abcde7890a1b2c3d4e5"
"secret_key": "PqRsT67890uVwXyZ/A1fB2gC3hD4iE5j\abc"
"endpoint": "https://us-west4.storage.googleapis.com"
"google-s3-bucket-on-asia-east2":
"dataiq_volume": "/gcp_on_asia_east_2"
"access_key": "12345abcde7890a1b2c3d4e5"
Plugin installation 51
"secret_key": "PqRsT67890uVwXyZ/A1fB2gC3hD4iE5j\abc"
"endpoint": "https://asia-east2.storage.googleapis.com"
Example for Data Mover external worker nodes: (... other than the heading "value:", all other values should match the DataIQ host)
"S3 buckets:"
"amazon-s3-bucket":
"dataiq_volume": "/amazon_s3"
"access_key": "123234345456567678789890"
"secret_key": "ABCDE12345"
"endpoint": "s3.ap-northeast-1.amazonaws.com"
(...etc...)
See previews
When Previewer is installed, DataIQ can show thumbnail-sized preview images for graphics and videos.
About this task
Supported file extensions:
● Graphics: ".jpg", ".jpeg", ".tiff", ".tif", ".dpx", ".bmp", ".png", ".gif", ".tga", ".targa", ".exr", ".pcx", ".pict", ".ico"
● Video: ".mov", ".mp4", ".mpeg", ".mpg", ".ts", ".avi", ".mkv", ".wmf", ".wmv", ".mxf", ".ogv"
Steps
1. From DataIQ, select Data Management , and then use either the Search or Browse features to find a photo or video.
2. In the right pane, open the Preview drop-down frame.
Results
DataIQ shows a preview image and details about the file such as Codec information or image resolution or pixel size. For videos,
Previewer shows a watermarked thumbnail-size image from the video.
52 Plugin installation
Download a plugin from Dell Support
Several plugins are available from Dell Support. Some plugins are free, others are paid.
Steps
Get a copy of the plugin, and if available, its Proof of Authenticity.
The following plugins are available from Dell Support, at https://www.dell.com/support/home/product-support/product/dataiq for DataIQ 2.2:
Plugin name
Audited
Delete
Data
Mover
File name audited-delete-2.9installer.tar.gz
data-mover-2.0-87installer.tar.gz
Proof of authenticity file name audited-delete-2.9installer.tar.gz.sign
data-mover-2.0-87installer.tar.gz.sign
Data
Mover
External
Worker data_mover_workers-2.0-87.x
86_64.rpm
data_mover_workers-2.0
-87.x86_64.rpm.sign
Duplicate s duplicates-2.16-installer.tar.gz
duplicates-2.16installer.tar.gz.sign
Previewer previewer-2.17-installer.tar.gz
previewer-2.17installer.tar.gz.sign
Folder created by the extractor
/audited-delete-2.9
/data-mover-2.0-87
(no separate extractor)
/duplicates-2.16
/previewer-2.17
Command to start the plugin python audited_delete.py
start python data_mover.py
start rpm -Uvh data_mover_workers-2.0
-87.x86_64.rpm
python duplicates.py start python previewer.py start
Install and enable a plugin
These instructions describe how to install and enable a plugin.
Steps
1. Optional: If you have previously installed this plugin, check to see if this plugin still exists on the system.
cd /opt/dataiq/maunakea/data/plugins/<plugin>
If the plugin does still exist, either use the instructions to remove the plugin or the instructions to upgrade the plugin.
2. Transfer the plugin tar.gz file to a folder on the DataIQ host, for example, /root . Optional: Transfer in the proof of authenticity file, in the same folder location.
cp /mydrive/previewer-2.17-installer.tar.gz /root/ cp /mydrive/previewer-2.17-installer.tar.gz.sign /root/
3. Optional: Verify the authenticity of the tar.gz file:
./verify.sh previewer-2.17-installer.tar.gz
Verified OK
4. Extract the tar.gz file. Example: cd /root tar -zxvf previewer-2.17-installer.tar.gz
Plugin installation 53
5. Navigate to the folder that contains the Python script. Example: cd /root/previewer-2.17/
6. Install or upgrade the plugin. Examples: python previewer.py install
7. Optional: Check the status: kubectl get pods -ndataiq
The plugin should appear in the list as Running .
NOTE: If there is an issue with any of the pods, contact Dell Support.
8. Enable the plugin. From the DataIQ UI, select Settings > Data management configuration . In the Plugins section, select the ⋮ (vertical ellipses) icon next to the plugin and select Enable .
Install or upgrade Data Mover External Worker
To create more Data Mover Workers, repeat this process on each new server. Use the same installer package to install or upgrade Data Mover External Workers from a previous version of DataIQ.
Steps
1. On the DataIQ host, configure the firewall to allow the port:
● TCP 52100
2. Download the Data Mover External Worker for DataIQ 2.x
installer (example, data_mover_workers-2.0-88.x86_64.rpm
) from Dell Support, at https://www.dell.com/support/home/productsupport/product/data-iq .
3. Copy the installer file to a folder on the new server, for example, /root .
4. From the command-line on the new server, install or upgrade Data Mover External Worker: cd /root/ rpm -Uvh data_mover_workers-2.0-88.x86_64.rpm
After the installer completes, the Data Mover External Worker service starts.
Upgrade a plugin
Use the plugin installation package to update your plugins.
Steps
1. Transfer the file to a folder on the DataIQ host, for example, /root .
cp /mydrive/previewer-2.17-installer.tar.gz /root/
2. Extract the tar.gz file. Example: cd /root tar -zxvf previewer-2.17-installer.tar.gz
3. Navigate to the folder that contains the Python script. Example: cd /root/previewer-2.17/
54 Plugin installation
4. Upgrade the plugin using a Python script, for example: python previewer.py upgrade
This stops the plugin, removes the old Docker image, and installs and updates the new Docker image. Configuration history and outputs are preserved.
5. For Data Mover and custom plugins that use more than one Kubernetes pod: Stop and restart the plugin, for example: python datamover.py stop python datamover.py start
6. Optional: Check the status: kubectl get pods -ndataiq
The plugin should appear in the list as Running .
7. Optional: Review configuration differences: review the V1 file: /opt/dataiq/maunakea/data/plugins/<pluginname>/ca.control.old
, and compare it with the version in the DataIQ UI.
NOTE: Do not edit the active version of ca.control
file from the command-line interface directly, because DataIQ may not notice the write, and in some cases this may cause a crash. DataIQ plugins keep a history of configuration changes to allow a configuration rollback in the event of a crash.
8. Enable the plugin from the DataIQ UI: Select Settings > Data management configuration . In the Plugins section, select the ⋮ (vertical ellipses) icon next to each plugin, and select Enable .
Check plugin version
Check which version a plugin is installed in DataIQ from the command line.
About this task
Note: The Data Mover configuration file may show an incorrect version. Check from the command line instead.
Steps
From the DataIQ command line, view the ca.control file for the plugin, for example: grep -i version /opt/dataiq/maunakea/data/plugins/data-mover/ca.control
Alternate method:
● If you still have the installer files, check the version number in the directory name, for example, data-mover-2.0-84installer.tar.gz
.
Modify who can use a plugin feature by their user group
Some plugins allow admins to grant access to features based on DataIQ groups.
Prerequisites
Install the plugin. After the plugin is installed, you can configure or enable it.
About this task
● For many plugins, such as Data Mover, Audited Deletes, Previewer, Duplicate Finder, and plugins that have been converted from legacy ClarityNow! custom context menus, you can modify files through a configuration file.
Plugin installation 55
● For custom plugins , configurations may be managed through the UI for the plugin itself.
NOTE:
Use these instructions to modify configurations through the DataIQ UI. Do not modify the configuration file directly from the command line while DataIQ is running, because the changes may not be applied and could break the plugin functionality.
Steps
1. From the DataIQ browser: Select Settings > Data management configuration .
2. Open the plugin configuration file: In the Plugins section, select the ⋮ (vertical ellipses) icon next to the plugin, and select
Edit configuration .
3. Modify which groups have access to the actions within the plugin. Group names are case sensitive. Each line should start with a dash and space ( - ) aligned with the groups category. In this example, three groups are allowed access to the
Preview function in the "Previewer" app:
#
"Plugin Name": "Previewer"
"Version": "2.17"
"Actions":
"Preview":
endpoint: "/execute/"
filter:
groups:
- DataIQ Administrators
- Data Managers
- MyCompany IT Leads
Note, when troubleshooting whether individuals should have access to plugins, check the user profile settings, because users may belong to multiple groups.
Disable or enable a plugin
Disable a plugin to make it temporarily unavailable to users.
Steps
Disable the plugin:
Option Description
From the DataIQ UI: Select Settings > Data management configuration . In the Plugins section, select the ⋮ (vertical ellipses) icon next to the plugin you want to disable, and select Disable .
From the command line a. Disable the plugin: python duplicates.py stop b. Check the status: the plugin should either disappear from the list, or appear as Terminating : kubectl get pods -ndataiq
Uninstall a plugin
Disable the plugin first, then uninstall it.
Steps
1. From a command line, navigate to the folder where you installed the plugin. Example for the Duplicate Finder plugin: cd /root/duplicates-2.17.1/
56 Plugin installation
NOTE: If this directory is no longer there, you can download a new copy of the plugin, and extract it to a folder.
2. Delete the plugin using the Python installer script: python duplicates.py delete
If the plugin has a custom Docker image, it removes the image from the Docker repo.
Plugin installation 57
8
Secure Remote Services
DataIQ can communicate with Dell Support using Secure Remote Services (SRS). With SRS, DataIQ can send telemetry data, and can provide remote support.
The SRS application architecture is a secure, asynchronous messaging system designed to support the functions of secure encrypted file transfer, monitoring of device status, and remote execution of diagnostic activities. This distributed solution is designed to provide a scalable, fault-tolerant, and minimally intrusive extension to your system support environment.
When SRS is installed, security tools are always launched and running. These tools protect sensitive information, such as username, passwords, directory names, and so on.
DataIQ can send logs, configurations, and managed device status to Dell Support, either periodically for proactive support, or on-demand as needed.
When needed, a DataIQ administrator can provide Dell Support temporary access to securely tunnel into your system to investigate issues and support requests.
By default, DataIQ does not share information to Dell Technologies. Users can disable SRS at any time.
You can use the same SRS server to support DataIQ and your Dell EMC PowerScale and Isilon systems.
About this task (E-EULA)
Read the following E-EULA before proceeding with the changes.
TELEMETRY SOFTWARE NOTICE
If you are acting on behalf of a U.S. Federal Government agency or if Customer has an express written agreement in place stating that no remote support shall be performed for this machine, please stop attempting to enable the Software and contact your sales account representative. By continuing to install this Software, you acknowledge that you understand the information stated below and accept it. Privacy Dell, Inc and its group of companies may collect, use and share information, including limited personal information from our customers in connection with the deployment of this telemetry software ("Software"). We will collect limited personal data when you register the Software and provide us with your contact details such as name, contact details and the company you work for. For more information on how we use your personal information, including how to exercise your data subject rights, please refer to our Dell Privacy Statement which is available online at https://www.dell.com/learn/ policies-privacy-country-specific-privacy-policy .
Telemetry Software
This Software gathers system information related to this machine, such as diagnostics, configurations, usage characteristics, performance, and deployment location (collectively, "System Data"), and it manages the remote access and the exchange of the
System Data with Dell Inc. or its applicable subsidiaries (together, "Dell"). By using the Software, Customer consents to Dell's connection to and remote access of the machine and acknowledges that Dell will use the System Data transmitted to Dell via the Software as follows ("Permitted Purposes"):
● remotely access the machine and Software to install, maintain, monitor, remotely support, receive alerts and notifications from, and change certain internal system parameters of this machine and the Customer's environment, in fulfillment of applicable warranty and support obligations;
● provide Customer with visibility to its actual usage and consumption patterns of the machine;
● utilize the System Data in connection with predictive analytics and usage intelligence to consult with and assist Customer, directly or through a reseller, to optimize Customer's future planning activities and requirements; and
● "anonymize" (i.e., remove any reference to a specific Customer) and aggregate System Data with that from machines of other Customers and use such data to develop and improve products.
Customer may disable the Software at any time, in which case all the above activities will stop. Customer acknowledges that this will limit Dell's ability and obligations (if any) to support the machine.
58 Secure Remote Services
The Software does not enable Dell or their service personnel to access, view, process, copy, modify, or handle Customer's business data stored on or in this machine. System Data does not include personally identifiable data relating to any individuals.
Install SRS
Set up Secure Remote Services (SRS), and then configure it to work with DataIQ.
Installation
Secure Remote Services (SRS) and DataIQ must be installed on separate hosts or VMs.
DataIQ supports SRS Docker Edition (DE) or Virtual Edition (VE), version 3.40.00.14 and later. To enable remote support sessions requires SRS version 3.44.00.08 and later.
DataIQ requires the Docker-supported 64-bit Linux distribution and the Docker Engine (Docker runtime).
Note the SRS requirements, for example, SRS requires 64 GB of disk space.
During installation, SRS creates a separate account, which is not the root account of the host.
See https://www.dell.com/support/home/en-us/product-support/product/emc-secure-remote-services/docs and search for the latest instructions on downloading and installing SRS.
For assistance with setup and maintenance with SRS, contact your Dell EMC Global Services representatives.
Network requirements
For both the DataIQ and SRS hosts, the following ports must be enabled:
● Passive FTP 21 (Sends "Connect home" messages)
● SMTP 25 (Processes "Connect home" files, sends email notifications).
● HTTPS 443 (service notifications and initial setup)
● HTTP 443 and 8443 (remote support)
● HTTP 9443 (managed file transfer)
In addition:
● SRS Docker Edition (SRS DE) and SRS Embedded Virtual Edition (SRS eVE) require that the DataIQ host has a valid IPv4 address. For IPV6 environments, use SRS Virtual Edition (SRS VE).
● Configure your firewall rules to include cloud provider targets
● Check to ensure the default gateway is properly configured.
Create a remote service password for Dell Support
Create a password so that a Dell Support agent can connect remotely to DataIQ through Secure Remote Services (SRS).
About this task
When you are working with Dell Support to troubleshoot a support issue, the support agent may ask you to create an SRS password. This password allows the support agent to access the DataIQ server remotely through a secure tunnel.
Steps
1. Log into the DataIQ CLI using a tool such as PuTTY into the root account.
2. From the command line, create a temporary password or set a new customized password:
Option
Recommended: Create a temporary password.
Description
/opt/dataiq/bin/svc_dataiq_get_service_account_password.sh
This script changes the password to a random password that expires after 24 hours.
Secure Remote Services 59
Option
Alternate method:
Create your own password.
Description sudo passwd remoteservice
Type in the new password.
This password does not automatically expire. Keep the password safe using recommended security practices for your organization.
3. When requested, provide this password to a Dell Support agent.
60 Secure Remote Services
9
Logs and troubleshooting
This section discusses support resources and troubleshooting techniques.
Scan logs and error logs
The Logs pane lets you see scan logs from your managed volumes. To get to this page, from the DataIQ Data Management page, select Logs on the left side.
Scan logs
Select the Scan logs tab to get information from scans that have run on your volumes.
Default columns include Path , Status , Issuer , Started , Completed/Cancelled , Duration , Files , and Errors . Select the ⋮
(vertical ellipses) to choose other ways to sort, including % complete and Size . Sort columns by selecting the icon next to the column name.
Error logs
To see just the errors, select the Error logs tab to see rule violations, scan errors, and other general errors encountered on the volumes, including expiration errors, size limit errors, scan errors, tag limit errors, or connectivity errors.
You can sort by Timestamp , Severity , or Message .
Use the Search errors box to find anything in the log, including error messages or volume information.
Troubleshooting logs and diagnostic script
DataIQ includes logfiles and a diagnostic script for troubleshooting issues.
DataIQ log files and locations
Log file
/opt/dataiq/backup/timescale/log/*
/opt/dataiq/maunakea/claritynow/log/*
/opt/dataiq/maunakea/data/timescale/pg_log/
*.log
/opt/dataiq/imanager/srs/log/telemetry/
/opt/dataiq/install/playbook.log
/opt/dataiq/install/dataiq_install.log
/opt/dataiq/install/dataiq_upgrade.log
/var/log/dataiq/alertmanager.log/*
/var/log/containers/*
Description
Storage management data backup.
Server processes, including launching, running, and informational messages.
TimescaleDB error logs
SRS: Removal of personally-identification information (PII).
Install, upgrade, and uninstall.
Install.
Upgrade.
Alert manager.
Variety of issues, including authentication, Active Directory, and LDAP.
Logs and troubleshooting 61
Log file
/var/log/dataiq/clew.log/*
/var/log/dataiq/collector.log/*
/var/log/dataiq/data-mover.log/*
/var/log/dataiq/dataiq-plugin-<plugin name>.log/*
/var/log/dataiq/grafana.log/*
/var/log/dataiq/imanager.log/*
/var/log/dataiq/ixui.log/*
/var/log/dataiq/keycloak.log/*
/var/log/dataiq/keycloakdb.log/*
/var/log/dataiq/kube-state-metrics.log/*
/var/log/dataiq/metadata.log/*
/var/log/dataiq/node-exporter.log/*
/var/log/dataiq/timescale.log/*
/var/log/dataiq/tsdb-data-migration.log/*
/var/log/dataiq/tsdb-down-sampling.log/*
/var/log/dataiq/tsdb-schema-upgrade.log/*
DataIQ diagnostic script
1. On the DataIQ server, to detect potential issues in DataIQ:
/opt/dataiq/bin/svc_dataiq_diagnostic.sh
Description
Clew certification and API gateway.
Minotaur for storage monitoring.
Data Mover plugin.
DataIQ plugin (Plugins appear in Kubernetes with labels like dataiq-plugin-audited-delete .)
Grafana shows UI for storage management.
IManager, includes backup, collecting Kubernetes info.
Web-based UI.
Keycloak.
Keycloak database.
Kubernetes metrics.
Metadata, Grafana and protocol latency alerts for storage monitoring.
Node exporter.
TimescaleDB storage monitoring.
TimescaleDB data migration.
TimescaleDB down sampling.
TimescaleDB schema upgrade.
Add a new drive to a volume (LVM)
If a volume group created using a logical volume manager (LVM) is running out of space, you can expand it by adding a new drive to the volume group, then expanding the existing volume group partition onto the new drive.
Prerequisites
This procedure requires that the volume was originally created as an LVM volume group. The application volume on a DataIQ virtual machine created from a DataIQ OVA is preconfigured as an LVM volume group. Standard Linux volumes can also be expanded, however, the process is more complex and is outside of the scope of this document.
About this task
NOTE: A common cause of the DataIQ application volume running out of space is that DataIQ is configured to store a large amount of historical storage monitoring data. As an alternative to expanding the drive, you can instead change the storage monitoring data retention policy.
Steps
1. Log into the DataIQ server and open a command prompt.
62 Logs and troubleshooting
2. Optional: Confirm that the size of the logical volume and note how full it is: df -h /
Sample output:
[root@centos76 ~]# df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos-root 26G 22G 4G 85% /
3. Add a physical drive to the server.
Linux adds the drive using a disk name, for example, /dev/sdb.
4. Identify the disk name for the drive you just added.
fdisk -l
In this sample output, the disk name is /dev/sdb :
# fdisk -l
Disk /dev/sda: 32.2 GB, 32212254720 bytes, 62914560 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000bde50
Device Boot Start End Blocks Id System
/dev/sdal * 2048 2099199 1048576 83 Linux
/dev/sda2 2099200 62914559 30407680 8e Linux LVM
Disk /dev/mapper/centos-root : 27.9 GB, 27913093120 bytes, 54517760 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/centos-swap: 3221 MB, 3221225472 bytes, 6291456 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/sdb: 53.7 GB, 53687091200 bytes, 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
(Also note in this example, /dev/sda2 is the primary Linux LVM physical volume in the volume group.)
5. Create a new partition on the new drive and configure it: a. Open the fdisk console, targeting the new drive: fdisk /dev/sdb b. Create a new primary partition, using the default partition values: n > p > press Enter 3 times to accept the defaults
(partition = 1, use all available space on the drive).
c. Change the partition type to Linux LVM: type t > 8e .
d. Review the partition info that will be written: type p .
Sample output:
Disk /dev/sdb: 53.7 GB, 53687091200 bytes, 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0xa883580d
Logs and troubleshooting 63
Device Boot Start End Blocks Id System
/dev/sdbl 2048 104857599 52427776 8e Linux LVM e. Write the partition table and exit: type w .
Sample output:
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
6. Create a new physical volume on the new partition, for example, /dev/sdb1 .
pvcreate /dev/sdb1
7. Confirm that the physical volume has been created: pvdisplay
In the sample output, the original physical volume, /dev/sda2, and the new physical volume, /dev/sdb1, are shown:
[root@centos ~]# pvdisplay
--- Physical volume ---
PV Name /dev/sda2
VG Name centos
PV Size <23.80 GiB / not usable 3.00 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 7423
Free PE 0
Allocated PE 7423
PV UUID 4g3yVT-3i7e-s xgH-EOGx-aykB-83xs-JxLCj6
"/dev/sdbl" is a new physical volume of "<50.00 GiB"
--- NEW Physical volume ---
PV Name /dev/sdbl
VG Name <50.00 GiB
PV Size NO
Allocatable 0
PE Size 0
Total PE 0
Free PE 0
Allocated PE 0
PV UUID qa7tv0-ryIa-kTec-E81o-RSxa-HpLz-wlPHOk
8. Identify the volume group name (VG Name) of the volume group that you want to expand: vgdisplay
For example, in this output, the only volume group is centos :
[ root@centos76 -]# vgdisplay
- Volume group —
VG Name centos
System ID
Format Lvmz
Metadata Areas 1
Metadata Sequence No 3
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 2
Open LV 2
Max PV 0
Cur PV 1
Act PV 1
VG Size <29.00 GiB
64 Logs and troubleshooting
PE Size 4.00 MiB
Total PE 7423
Alloc PE / Size 7423 / <29.00 GiB
Free PE / Size 0/0
VG UUID omT41C-L5Oq-393P-LxHN-XO6q-0oi8-2blglP
9. Add the physical volume which we created earlier to the volume group: vgextend centos /dev/sdb1
10. Confirm the logical volume path (LV Path) lvdisplay
In this sample output, the logical volume path is /dev/centos/root .
LV Path /dev/centos/swap
LV Name swap
VG Name centos
LV UUID wC9BlK-qeG5-tj2f-oQtN-AKTP-xfGA-fxk4JT
LV Write Access read/write
LV Creation host, time sarje-dpcwin2kl2r2.dpc.sso, 2019-07-02 05:43:34 -0400
LV Status available
# open 2
LV Size 3.00 GiB
Current LE 768
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 8192
Block device 253:1
--- Logical volume ---
LV Path /dev/centos/root
LV Name root
VG Name centos
LV UUID XKVxr8-GFfq-XoKa-ytkb-Mloe-nVig-0j01mf
LV Write Access read/write
LV Creation host, time sarje-dpcwin2kl2r2.dpc.sso, 2019-07-02 05:43:35 -0400
LV Status available
# open 1
LV Size <26.00 GiB
Current LE 6655
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 8192
Block device 253:0
11. Extend the logical volume using the space in the newly-created physical volume: lvextend /dev/centos/root /dev/sdb1
12. Resize the file system to make use of the space: xfs_growfs /dev/centos/root
13. Confirm that the size of the logical volume has changed: df -h /
Sample output:
[root@centos76 ~]# df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos-root 76G 22G 54G 29% /
Logs and troubleshooting 65
Data management server status
Check the status of memory used, file types, files, and folders.
Steps
In the DataIQ UI, in the Settings window, select Data management configuration > View data management server status .
Check datastore disk usage
Check how full the storage monitoring database (TimescaleDB) is.
Steps
1. In the DataIQ UI, in the Settings window, select Storage monitoring datastore management .
2. Review the disk usage.
Reduce the amount of OneFS data collected in DataIQ
To reduce network traffic between DataIQ and OneFS, or to better see specific OneFS statistics and monitoring details, you can enable and disable the collection of specific sets of details.
About this task
You can enable or disable collection across all clusters for a specific set of data.
Steps
1. Create a copy of the deployed collector-settings configuration file: cp /opt/dataiq/k8s/collector-config.yaml /opt/dataiq/k8s/modified-collectorconfig.yaml
2. Edit the modified-collector-config.yaml file to enable/disable the desired set of data: vi /opt/dataiq/k8s/modified-collector-config.yaml
Example: apiVersion: v1 data:
collector_settings.json: |
{
"capacity": {"enabled": true},
"clients": {"enabled": true},
"cloud_pools": {"enabled": true},
"dedupe": {"enabled": true},
"disk": {"enabled": true},
"events": {"enabled": false},
"events_specific": {"enabled": true},
"filesystem": {"enabled": true},
"filesystem_policy": {"enabled": true},
"health": {"enabled": true},
"jobs": {"enabled": true},
"ndmp": {"enabled": true},
"network": {"enabled": true},
"network_interface": {"enabled": true},
"node_detail": {"enabled": true},
"protocol": {"enabled": true},
"synciq": {"enabled": true},
"quotas": {"enabled": true},
"snapshots": {"enabled": true},
66 Logs and troubleshooting
"storage_pools": {"enabled": true},
"srs": {"enabled": true},
"users": {"enabled": true}
} kind: ConfigMap metadata:
name: collector-settings
namespace: dataiq
labels:
app: collector
app.kubernetes.io/name: collector-settings
app.kubernetes.io/component: collector-config
app.kubernetes.io/part-of: collector
NOTE: Using VIM: Press i to enter insert mode, press Esc to escape. Press :wq to write changes and quit. To learn more, see A Great VIM Cheat Sheet .
3. Apply the changes: sudo kubectl apply -f /opt/dataiq/k8s/modified-collector-config.yaml
4. Wait 2-3 minutes for the changes to be realized during collection.
5. Confirm that unwanted details are disabled in the collector logs: sudo kubectl logs -n dataiq collector-<unique-collector-id>
Example
2020-05-28 00:08:03,114 [1] DEBUG minotaur: Disabled task EventsTask
Logs and troubleshooting 67
تنزيل
تم تحديث الرابط العام
رابط عام للمستند
إعلان