IBM Storage News - Neues aus der Storwize Familie

IBM Storage News - Neues aus der Storwize Familie
IBM Storage News
Neues aus der Storwize Familie
Manuel Schweiger
Master Certified IT Specialist
© 2014 IBM Corporation
Meet The Storwize Family
A comprehensive range of fully virtualized
storage systems!
! 
! 
! 
! 
One code base on all platforms!
One set of functions (selectively licensed)!
One user interface!
One CLI!
SAN Volume
Controller
Flex System
V7000
Storwize V7000
Unified
Storwize V7000 GEN2
Storwize V5000
Storwize V3700
© 2014 IBM Corporation
SVC Supported Environments (sample view*) – applies to Storwize
IBM
z/VSE
z/VM
z/Linux
Novell
NetWare VMware
vSphere
OES2
Microsoft
Windows
Hyper-V
HP-UX 11i
IBM Power
Tru64
IBM AIX
Solaris OpenVMS
IBM I
iSCSI or FCOE
Point-in-time Copy
1Gb or 10Gb
Full volume, Copy on write
256 targets,
Incremental, Cascaded, Reverse,
Snapshots, FlashCopy Mgr
Thin Provisioning
Compellent
Fluid Data
IBM
DS
Easy Tier
IBM
XIV
DS3000, DS4000
DS5000, DS6000
DS8000
Violin
Hitachi
N series
Lightning
Thunder
FlashSystem
Storwize Family TagmaStore
AMS, WMS,
USP, USP-V
VSP
MA, EMA
MSA, EVA,
XP, P9500,
3Par
EMC
IBM TS7650G IBM
BladeCenter
Flex System
PureFlex
Continuous Copy
Multiple Cluster Mirror
Metro/Global Mirror
Stretched Cluster
Real-time Compression
HP
Citrix Xen
Server
Apple ProtecTIER
Mac OSGateway
SGI IRIX
8Gbps SAN fabric
Virtual Disk Mirroring
IBM
Linux
(Intel/Power/z)
RHEL 4/5/6
SUSE 9/10/11
Sun
CLARiiON
StorageTek
CX4,
StorEdge
Symmetrix,
DMX, VMAX,
VNX, VNXe
1024
Hosts
SAN
Internal SSD
NetApp NEC
Fujitsu
TMS
Eternus
RamSAN
XioTech Bull
NexSAN
Pillar
FAS
Emprise
iStorage
Storeway
SATABeast
Axiom
* Confirm all supported configurations at ibm.com/storage/support/2145 and click on Support
3
© 2014 IBM Corporation
What have we announced?
"  New SVC engine
"  New Storwize V7000
– Including new 600GB 15K RPM HDD
"  Statements of Direction
"  Storwize Family Software V7.3
– Easy Tier 3
– Storage Pool Balancing
– Simplified licensing
"  Storwize V7000 Unified File Module Software V1.5
– NFS v4, SMB2.1
– Multi-tenancy
– ACE multi-writer
– Automated ID mapping
"  Increased expansion for Storwize V3700 and Storwize V5000
4
© 2014 IBM Corporation
SVC DH8 – Front View
Boot drives
2 – 300GB 10K SAS
5
System indicators
Battery 1
Battery 2
© 2014 IBM Corporation
SVC DH8 – Internal View
PCIe Riser cards
PSUs
DIMMs
CPU
Fans
Boot drives
Batteries
6
© 2014 IBM Corporation
SVC DH8 – Hardware Overview
" 8-core CPU with 32GB memory for SVC
"  Minimum 1 HIC for I/O
"  Can add a 2nd I/O HIC, and SAS HIC on this
CPU
"  2nd CPU is optional
" 
" 
" 
" 
7
Comes with extra 32GB memory
Required to access 3rd I/O HIC
Required for RTC
At least 1 Compression Accelerator card
required for RTC
system battery
CPU1
Boot / dump drives
QPI
CPU2
SVC
RTC
PCIe Gen3
PCIe Gen3
© 2014 IBM Corporation
SVC DH8 – Rear View
PCIe expansion slots
Mgmt ports
Slot 1
Slot 2
Slot 3
1 Gb iSCSI ports
8
750W PSUs
Slot 4
Slot 5
Slot 6
Technician Port
© 2014 IBM Corporation
More I/O ports
"  Prior to v7.2, we only supported 6 FC ports for backend storage virtualisation
"  In 7.2.0, we extended the max local port count for backend storage virtualisation to 8
– 2nd 4 port HBA or FCoE (2 port 10Gb card)
"  Because we support up to 3 I/O cards in the SVC DH8, the maximum local I/O port
count for backend storage virtualisation is extended to 12
– maximum local I/O ports is also 12 on SVC DH8
"  Check backend controller spec for how many initiator ports from one host a storage
controller can handle
– Some of the older controller may have a limit for this.
•  Use zoning to limit the initiator ports according to the limit of the backend
controller
– Should be able to set up a host group or multiple hosts in the backend controller
to work around this
9
© 2014 IBM Corporation
10Gb Ethernet Card (FCoE and iSCSI)
" The new 4x port 10GbE card will only be supported in the new SVC DH8
and in the Storwize V7000 2076 -524
" The card is delivered with the SFPs fitted, unless it is a FRU.
" In SVC 7.3.0 we will only support 1 x 10GbE adapter
in each of the platforms (see above)
" Only IBM supported 10Gb SFPs
should be used
" Each adapter port has amber and green
coloured LED to indicate port status
– fault LED is not used in 7.3.0.
10
Green LED
Meaning
On
Link established
Off
No link
© 2014 IBM Corporation
Compression accelerator adapter
"  Up to a total of 2 compression accelerator adapters can be installed, each
additional adapter installed will improve I/O performance when using compressed
volumes
–  Intel QuickAssist technology is used. IBM is the first in the industry to integrate this
technology into our products
•  second generation Intel Communications Chipset 89xx Series (Coleto Creek)
nd
–  2 CPU and extra 32 GB of memory are mandatory with the compression accelerator
adapter
"  The use of compression accelerator adapters is mandatory (at least one) if
users wish to use compression on SVC DH8.
–  For an I/O group containing a SVC DH8 with
no compression accelerator, an attempt to create
first compressed volume will fail.
–  The addnode command will also fail, if trying
to add a SVC DH8 without a compression
accelerator, to an I/O group which has
compressed volumes
11
© 2014 IBM Corporation
SVC DH8 – Expansion Enclosure 2145 24F
"  With Release 1 and 7.30 Code – support 2 expansion enclosures per I/O Group
"  Ports 1 and 3 of the 12 Gb SAS card can be used to attach 2U24 Expansion
enclosures of flash drives
"  Expansion enclosures are physically identical to the V7000 Gen2 expansion
enclosures, but will have a different product ID
– SVC DH8 cannot use the V7000 Gen2 expansion enclosures, V7000 Gen2
cannot use the SVC DH8 expansion enclosure
12
© 2014 IBM Corporation
SVC CG8 vs DH8
13
Attribute
(per node)
SVC CG8
SVC DH8 R1
CPU
2X 6 cores Westmere
2x 8- cores Ivy Bridge
Controller memory
24GB to 48GB
32GB to 64GB
Host I/O
2x 1GbE
4x to 8x - 8Gb FC
2x 10GbE
(2 card max)
3x 1GbE
0 to 12x - 8Gb FC
0 to 4x – 10GbE
(3 I/O card max)
Compression
resources
8 cores (with 2nd CPU fitted) 8 cores (with 2nd CPU fitted)
1 or 2 Compression Accelerator
Card
Drive expansion
4 flash drive local to node
(RAID 0,1,10 only)
48 flash drives shared by 2 nodes
(RAID 0,1,5,6,10)
SAS fabric
6Gb SAS
12Gb SAS
© 2014 IBM Corporation
SVC Investment Protection
" New SVC engines may be added to existing SVC systems
without disruption
" New SVC engines may replace existing engines in SVC systems
without disruption
" All engines in an SVC system may access flash expansion
capacity
" SVC system must be running Version 7.3 software
14
© 2014 IBM Corporation
Storwize V7000 Hardware Refresh: 2076-524 Control Enclosure
"  Control enclosure is 2U, same physical size as previous model
"  Front view looks identical to V5000 and only comes in 24 drive SFF configuration for the
control enclosure and both SFF and LFF configurations for expansion enclosures
"  Back layout is very different to make room for the more powerful canisters
PSU 1
15
Canister 1
Canister 2
PSU 2
© 2014 IBM Corporation
New Storwize V7000 Control Enclosure
"  Available only in 2.5-inch drive model
– 2076-524
Slot Supported cards
1
"  Three expansion slots numbered 1-3
left to right when viewed from the rear
Compression pass-through
Compression Acceleration card
2
None, 8Gb FC*, 10GbE**
3
None, 8Gb FC*, 10GbE**
"  Ports on a card are numbered top to
bottom starting with 1
"  Supported expansion cards
– 4x 8Gb FC
– 4x 10GbE iSCSI / FCoE
– Compression pass-through comes
standard with system to enable
on-board compression engine
– Compression Acceleration card
"  Technician IP port replaces USB key
configuration
"  Same funcionality as on SVC DH8
* Statement of Direction for 16Gb FC announced
** Only one 10GbE card supported per node canister
4x 1GbE ports
SAS expansion ports
Compression
Host interface slots
accelerator slot
PSU
Technician
IP port
PSU
Dual controller canisters
16
© 2014 IBM Corporation
Storwize V7000 Hardware Refresh: 2076-24/12F Expansion
Enclosure
"  Available in 2.5- and 3.5-inch drive models
–  2076 Models 24F and 12F respectively
"  Attach to new control enclosure using 12Gbps SAS
"  Mix drive classes within enclosure including different drive SAS interface speeds
"  Mix new enclosure models in a system even on same SAS chain
"  All drives dual ported and hot swappable
2.5-inch Flash Drives
200GB, 400GB, 800GB
Flash
12Gbps
2.5-inch High Performance Drives
300GB, 600GB
SAS, 15K RPM
12Gbps
600GB, 900GB, 1.2TB
SAS, 10K RPM
6Gbps
1TB
NL-SAS, 7.2K RPM
6Gbps
3.5-inch High Capacity Drives
2TB, 3TB, 4TB
17
NL-SAS, 7.2K RPM
6Gbps
© 2014 IBM Corporation
Storwize V7000 Hardware Refresh: Expansion Enclosure Cabling
18
© 2014 IBM Corporation
Storwize V7000 Hardware Refresh: SAS Chain Layout
"  Each control enclosure supports two
expansion chains and each can
connect up to 10 enclosures
– Max. 1056 drives allowed
‘SAS port 0’
Chain 0
"  SSDs can also go in other enclosures if
more then 24 required for capacity
reasons
"  HDDs can go in control enclosure if
desired
– Mix of SSDs and HDDs is fine too
19
Internal
SAS links
SAS Adapter
Control
"  Internal double-width high-speed link to
the control enclosure
–  SSDs should be installed in control
enclosure
–  Same SAS bandwidth in control
enclosure dedicated to these 24 slots as
there is to other two chains combined
Node Canister
SAS port 1
Chain 1
SAS port 2
Chain 2
Expansion
Expansion
Expansion
Expansion
8 more
8 more
© 2014 IBM Corporation
V7000 Gen 1 vs. Gen 2
Storwize
V7000 GEN1
V7000 GEN2
CPU
Intel Jasper Forest 4-core
Intel Ivy Bridge 8-core
Controller memory
(per node)
8GB
16GB-64GB configurable
Host I/O (per node)
2x 1GbE
4x 8Gb FC
2x 10GbE (some models)
3x 1GbE
Mix of: 4 or 8 ports 8GGb FC
4 or 8 ports 10GbE
Compression
resources (per node)
3 CPU cores
On-board Compression Assist
Engine
Optional second Compression
Assist Engine
Drive expansion
Up to 9 expansions (240
drives per controller)
Up to 20 expansions (504 drives
per controller)
Drive fabric
6Gb SAS
12Gb SAS
20
© 2014 IBM Corporation
New Storwize V7000
Migration and Investment Protection
"  Build Storwize V7000 Unified systems using new Storwize V7000 and Storwize V7000
File Modules (2073-720)
– Requires File Module software V1.5
"  Can mix new and existing Storwize V7000 systems in a cluster
"  Provides complete protection for existing Storwize V7000 investments
– All existing and new storage can be accessed by all systems
"  Migration from existing system with no downtime at all
– For systems that support nondisruptive volume move
– No competitive system can make similar claim
"  SVC and new Storwize V7000 can virtualize existing Storwize V7000 or any other
installed disk system
– Provides conventional Storwize family migration using virtualization
Cluster
Existing V7000
21
New V7000 System
Existing
Virtualize
Existing
Replicate
V7000
V7000
© 2014 IBM Corporation
Easy Tier Evolution
"  Easy Tier 1 (DS8700 R5.1 and Storwize Family)
–  Automated cross-tier performance management for SSD/HDD hybrid pools
–  Manual mode management support for dynamic extent pool merge and dynamic volume relocation
"  Easy Tier 2 (DS8700/DS8800 R6.1)
–  Automated cross-tier performance or storage economics management for hybrid pools with any 2 tiers
(SSD/ENT, SSD/NL or ENT/NL)
–  Automated intra-tier performance management (auto-rebalance) in hybrid pools
–  Manual mode management support for rank depopulation and optimized volume restriping within nonmanaged pools (manual volume rebalance)
"  Easy Tier 3 (DS8700/DS8800 R6.2 and Storwize Family 7.3)
–  Automated cross-tier performance and storage economics management for hybrid pools with 3
tiers (SSD/ENT/NL)
–  Automated intra-tier performance management in both hybrid (multi-tier) as well as homogenous
(single tier) pools (auto-rebalance)
–  Thin Provisioning support for Extent Space Efficient (ESE) Volumes
"  Easy Tier 4 (DS8800 R6.3 / DS8870 R7.0)
–  Support for encryption capable environments
•  DS8800 and DS8870 have FDE capable SSD and nearline drives
"  Easy Tier 5 (DS8870 R7.1)
22
–  Server
–  Application
–  Heat map transfer
© 2014 IBM Corporation
Easy Tier 3
" Now supports two or three tiers
– Flash + 2 disk tiers
– Flash + 1 disk tier
– 2 disk tiers
" Deploy flash and enterprise disk for
performance
Flash Arrays
Less Active Data
Migrates Down
Active Data
Migrates Up
" Grow capacity with low cost disk
" Storwize V7000 moves data
automatically between tiers
HDD Arrays
" Available on all systems licensed for
Easy Tier
– No additional charge
23
© 2014 IBM Corporation
Possible Hotspot creation with Software Release 7.2 and older
Workload
"  Extents are distributed
based on free capacity on
mdisks.
vdisk
Extent distribution
SVC
mdiskgrp
mdisk0
mdisk1
mdisk2
mdisk3
LUN0
LUN1
LUN2
LUN3
Storage
Subsystem
300GB 7+P 300GB 7+P
24
600GB 7+P 600GB 7+P
200 IOPS / Extent
100 IOPS / Extent
© 2014 IBM Corporation
EasyTier 3 - Storage Pool Balancing (new feature in 7.3)
Workload
"  Extents are distributed
based on workload /
performance
vdisk
Extent distribution
SVC
mdiskgrp
mdisk0
mdisk1
mdisk2
mdisk3
LUN0
LUN1
LUN2
LUN3
Storage
Subsystem
300GB 7+P 300GB 7+P
25
600GB 7+P 600GB 7+P
200 IOPS / Extent
100 IOPS / Extent
© 2014 IBM Corporation
Automated Storage Pool Balancing in Action
"  Configuration
Drive
MDisk
Volume
Comments
24 - 300GB 15K RPM Drives
3 - RAID-5 arrays
Vol_0, Vol_1, Vol_2, Vol_3 each
32GB capacity
Total MDisk size 5.44TB
Total Volume size 128GB
All Volumes are created on MDisk0 initially
"  Performance improved by balancing workload across all 3 MDisks:
Improvement
Throughput
6000 vs 12.000 IOps
Latency:
45 vs 20 msec
"  Automated Storage Pool Balancing is included in the Base Licensing. No
additional EasyTier license required to use it.
"  Enabled by default
26
© 2014 IBM Corporation
Using the STAT Tool
" Download the heatfile
" Run the STAT Tool
" Click on index.html
28
© 2014 IBM Corporation
Workload skew drives Easy Tier benefits
100
90
50% of the extents do
10% of the MB and
virtually no random IOPS!
80
Percent of workload
70
60
50
40
58% of the random IOPS and
33% of the MB from about
5% of the extents!
30
20
10
0
0
5
10
15
20
25
30
35
40
45
50
55
60
65
70
75
80
85
90
95
100
Percent of extents
Percent of small Ios
Percent of MB
© 2014 IBM Corporation
Why re-architect Cache?
"  More scalable for the future
–  Required for supporting beyond 8K volumes
–  Required for support beyond 8 node clusters
–  Required for 64 bit user addressing beyond 28 GB
•  SVC code only uses 28 GB max today
"  Required for larger memory sizes in nodes/canisters
"  Required for more CPU cores
"  Reduces # of IOPs copy services do directly to the back end storage
"  Required for flush-less FlashCopy prepare
–  Allows near CDP like capability
"  RtC benefits from the cache underneath
"  Algorithmic independence
–  Allows changes to pre-fetch and destage algorithms without touching the rest of the
cache
"  Improved debugging capability
34
© 2014 IBM Corporation
Cache Architecture pre-V7.3.x
FWL = Forwarding Layer
Host I/O
FWL
Front End
TP/RtC
FWL
Remote Copy
Cache
Volume Mirror
FWL
Virtualization
FWL RAID 1/5/6/10
Backend
TP/RtC
Virtualization
RAID 1/5/6/10
Backend
FlashCopy
35
© 2014 IBM Corporation
Cache Architecture V7.3.x
FWL = Forwarding Layer
Host I/O
FWL
Front End
TP/RtC
FWL
Remote Copy
Upper Cache
Volume Mirror
FWL
Lower Cache
Lower Cache
Virtualization
Virtualization
FWL RAID 1/5/6/10
FlashCopy
36
TP/RtC
Backend
RAID 1/5/6/10
Backend
© 2014 IBM Corporation
New SVC Data Engine
Power to Virtualize
" Virtualized server environments demand virtualized storage
" 2x performance: up to 128 cores
" 50% more connectivity: up to 96 FC ports
" 2.5x cache: up to 512GB
" 6x integrated flash: up to 192 drives
39
© 2014 IBM Corporation
SVC Maximum Performance (One I/O Group)
Uncompressed
SVC CG8
New SVC DH8
Read Hit IOPS
1,000,000
1,500,000
Read Miss IOPS
360,000
630,000
Write Miss IOPS
125,000
200,000
“DB-like”
200,000
385,000
SVC CG8
New SVC DH8
Read Miss IOPS
2,600-50,000
71,000-175,000
Write Miss IOPS
1,200-16,000
28,000-115,000
“DB-like”
2,200-40,000
59,000-149,000
Compressed
" Compressed performance shows a range depending on I/O distribution
" Compressed performance is better than uncompressed in some cases
because of fewer I/Os to drives and additional cache benefits
Preliminary data: Subject to change before GA
© 2014 IBM Corporation
New Storwize V7000
Power and Scalability for All Your Data
" 2x scalability: up to 4PB
" 2x performance: up to 64 cores
" 2x infrastructure speed: 12Gbps SAS
" 2x connectivity: up to 64 FC ports
" 8x cache: up to 512GB
41
© 2014 IBM Corporation
Storwize V7000 Maximum Performance (One Control Enclosure)
Uncompressed
Previous Storwize V7000
New Storwize V7000
Read Hit IOPS
850,000
1,300,000
Read Miss IOPS
125,000
238,000
Write Miss IOPS
25,000
50,000
“DB-like”
52,000
100,000
Previous Storwize V7000
New Storwize V7000
Read Miss IOPS
2,000-44,000
39,000-149,000
Write Miss IOPS
1,100-17,000
22,500-78,000
“DB-like”
1,500-32,700
41,000-115,000
Compressed
" Compressed performance shows a range depending on I/O distribution
" Compressed performance is better than uncompressed in some cases
because of fewer I/Os to drives and additional cache benefits
Preliminary data: Subject to change before GA
© 2014 IBM Corporation
New Simplified Storwize Licensing Structure for V5000 & V7000
License Per Enclosure
Option 1
FLEXIBLE OPTIONS
Controller
Expansion
External
$  Base
Advanced Functions
#  Easy Tier
#  Flashcopy
#  Remote Mirror
#  Compression*
Option 2
FULL BUNDLE
Controller
Expansion
External
$  Base
$  Full Bundle
$  Easy Tier
$  Flashcopy
$  Remote Mirror
$  Compression*
Summary
44
•  Licenses: Controller, Expansion, External Data Virtualization
•  Advanced Functions are identically priced for Controller, Expansion & External
•  Advanced function licenses, when selected, are required for all enclosures
•  No capping
•  No complexity
* Storwize V7000 Only
© 2014 IBM Corporation
Statements of Direction
"  IBM intends to enhance the new SVC engine and new Storwize V7000 to support 16 Gb
Fibre Channel connectivity
"  The cache upgrade feature on the new Storwize V7000 provides performance benefit only
when Real-time Compression is used. IBM intends to enhance IBM Storwize Family
Software for Storwize V7000 to extend support of this feature to also benefit uncompressed
workloads.
"  The second CPU with 32 GB memory feature on SVC Storage Engine Model DH8 provides
performance benefit only when Real-time Compression is used. IBM intends to enhance IBM
Storwize Family Software for SVC to extend support of this feature to also benefit
uncompressed workloads.
45
© 2014 IBM Corporation
Summary - Storwize Family Software Version 7.3
" Supports new Storwize V7000 and new SVC engine
" Supports additional expansion for Storwize V3700 and Storwize V5000
– Both systems now support up to 9 expansion enclosures
" Easy Tier 3
" Storage Pool Balancing
" Improved licensing model for Storwize V7000 and Storwize V5000
46
© 2014 IBM Corporation
IBM Storage News
IBM FlashSystem 840
© 2014 IBM Corporation
IBM FlashSystem 840 Components
Flash Modules (12)
Batteries (2)
RAID Controllers (2)
Interface Modules (4)
Canisters (2)
Fans (4)
Management Modules (2)
Power Supplies (2)
© 2014 IBM Corporation
IBM FlashSystem 840 Physical Overview
IBM FlashSystem 840 Front View
Battery Modules
Flash Modules
"  Small 2U form factor can support up to twelve flash modules. 2 TB and 4 TB
flash modules available.
"  Hot swappable hardware components easily accessible from the front or back
of unit.
"  All hot swappable FRU and CRU can be replaced without removing unit from
rack. Stacked units do not have to be removed for FRU CRU replacement.
"  Capacity from 4TB up to 40TB
© 2014 IBM Corporation
IBM FlashSystem 840 Performance
Performance
Criteria1
Max Capacity
(12 modules)
Half Capacity
(8 modules)
Min Capacity
(4 modules)
Recommended
100% Read IOPS
1.1 M IOPS
1.1 M IOPS
1.0 M IOPS
10/15/13 Final
100% Write IOPS
(burst)
600.0 k IOPS
400.0 kI OPS
225.0 k IOPS
10/15/13 Final
70% / 30% Mix
775.0 k IOPS
500.0 k IOPS
225.0 k IOPS
10/15/13 Final
100% Large Block
Sequential Read
8.00 GB/s
8.00 GB/s
4.00 GB/s
10/15/13 Final
100% Large Block
Sequential Write
4.00 GB/s
2.50 GB/s
1.00 GB/s
10/15/13 Final
Min. Read Latency
135 us
135 us
135 us
10/15/13 Final
Min. Write Latency
90 us
90 us
90 us
10/15/13 Final
1Data gathered using an Oakgate
2All measurements are made in a
storage test appliance and a fiber channel protocol analyzer
RAID 5 configuration, 4 TB cards, 90% of usable capacity
© 2014 IBM Corporation
IBM FlashSystem 840 Maximum Interface Port Counts
" Fibre Channel 16Gb/s = maximum 8 ports.
" Fibre Channel 8Gb/s = maximum 16 ports
" Fibre Channel over Ethernet 10Gb/s = maximum 16 ports
" Infiniband 40Gb/s = maximum 8 ports.
" All hosts must have at least one connection to each RAID
module to eliminate an outage during concurrent code load.
" Note – 16Gb/s ports can also operate at 8Gb/s or 4Gb/s
" Note – 8Gb/s ports can also operate at 4Gb/s
© 2014 IBM Corporation
IBM FlashSystem 840 GUI Preview - Monitoring
© 2014 IBM Corporation
Performance Scenario: Oracle RAC, 4 Nodes
© 2014 IBM Corporation
SVC VDISK Mirroring …across locations
© 2014 IBM Corporation
Question & Answer Time
innovation that matters
55
© 2014 IBM Corporation
Was this manual useful for you? yes no
Thank you for your participation!

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Download PDF

advertising