Tips for Successful Oracle Exadata Management with Oracle Enterprise Manager 12 c

Tips for Successful Oracle Exadata Management with Oracle Enterprise Manager 12 c
CON8245
Tips for Successful Oracle Exadata
Management with Oracle Enterprise
Manager 12c
Ashish Agrawal
Consulting Product Manager
Oracle Corporation, Product Management
Roland Schiller
Expert System Engineer,
Swiss Reinsurance Company Ltd
Akash Guha
Database Administration Specialist, eBay Inc
September, 2014
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features or
functionality described for Oracle’s products remains at the sole discretion of Oracle.
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Program Agenda
1
What & How is Exadata Monitored?
2
What's New in Exadata Management?
3
Real World Experience
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Program Agenda
1
What & How is Exadata Monitored?
2
What's New in Exadata Management?
3
Real World Experience
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
What is monitored?
Database
Storage Server
Infiniband Network
KVM, PDU, ILOM, CISCO SWITCH
Copyright © 2012, Oracle
and/or
its affiliates.
All rights
reserved..
Copyright
© 2014,
Oracle and/or
its affiliates.
All rights reserved. |
Oracle Confidential – Internal Only
6
How is it monitored?
ORACLE DATABASE MACHINE
OEM Agent deployed on the Compute Node
COMPUTE NODE #1
DATABASE SERVER
Exadata Plug-in deployed with the Agent
1
5
Oracle Enterprise Manager 12c Agent
OMS
2
Exadata Plug-in
Agent communicates with Storage Server and
Infiniband Switch targets using ssh
4
3
Exadata Plug-in monitors the other DBM
components
OEM Agent collects data and uploads it to the
Oracle Management Server (OMS)
Exadata Storage
Server
Exadata Infiniband
Switches and Network
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Other DBM Devices
PDU
KVM
ILOM
CISCO S/W
Exadata Machine Schematic
Simplifies monitoring
View all critical issues in the system,
monitor performance
Drill down to individual targets
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Exadata Management
Integrated View of Hardware and Software
• Hardware view
• Schematic of hardware components:
Storage Servers, Compute Nodes, IB
Switches etc
• Hardware components status & alerts
• Software/system view
• Performance, availability, usage by
databases, services, clusters
• Software Alerts for db, Cluster, ASM
• Topology view of DB systems/clusters
• Configuration view
• Version summary of all components
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Storage Cell Management
Storage Cell monitoring and administration
support
• Cell Home page and performance pages
• Actions supported: Start/stop Cell, verify
connectivity, setup SSH
Automatic discovery of Exadata cells
Management by Cell Group
• All cells used by a database automatically
placed in a group
• Cell Group level administration operations
Navigation: Targets Exadata DB Machine
Exadata Grid Name
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Storage Cell Management
Perform Cell Administration tasks
• Execute Cellcli commands on a set of cells or all cells
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Setup IORM using EM 12c
Allocate I/O resources across databases by
means of an IORM plan configured on
each storage cell
IORM plans should be identically
configured on each storage cell
Advantage with Exadata Plugin:Implement IORM plan across all
cells with one click
Navigation: Exadata Storage Server Administration
IO Resource
Manage
CellCLI> ALTER IORMPLAN –
dbPlan = ( (name=DBM, level=1 allocation=60),
(name=CRM, level=2 allocation=80),
(name=other, level =3 allocation=100))
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
12
Infiniband Network Management
Infiniband network and switches as GC targets
Full monitoring
• Automatic discovery
• Alerts (switch generated and EM generated)
• Performance metrics, collected every 60
seconds
• Configuration metrics – detect and notify
configuration changes/best practice
violations
Network home page and performance page
• Real time and historical
Topology view of Network with switch and port level
details
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Infiniband Network Management
Perform Infiniband Administration tasks
•
•
•
•
Enable Port
Disable Port
Clear Performance counters
Clear Error counters
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Integration with the Database Performance Page
• Drill down from database
Performance page
Provides composite view
of all health indicators of
a cell or cell group
Helps triage
Navigation: Targets Databases Target Database
Performance Performance Home
Database Instance
I/O
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
• Load imbalance
• ASM related problems
• Cell software or
hardware failures
• Cell configuration
issues
• Network related
failures
Monitoring: ILOM, Cisco Switch, KVM
Common metrics monitored
Power supply failure
Fan failure
Temperature out of range
Specific metrics monitored
• Cisco Switch
• Configuration change tracking and reporting
• Unauthorized SNMP access
• Keyboard, Video, Mouse (KVM)
• Server connected to KVM added/removed, powered on/off
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Exadata Database Machine Fault Monitoring
Storage Server Fault Monitoring
Compute Node Fault Monitoring
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Program Agenda with Highlight
1
What & How is Exadata Monitored?
2
What's New in Exadata Management?
3
Real World Experience
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
What's New in Exadata Management?
EM Exadata Plug-in 12.1.0.6.0
Full support for Oracle Database Multitenant
option
1
2
CDB level IO workload summary with PDB level
details breakdown
Visualization of IO utilization for CDB & PDB
3
IORM for multi-tenancy database (CDB/PDB)
environment
Support for Exadata Server
Version:11.2.3.3.1,12.1.1.1.1 & Exadata Hardware
X4-2,X4-8
Navigation: Exadata Storage Server
Administration Manage IO Resource
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
19
What's New in Exadata Management?
Enhanced Grid Home Page in 12.1.0.6.0 Plug-in
New Fine-grained
performance
summary for flash
and hard disk
Detailed summary of flash space usage
New
I/O time distribution
by flash and hard disk.
IORM wait per database
New Usage Statistics
Highlights
Flash cache efficiency
Smart Scan efficiency
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
20
What's New in Exadata Management?
Enhanced Grid Home Page in 12.1.0.6.0 Plug-in
Usage Statistics by Hard Disk
& Flash
Note Utilization, IOPS & MBPS
Provides insight into storage
capacity utilization
Numbers will be workload
dependent
Any deviation from normal
baselines should be
investigated
Easier to monitor & compare
performance
of flash and hard disk
New usage statistics highlights Flash
Cache
& Smart Scan efficiency
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
21
What's New in Exadata Management?
Enhanced Grid Home Page in 12.1.0.6.0 Plug-in
New detailed breakdown of Flash Space Usage
Easily identify how Flash Cache is used
Visually see Keep, Non-Keep and Unused area
New IO time distribution by Flash, Hard disk & IORM
Wait per DB
Visualization for:
I/O time distribution by flash and hard disk
IORM wait per database
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
22
What's New in Exadata Management?
Storage Cell Performance Page in 12.1.0.6.0 Plug-in
Flash & Hard disk Utilization chart to identify if workload is reaching hardware limit
New side-by-side comparison of Flash & Hard disk performance charts
Average IO Utilization
Maximum Cell Disk Limit Line
For IOPS
Maximum Cell Disk Limit Line
For MBPS
Check the Average IO utilization in conjunction with the Maximum Cell Disk Limit
Identify if workload reaching hardware limit
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Check Datasheet
For Exadata hardware for limits
Oracle Confidential – Internal/Restricted/Highly Restricted
23
What's New in Exadata Management?
Storage Cell Performance Page in 12.1.0.6.0 Plug-in
Drill down charts to compare performance between multiple Cells
Performance comparison between multiple Exadata Storage Servers
Identify load imbalance and outliers
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Enterprise Manager Exadata plug-in Support
Doc ID 1626579.1- Exadata Storage software versions supported by the Oracle
Enterprise Manager Exadata plug-in
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Exadata Monitoring with OEM12c
Roland Schiller, Swiss Re, Sep 2014
Swiss Re is a leading and highly diversified global
re/insurance company
•
151 years of experience in providing wholesale re/insurance and risk
management solutions
•
We deliver both traditional and innovative offerings
in Property & Casualty and Life & Health that meet our clients' needs
•
A pioneer in insurance-based capital market solutions, we combine
financial strength and unparalleled expertise for the benefit of our clients
•
Our financial strength is currently rated:
Standard & Poor’s: AA-/stable; Moody’s: Aa3/stable;
A.M. Best: A+/stable
Headquarters, Zurich
Armonk, New York
The “Gherkin”, London
Exadata Management with OEM12c | Swiss Re | Sep 2014
27
We enable risk-taking that is
essential to enterprise and progress
We identify and
evaluate risks
Climate change identified as emerging risk almost 20 years ago
We select and
take risks
Insurance of single risks
(eg industrial plants) or entire portfolios of risk
(eg motor insurance portfolio)
We transfer and
trade risks
Securitisation of earthquake,
hurricane or life risks
We educate and consult on
risks
Over 30 risk-related publications
during the last 12 months
Exadata Management with OEM12c | Swiss Re | Sep 2014
28
Exadata Strategy until 2015
•
•
•
General move to Exadata
150 often highly shared databases
supporting 350 different applications
1PB Storage
Oracle 12c with the Multitenant
Option
Active Data Guard
•
OEM12c for
•
•
Risk DB – 65TB
x4-2
2014
DWH DB – 8TB
x2-2
2012
Monitoring + alerting
Job scheduling
Backup
Data pump
DB provisioning
Active Data Guard
GDS
Exadata Management with OEM12c | Swiss Re | Sep 2014
29
Hundreds of Exadata Metrics to collect, monitor and alert for …all in one
place!
Metrics for
•
•
•
•
•
•
Storage cells
Exadata hosts
Databases
Infiniband
PDU
Exachk
Configure
•
•
•
•
Warning Thresholds
Critical Thresholds
Corrective Actions
Add your own Metric
Extensions
Exadata Management with OEM12c | Swiss Re | Sep 2014
30
Metrics by Component
=> Monitoring Templates for Hosts, DB, ILOM, PDU etc
• You may need different templates per SLA level
• Have an Enterprise User own the objects
Exadata Management with OEM12c | Swiss Re | Sep 2014
31
Templates organised by Environment
=> Template Collections for Prod / Non-Prod
Exadata Management with OEM12c | Swiss Re | Sep 2014
32
Administration Groups
Keep it Simple!
• Administration Groups are a special type of group used to fully automate the application of management settings
(monitoring settings, compliance standards, and cloud policies) to targets upon joining the group.
Exadata Management with OEM12c | Swiss Re | Sep 2014
33
Set Lifecycle during Discovery ->
All Monitoring/Alerting Settings applied automatically
• When a target is added to an administration group, Enterprise Manager automatically applies
management settings associated with the group to the newly added target.
Exadata Management with OEM12c | Swiss Re | Sep 2014
34
Dynamic Groups for Exadata
Availability and Incident Dashboard
•
•
•
A dynamic group allows administrators to create a group of targets by specifying its membership criteria instead of adding
targets directly into the group.
Membership criteria are based on target properties such as Lifecycle Status, Line of Business, Target Type, etc.
When multiple criteria are defined for a dynamic group, a target must match all criteria before it is automatically added to the
dynamic group.
Exadata Management with OEM12c | Swiss Re | Sep 2014
35
Dynamic group "Exadata Production"
Exadata Management with OEM12c | Swiss Re | Sep 2014
36
Drill down into any target member of "Exadata Production" for detailed analysis
Exadata Management with OEM12c | Swiss Re | Sep 2014
37
Compare Database Workload on the Storage Grid
Exadata Management with OEM12c | Swiss Re | Sep 2014
38
Detailed Cell Metrics
Compare I/O Throughput for DB on Cell Targets
Exadata Management with OEM12c | Swiss Re | Sep 2014
39
IORM Utilization of the Production Standby
Compare DB Keys
Exadata Management with OEM12c | Swiss Re | Sep 2014
40
Support for Swiss Re's Strategic IT Objectives
•Availability
•Performance
•SLA compliance
•Standardisation
•Simplification by
removing 'homemade' complexity
•Centralisation
•Very short time to
market
Positive
Customer
Experience
Agility
Productivity
Cloud
Transformation
•Full Integration with
Cloud Self Service
(PDBaaS)
Exadata Management with OEM12c | Swiss Re | Sep 2014
41
Exadata Management with OEM12c | Swiss Re | Sep 2014
42
Legal notice
©2014 Swiss Re. All rights reserved. You are not permitted to create any modifications
or derivative works of this presentation or to use it for commercial or other public purposes
without the prior written permission of Swiss Re.
The information and opinions contained in the presentation are provided as at the date of
the presentation and are subject to change without notice. Although the information used
was taken from reliable sources, Swiss Re does not accept any responsibility for the accuracy
or comprehensiveness of the details given. All liability for the accuracy and completeness
thereof or for any damage or loss resulting from the use of the information contained in this
presentation is expressly excluded. Under no circumstances shall Swiss Re or its Group
companies be liable for any financial or consequential loss relating to this presentation.
Exadata Management with OEM12c | Swiss Re | Sep 2014
43
Program Agenda with Highlight
1
What & How is Exadata Monitored?
2
What's New in Exadata Management?
3
Real World Experience
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Monitoring Exadata with Oracle Enterprise Manager 12c
September 09, 2014
Agenda
• Company Background
• Architecture
• EM 12c - Real Experience
• Benefits of Oracle Enterprise Manager
• Useful Tips
Monitoring Exadata with Oracle Enterprise Manager 12c
46
Company Background
Company
– Ebay Inc: Global commerce platform and payments leader
Who Am I
– Akash Guha, Database Engineer, MTS2
• Oracle Certified Professional with many years of experience starting with
Oracle 8
• Oracle RAC and Performance Tuning Expert
• Working On Exadata for the past 2 years
Monitoring Exadata with Oracle Enterprise Manager 12c
47
PayPal Exadata Architecture
Primary Data Center
Mission-critical
OLTP Databases
Production Database
• 2 X Exadata X2-8 (HP-600GB Disk)
• 2 X Full Storage Expansion (HP-600GB
Disk)
Cluster A
Cluster B
Cluster C
DR Data Center
Active Dataguard
Down Stream Users
DR Database
• 2 X Exadata X2-8 (HP600GB Disk)
• 2 X Full Storage
Expansion (HP-600GB
Disk)
RW DW System
Cluster A
Cluster B
Cluster C
Analytical Database
• 1X Exadata X2-8
(HC-3TB disk)
Monitoring Exadata with Oracle Enterprise Manager 12c
48
Characteristics
Hardware Characteristics
• 13 x X2-8
• 12 X Full Storage Expansion
• 384 Storage Server with 600GB HP Disk
Database Characteristics
• Redo 200MB/s across the clusters
• 300k Executions/s
• 90k Connections from Application
Application Characteristics
• Mix of JDBC and C++
• Over 400 application servers
• SLA of 250ms for execution of each SQL
Monitoring Exadata with Oracle Enterprise Manager 12c
PayPal OEM/ASR/Ops Center Primary Data Center
Primary Data Center
Other Assets
OEM LB
DBnode/Cell OS
Exadat
a
ILOMs/Cellcli
Solaris OS
Solaris ILOM
Back Tier
OC Back
Tier Proxy1
T4-1
OC Back
Tier Proxy2
T4-1
DR Site Extranet Proxy
To DR OEM LB
ASR
Server
OEM RAC1
OEM RAC2
DL360G8
DL360G8
DB SAN Storage
OEM/OC
OCR on ASM
DB
s/w repo NAS
OpsCenter1/2 EC
2-node cluster
2x T4-2
s/w repo NAS
DCN
Dataguar
d to DR
OC DCN
Proxy1 T41
OC DCN
Proxy2 T41
Oracle GCS
ASR Central
Rules open
SR
Manual SR
NOC
Console
Monitoring Exadata with Oracle Enterprise Manager 12c
Enterprise Manager Configuration
• 7000+ targets
• RAC cluster for OMR on Primary and Standby
• 2x active OMS behind LB on primary, 2x passive OMS behind LB on
standby
• OpsCenter also on RAC clusters
• ASR opening SRs via proxy
• Alerts visible on NOC Console for 24x7x365 monitoring
Monitoring Exadata with Oracle Enterprise Manager 12c
51
Exadata Plugin for Enterprise Manager 12c
Monitoring Exadata with Oracle Enterprise Manager 12c
52
Exadata Plugin – Alert
stxxx.payapal.com
Monitoring Exadata with Oracle Enterprise Manager 12c
53
Enterprise Manager 12c – Real Experience
Problem Description
• Intermittent Session Spike observed on all three primary clusters
• Application SLA miss by 5% causing significant loss to business
• Sub-optimal Query performance- SQLs taking more than 2 seconds
Triage Using Enterprise Manager 12c
• Ash Analytics
• Metric Extension
Monitoring Exadata with Oracle Enterprise Manager 12c
Monitoring Exadata with Oracle Enterprise Manager 12c
55
Enterprise Manager 12c – Metric Extension
Monitoring Exadata with Oracle Enterprise Manager 12c
56
Metric Extension: Monitoring HCA Port Throughput
• Measuring amount of data Received and Transmitted through the Host Channel
Adaptor Ports.
• Implementation: Multi column Host script run by root from OEM (courtesy Oracle)
Imbalance in Traffic
between Ports
Monitoring Exadata with Oracle Enterprise Manager 12c
57
Observations
• Ash Analytics page shows Cluster Waits as the top Wait Class
• The Drill down page points out the detailed GC Waits and the top SQLs
contributing to it
• Metric Extension on HCA throughput data confirms imbalance in traffic between
the InfiniBand cards
• The output of RDS-ping shows high outliers and latencies(via ExaWatcher)
Monitoring Exadata with Oracle Enterprise Manager 12c
58
58
Solutions Applied
• Adjust Cluster Interconnect parameter to use 4 IB cards
cluster_interconnects=192.168.10.1:192.168.10.2:192.168.10.3:192.168.10.4
• Applied Bundle Patch 22 (pre-requisite for patch 12811198)
• Applied Patch 12811198: ASM ASYNC REBALANCE NOT MAXIMIZING IB
THROUGHPUT
Balanced Traffic
across all Ports
Monitoring Exadata with Oracle Enterprise Manager 12c
59
Benefits of Oracle Enterprise Manager 12c
• Automation of Hardware and Software Monitoring for Exadata
• Standardized Monitoring Template
• Use of ASR and minimize human intervention
• Centralized Repository for database metrics
• Extensible Metric Framework
• Easy Interface for troubleshooting Performance problem
• Rich Reporting Framework and enables comparison between Targets
Monitoring Exadata with Oracle Enterprise Manager 12c
60
Tips from our Experience
• Implement High Availability solutions for Enterprise Manger
• Keep Enterprise manager up-to-date with current release to benefit from
new features and latest fixes
• Configure ASR for fast restoration of Hardware Faults
• Implement Monitoring Templates to improve standardization
• Use OMS Repository for enterprise wise Metrics data – this enables easy
comparison between various Targets and Metrics
Monitoring Exadata with Oracle Enterprise Manager 12c
61
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Was this manual useful for you? yes no
Thank you for your participation!

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Download PDF

advertising