Oracle Enterprise Manager Cloud Control Administrators Guide

Oracle Enterprise Manager Cloud Control Administrators Guide
OracleВ® Enterprise Manager
Cloud Control Administrator’s Guide
12c Release 4 (12.1.0.4)
E24473-33
November 2014
Oracle Enterprise Manager Cloud Control Administrator’s Guide, 12c Release 4 (12.1.0.4)
E24473-33
Copyright В© 2014, Oracle and/or its affiliates. All rights reserved.
This software and related documentation are provided under a license agreement containing restrictions on
use and disclosure and are protected by intellectual property laws. Except as expressly permitted in your
license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license,
transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse
engineering, disassembly, or decompilation of this software, unless required by law for interoperability, is
prohibited.
The information contained herein is subject to change without notice and is not warranted to be error-free. If
you find any errors, please report them to us in writing.
If this is software or related documentation that is delivered to the U.S. Government or anyone licensing it
on behalf of the U.S. Government, the following notice is applicable:
U.S. GOVERNMENT RIGHTS Programs, software, databases, and related documentation and technical data
delivered to U.S. Government customers are "commercial computer software" or "commercial technical data"
pursuant to the applicable Federal Acquisition Regulation and agency-specific supplemental regulations. As
such, the use, duplication, disclosure, modification, and adaptation shall be subject to the restrictions and
license terms set forth in the applicable Government contract, and, to the extent applicable by the terms of
the Government contract, the additional rights set forth in FAR 52.227-19, Commercial Computer Software
License (December 2007). Oracle USA, Inc., 500 Oracle Parkway, Redwood City, CA 94065.
This software or hardware is developed for general use in a variety of information management
applications. It is not developed or intended for use in any inherently dangerous applications, including
applications that may create a risk of personal injury. If you use this software or hardware in dangerous
applications, then you shall be responsible to take all appropriate fail-safe, backup, redundancy, and other
measures to ensure its safe use. Oracle Corporation and its affiliates disclaim any liability for any damages
caused by use of this software or hardware in dangerous applications.
Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks
of their respective owners.
This software and documentation may provide access to or information on content, products, and services
from third parties. Oracle Corporation and its affiliates are not responsible for and expressly disclaim all
warranties of any kind with respect to third-party content, products, and services. Oracle Corporation and
its affiliates will not be responsible for any loss, costs, or damages incurred due to your access to or use of
third-party content, products, or services.
Contents
Preface ............................................................................................................................................................. xxv
Audience...................................................................................................................................................
Documentation Accessibility .................................................................................................................
Related Documents .................................................................................................................................
Conventions .............................................................................................................................................
xxv
xxv
xxv
xxvi
What's Changed in this Guide? ....................................................................................................... xxvii
Part I
Monitoring and Managing Targets
1 Enterprise Monitoring
1.1
1.2
1.3
1.3.1
1.3.2
1.3.3
1.3.4
1.3.5
1.3.6
1.3.7
1.4
1.4.1
1.4.2
1.4.3
1.5
1.5.1
1.6
1.6.1
1.6.2
1.6.3
1.7
Monitoring Overview................................................................................................................. 1-1
Comprehensive Out-of-Box Monitoring ................................................................................. 1-1
Monitoring: Basics ...................................................................................................................... 1-2
Metric Thresholds: Determining When a Monitored Condition is an Issue............... 1-3
Metric Baselines: Determining Valid Metric Thresholds ............................................... 1-3
Advanced Threshold Management................................................................................... 1-4
Events: Defining What Conditions are of Interest .......................................................... 1-5
Corrective Actions: Resolving Issues Automatically ..................................................... 1-5
Metric Extensions: Customizing Monitoring................................................................... 1-6
Blackouts ............................................................................................................................... 1-6
Monitoring: Advanced Setup.................................................................................................... 1-7
Monitoring Templates......................................................................................................... 1-7
Administration Groups and Template Collections ........................................................ 1-8
Customizing Alert Messages ............................................................................................. 1-8
Notifications.............................................................................................................................. 1-10
Customizing Notifications .............................................................................................. 1-11
Managing Events, Incidents, and Problems......................................................................... 1-11
Incident Manager.............................................................................................................. 1-12
Incident Rules and Rule Sets........................................................................................... 1-13
Connectors ......................................................................................................................... 1-14
Accessing Monitoring Information ....................................................................................... 1-14
2 Discovering, Promoting, and Adding Targets
2.1
About Discovering, Promoting, and Adding Targets ........................................................... 2-1
iii
2.1.1
About Discovering, Promoting, and Monitoring Hosts and Targets........................... 2-1
2.1.1.1
What are Targets and Managed Targets? ................................................................. 2-1
2.1.1.2
What is Discovery?....................................................................................................... 2-1
2.1.1.3
What is Promotion? ...................................................................................................... 2-3
2.1.1.4
What is Monitoring?..................................................................................................... 2-3
2.1.2
Discovery and Monitoring in Enterprise Manager Lifecycle ........................................ 2-4
2.1.3
Discovery and Monitoring Process ................................................................................... 2-4
2.2
Discovering and Promoting All Target Types........................................................................ 2-5
2.2.1
Discovering and Promoting All Target Types Using the Autodiscovery Process ..... 2-6
2.2.1.1
Meeting the Prerequisites ............................................................................................ 2-6
2.2.1.2
Discovering and Promoting All Target Types ......................................................... 2-7
2.2.2
Discovering and Adding All Target Types Using the Guided Discovery Process . 2-13
2.2.2.1
Step 1: Identifying Unmanaged Hosts ................................................................... 2-13
2.2.2.2
Step 2: Converting Unmanaged Hosts to Managed Hosts.................................. 2-14
2.2.2.3
Step 3: Adding Targets ............................................................................................. 2-14
2.2.3
Discovering and Adding All Target Types By Specifying Target Monitoring Properties
2-15
2.2.3.1
Step 1: Indentifying Unmanaged Hosts ................................................................. 2-15
2.2.3.2
Step 2: Converting Unmanaged Hosts to Managed Hosts.................................. 2-16
2.2.3.3
Step 3: Adding Targets ............................................................................................. 2-16
2.2.4
Retrieving Deleted Targets.............................................................................................. 2-17
2.2.4.1
Retrieving Deleted Target Types............................................................................. 2-17
2.2.4.2
Retrieving Deleted Host and Corresponding Management Agent Targets ..... 2-18
2.3
Discovering and Promoting Oracle Homes ......................................................................... 2-19
2.4
Discovering, Promoting, and Adding Database Targets ................................................... 2-21
2.4.1
Discovering Container Database and Pluggable Database Targets .......................... 2-21
2.4.1.1
Discovering and Promoting CDB and PDB Targets Using Autodiscovery ...... 2-22
2.4.1.2
Discovering and Adding CDB and PDB Targets Using the Guided Discovery
Process 2-24
2.4.1.3
Discovering and Adding CDB and PDB Targets By Specifying Target Monitoring
Properties 2-27
2.4.2
Discovering Cluster Database Targets........................................................................... 2-28
2.4.2.1
Discovering and Promoting Cluster Database Targets Using Autodiscovery . 2-29
2.4.2.2
Discovering and Adding Cluster Database Targets Using the Guided Discovery
Process 2-31
2.4.2.3
Discovering and Adding Cluster Database Targets By Specifying Target
Monitoring Properties 2-33
2.4.3
Discovering Single Instance Database Targets ............................................................. 2-35
2.4.3.1
Discovering and Promoting Single Instance Database Targets Using
Autodiscovery 2-35
2.4.3.2
Discovering and Adding Single Instance Database Targets Using Guided
Discovery Process 2-37
2.4.3.3
Discovering and Adding Single Instance Database Targets By Specifying Target
Monitoring Properties 2-39
2.4.4
Discovering Cluster Targets............................................................................................ 2-41
2.4.4.1
Discovering and Promoting Cluster Targets Using Autodiscovery .................. 2-41
2.4.4.2
Discovering and Adding Cluster Targets Using the Guided Discovery Process .......
2-43
iv
2.4.4.3
2.4.5
2.4.5.1
2.4.5.2
2.4.5.3
2.4.6
2.4.6.1
2.4.6.2
2.4.6.3
2.4.7
2.5
2.5.1
2.5.1.1
2.5.1.2
2.5.1.3
2.5.2
2.5.2.1
2.5.2.2
2.5.3
2.5.3.1
2.5.3.2
2.5.4
2.5.4.1
2.5.4.2
2.5.5
Discovering and Adding Cluster Targets By Specifying Target Monitoring
Properties 2-44
Discovering Single Instance High Availability Service Targets ................................ 2-46
Discovering and Promoting Single Instance High Availability Service Targets
Using Autodiscovery 2-46
Discovering and Adding Single Instance High Availability Service Targets Using
the Guided Discovery Process 2-47
Discovering and Adding Single Instance High Availability Service Targets By
Specifying Target Monitoring Properties 2-49
Discovering Cluster Automatic Storage Management Targets ................................. 2-50
Discovering and Promoting Cluster ASM Targets Using Autodiscovery ........ 2-50
Discovering and Adding Cluster ASM Targets Using the Guided Discovery
Process 2-52
Discovering and Adding Cluster ASM Targets By Specifying Target Monitoring
Properties 2-54
Enabling Autodiscovery of Database Targets .............................................................. 2-55
Discovering, Promoting, and Adding Middleware Targets.............................................. 2-56
Discovering Weblogic 9.x or 10.x Domains .................................................................. 2-56
Discovering and Promoting Weblogic Domains Using Autodiscovery............ 2-57
Discovering and Adding WebLogic 9.x or 10.x Domains Using the Guided
Discovery Process 2-64
Adding Multiple WebLogic Domains Using EM CLI.......................................... 2-69
Discovering New or Modified Domain Members ....................................................... 2-69
Enabling Automatic Discovery of New Domain Members ................................ 2-70
Manually Checking for New or Modified Domain Members ............................ 2-70
Discovering a Standalone Oracle HTTP Server............................................................ 2-75
Meeting the Prerequisites......................................................................................... 2-75
Discovering a Standalone Oracle HTTP Server Using the Guided Discovery
Process 2-76
Discovering Exalytics Targets......................................................................................... 2-77
Meeting the Prerequisites......................................................................................... 2-78
Discovering and Adding Exalytics System Targets Using the Guided Discovery
Process 2-78
Removing Middleware Targets ...................................................................................... 2-80
3 Using Incident Management
3.1
Management Concepts............................................................................................................... 3-3
3.1.1
Event Management.............................................................................................................. 3-3
3.1.2
Incident Management ......................................................................................................... 3-6
3.1.2.1
Working with Incidents ............................................................................................... 3-7
3.1.2.2
Incident Composed of a Single Event........................................................................ 3-9
3.1.2.3
Incident Composed of Multiple Events ................................................................. 3-10
3.1.2.4
How are Incidents Created?..................................................................................... 3-11
3.1.3
Problem Management ...................................................................................................... 3-11
3.1.4
Rule Sets ............................................................................................................................. 3-12
3.1.4.1
Out-of-Box Rule Sets ................................................................................................. 3-13
3.1.4.2
Rule Set Types ............................................................................................................ 3-14
3.1.4.3
Rules ............................................................................................................................ 3-15
v
3.1.5
Incident Manager.............................................................................................................. 3-19
3.1.5.1
Views ........................................................................................................................... 3-21
3.1.6
Summing Up ..................................................................................................................... 3-21
3.2
Setting Up Your Incident Management Environment ....................................................... 3-22
3.2.1
Setting Up Your Monitoring Infrastructure.................................................................. 3-23
3.2.1.1
Rule Set Development............................................................................................... 3-23
3.2.2
Setting Up Administrators and Privileges .................................................................... 3-26
3.2.3
Monitoring Privileges....................................................................................................... 3-30
3.2.4
Setting Up Rule Sets ......................................................................................................... 3-33
3.2.4.1
Creating a Rule Set .................................................................................................... 3-33
3.2.4.2
Creating a Rule to Create an Incident..................................................................... 3-34
3.2.4.3
Creating a Rule to Manage Escalation of Incidents .............................................. 3-34
3.2.4.4
Creating a Rule to Escalate a Problem.................................................................... 3-36
3.2.4.5
Testing Rule Sets........................................................................................................ 3-37
3.2.4.6
Subscribing to Receive Email from a Rule ........................................................... 3-39
3.2.4.7
Receiving E-mail for Private Rules ......................................................................... 3-39
3.3
Working with Incidents .......................................................................................................... 3-40
3.3.1
Finding What Needs to be Worked On ......................................................................... 3-41
3.3.2
Searching for Incidents .................................................................................................... 3-44
3.3.3
Setting Up Custom Views ............................................................................................... 3-45
3.3.4
Sharing/Unsharing Custom Views ............................................................................... 3-46
3.3.5
Responding and Working on a Simple Incident .......................................................... 3-46
3.3.6
Responding to and Managing Multiple Incidents, Events and Problems in Bulk.. 3-47
3.3.7
Searching My Oracle Support Knowledge ................................................................... 3-49
3.3.8
Open Service Request (Problems-only) ......................................................................... 3-50
3.3.9
Suppressing Incidents and Problems............................................................................. 3-50
3.3.10
Managing Workload Distribution of Incidents ............................................................ 3-50
3.3.11
Reviewing Events on a Periodic Basis ........................................................................... 3-51
3.3.11.1
Creating an Incident Manually................................................................................ 3-51
3.4
Advanced Topics...................................................................................................................... 3-52
3.4.1
Automatic Diagnostic Repository (ADR): Incident Flood Control ........................... 3-52
3.4.1.1
Working with ADR Diagnostic Incidents Using Incident Manager .................. 3-52
3.4.1.2
Incident Flood Control.............................................................................................. 3-52
3.4.2
Defining Custom Incident Statuses................................................................................ 3-54
3.4.2.1
Creating a New Resolution State ............................................................................ 3-54
3.4.2.2
Modifying an Existing Resolution State................................................................. 3-55
3.4.3
Clearing Stateless Alerts for Metric Alert Event Types............................................... 3-55
3.4.4
Automatically Clearing "Manually Clearable" Events ................................................ 3-56
3.4.5
User-reported Events ....................................................................................................... 3-57
3.4.5.1
Format ......................................................................................................................... 3-58
3.4.5.2
Options........................................................................................................................ 3-58
3.4.5.3
Examples..................................................................................................................... 3-59
3.4.6
Additional Rule Applications ......................................................................................... 3-60
3.4.6.1
Setting Up a Rule to Send Different Notifications for Different Severity States of an
Event 3-60
3.4.6.2
Creating a Rule to Notify Different Administrators Based on the Event Type 3-61
3.4.6.3
Creating a Rule to Create a Ticket for Incidents ................................................... 3-62
3.4.6.4
Creating a Rule to Send SNMP Traps to Third Party Systems ........................... 3-63
vi
3.4.7
Event Prioritization ..........................................................................................................
3.4.8
Root Cause Analysis (RCA) and Target Down Events ...............................................
3.4.8.1
How RCA Works .......................................................................................................
3.4.8.2
Leveraging RCA Results in Incident Rule Sets .....................................................
3.4.8.3
Leveraging RCA Results in Incident Manager......................................................
3.4.8.4
Leveraging RCA Results in the System Dashboard .............................................
3.4.8.5
Creating a Rule to Update Incident Priority for Non-symptom Events............
3.4.8.6
Creating Incidents On Non-symptom Events .......................................................
3.4.8.7
Introducing a Time Delay.........................................................................................
3.5
Moving from Enterprise Manager 10/11g to 12c................................................................
3-64
3-65
3-65
3-67
3-68
3-70
3-71
3-71
3-73
3-74
Monitoring: Common Tasks................................................................................................... 3-75
Setting Up an Email Gateway ......................................................................................... 3-76
Sending Email for Metric Alerts ..................................................................................... 3-78
Sending SNMP Traps for Metric Alerts......................................................................... 3-82
Sending Events to an Event Connector ......................................................................... 3-87
Sending Email to Different Email Addresses for Different Periods of the Day ...... 3-92
Disabling Out-of-Box Rulesets........................................................................................ 3-94
4 Using Notifications
4.1
4.1.1
4.1.2
4.1.2.1
4.1.2.2
4.1.2.3
4.1.3
4.1.4
4.1.4.1
4.1.5
4.2
4.3
4.3.1
4.3.2
4.3.2.1
4.3.2.2
4.3.2.3
4.3.2.4
4.3.2.5
4.4
4.4.1
4.4.2
4.4.2.1
4.4.2.2
4.4.2.3
4.5
4.5.1
Setting Up Notifications............................................................................................................. 4-2
Setting Up a Mail Server for Notifications ....................................................................... 4-2
Setting Up E-mail for Yourself........................................................................................... 4-5
Defining E-mail Addresses ......................................................................................... 4-5
Setting Up a Notification Schedule ............................................................................ 4-6
Subscribe to Receive E-mail for Incident Rules........................................................ 4-7
Setting Up E-mail for Other Administrators ................................................................... 4-9
E-mail Customization....................................................................................................... 4-10
E-mail Customization Reference ............................................................................. 4-11
Setting Up Repeat Notifications ..................................................................................... 4-14
Extending Notification Beyond E-mail................................................................................. 4-15
Sending Notifications Using OS Commands and Scripts .................................................. 4-16
Script Examples................................................................................................................. 4-18
Migrating pre-12c OS Command Scripts ...................................................................... 4-21
Migrating Metric Alert Event Types....................................................................... 4-21
Migrating Target Availability Event Types ........................................................... 4-22
Migrating Job Status Change Event Types ............................................................ 4-22
Migrating Corrective Action-Related OS Scripts.................................................. 4-23
Notification Type Mapping...................................................................................... 4-24
Sending Notifications Using PL/SQL Procedures.............................................................. 4-24
Defining a PL/SQL-based Notification Method .......................................................... 4-24
Migrating Pre-12c PL/SQL Advanced Notification Methods ................................... 4-32
Mapping for MGMT_NOTIFY_SEVERITY ........................................................... 4-32
Mapping for MGMT_NOTIFY_JOB........................................................................ 4-37
Mapping for MGMT_NOTIFY_CORRECTIVE_ACTION .................................. 4-37
Sending SNMP Traps to Third Party Systems..................................................................... 4-38
SNMP Version 1 Versus SNMP Version 3 .................................................................... 4-39
vii
4.5.2
Working with SNMP V3 Trap Notification Methods..................................................
4.5.2.1
Configuring the OMS to Send SNMP Trap Notifications ...................................
4.5.2.2
Creating/Editing an SNMP V3 Trap Notification Method.................................
4.5.2.3
Editing a User Security Model Entry......................................................................
4.5.2.4
Viewing Available SNMP V3 Trap Notification Methods ..................................
4.5.2.5
Deleting an SNMP V3 Trap Notification Method ................................................
4.5.3
Creating an SNMP V1 Trap.............................................................................................
4.5.4
SNMP Traps: Moving from Previous Enterprise Manager Releases to 12c ............
4.6
Management Information Base (MIB)...................................................................................
4.6.1
About MIBs........................................................................................................................
4.6.2
MIB Definition...................................................................................................................
4.6.3
Reading the MIB Variable Descriptions ........................................................................
4.6.3.1
Variable Name ...........................................................................................................
4.7
Passing Corrective Action Status Change Information......................................................
4.7.1
Passing Corrective Action Execution Status to an OS Command or Script ............
4.7.2
Passing Corrective Action Execution Status to a PLSQL Procedure........................
4.8
Passing Job Execution Status Information...........................................................................
4.8.1
Passing Job Execution Status to a PL/SQL Procedure ...............................................
4.8.2
Passing Job Execution Status to an OS Command or Script......................................
4.9
Passing User-Defined Target Properties to Notification Methods ...................................
4.10
Notification Reference .............................................................................................................
4.10.1
EMOMS Properties...........................................................................................................
4.10.2
Passing Event, Incident, Problem Information to an OS Command or Script.........
4.10.2.1
Environment Variables Common to Event, Incident and Problem ..................
4.10.2.2
Event Notification-Specific Environment Variables.............................................
4.10.2.3
Environment Variables Specific to Event Types ...................................................
4.10.2.4
Environment Variables Specific to Incident Notifications ..................................
4.10.2.5
Environment Variables Specific to Problem Notifications ..................................
4.10.2.6
Environment Variables Common to Incident and Problem Notifications........
4.10.3
Passing Information to a PL/SQL Procedure...............................................................
4.10.3.1
Notification Payload Elements Specific to Event Types ......................................
4.10.4
Troubleshooting Notifications ........................................................................................
4.10.4.1
General Setup .............................................................................................................
4.10.4.2
Notification System Errors .......................................................................................
4.10.4.3
Notification System Trace Messages ......................................................................
4.10.4.4
E-mail Errors ..............................................................................................................
4.10.4.5
OS Command Errors .................................................................................................
4.10.4.6
SNMP Trap Errors .....................................................................................................
4.10.4.7
PL/SQL Errors ...........................................................................................................
4-39
4-39
4-41
4-43
4-44
4-44
4-44
4-47
4-48
4-48
4-48
4-49
4-49
4-50
4-50
4-51
4-52
4-52
4-54
4-55
4-55
4-56
4-59
4-60
4-61
4-63
4-65
4-67
4-68
4-69
4-77
4-80
4-80
4-81
4-81
4-83
4-83
4-84
4-84
5 Using Blackouts
5.1
Working with Blackouts ............................................................................................................
5.1.1
Creating a Blackout .............................................................................................................
5.1.2
Editing a Blackout................................................................................................................
5.1.3
Viewing Blackouts ...............................................................................................................
5.1.3.1
Viewing Blackouts on Targets Monitored by a Specific Management Agent .....
5.1.3.2
Viewing Blackouts from Target Home Pages...........................................................
viii
5-1
5-1
5-2
5-2
5-3
5-3
5.1.3.3
Viewing Blackouts from Groups and Systems Target Administration Pages.....
5.1.4
Purging Blackouts that have Ended..................................................................................
5.2
Controlling Blackouts Using the Command Line Utility......................................................
5.3
About Blackouts Best Effort.......................................................................................................
5.3.1
When to Use Blackout Best Effort .....................................................................................
5-3
5-4
5-4
5-6
5-6
6 Managing Groups
6.1
6.1.1
6.1.2
6.1.3
6.1.4
6.1.5
6.2
6.2.1
6.2.2
6.2.3
6.2.4
6.2.5
6.2.6
6.2.7
6.2.8
6.2.9
6.3
Introduction to Groups .............................................................................................................. 6-1
Overview of Groups............................................................................................................ 6-2
Overview of Privilege Propagating Groups .................................................................... 6-2
Overview of Dynamic Groups........................................................................................... 6-3
Overview of Administration Groups................................................................................ 6-3
Choosing Which Type of Group To Use .......................................................................... 6-4
Managing Groups ....................................................................................................................... 6-4
Creating and Editing Groups............................................................................................. 6-5
Creating Dynamic Groups ................................................................................................. 6-6
Adding Members to Privilege Propagating Groups ...................................................... 6-8
Converting Conventional Groups to Privilege Propagating Groups .......................... 6-8
Viewing and Managing Groups ........................................................................................ 6-9
Overview of Group Charts.............................................................................................. 6-10
Overview of Group Members ......................................................................................... 6-10
Viewing Group Status History ....................................................................................... 6-11
About the System Dashboard ......................................................................................... 6-11
Using Out-of-Box Reports ...................................................................................................... 6-12
7 Using Administration Groups
7.1
7.1.1
7.2
7.3
7.3.1
7.3.1.1
7.3.1.2
7.3.1.3
7.3.1.4
7.4
7.4.1
7.4.2
7.4.3
7.4.4
What is an Administration Group? .......................................................................................... 7-1
Developing an Administration Group ............................................................................. 7-3
Planning........................................................................................................................................ 7-3
Implementing Administration Groups and Template Collections ..................................... 7-9
Creating the Administration Group Hierarchy............................................................ 7-10
Accessing the Administration Group Home Page ............................................... 7-11
Defining the Hierarchy ............................................................................................. 7-11
Defining Template Collections ................................................................................ 7-15
Associating Template Collections with Administration Groups ....................... 7-18
Changing the Administration Group Hierarchy................................................................. 7-28
Adding a New Hierarchy Level ..................................................................................... 7-29
Removing a Hierarchy Level .......................................................................................... 7-29
Merging Administration Groups ................................................................................... 7-29
Removing Administration Groups ................................................................................ 7-32
8 Using Monitoring Templates
8.1
8.2
8.3
8.4
About Monitoring Templates....................................................................................................
Definition of a Monitoring Template .......................................................................................
Default Templates (Auto Apply Templates) ..........................................................................
Viewing a List of Monitoring Templates.................................................................................
8-1
8-2
8-2
8-2
ix
8.5
8.6
8.7
8.7.1
8.7.2
8.7.2.1
8.7.2.2
8.8
8.8.1
8.9
8.10
8.11
8.12
Creating a Monitoring Template .............................................................................................. 8-3
Editing a Monitoring Template ................................................................................................ 8-4
Applying Monitoring Templates to Targets ........................................................................... 8-4
Applying a Monitoring Template ..................................................................................... 8-4
Monitoring Template Application Options ..................................................................... 8-5
Apply Options............................................................................................................... 8-5
Metrics with Key Value Settings ................................................................................ 8-5
Comparing Monitoring Templates with Targets ................................................................... 8-7
When is a metric between a template and a target considered "different"? ............... 8-7
Comparing Metric Settings Using Information Publisher .................................................... 8-8
Exporting and Importing Monitoring Templates .................................................................. 8-9
Upgrading Enterprise Manager: Comparing Monitoring Templates .............................. 8-10
Changing the Monitoring Template Apply History Retention Period ............................ 8-10
9 Using Metric Extensions
9.1
What are Metric Extensions? ..................................................................................................... 9-1
9.2
Metric Extension Lifecycle......................................................................................................... 9-3
9.3
Working with Metric Extensions .............................................................................................. 9-6
9.3.1
Administrator Privilege Requirements ............................................................................ 9-6
9.3.2
Granting Create Metric Extension Privilege .................................................................... 9-7
9.3.3
Managing Administrator Privileges ................................................................................. 9-7
9.3.4
Managing Administrator Access to Metric Extensions.................................................. 9-8
9.3.4.1
Granting Full/Edit Privileges on a Metric Extension ............................................. 9-8
9.3.4.2
Revoking Access Privileges on a Metric Extension ................................................. 9-8
9.3.4.3
Transferring Metric Extension Ownership ............................................................... 9-9
9.3.5
Creating a New Metric Extension ..................................................................................... 9-9
9.3.6
Creating a New Metric Extension (Create Like) .......................................................... 9-14
9.3.7
Editing a Metric Extension .............................................................................................. 9-14
9.3.8
Creating the Next Version of an Existing Metric Extension....................................... 9-15
9.3.9
Importing a Metric Extension ......................................................................................... 9-15
9.3.10
Exporting a Metric Extension.......................................................................................... 9-16
9.3.11
Deleting a Metric Extension ............................................................................................ 9-16
9.3.12
Deploying Metric Extensions to a Group of Targets ................................................... 9-16
9.3.13
Creating an Incident Rule to Send Email from Metric Extensions ............................ 9-17
9.3.14
Updating Older Versions of Metric Extensions Already deployed to a Group of
Targets 9-17
9.3.15
Creating Repository-side Metric Extensions ................................................................ 9-18
9.4
Adapters .................................................................................................................................... 9-21
9.4.1
OS Command Adapter - Single Column....................................................................... 9-22
9.4.2
OS Command Adapter- Multiple Values...................................................................... 9-25
9.4.3
OS Command Adapter - Multiple Columns................................................................. 9-26
9.4.4
SQL Adapter...................................................................................................................... 9-27
9.4.5
SNMP (Simple Network Management Protocol) Adapter ......................................... 9-28
9.4.6
JMX Adapter...................................................................................................................... 9-28
9.5
Converting User-defined Metrics to Metric Extensions..................................................... 9-29
9.5.1
Overview............................................................................................................................ 9-30
9.5.2
Commands......................................................................................................................... 9-30
x
9.6
Metric Extension Command Line Verbs............................................................................... 9-34
10 Advanced Threshold Management
10.1
Accessing the Advanced Threshold Management Page ....................................................
10.2
Adaptive Thresholds ...............................................................................................................
10.2.1
Registering Adaptive Threshold Metrics ......................................................................
10.2.1.1
Standard Registration Method ................................................................................
10.2.1.2
Quick Configuration Method ..................................................................................
10.2.2
Configuring Adaptive Thresholds .................................................................................
10.2.3
Determining whether Adaptive Thresholds are Correct ............................................
10.2.4
Testing Adaptive Metric Thresholds .............................................................................
10.2.5
Deregistering Adaptive Threshold Metrics ..................................................................
10.3
Time-based Static Thresholds ..............................................................................................
10.3.1
Registering Time-based Static Thresholds ..................................................................
10.3.2
Deregistering Time-based Static Thresholds ..............................................................
10.4
Determining What is a Valid Metric Threshold ................................................................
10-1
10-2
10-3
10-3
10-6
10-6
10-7
10-9
10-9
10-10
10-10
10-12
10-12
11 Utilizing the Job System and Corrective Actions
11.1
Job System Purpose and Overview .......................................................................................
11.1.1
What Are Job Executions and Job Runs?.......................................................................
11.1.1.1
Job Executions ............................................................................................................
11.1.1.2
Job Runs ......................................................................................................................
11.1.2
Operations on Job Executions and Job Runs ................................................................
11.2
Preliminary Considerations....................................................................................................
11.2.1
Administrator Roles .........................................................................................................
11.2.2
Creating Scripts .................................................................................................................
11.2.3
Sharing Job Responsibilities ............................................................................................
11.2.4
Submitting Jobs for Groups.............................................................................................
11.3
Creating Jobs.............................................................................................................................
11.3.1
Selecting a Job Type..........................................................................................................
11.3.2
Creating an OS Command Job........................................................................................
11.3.2.1
Specifying a Single Operation................................................................................
11.3.2.2
Specifying a Script ...................................................................................................
11.3.2.3
Access Level Rules...................................................................................................
11.3.3
Creating a SQL Script Job ..............................................................................................
11.3.3.1
Specifying Targets ...................................................................................................
11.3.3.2
Specifying Options for the Parameters Page .......................................................
11.3.3.3
Specifying Host and Database Credentials..........................................................
11.3.3.4
Returning Error Codes from SQL Script Jobs......................................................
11.3.4
Creating a Multi-task Job...............................................................................................
11.3.4.1
Job Capabilities ........................................................................................................
11.3.4.2
Specifying Targets for a Multi-task Job ................................................................
11.3.4.3
Adding Tasks to the Job..........................................................................................
11.4
Viewing and Analyzing Job Status......................................................................................
11.5
Generating Job Event Criteria ..............................................................................................
11.5.1
Enabling Events For Job Status, Status Severity, and Targetless Jobs ....................
11-1
11-2
11-2
11-3
11-3
11-3
11-4
11-4
11-4
11-4
11-5
11-5
11-5
11-11
11-11
11-12
11-13
11-13
11-13
11-14
11-14
11-15
11-15
11-16
11-16
11-16
11-20
11-20
xi
11.5.2
Adding Targets To Generate Events For Job Status .................................................
11.6
Creating Event Rules For Job Status Change.....................................................................
11.6.1
Creating Job Status Change Event Rules For Jobs .....................................................
11.6.2
Creating Job Status Change Event Rules For Targets ...............................................
11.7
Using Diagnostic Tools .........................................................................................................
11.7.1
Enabling Job Logging.....................................................................................................
11.7.2
Viewing Job Logging......................................................................................................
11.7.3
Debugging a Failed Job .................................................................................................
11.7.4
Checking for Incidents Related to a Failed Job...........................................................
11.7.5
Packaging an Incident Generated by a Job Step.........................................................
11.7.6
Viewing Remote Log Files.............................................................................................
11.7.7
Diagnosing Problems with Cloud Control Management Tools ..............................
11.7.7.1
Health Overview .....................................................................................................
11.7.7.2
Repository Home Page ...........................................................................................
11.7.7.3
Management Services and Repository: All Metrics............................................
11.7.7.4
OMS and Repository: Diagnostic Metrics............................................................
11.7.7.5
OMS and Repository: Charts .................................................................................
11.7.7.6
Management Servers and Job Activity Details Pages ........................................
11.7.7.7
Job System Reports..................................................................................................
11.8
Creating Corrective Actions .................................................................................................
11.8.1
Providing Credentials ....................................................................................................
11.8.2
Creating Corrective Actions for Metrics......................................................................
11.8.3
Creating a Library Corrective Action ..........................................................................
11.8.4
Specifying Access to Corrective Actions .....................................................................
11.8.4.1
Defining or Modifying Access ...............................................................................
11.8.4.2
Access Level Rules...................................................................................................
11.8.5
Setting Up Notifications for Corrective Actions ........................................................
11.8.6
Providing Agent-side Response Actions.....................................................................
11.8.6.1
Specifying Commands and Scripts .......................................................................
11.8.6.2
Using Target Properties in Commands................................................................
11.8.6.3
Using Advanced Capabilities ................................................................................
11.8.7
Viewing the Details of a Corrective Action Execution..............................................
11-21
11-21
11-21
11-25
11-28
11-28
11-28
11-29
11-30
11-31
11-32
11-33
11-33
11-35
11-35
11-37
11-38
11-38
11-39
11-40
11-40
11-40
11-42
11-42
11-42
11-43
11-43
11-44
11-44
11-45
11-45
11-46
Part II Administering Cloud Control
12 Maintaining Enterprise Manager
12.1
12.2
12.2.1
12.2.2
12.3
12.3.1
12.3.2
12.3.3
12.4
12.4.1
12.4.2
xii
Overview: Managing the Manager .......................................................................................
Health Overview......................................................................................................................
Viewing Enterprise Manager Topology and Charts....................................................
Determining Enterprise Manager Page Performance .................................................
Repository ...............................................................................................................................
Repository Tab ................................................................................................................
Metrics Tab ......................................................................................................................
Schema Tab ......................................................................................................................
Controlling and Configuring Management Agents..........................................................
Manage Cloud Control Agents Page ...........................................................................
Agent Home Page...........................................................................................................
12-1
12-2
12-4
12-6
12-12
12-12
12-16
12-19
12-20
12-20
12-21
12.4.3
Controlling a Single Agent ............................................................................................
12.4.4
Configuring Single Management Agents....................................................................
12.4.5
Controlling Multiple Management Agents.................................................................
12.4.6
Configuring Multiple Agents........................................................................................
12.4.7
Upgrading Multiple Management Agents..................................................................
12.5
Management Servers .............................................................................................................
12-22
12-23
12-23
12-24
12-25
12-26
13 Maintaining and Troubleshooting the Management Repository
13.1
Management Repository Deployment Guidelines ............................................................. 13-1
13.2
Management Repository Data Retention Policies............................................................... 13-2
13.2.1
Management Repository Default Aggregation and Purging Policies....................... 13-2
13.2.2
Management Repository Default Aggregation and Purging Policies for Other
Management Data 13-4
13.2.3
Modifying the Default Aggregation and Purging Policies......................................... 13-4
13.2.4
How to Modify the Retention Period of Job History................................................... 13-5
13.2.5
DBMS_SCHEDULER Troubleshooting ......................................................................... 13-7
13.3
Dropping and Recreating the Management Repository .................................................... 13-8
13.3.1
Dropping the Management Repository......................................................................... 13-9
13.3.2
Recreating the Management Repository ..................................................................... 13-10
13.3.2.1
Using a Connect Descriptor to Identify the Management Repository Database ........
13-10
13.4
Troubleshooting Management Repository Creation Errors ............................................ 13-11
13.4.1
Package Body Does Not Exist Error While Creating the Management Repository...........
13-11
13.4.2
Server Connection Hung Error While Creating the Management Repository...... 13-11
13.4.3
General Troubleshooting Techniques for Creating the Management Repository 13-11
13.5
Cross Platform Enterprise Manager Repository Migration............................................. 13-13
13.5.1
Common Prerequisites................................................................................................... 13-13
13.5.2
Methodologies................................................................................................................. 13-14
13.5.2.1
Cross Platform Transportable Database............................................................... 13-14
13.5.2.2
Migration Using Physical Standby ....................................................................... 13-18
13.5.3
Post Migration Verification ........................................................................................... 13-20
14 Updating Cloud Control
14.1
14.1.1
14.2
14.2.1
14.2.2
14.2.3
14.2.4
14.2.5
14.2.6
14.3
14.3.1
14.3.2
14.4
Using Self Update ....................................................................................................................
What Can Be Updated? ...................................................................................................
Setting Up Self Update ............................................................................................................
Setting Up Enterprise Manager Self Update Mode .....................................................
Assigning Self Update Privileges to Users....................................................................
Setting Up the Software Library .....................................................................................
Setting My Oracle Support Preferred Credentials .......................................................
Registering the Proxy Details for My Oracle Support.................................................
Setting Up the EM CLI Utility (Optional) .....................................................................
Applying an Update ................................................................................................................
Applying an Update in Online Mode............................................................................
Applying an Update in Offline Mode............................................................................
Accessing Informational Updates .........................................................................................
14-1
14-1
14-2
14-2
14-3
14-3
14-3
14-3
14-4
14-5
14-5
14-6
14-7
xiii
14.5
Acquiring or Updating Management Agent Software....................................................... 14-8
15 Configuring a Software Library
15.1
Overview of Software Library ...............................................................................................
15.2
Users, Roles, and Privileges....................................................................................................
15.3
What’s New in Software Library ...........................................................................................
15.4
Performing Software Library Tasks Using EM CLI Verbs or in Graphical Mode .........
15.5
Software Library Storage ........................................................................................................
15.5.1
Upload File Locations ......................................................................................................
15.5.2
Referenced File Location................................................................................................
15.6
Prerequisites for Configuring Software Library................................................................
15.7
Configuring Software Library Storage Location ...............................................................
15.7.1
Configuring an OMS Shared Filesystem Location.....................................................
15.7.2
Configuring an OMS Agent Filesystem Location ......................................................
15.7.3
Configuring a Referenced File Location......................................................................
15.8
Configuring Software Library on a Multi-OMS System ..................................................
15.9
Using Software Library Entities...........................................................................................
15.10 Tasks Performed Using the Software Library Home Page ..............................................
15.10.1
Organizing Entities.........................................................................................................
15.10.2
Creating Entities..............................................................................................................
15.10.2.1
Creating Generic Components ..............................................................................
15.10.2.2
Creating Directives..................................................................................................
15.10.3
Customizing Entities ......................................................................................................
15.10.4
Managing Entities ...........................................................................................................
15.10.4.1
Accessing Software Library Home Page ..............................................................
15.10.4.2
Accessing Software Library Administration Page..............................................
15.10.4.3
Granting or Revoking Privileges...........................................................................
15.10.4.4
Moving Entities........................................................................................................
15.10.4.5
Changing Entity Maturity ......................................................................................
15.10.4.6
Adding Notes to Entities ........................................................................................
15.10.4.7
Adding Attachments to Entities ............................................................................
15.10.4.8
Viewing, Editing, and Deleting Entities...............................................................
15.10.4.9
Purging Deleted Entities.........................................................................................
15.10.4.10
Searching Entities ....................................................................................................
15.10.4.11
Exporting Entities ....................................................................................................
15.10.4.12
Importing Entities....................................................................................................
15.10.4.13
Staging Files Associated With an Entity ..............................................................
15.11 Maintaining Software Library..............................................................................................
15.11.1
Periodic Maintenance Tasks..........................................................................................
15.11.2
Re-Importing Oracle Owned Entity Files....................................................................
15.11.3
Removing (and Migrating) Software Library Storage Location ..............................
15.11.4
Removing a Referenced Storage Location...................................................................
15.11.5
Deactivating and Activating a Storage Location........................................................
15.11.6
Scheduling Purge Job .....................................................................................................
15.11.7
Backing Up Software Library........................................................................................
xiv
15-1
15-3
15-5
15-5
15-7
15-8
15-10
15-11
15-11
15-11
15-12
15-13
15-15
15-15
15-17
15-17
15-18
15-18
15-20
15-22
15-23
15-23
15-23
15-23
15-24
15-24
15-25
15-25
15-25
15-26
15-26
15-28
15-29
15-29
15-30
15-31
15-31
15-31
15-33
15-33
15-34
15-34
16 Managing Plug-Ins
16.1
Getting Started.......................................................................................................................... 16-1
16.2
Introduction to Plug-ins .......................................................................................................... 16-2
16.2.1
Enterprise Manager 12c Extensibility Paradigm.......................................................... 16-2
16.2.2
Plug-Ins .............................................................................................................................. 16-3
16.2.3
Plug-Ins Deployed by Default ........................................................................................ 16-3
16.2.4
Plug-In Releases ................................................................................................................ 16-4
16.2.5
Roles Required to Manage Plug-Ins............................................................................... 16-4
16.3
Workflow of Plug-In Deployment......................................................................................... 16-4
16.4
Introduction to Plug-In Manager........................................................................................... 16-8
16.4.1
Accessing Plug-In Manager............................................................................................. 16-8
16.4.2
Performing Operations Using Plug-In Manager.......................................................... 16-9
16.5
Knowing Your Plug-Ins .......................................................................................................... 16-9
16.5.1
Customizing Your View ................................................................................................ 16-10
16.5.1.1
Customizing Displayed Plug-Ins ......................................................................... 16-10
16.5.1.2
Customizing Displayed Columns ......................................................................... 16-10
16.5.2
Checking the Availability of Plug-Ins.......................................................................... 16-11
16.5.3
Viewing Information about Plug-Ins ........................................................................... 16-11
16.5.3.1
Differentiating Plug-In Releases from Enterprise Manager Platform Releases ..........
16-11
16.5.3.2
Identifying Plug-In ID............................................................................................. 16-12
16.5.3.3
Viewing Targets and Operating Systems Certified for Deployed Plug-Ins.... 16-12
16.5.3.4
Viewing Plug-In Dependencies ............................................................................. 16-12
16.5.3.5
Verifying Deployed Plug-Ins ................................................................................. 16-12
16.6
Downloading, Deploying, and Upgrading Plug-Ins ........................................................ 16-14
16.6.1
Downloading Plug-Ins ................................................................................................... 16-14
16.6.1.1
Downloading Plug-Ins in Online Mode............................................................... 16-14
16.6.1.2
Downloading Plug-Ins in Offline Mode .............................................................. 16-14
16.6.1.3
Importing Catalog Archives .................................................................................. 16-15
16.6.1.4
Importing Plug-In Archives ................................................................................... 16-16
16.6.2
Deploying Plug-Ins to Oracle Management Service (Reduce OMS Restart time and
Downtime) 16-17
16.6.2.1
Tracking the Deployment Status of Plug-Ins on Oracle Management Service ...........
16-20
16.6.3
Upgrading Plug-Ins Deployed to Oracle Management Service .............................. 16-20
16.6.3.1
Upgrading Across Plug-In Versions Deployed to Oracle Management Service ........
16-20
16.6.3.2
Upgrading Across Plug-In Revisions Within a Plug-In Version Deployed to Oracle
Management Service 16-20
16.6.4
Deploying Plug-Ins on Oracle Management Agent .................................................. 16-21
16.6.4.1
Tracking the Deployment Status of Plug-Ins on Oracle Management Agent 16-22
16.6.5
Upgrading Plug-Ins Deployed to Oracle Management Agent ................................ 16-22
16.7
Undeploying Plug-Ins ........................................................................................................... 16-22
16.7.1
Undeploying Plug-Ins from Oracle Management Service........................................ 16-22
16.7.2
Undeploying Plug-Ins from Oracle Management Agent ......................................... 16-23
16.8
Advanced Operations with Plug-Ins .................................................................................. 16-24
16.8.1
Re-deploying Plug-Ins on Oracle Management Agent ............................................. 16-24
xv
Deploying Plug-In Patches While Deploying or Upgrading Management Agent
(Create Custom Plug-In Update) 16-24
16.8.2.1
Creating Custom Plug-In Update Using EMCLI ................................................
16.8.2.2
Creating Custom Plug-In Update Using EDK ....................................................
16.9
Troubleshooting .....................................................................................................................
16.9.1
Understanding Plug-In Homes.....................................................................................
16.9.2
Troubleshooting OMS Plug-In Deployment and Upgrade Issues ..........................
16.9.2.1
Troubleshooting OMS Plug-In Deployment Issues............................................
16.9.2.2
Rollback and Resume OMS Plug-In Upgrade.....................................................
16.9.3
Troubleshooting Management Agent Plug-In Deployment and Upgrade Issues
16.9.3.1
Troubleshooting Management Agent Plug-In Deployment Issues .................
16.9.3.2
Troubleshooting Management Agent Plug-In Upgrade Issues........................
16.8.2
16-25
16-27
16-28
16-28
16-30
16-30
16-31
16-31
16-31
16-32
17 Patching Oracle Management Service and the Repository
17.1
17.1.1
17.1.2
17.1.3
17.1.4
17.2
17.2.1
17.3
17.4
17.4.1
17.4.2
17.4.3
17.4.4
17.4.5
17.4.6
OPatch Automation .................................................................................................................
Supported OMS Configurations and OPatchauto Patchability .................................
OUI Inventory Configurations .......................................................................................
Supported Patch Format..................................................................................................
Supported Patching Methodologies ..............................................................................
Required OPatchauto Parameters .........................................................................................
Creating a Property File...................................................................................................
Prerequisites for Running OPatchauto .................................................................................
Using OPatchauto ....................................................................................................................
My Oracle Support: Searching for Patches ................................................................
Running opatchauto apply ....................................................................................
Running opatchauto rollback .............................................................................
Running opatchauto lspatches...........................................................................
Running opatchauto version................................................................................
Patching a Standby OMS System .................................................................................
17-1
17-1
17-2
17-3
17-3
17-4
17-4
17-6
17-9
17-10
17-14
17-15
17-15
17-16
17-16
OPatchauto Command Syntax............................................................................................. 17-17
Apply ................................................................................................................................ 17-19
Rollback ............................................................................................................................ 17-22
lspatches ........................................................................................................................... 17-25
version .............................................................................................................................. 17-26
checkApplicable .............................................................................................................. 17-27
saveConfigurationSnapshot ......................................................................................... 17-28
Standby OMS Patching ......................................................................................................... 17-30
Troubleshooting ..................................................................................................................... 17-31
OPatchauto Troubleshooting Architecture ................................................................. 17-32
OPatchauto Log Management Architecture ............................................................... 17-33
Logs for Oracle Support................................................................................................. 17-36
OPatchauto: Cases Analysis, Error Codes, and Remedies/Suggestions................ 17-37
OPatchauto: External Utilities Error Codes ................................................................ 17-39
Special Error Cases for OPatchauto OMS Automation............................................. 17-40
Multi-OMS Execution for UNIX based Systems ....................................................... 17-43
xvi
Features in OPatchauto Release 11.1.0.11.0 and Above ................................................... 17-46
Resume capability in Single-OMS Configuration ...................................................... 17-47
Resume Capability in Multi-OMS Configuration ...................................................... 17-51
18
Patching Oracle Management Agents
18.1
Overview...................................................................................................................................
18.2
Automated Management Agent Patching Using Patch Plans (Recommended) ............
18.2.1
Advantages of Automated Management Agent Patching .........................................
18.2.2
Accessing the Patches and Updates Page .....................................................................
18.2.3
Viewing Patch Recommendations .................................................................................
18.2.4
Searching for Patches .......................................................................................................
18.2.4.1
Searching for Patches On My Oracle Support.......................................................
18.2.4.2
Searching for Patches in Software Library.............................................................
18.2.5
Applying Management Agent Patches..........................................................................
18.2.6
Verifying the Applied Management Agent Patches....................................................
18.2.7
Management Agent Patching Errors .............................................................................
18.2.7.1
Oracle Home Credentials Are Not Set ...................................................................
18.2.7.2
Management Agent Target Is Down ....................................................................
18.2.7.3
Patch Conflicts Are Detected .................................................................................
18.2.7.4
User Is Not a Super User ........................................................................................
18.2.7.5
Patch Is Not Staged or Found ................................................................................
18.3
Manual Management Agent Patching ................................................................................
18-1
18-1
18-2
18-2
18-3
18-3
18-3
18-4
18-5
18-9
18-9
18-9
18-10
18-10
18-11
18-12
18-12
19 Personalizing Cloud Control
19.1
19.2
19.3
20
Personalizing a Cloud Control Page ..................................................................................... 19-1
Customizing a Region ............................................................................................................. 19-2
Setting Your Homepage.......................................................................................................... 19-3
Starting and Stopping Enterprise Manager Components
20.1
Controlling the Oracle Management Agent......................................................................... 20-1
20.1.1
Starting, Stopping, and Checking the Status of the Management Agent on UNIX 20-1
20.1.2
Starting and Stopping the Management Agent on Windows .................................... 20-3
20.1.3
Checking the Status of the Management Agent on Windows ................................... 20-4
20.1.4
Troubleshooting Management Agent Startup Errors.................................................. 20-4
20.1.4.1
Management Agent starts up but is not ready...................................................... 20-4
20.1.4.2
Management Agent fails to start due to time zone mismatch between agent and
OMS 20-5
20.1.4.3
Management Agent fails to start due to possible port conflict .......................... 20-5
20.1.4.4
Management Agent fails to start due to failure of securing or unsecuring ...... 20-5
20.2
Controlling the Oracle Management Service....................................................................... 20-5
20.2.1
Controlling the Management Service on UNIX ........................................................... 20-6
20.2.2
Controlling the Management Service on Windows..................................................... 20-7
20.2.3
Troubleshooting Oracle Management Service Startup Errors ................................... 20-7
20.3
Setting Java Memory Arguments for Oracle Management Service ................................. 20-8
20.4
Controlling JVMD and ADP Engines ................................................................................... 20-8
20.4.1
Controlling JVMD Engines.............................................................................................. 20-8
xvii
20.4.2
Controlling ADP Engines ................................................................................................
20.5
Guidelines for Starting Multiple Enterprise Manager Components on a Single Host
20.6
Starting and Stopping Oracle Enterprise Manager 12c Cloud Control .........................
20.6.1
Starting Cloud Control and All Its Components .......................................................
20.6.2
Stopping Cloud Control and All Its Components .....................................................
20.7
Additional Management Agent Commands .....................................................................
20.7.1
Uploading and Reloading Data to the Management Repository ............................
20.7.2
Specifying New Target Monitoring Credentials ........................................................
20.7.3
Listing the Targets on a Managed Host.......................................................................
20.7.4
Changing the Management Agent Time Zone ...........................................................
20.7.5
Reevaluating Metric Collections...................................................................................
20-9
20-10
20-10
20-10
20-11
20-12
20-12
20-13
20-13
20-14
20-14
21 Enterprise Manager Command Line Utility Commands
21.1
Executing EMCTL Commands ..............................................................................................
21.2
EMCTL Commands for OMS.................................................................................................
21.3
EMCTL Commands for Management Agent.......................................................................
21.4
EMTCL Security Commands .................................................................................................
21.4.1
EMCTL Secure Commands ...........................................................................................
21.4.2
Security diagnostic commands .....................................................................................
21.4.3
EMCTL EM Key Commands ........................................................................................
21.4.4
Configuring Authentication..........................................................................................
21.4.4.1
Configuring OSSO Authentication .......................................................................
21.4.4.2
Configuring OAM Authentication........................................................................
21.4.4.3
Configuring LDAP (OID and AD) Authentication ............................................
21.4.4.4
Configuring Repository Authentication (Default Authentication)..................
21.5
EMCTL HAConfig Commands ...........................................................................................
21.6
EMCTL Resync Commands .................................................................................................
21.7
EMCTL Connector Command .............................................................................................
21.8
EMCTL Patch Repository Commands................................................................................
21.9
EMCTL Commands for Windows NT ................................................................................
21.10 EMCTL Partool Commands .................................................................................................
21.11 EMCTL Plug-in Commands .................................................................................................
21.12 Syncing with OPSS Policy Store ..........................................................................................
21.13 Using emctl.log File to Troubleshoot ..................................................................................
22
Locating and Configuring Enterprise Manager Log Files
22.1
Managing Log Files .................................................................................................................
22.1.1
Viewing Log Files and Their Messages .........................................................................
22.1.1.1
Restricting Access to the View Log Messages Menu Item and Functionality ..
22.1.2
Searching Log Files...........................................................................................................
22.1.2.1
Searching Log Files: Basic Searches ........................................................................
22.1.2.2
Searching Log Files: Advanced Searches ...............................................................
22.1.3
Downloading Log Files....................................................................................................
22.2
Managing Saved Searches ......................................................................................................
22.2.1
Saving Searches .................................................................................................................
22.2.2
Retrieving Saved Searches...............................................................................................
22.2.3
Managing Saved Searches ...............................................................................................
xviii
21-1
21-1
21-7
21-9
21-10
21-12
21-13
21-14
21-15
21-15
21-16
21-16
21-16
21-18
21-18
21-19
21-19
21-19
21-20
21-21
21-21
22-1
22-3
22-4
22-5
22-5
22-6
22-7
22-8
22-8
22-9
22-9
22.3
Locating Management Agent Log and Trace Files ........................................................... 22-10
22.3.1
About the Management Agent Log and Trace Files.................................................. 22-10
22.3.1.1
Structure of Agent Log Files .................................................................................. 22-10
22.3.2
Locating the Management Agent Log and Trace Files.............................................. 22-11
22.3.3
Setting Oracle Management Agent Log Levels.......................................................... 22-11
22.3.3.1
Modifying the Default Logging Level .................................................................. 22-12
22.3.3.2
Setting gcagent.log .................................................................................................. 22-13
22.3.3.3
Setting gcagent_error.log........................................................................................ 22-13
22.3.3.4
Setting the Log Level for Individual Classes and Packages.............................. 22-13
22.3.3.5
Setting gcagent_mdu.log ........................................................................................ 22-13
22.3.3.6
Setting the TRACE Level ........................................................................................ 22-15
22.4
Locating and Configuring Oracle Management Service Log and Trace Files .............. 22-16
22.4.1
About the Oracle Management Service Log and Trace Files ................................... 22-16
22.4.2
Locating Oracle Management Service Log and Trace Files...................................... 22-17
22.4.3
Controlling the Size and Number of Oracle Management Service Log and Trace Files...
22-17
22.4.4
Controlling the Contents of the Oracle Management Service Trace File ............... 22-18
22.4.5
Controlling the Oracle WebLogic Server and Oracle HTTP Server Log Files ....... 22-19
22.5
Monitoring Log Files ............................................................................................................. 22-21
22.5.1
About Log Viewer .......................................................................................................... 22-21
22.5.2
Overview of WebLogic Server and Application Deployment Log File Monitoring .........
22-22
22.5.3
Enabling Log File Monitoring....................................................................................... 22-23
22.5.4
Configuring Log File Monitoring ................................................................................. 22-23
22.5.5
Viewing Alerts from Log File Monitoring .................................................................. 22-25
22.6
Configuring Log Archive Locations.................................................................................... 22-25
23 Configuring and Using Services
23.1
Introduction to Services ..........................................................................................................
23.1.1
Defining Services in Enterprise Manager......................................................................
23.2
Creating a Service ....................................................................................................................
23.2.1
Creating a Generic Service - Test Based ........................................................................
23.2.2
Creating a Generic Service - System Based...................................................................
23.2.3
Creating an Aggregate Service .......................................................................................
23.3
Monitoring a Service................................................................................................................
23.3.1
Viewing the Generic / Aggregate Service Home Page...............................................
23.3.2
Viewing the Performance / Incidents Page..................................................................
23.3.3
Viewing the SLA Dashboard ..........................................................................................
23.3.4
Viewing the Test Summary .............................................................................................
23.3.5
Viewing the Service Topology ........................................................................................
23.3.6
Sub Services .......................................................................................................................
23.4
Configuring a Service ..............................................................................................................
23.4.1
Availability Definition (Generic and Aggregate Service) ...........................................
23.4.2
Root Cause Analysis Configuration...............................................................................
23.4.2.1
Getting the Most From Root Cause Analysis ........................................................
23.4.3
System Association.........................................................................................................
23.4.4
Monitoring Settings .......................................................................................................
23-1
23-2
23-2
23-3
23-4
23-4
23-5
23-5
23-6
23-6
23-6
23-7
23-7
23-7
23-8
23-9
23-9
23-10
23-11
xix
23.4.5
Service Tests and Beacons .............................................................................................
23.4.5.1
Defining Additional Service Tests ........................................................................
23.4.5.2
Deploying and Using Beacons...............................................................................
23.4.5.3
Configuring the Beacons ........................................................................................
23.4.5.4
Creating an ATS Service Test Using OATS Load Script ....................................
23.4.6
Performance Metrics ......................................................................................................
23.4.6.1
Rule Based Target List ............................................................................................
23.4.6.2
Static Based Target List...........................................................................................
23.4.7
Usage Metrics ..................................................................................................................
23.5
Using the Transaction Recorder...........................................................................................
23.6
Setting Up and Using Service Level Agreements .............................................................
23.6.1
Actionable Item Rules for SLAs....................................................................................
23.6.2
Creating a Service Level Objective...............................................................................
23.6.3
Lifecycle of an SLA .........................................................................................................
23.6.4
Viewing the Status of SLAs for a Service ....................................................................
23.6.5
Defining Custom SLA Business Calendars.................................................................
23.7
Using the Services Dashboard .............................................................................................
23.7.1
Viewing the All Dashboards Page ..............................................................................
23.7.2
Viewing the Dashboard Details Page ..........................................................................
23.7.3
Customizing and Personalizing the Dashboard ........................................................
23.7.4
Viewing the Dashboard Service Details Page.............................................................
23.8
Using the Test Repository.....................................................................................................
23.8.1
Viewing the Test Repository .........................................................................................
23.8.2
Editing an ATS Script.....................................................................................................
23.9
Configuring Service Levels...................................................................................................
23.9.1
Defining Service Level Rules ........................................................................................
23.9.2
Viewing Service Level Details ......................................................................................
23.10 Configuring a Service Using the Command Line Interface.............................................
23.11 Troubleshooting Service Tests .............................................................................................
23.11.1
Verifying and Troubleshooting Forms Transactions.................................................
23.11.1.1
Troubleshooting Forms Transaction Playback....................................................
23.11.1.2
Troubleshooting Forms Transaction Recording .................................................
23.11.2
Verifying and Troubleshooting Web Transactions....................................................
23-11
23-12
23-13
23-15
23-16
23-22
23-24
23-24
23-24
23-25
23-25
23-27
23-28
23-29
23-30
23-31
23-31
23-32
23-32
23-33
23-33
23-34
23-35
23-35
23-36
23-37
23-37
23-38
23-41
23-41
23-41
23-43
23-44
24 Introducing Enterprise Manager Support for SNMP
24.1
24.2
24.3
24.4
24.4.1
24.5
24.5.1
24.5.2
24.6
Part III
xx
Benefits of SNMP Support......................................................................................................
About the SNMP Management Station ................................................................................
How Enterprise Manager Supports SNMP..........................................................................
Sending SNMP Trap Notifications ........................................................................................
About the Management Information Base (MIB).........................................................
Monitoring External Devices Using SNMP .........................................................................
About SNMP Receivelets.................................................................................................
About SNMP Fetchlets.....................................................................................................
About Metric Extensions.........................................................................................................
Security
24-1
24-2
24-2
24-4
24-5
24-5
24-5
24-6
24-6
25 Configuring Security
Part IV Generating Reports
26 Using Information Publisher
26.1
26.2
26.3
26.3.1
26.3.2
26.3.3
26.4
26.4.1
26.4.2
26.4.3
26.5
About Information Publisher .................................................................................................
Out-of-Box Report Definitions ...............................................................................................
Custom Reports........................................................................................................................
Creating Custom Reports ................................................................................................
Report Parameters ............................................................................................................
Report Elements ................................................................................................................
Scheduling Reports..................................................................................................................
Flexible Schedules.............................................................................................................
Storing and Purging Report Copies ...............................................................................
E-mailing Reports .............................................................................................................
Sharing Reports ........................................................................................................................
26-1
26-2
26-2
26-2
26-3
26-3
26-3
26-3
26-4
26-4
26-4
27 Creating Usage Tracking Reports
27.1
Usage Tracking Reports ..........................................................................................................
27.2
Collecting Data for Database Usage Tracking .....................................................................
27.2.1
Setting Database Usage Tracking Credentials..............................................................
27.2.2
Enabling/Disabling the Metric Collection using Monitoring Templates ................
27.2.3
Enabling/Disabling the Metric Collection using the Command Line Interface .....
27.2.3.1
Setting up EM CLI login ...........................................................................................
27.2.3.2
Enabling/disabling the metric collection...............................................................
27.2.3.3
Using EM CLI to list all the database targets ........................................................
27.2.3.4
Using SQL to verify collection status......................................................................
27.2.4
Creating a Database Usage Tracking Report ..............................................................
27.3
Generating Database Usage Tracking Report....................................................................
27.3.1
Configuring Business Intelligence Publisher (BI Publisher) ....................................
27.3.2
Running Usage Tracking Reports: ...............................................................................
27.4
Database Usage Tracking Summary Report ......................................................................
27.5
Fusion Middleware Usage Tracking Summary Report....................................................
27.6
Host Usage Tracking Reports ..............................................................................................
27.6.1
Host Usage Tracking Summary Report.......................................................................
27.6.2
Host Usage Tracking Details Report............................................................................
27-1
27-2
27-2
27-3
27-7
27-7
27-7
27-9
27-9
27-10
27-11
27-11
27-14
27-15
27-17
27-19
27-19
27-20
Part V Accessing Enterprise Manager via Mobile Devices
28 Remote Access To Enterprise Manager
28.1
28.2
28.3
28.4
28.5
Reviewing System Requirements ..........................................................................................
Performing Initial Setup..........................................................................................................
Connecting the First Time ......................................................................................................
Encountering the Login Screen .............................................................................................
Managing Settings ...................................................................................................................
28-1
28-1
28-2
28-2
28-3
xxi
28.6
28.7
28.7.1
28.7.2
28.7.3
28.8
28.9
Using Cloud Control Mobile in Incident Manager .............................................................
Working in Cloud Control Mobile ........................................................................................
Viewing Incidents and Problems ...................................................................................
Changing Views................................................................................................................
Performing Actions ..........................................................................................................
Learning Tips and Tricks ........................................................................................................
Connecting to Enterprise Manager Desktop Version.........................................................
Part VI
Configuring Enterprise Manager for High Availability
Part VII
Appendixes
28-4
28-6
28-6
28-8
28-8
28-8
28-9
A.1
oraEMNGEvent.......................................................................................................................... A-1
A.1.1
oraEMNGEventIndex......................................................................................................... A-3
A.1.2
oraEMNGEventNotifType ................................................................................................ A-4
A.1.3
oraEMNGEventMessage ................................................................................................... A-4
A.1.4
oraEMNGEventMessageURL ........................................................................................... A-4
A.1.5
oraEMNGEventSeverity .................................................................................................... A-5
A.1.6
oraEMNGEventSeverityCode........................................................................................... A-5
A.1.7
oraEMNGEventRepeatCount ........................................................................................... A-5
A.1.8
oraEMNGEventActionMsg ............................................................................................... A-6
A.1.9
oraEMNGEventOccurrenceTime ..................................................................................... A-6
A.1.10
oraEMNGEventReportedTime ......................................................................................... A-6
A.1.11
oraEMNGEventCategories................................................................................................ A-7
A.1.12
oraEMNGEventCategoryCodes ....................................................................................... A-7
A.1.13
oraEMNGEventType.......................................................................................................... A-7
A.1.14
oraEMNGEventName........................................................................................................ A-8
A.1.15
oraEMNGAssocIncidentId ................................................................................................ A-8
A.1.16
oraEMNGAssocIncidentOwner........................................................................................ A-9
A.1.17
oraEMNGAssocIncidentAcked ........................................................................................ A-9
A.1.18
oraEMNGAssocIncidentStatus ......................................................................................... A-9
A.1.19
oraEMNGAssocIncidentPriority .................................................................................... A-10
A.1.20
oraEMNGAssocIncidentEscLevel .................................................................................. A-10
A.1.21
oraEMNGEventTargetName .......................................................................................... A-10
A.1.22
oraEMNGEventTargetNameURL .................................................................................. A-11
A.1.23
oraEMNGEventTargetType ............................................................................................ A-11
A.1.24
oraEMNGEventHostName ............................................................................................. A-11
A.1.25
oraEMNGEventTargetOwner......................................................................................... A-12
A.1.26
oraEMNGEventTgtLifeCycleStatus ............................................................................... A-12
A.1.27
oraEMNGEventTargetVersion ....................................................................................... A-12
A.1.28
oraEMNGEventUserDefinedTgtProp............................................................................ A-12
A.1.29
oraEMNGEventSourceObjName.................................................................................... A-13
A.1.30
oraEMNGEventSourceObjNameURL ........................................................................... A-13
A.1.31
oraEMNGEventSourceObjType ..................................................................................... A-13
A.1.32
oraEMNGEventSourceObjSubType............................................................................... A-14
A.1.33
oraEMNGEventSourceObjOwner .................................................................................. A-14
A.1.34
oraEMNGEventCAJobName .......................................................................................... A-14
A.1.35
oraEMNGEventCAJobStatus .......................................................................................... A-15
xxii
A.1.36
oraEMNGEventCAJobOwner......................................................................................... A-15
A.1.37
oraEMNGEventCAJobStepOutput ................................................................................ A-15
A.1.38
oraEMNGEventCAJobType ............................................................................................ A-16
A.1.39
oraEMNGEventRuleSetName ........................................................................................ A-16
A.1.40
oraEMNGEventRuleName.............................................................................................. A-16
A.1.41
oraEMNGEventRuleOwner ............................................................................................ A-17
A.1.42
oraEMNGEventSequenceId ............................................................................................ A-17
A.1.43
oraEMNGEventRCADetails............................................................................................ A-17
A.1.44
oraEMNGEventContextAttrs.......................................................................................... A-18
A.1.45
oraEMNGEventUserComments ..................................................................................... A-18
A.1.46
oraEMNGEventUpdates.................................................................................................. A-18
A.1.47
oraEMNGEventTypeAttr(1-71) ...................................................................................... A-19
A.2
oraEM4AlertTable.................................................................................................................... A-22
A.2.1
oraEM4AlertTargetName ................................................................................................ A-23
A.2.2
oraEM4AlertTargetType.................................................................................................. A-24
A.2.3
oraEM4AlertHostName ................................................................................................... A-24
A.2.4
oraEM4AlertMetricName ................................................................................................ A-24
A.2.5
oraEM4AlertKeyName .................................................................................................... A-25
A.2.6
oraEM4AlertKeyValue..................................................................................................... A-25
A.2.7
oraEM4AlertTimeStamp.................................................................................................. A-26
A.2.8
oraEM4AlertSeverity........................................................................................................ A-26
A.2.9
oraEM4AlertMessage ....................................................................................................... A-27
A.2.10
oraEM4AlertRuleName ................................................................................................... A-27
A.2.11
oraEM4AlertRuleOwner.................................................................................................. A-28
A.2.12
oraEM4AlertMetricValue ................................................................................................ A-28
A.2.13
oraEM4AlertContext ........................................................................................................ A-28
A.2.14
oraEM4AlertCycleGuid ................................................................................................... A-29
A.2.15
oraEM4AlertRepeatCount ............................................................................................... A-29
A.2.16
oraEM4AlertUDTargetProperties .................................................................................. A-30
A.2.17
oraEM4AlertAck ............................................................................................................... A-30
A.2.18
oraEM4AlertAckBy .......................................................................................................... A-31
A.2.19
oraEM4AlertNotifType.................................................................................................... A-31
A.2.20
oraEM4AlertViolationGuid............................................................................................. A-32
A.3
oraEM4JobAlertTable .............................................................................................................. A-32
A.3.1
oraEM4JobAlertJobName ................................................................................................ A-33
A.3.2
oraEM4JobAlertJobOwner .............................................................................................. A-33
A.3.3
oraEM4JobAlertJobType.................................................................................................. A-34
A.3.4
oraEM4JobAlertJobStatus ................................................................................................ A-34
A.3.5
oraEM4JobAlertTargets ................................................................................................... A-34
A.3.6
oraEM4JobAlertTimeStamp ............................................................................................ A-35
A.3.7
oraEM4JobAlertRuleName.............................................................................................. A-35
A.3.8
oraEM4JobAlertRuleOwner ........................................................................................... A-36
A.3.9
oraEM4JobAlertMetricName .......................................................................................... A-36
A.3.10
oraEM4JobAlertMetricValue .......................................................................................... A-37
A.3.11
oraEM4JobAlertContext................................................................................................... A-37
A.3.12
oraEM4JobAlertKeyName............................................................................................... A-38
A.3.13
oraEM4JobAlertKeyValue ............................................................................................... A-38
xxiii
A.3.14
A.3.15
A.3.16
B.1
C.1
C.2
C.3
C.4
C.5
D.1
D.2
Index
xxiv
oraEM4JobAlertSeverity .................................................................................................
oraEM4JobAlertJobId .......................................................................................................
oraEM4JobAlertJobExecId...............................................................................................
MIB Definition ............................................................................................................................
Pre-12c Enterprise Manager Metric Alerts .............................................................................
Pre-12C Target Availability Alerts .........................................................................................
Pre-12C Corrective Action Results for Metric Alerts............................................................
Corrective Action Results for Target Availability.................................................................
Job Status Change ......................................................................................................................
Target Availability State Changes ...........................................................................................
Target Status: Real-time Updates ............................................................................................
A-38
A-39
A-39
B-1
C-1
C-2
C-2
C-3
C-4
D-1
D-3
Preface
This guide describes how to use Oracle Enterprise Manager Cloud Control 12c core
functionality.
The preface covers the following:
в– Audience
в– Documentation Accessibility
в– Related Documents
в– Conventions
Audience
This document is intended for Enterprise Manager administrators and developers who
want to manage their Enterprise Manager infrastructure.
Documentation Accessibility
For information about Oracle's commitment to accessibility, visit the Oracle
Accessibility Program website at
http://www.oracle.com/pls/topic/lookup?ctx=acc&id=docacc
Access to Oracle Support
Oracle customers have access to electronic support through My Oracle Support. For
information, visit http://www.oracle.com/pls/topic/lookup?ctx=acc&id=info or
visit http://www.oracle.com/pls/topic/lookup?ctx=acc&id=trs if you are hearing
impaired.
Related Documents
For the latest releases of these and other Oracle documentation, check the Oracle
Technology Network at:
http://www.oracle.com/technetwork/documentation/index.html#em
Oracle Enterprise Manager also provides extensive Online Help. From the user menu
at the top of any Enterprise Manager page, select Help to display the online help
window.
xxv
Conventions
The following text conventions are used in this document:
xxvi
Convention
Meaning
boldface
Boldface type indicates graphical user interface elements associated
with an action, or terms defined in text or the glossary.
italic
Italic type indicates book titles, emphasis, or placeholder variables for
which you supply particular values.
monospace
Monospace type indicates commands within a paragraph, URLs, code
in examples, text that appears on the screen, or text that you enter.
What's Changed in this Guide?
The revisions listed below identify information updates, structural changes, as well as
relocation of information to other guides.
Since the last revision, the following changes have been made:
в– Modified: Chapter on Advanced Thrreshold Management
в– Added: Customizing alert messages
в– Modified: Various technical and procedural updates.
xxvii
xxviii
Part I
Part I
Monitoring and Managing Targets
This section contains the following chapters:
в– Enterprise Monitoring
в– Discovering, Promoting, and Adding Targets
в– Using Incident Management
в– Using Notifications
в– Using Blackouts
в– Managing Groups
в– Using Administration Groups
в– Using Monitoring Templates
в– Using Metric Extensions
в– Advanced Threshold Management
в– Utilizing the Job System and Corrective Actions
1
Enterprise Monitoring
1
This chapter covers the following topics:
в– Monitoring Overview
в– Monitoring: Basics
в– Monitoring: Advanced Setup
в– Notifications
в– Managing Events, Incidents, and Problems
в– Accessing Monitoring Information
1.1 Monitoring Overview
Enterprise Manager Cloud Control monitoring functionality permits unattended
monitoring of your IT environment. Enterprise Manager comes with a comprehensive
set of performance and health metrics that allows monitoring of key components in
your environment, such as applications, application servers, databases, as well as the
back-end components on which they rely (such as hosts, operating systems, storage).
The Management Agent on each monitored host monitors the status, health, and
performance of all managed components (targets) on that host. If a target goes down,
or if a performance metric crosses a warning or critical threshold, an event is triggered
and sent to Enterprise Manager. Administrators or any interested party can be notified
of the triggered event through the Enterprise Manager notification system.
Adding targets to monitor is simple. Enterprise Manager provides you with the option
of either adding targets manually or automatically discovering all targets on a host.
Enterprise Manager can also automatically and intelligently apply monitoring settings
for newly added targets. For more information, see Section 1.4.2, "Administration
Groups and Template Collections"). While Enterprise Manager provides a
comprehensive set of metrics used for monitoring, you can also use metric extensions
(see Section 1.3.6, "Metric Extensions: Customizing Monitoring") to monitor conditions
that are specific to your environment. As your data center grows, it will become more
challenging to manage individual targets separately, thus you can use Enterprise
Manager's group management functionality to organize large sets of targets into
groups, allowing you to monitor and manage many targets as one.
1.2 Comprehensive Out-of-Box Monitoring
Monitoring begins as soon as you install Enterprise Manager Cloud Control 12c.
Enterprise Manager's Management Agents automatically start monitoring their host’s
systems (including hardware and software configuration data on these hosts) as soon
Enterprise Monitoring 1-1
Monitoring: Basics
as they are deployed and started. Enterprise Manager provides auto-discovery scripts
that enable these Agents to automatically discover all Oracle components and start
monitoring them using a comprehensive set of metrics at Oracle-recommended
thresholds.
This monitoring functionality includes other components of the Oracle ecosystem such
as NetApp Filer, BIG-IP load balancers, Checkpoint Firewall, and IBM WebSphere.
Metrics from all monitored components are stored and aggregated in the Management
Repository, providing administrators with a rich source of diagnostic information and
trend analysis data. When critical alerts are detected, notifications are sent to
administrators for rapid resolution.
Out-of-box, Enterprise Manager monitoring functionality provides:
в– в– в– в– в– In-depth monitoring with Oracle-recommended metrics and thresholds.
Monitoring of all components of your IT infrastructure (Oracle and non-Oracle) as
well as the applications and services that are running on them.
Access to real-time performance charts.
Collection, storage, and aggregation of metric data in the Management Repository.
This allows you to perform strategic tasks such as trend analysis and reporting.
E-mail and pager notifications for detected critical events.
Enterprise Manager can monitor a wide variety of components (such as databases,
hosts, and routers) within your IT infrastructure.
Some examples of monitored metrics are:
в– Archive Area Used (Database)
в– Component Memory Usage (Application Server)
в– Segments Approaching Maximum Extents Count (Database)
в– Network Interface Total I/O Rate (Host)
Monitoring Without Management Agents
When it is not practical to have a Management Agent present to monitor specific
components of your IT infrastructure, as might be the case with an IP traffic controller
or remote Web application, Enterprise Manager provides Extended Network and
Critical URL Monitoring functionality. This feature allows the Beacon functionality of
the Agent to monitor remote network devices and URLs for availability and
responsiveness without requiring an Agent to be physically present on that device.
You simply select a specific Beacon, and add key network components and URLs to
the Network and URL Watch Lists. More information about using this feature is
available in the Enterprise Manager online help and from the Oracle Technology
Network Web site. Enterprise Manager monitoring concepts and the underlying
subsystems that support this functionality are discussed in the following sections.
1.3 Monitoring: Basics
Enterprise Manager Cloud Control 12c comes with a comprehensive set of predefined
performance and health metrics that enables automated monitoring of key
components in your environment, such as applications, application servers, databases,
as well as the back-end components on which they rely, such as hosts, operating
systems, storage. While Enterprise Manager can monitor for many types of conditions
(events), the most common use of its monitoring capability centers around the basics
of monitoring for violation of acceptable performance boundaries defined by metric
1-2 OracleВ® Enterprise Manager Administration
Monitoring: Basics
values. The following sections discuss the basic concepts and Enterprise Manger
functionality that supports monitoring of targets.
1.3.1 Metric Thresholds: Determining When a Monitored Condition is an Issue
Some metrics have associated predefined limiting parameters called thresholds that
cause metric alerts (specific type of event) to be triggered when collected metric values
exceed these limits. Enterprise Manager allows you to set metric threshold values for
two levels of alert severity:
в– в– Warning - Attention is required in a particular area, but the area is still functional.
Critical - Immediate action is required in a particular area. The area is either not
functional or indicative of imminent problems.
Hence, thresholds are boundary values against which monitored metric values are
compared. For example, for each disk device associated with the Disk Utilization (%)
metric, you might define a warning threshold at 80% disk space used and critical
threshold at 95%.
Not all metrics need a threshold: If the values do not make
sense, or are not needed in a particular environment, they can be
removed or simply not set.
Note:
While the out-of-box predefined metric threshold values will work for most
monitoring conditions, your environment may require that you customize threshold
values to more accurately reflect the operational norms of your environment. Setting
accurate threshold values, however, may be more challenging for certain categories of
metrics such as performance metrics.
For example, what are appropriate warning and critical thresholds for the Response
Time Per Transaction database metric? For such metrics, it might make more sense to be
alerted when the monitored values for the performance metric deviates from normal
behavior. Enterprise Manager provides features to enable you to capture normal
performance behavior for a target and determine thresholds that are deviations from
that performance norm.
Enterprise Manager administrators must be granted Manage
Target Metrics or greater privilege on a target in order to perform any
metric threshold changes.
Note:
1.3.2 Metric Baselines: Determining Valid Metric Thresholds
Determining what metric threshold values accurately reflect the performance
monitoring needs of your environment is not trivial. Rather than relying on trial and
error to determine the correct values, Enterprise Manager provides metric baselines.
Metric baselines are well-defined time intervals (baseline periods) over which
Enterprise Manager has captured system performance metrics, creating statistical
characterizations of system performance over specific time periods. This historical
data greatly simplifies the task of determining valid metric threshold values by
providing normalized views of system performance. Baseline normalized views of
metric behavior help administrators explain and understand event occurrences.
The underlying assumption of metric baselines is that systems with relatively stable
performance should exhibit similar metric observations (values) over times of
comparable workload. Two types of baseline periods are supported:
Enterprise Monitoring 1-3
Monitoring: Basics
в– в– Moving Window Baseline Periods: Moving window baseline periods are defined
as some number of days prior to the current date (Example: Last 7 days). This
allows comparison of current metric values with recently observed history.
Moving window baselines are useful for operational systems with predictable
workload cycles (Example: OLTP days and batch nights).
Static Baseline Periods: Static baselines are periods of time you define that are of
particular interest to you (Example: End of the fiscal year). These baselines can be
used to characterize workload periods for comparison against future occurrences
of that workload (Example: Compare the end of the fiscal year from one calendar
year to the next).
1.3.3 Advanced Threshold Management
While metric baselines are generally useful for determining valid target alert
thresholds, these thresholds are static and are not able to account for expected
performance variation. There are monitoring situations in which different work loads
for a target occur at regular (expected) intervals. Here, a static alert threshold would
prove to be inaccurate. For example, the alert thresholds for a database performing
Online Transaction Process (OLTP) during the day and batch processing at night
would be different. Similarly, database workloads can change based purely on
different time periods, such as weekday versus weekend. Thus, fixed static values for
thresholds might result in false alert reporting, and with excessive alerting could
generate excessive overhead with regard to performance management. For this OLTP
example, using static baselines to determine accurate alert thresholds fails to account
for expected cyclic variations in performance, adversely affecting problem detection.
Static baselines introduce the following configuration issues:
в– в– Baselines configured for Batch performance may fail to detect OLTP performance
degradation.
Baselines configured for OLTP performance may generate excessive alerts during
Batch cycles
Beginning with Enterprise Manager Release 12.1.0.4, Advanced Threshold
Management can be used to compute thresholds using baselines that are either
adaptive (self-adjusting) or time-based (user-defined).
в– в– Adaptive Thresholds: Allows Enterprise Manager to statistically compute threshold
that are adaptive in nature. Adaptive thresholds apply to all targets (both Agent
and repository monitored).
Time-based Thresholds: Allows you to define a specific threshold values to be used
at different times to account for changing workloads over time.
A convenient UI allows you to create time-based and adaptive thresholds. From a
target home page (a host, for example), navigate to the Metric Collection and Settings
page. Click Advanced Threshold Management in the Related Links region.
1-4 OracleВ® Enterprise Manager Administration
Monitoring: Basics
Figure 1–1 Advanced Threshold Management Page
Only numeric and View Collect metrics can be registered as adaptive thresholds. In
addition, only the following types of metrics are permitted:
в– Load
в– Load Type
в– Utilization and Response
1.3.4 Events: Defining What Conditions are of Interest
When a metric threshold value is reached, a metric alert is raised. A metric alert is a
type of event. An event is a significant occurrence that indicates a potential problem;
for example, either a warning or critical threshold for a monitored metric has been
crossed. Other examples of events include: database instance is down, a configuration
file has been changed, job executions ended in failure, or a host exceeded a specified
percentage CPU utilization. Two of the most important event types used in enterprise
monitoring are:
в– Metric Alert
в– Target Availability
For more information on events and available event types for which you can monitor,
see Chapter 3, "Using Incident Management".
1.3.5 Corrective Actions: Resolving Issues Automatically
Corrective actions allow you to specify automated responses to metric alerts, saving
administrator time and ensuring issues are dealt with before they noticeably impact
users. For example, if Enterprise Manager detects that a component, such as the
SQL*Net listener is down, a corrective action can be specified to automatically start it
back up. A corrective action is, therefore, any task you specify that will be executed
when a metric triggers a warning or critical alert severity. In addition to performing a
corrective task, a corrective action can be used to gather more diagnostic information,
if needed. By default, the corrective action runs on the target on which the event has
been raised.
Enterprise Monitoring 1-5
Monitoring: Basics
A corrective action can also consist of multiple tasks, with each task running on a
different target. Administrators can also receive notifications for the success or failure
of corrective actions. A corrective action can also consist of multiple tasks, with each
task running on a different target.
Corrective actions for a target can be defined by all Enterprise Manager administrators
who have been granted Manage Target Metrics or greater privilege on the target. For
any metric, you can define different corrective actions when the metric triggers at
warning severity or at critical severity.
Corrective actions must run using the credentials of a specific Enterprise Manager
administrator. For this reason, whenever a corrective action is created or modified, the
credentials that the modified action will run with must be specified.
1.3.6 Metric Extensions: Customizing Monitoring
Metric Extensions let you extend Enterprise Manager’s monitoring capabilities to
cover conditions specific to your IT environment, thus providing you with a complete
and comprehensive view of your monitored environment.
Metric extensions allow you to define new metrics on any target type that utilize the
same full set of data collection mechanisms used by Oracle provided metrics. For
example, some target types you can create metrics on are:
в– Hosts
в– Databases
в– IBM Websphere
в– Oracle Exadata Databases and Storage Servers
в– Oracle Business Intelligence Components
Once these new metrics are defined, they are used like any other Enterprise Manager
metric. For more information about metric extensions, see Chapter 9, "Using Metric
Extensions".
User-Defined Metrics (Pre-12c)
If you upgraded your Enterprise Manager 12c site from an older version of Enterprise
Manager, then all user-defined metrics defined in the older version will also be
migrated to Enterprise Manager 12c. These user-defined metrics will continue to work,
however they will no longer be supported a future release. If you have existing
user-defined metrics, it is recommended that you migrate them to metric extensions as
soon as possible to prevent potential monitoring disruptions in your managed
environment. For information about the migration process, see Converting User-defined
Metrics to Metric Extensions in Chapter 9, "Using Metric Extensions"
1.3.7 Blackouts
Blackouts allow you to support planned outage periods to perform scheduled or
emergency maintenance. When a target is put under blackout, monitoring is
suspended, thus preventing unnecessary alerts from being sent when you bring down
a target for scheduled maintenance operations such as database backup or hardware
upgrade. Blackout periods are automatically excluded when calculating a target’s
overall availability.
A blackout period can be defined for individual targets, a group of targets or for all
targets on a host. The blackout can be scheduled to run immediately or in the future,
and to run indefinitely or stop after a specific duration. Blackouts can be created on an
1-6 OracleВ® Enterprise Manager Administration
Monitoring: Advanced Setup
as-needed basis, or scheduled to run at regular intervals. If, during the maintenance
period, you discover that you need more (or less) time to complete maintenance tasks,
you can easily extend (or stop) the blackout that is currently in effect. Blackout
functionality is available from both the Enterprise Manager console as well as via the
Enterprise Manager command-line interface (EM CLI). EM CLI is often useful for
administrators who would like to incorporate the blacking out of a target within their
maintenance scripts. When a blackout ends, the Management Agent automatically
re-evaluates all metrics for the target to provide current status of the target
post-blackout.
If an administrator inadvertently performs scheduled maintenance on a target without
first putting the target under blackout, these periods would be reflected as target
downtime instead of planned blackout periods. This has an adverse impact on the
target’s availability records. In such cases, Enterprise Manager allows Super
Administrators to go back and define the blackout period that should have happened
at that time. The ability to create these retroactive blackouts provides Super
Administrators with the flexibility to define a more accurate picture of target
availability.
1.4 Monitoring: Advanced Setup
Enterprise Manager greatly simplifies managing your monitored environment and
also allows you to customize and extend Enterprise Manager monitoring capabilities.
However, the primary advantage Enterprise Manager monitoring provides is the
ability to monitor and manage large-scale, heterogeneous environments. Whether you
are monitoring an environment with 10 targets or 10,000 targets, the following
Enterprise Manager advanced features allow you to implement and maintain your
monitored environment with the equal levels of convenience and simplicity.
1.4.1 Monitoring Templates
Monitoring Templates simplify the task of standardizing monitoring settings across
your enterprise by allowing you to specify your standards for monitoring in a
template once and apply them to monitored targets across your organization. This
makes it easy for you to apply specific monitoring settings to specific classes of targets
throughout your enterprise. For example, you can define one monitoring template for
test databases and another monitoring template for production databases.
A monitoring template defines all Enterprise Manager parameters you would
normally set to monitor a target, such as:
в– в– Target type to which the template applies.
Metrics (including user-defined metrics), thresholds, metric collection schedules,
and corrective actions.
When a change is made to a template, you can reapply the template across affected
targets in order to propagate the new changes. The apply operation can be automated
using Administration Groups and Template Collections. For any target, you can
preserve custom monitoring settings by specifying metric settings that can never be
overwritten by a template.
Enterprise Manager comes with an array of Oracle-certified templates that provide
recommended metric settings for various Oracle target types.
For more information about monitoring templates, see Chapter 8, "Using Monitoring
Templates".
Enterprise Monitoring 1-7
Monitoring: Advanced Setup
1.4.2 Administration Groups and Template Collections
Monitored environments are rarely static—new targets are constantly being added
from across your ecosystem. Enterprise Manager allows you to maintain control of this
dynamic environment through administration groups. Administration groups
automate the process of setting up targets for management in Enterprise Manager by
automatically applying management settings such as monitoring settings or
compliance standards. Typically, these settings are manually applied to individual
targets, or perhaps semi-automatically using monitoring templates (see Section 1.4.1,
"Monitoring Templates") or custom scripts. Administration groups combine the
convenience of applying monitoring settings using monitoring templates with the
power of automation.
Template collections contain the monitoring settings and other management settings
that are meant to be applied to targets as they join the administration group.
Monitoring settings for targets are defined in monitoring templates. Monitoring
templates are defined on a per target type basis, so you will need to create monitoring
templates for each of the different target types in your administration group. You will
most likely create multiple monitoring templates to define the appropriate monitoring
settings for an administration group.
Every target added to Enterprise Manager possesses innate attributes called target
properties. Enterprise Manager uses these target properties to add targets to the correct
administration group. Administration group membership is based on target properties
as membership criteria so target membership is dynamic. Once added to the
administration group, Enterprise Manager automatically applies the requisite
monitoring settings using monitoring templates that are part of the associated
template collection .
Administration groups use the following target properties to define membership
criteria:
в– Contact
в– Cost Center
в– Customer Support Identifier
в– Department
в– Lifecycle Status
в– Line of Business
в– Location
в– Target Version
в– Target Type
1.4.3 Customizing Alert Messages
Whenever a metric threshold is reached, an alert is raised along with a metric-specific
message. These messages are written to address generic metric alert conditions.
Beginning with Enterprise Manager Release 12.1.0.4, you can customize these
messages to suit the specific requirements of your monitored environment.
Customizing an alert message allows you to tailor the message to suit your monitoring
needs. You can tailor the message to include their operational context specific to your
environment such as IT error codes used in your data center, or add additional
information collected by Enterprise Manager such as:
в– Metric name for which the alert has been triggered
1-8 OracleВ® Enterprise Manager Administration
Monitoring: Advanced Setup
в– Severity level of the alert or violation
в– Threshold value for which warning or critical violation has been triggered
в– Number of Occurrences after which alert has been triggered
Alert message customization allows for more efficient alert management by increasing
message usability.
To customize a metric alert message:
1.
Navigate to a target homepage.
2.
From the target menu (host target type is shows in the graphic), select Monitoring
and then Metric and Collection Settings.
The Metric and Collection Settings page displays.
3.
In the metric table, find the specific metric whose message you want to change and
click the edit icon (pencil).
Enterprise Monitoring 1-9
Notifications
The Edit Advanced Settings page displays.
4.
In the Monitored Objects region, click Edit Alert Message.
5.
Modify the alert message as appropriate.
To change your revised message back to the original
Oracle-defined message at any time, click Reset Alert Message.
Note:
6.
Click Continue to return to the Metric and Collection Settings page.
7.
To modify additional metric alert messages, repeat steps three through six.
8.
Once you are finished, click OK to save all changes to the Enterprise Manager
Repository. Enterprise Manager will display a message indicating the updates
have succeeded.
9.
Click OK to dismiss the message and return to the target homepage.
1.5 Notifications
For a typical monitoring scenario, when a target becomes unavailable or if thresholds
for performance are crossed, events are raised and notifications are sent to the
appropriate administrators. Enterprise Manager supports notifications via email,
pager, SNMP traps, or by running custom scripts and allows administrators to control
these notification mechanisms through:
в– Notification Methods
в– Rules and Rule Sets
Notification Methods
A notification method represents a specific way to send notifications. Besides e-mail,
there are three types of notification methods: OS Command, PL/SQL, SNMP Traps.
When configuring a notification method, you need to specify the particulars associated
with a specific notification mechanism such as which SMTP gateway(s) to use for
e-mail or which custom OS script to run. Super Administrators perform a one-time
setup of the various types of notification methods available for use.
Rules
A rule instructs Enterprise Manager to take specific action when events or incidents
(entity containing one important event or related events) occur, such as notifying an
administrator or opening a helpdesk ticket (see Section 1.6, "Managing Events,
Incidents, and Problems"). For example, you can define a rule that specifies e-mail
should be sent to you when CPU Utilization on any host target is at critical severity, or
1-10 OracleВ® Enterprise Manager Administration
Managing Events, Incidents, and Problems
another rule that notifies an administrator’s supervisor if an incident is not
acknowledged within 24 hours.
1.5.1 Customizing Notifications
Notifications that are sent to Administrators can be customized based on message type
and on-call schedule. Message customization is useful for administrators who rely on
both e-mail and paging systems as a means for receiving notifications. The message
formats for these systems typically vary—messages sent to e-mail can be lengthy and
can contain URLs, and messages sent to a pager are brief and limited to a finite
number of characters. To support these types of mechanisms, Enterprise Manager
allows administrators to associate a long or short message format with each e-mail
address. E-mail addresses that are used to send regular e-mails can be associated with
the long format; pages can be associated with the short format. The long format contains
full details about the event/incident; the short format contains the most critical pieces
of information.
Notifications can also be customized based on an administrator's on-call schedule. An
administrator who is on-call might want to be contacted by both his pager and work
email address during business hours and only by his pager address during off hours.
Enterprise Manager offers a flexible notification schedule to support the wide variety
of on-call schedules. Using this schedule, an administrator defines his on-call schedule
by specifying the email addresses by which they should be contacted when they are
on-call. For periods where they are not on-call, or do not wish to receive notifications
for incidents, they simply leave that part of the schedule blank. All alerts that are sent
to an administrator automatically adhere to his specified schedule.
1.6 Managing Events, Incidents, and Problems
Enterprise Manager's monitoring functionality is built upon the precept of monitoring
by exception. This means it monitors and raises events when exception conditions
exist in your IT environment and allowing administrators to address them in a timely
manner. As discussed earlier, the two most commonly used event types to monitor for
are metric alert and target availability. Although these are the most common event
types for which Enterprise Manager monitors, there are many others. Available event
types include:
в– Target Availability
в– Metric Alert
в– Metric Evaluation Errors
в– Job Status Changes
в– Compliance Standard Rule Violations
в– Compliance Standard Score Violations
в– High Availability
в– Service Level Agreement Alerts
в– User-reported
в– Application Dependency and Performance Alert
в– JVM Diagnostics Threshold Violation
By definition, an incident is a unit containing a single, or closely correlated set of
events that identify an issue that needs administrator attention within your managed
Enterprise Monitoring 1-11
Managing Events, Incidents, and Problems
environment. So an incident might be as simple as a single event indicating available
space in a tablespace has fallen below a specified limit, or more complex such as an
incident consisting of multiple events relating to potential performance issue when a
server is running out of resources. Such an incident would contain events relating to
the usage of CPU, I/O , and memory resources. Managing by incident gives you the
ability to address issues that may consist of any number of causal factors. For an
in-depth discussion on incidents and events, see Chapter 3, "Using Incident
Management".
Although incidents can correspond to a single events, incidents more commonly
correspond to groups of related events. A large number of discrete events can quickly
become unmanageable, but handled as an assemblage of related events, incidents
allow you to manage large numbers of event occurrences more effectively.
Once an incident is created, Enterprise Manager makes available a rich set of incident
management workflow features that let you to manage and track the incident through
its complete lifecycle. Incident management features include:
в– Assign incident ownership.
в– Track the incident resolution status.
в– Set incident priority.
в– Set incident escalation level.
в– Ability to provide a manual summary.
в– Ability to add user comments.
в– Ability to suppress/unsuppress
в– Ability to manually clear the incident.
в– Ability to create a ticket manually.
Problems pertain to the diagnostic incidents and problems stored in Automatic
Diagnostic Repository (ADR), which are automatically raised by Oracle software when
it encounters critical errors in the software. When problems are raised for Oracle
software, Oracle has determined that the recommended recourse is to open a Service
Request (SR), send support the diagnostic logs, and eventually provide a solution from
Oracle. A problem represents the underlying root cause of a set of incidents. Enterprise
Manager provides features to track and manage the lifecycle of a problem.
1.6.1 Incident Manager
Enterprise Manager Cloud Control simplifies managing incidents through an intuitive
UI called Incident Manager. Incident Manager provides and easy-to-use interface that
allows you to search, view, manage, and resolve incidents and problems impacting
your environment. To access Incident Manager, from the Enterprise menu, select
Monitoring, and then Incident Manager.
1-12 OracleВ® Enterprise Manager Administration
Managing Events, Incidents, and Problems
Figure 1–2 Incident Manager
From the Incident Manager UI, you can:
в– Filter incidents, problems, and events by using custom views
в– Respond and work on an incident
в– в– в– Manage incident lifecycle including assigning, acknowledging, tracking its status,
prioritization, and escalation
Access (in context) My Oracle Support knowledge base articles and other Oracle
documentation to help resolve the incident.
Access direct in-context diagnostic/action links to relevant Enterprise Manager
functionality allowing you to quickly diagnose or resolve the incident.
1.6.2 Incident Rules and Rule Sets
An incident rule specifies criteria and actions that determine when a notification should
be sent and how it should be sent whenever an event or incident is raised. The criteria
defined within a rule can apply to attributes such as the target type, events and
severity states (clear, warning or critical) and the notification method that should be
used when an incident is raised that matches the rule criteria. Rule actions can be
conditional in nature. For example, a rule action can be defined to page a user when an
incident severity is critical or just send e-mail if it is warning.
A rule set is a collection of rules that apply to a common set of targets such as hosts,
databases, groups, jobs, metric extensions, or self updates and take appropriate actions
to automate the business processes underlying incident. Incident rule sets can be made
public for sharing across administrators. For example, administrators can subscribe to
the same rule set if they are interested in receiving notifications for the same criteria
defined in the rule. Alternatively, an Enterprise Manager Super Administrator can
assign incident rule sets to other administrators so that they receive notifications for
incidents as defined in the rule.
In addition to being used by the notification system (see Rules in Section 1.5,
"Notifications" ), rule sets can also instruct Enterprise Manager to perform other
Enterprise Monitoring 1-13
Accessing Monitoring Information
actions, such as creating incidents, updating incidents, or call into a trouble ticketing
system as discussed in Section 1.6.3, "Connectors".
1.6.3 Connectors
An Oracle Management Connector integrates third-party management systems with
Enterprise Manager. There are two types of connectors: Event connectors and helpdesk
connectors.
Using the event connector, you can configure Enterprise Manager to share events with
non-Oracle management systems. The connector monitors all events sent from Oracle
Enterprise Manager and automatically updates alert information in the third-party
management system. Event connectors support the following functions:
в– в– в– Sharing of event information from Oracle Enterprise Manager to the third-party
management system.
Customization of event to alert mappings between Oracle Enterprise Manager and
the third-party management system.
Synchronization of event changes in Oracle Enterprise Manager with the alerts in
the third-party management system.
Using the helpdesk connector, you can configure Enterprise Manager to create, update,
or close a ticket for any event created in Enterprise Manager. The ticket generated by
the connector contains the relevant information about the Enterprise Manager
incident, including a link to the Enterprise Manager console to enable helpdesk
analysts leverage Enterprise Manager's diagnostic and resolution features to resolve
the incident. In Enterprise Manger, the ticket ID, ticket status, and link to the
third-party ticketing system is the shown in the context of the incident. This provides
Enterprise Manager administrators with ticket status information and an easy way to
quickly access the ticket.
Available connectors include:
в– BMC Remedy Service Desk Connector
в– HP Service Manager Connector
в– CA Service Desk Connector
в– HP Operations Manager Connector
в– Microsoft Systems Center Operations Manager Connector
в– IBM Tivoli Enterprise Console Connector
в– IBM Tivoli Netcool/OMNIbus Connector
For more information about Oracle-built connectors, see the Enterprise Manager
Plug-ins Exchange.
http://www.oracle.com/goto/emextensibility
1.7 Accessing Monitoring Information
Enterprise Manager provides multiple ways to access monitoring information. The
primary focal point for incident management is the Incident Manager console,
however Enterprise Manager also provides other ways to access monitoring
information. The following figures show the various locations within Enterprise
Manager that display target monitoring information. The following figure shows the
1-14 OracleВ® Enterprise Manager Administration
Accessing Monitoring Information
Enterprise Manager Overview page that conveniently displays target status rollup and
rollup of incidents.
Figure 1–3 Enterprise Manager Console
The next figure shows the Incident Manager home page which displays incidents for a
system or target.
Figure 1–4 Incident Manager (in context of a system or target)
Monitoring information is also displayed on target home pages. In the following
figure, you can see target status as well as a rollup of incidents.
Enterprise Monitoring 1-15
Accessing Monitoring Information
Figure 1–5 Target Home Pages
1-16 OracleВ® Enterprise Manager Administration
2
Discovering, Promoting, and Adding Targets
2
Enterprise Manager Cloud Control (Cloud Control) enables you to discover, promote,
add, and then monitor software deployments across your network, using a single
GUI-rich console. This chapter introduces you to the concepts of discovery, promotion,
and monitoring, and describes how you can perform these tasks using Cloud Control.
In particular, this chapter covers the following:
в– About Discovering, Promoting, and Adding Targets
в– Discovering and Promoting All Target Types
в– Discovering and Promoting Oracle Homes
в– Discovering, Promoting, and Adding Database Targets
в– Discovering, Promoting, and Adding Middleware Targets
2.1 About Discovering, Promoting, and Adding Targets
в– About Discovering, Promoting, and Monitoring Hosts and Targets
в– Discovery and Monitoring in Enterprise Manager Lifecycle
в– Discovery and Monitoring Process
2.1.1 About Discovering, Promoting, and Monitoring Hosts and Targets
This section describes the following:
в– What are Targets and Managed Targets?
в– What is Discovery?
в– What is Promotion?
в– What is Monitoring?
2.1.1.1 What are Targets and Managed Targets?
Targets are entities such as host machines, databases, Fusion Middleware components,
that can be managed and monitored in Cloud Control.
Managed targets are entities that are actively being monitored and managed by Cloud
Control.
2.1.1.2 What is Discovery?
Discovery refers to the process of identifying unmanaged hosts and targets in your
environment. You can discover hosts and targets automatically or manually.
Discovering, Promoting, and Adding Targets 2-1
About Discovering, Promoting, and Adding Targets
Figure 2–1 illustrates the discovery process.
Figure 2–1 Discovery
Detecting New
Hosts in the
Network
OMS
OMS
Console
Monitored Hosts with Management Agents
Installed
You can discover targets using either of the following methods:
Autodiscovery Process
For discovery of a host, the autodiscovery process enables a Management Agent
running on the host to run an Enterprise Manager job that scans for unmanaged hosts.
You then convert these unmanaged hosts to managed hosts by deploying Management
Agents on these hosts. Next, you search for targets such as databases or other
deployed components or applications on these managed hosts, and finally you
promote these targets to managed status.
For discovery of targets, the autodiscovery process enables you to search for targets on
the host and then add these targets using Enterprise Manager.
The benefit of using this process is that as new components are added to your
infrastructure, they can be found and brought under management on a
regularly-scheduled basis.
Guided Discovery Process
The guided discovery process enables you to explicitly add a specific database target
as a target to bring under management. The discovery wizard guides you through the
process and most of the specifications required are filled by default.
The benefits of using this process are as follows:
в– в– в– в– You can find targets with less effort.
You can find a new database that has been added recently even if autodiscovery
has not been run.
You can find a non-promoted database that already exists in autodiscovery results,
but has a change in details. For example, the port.
You eliminate unnecessary consumption of resources on the Management Agent
when discovery is not needed.
2-2 OracleВ® Enterprise Manager Administration
About Discovering, Promoting, and Adding Targets
Specifying Target Monitoring Properties
Specifying target monitoring properties enables you to manually specify all the details
required to discover the database target, such as the host name and location, target
name and location, and other specific information. This process is generally used when
the autodiscovery process and guided discovery process fails to discover the target
that you want to add.
For more information on how to discover unmanaged hosts refer to Step 1:
Discovering Unmanaged Hosts Using Network Scan.
For more information on how to discover targets on managed hosts refer to Step 3:
Discovering Targets on Managed Hosts
2.1.1.3 What is Promotion?
Promotion refers to the process of converting unmanaged hosts and targets, which have
been discovered in your network, to managed hosts and targets in Cloud Control so
that they can be monitored and managed efficiently. While conversion of unmanaged
hosts to managed hosts involves deployment of a Management Agent on those hosts,
conversion of unmanaged targets running on those hosts to managed targets involves
only adding the targets as manageable entities in Cloud Control without deploying
any additional component on the hosts.
Figure 2–2 illustrates the promotion process.
Figure 2–2 Promotion
Install
Management
Agents
OMS
OMS
Console
2.1.1.4 What is Monitoring?
Monitoring refers to the process of gathering information and keeping track of activity,
status, performance, and health of targets managed by Cloud Control on your host. A
Management Agent deployed on the host in conjunction with plug-ins monitors every
managed target on the host.
Discovering, Promoting, and Adding Targets 2-3
About Discovering, Promoting, and Adding Targets
2.1.2 Discovery and Monitoring in Enterprise Manager Lifecycle
Figure 2–3 illustrates the lifecycle process of discovering and monitoring targets in
Cloud Control.
Figure 2–3 Discovery and Monitoring in Enterprise Manager Lifecycle
OTN/DVD
Download Enterprise
Manager 12c
Install Enterprise
Manager 12c
EM
Console
EM
Console
OMS
Maintain EM
Discover New
Hosts/Targets
Monitor
Promote
2.1.3 Discovery and Monitoring Process
Figure 2–4 illustrates the high level process of discovering and monitoring targets:
2-4 OracleВ® Enterprise Manager Administration
Discovering and Promoting All Target Types
Figure 2–4 Discovery and Monitoring Process
Automatic Process
Discover new hosts by
using IP scan
1
Manual Process
Discover New Hosts
Discover new hosts by getting
a list from your Administrator
OMS Console
OMS
T
LIS
T
LIS
2
Install Management Agent on
Unmanaged Hosts
Install using Add Host
Wizard Target
IInstall using Add Host Wizard
Target or by Silent Method
OMS
OMS
CONSOLE
3
Discover Targets using Configure
Auto Discovery
Discover Targets on
Managed Hosts
Discover Targets using All Discovery
Modules in Auto Discovery Table
OMS
OMS
Console
4
Monitor Targets
Monitor Targets
EM Console
EM Console
Monitor Targets
EM Console
2.2 Discovering and Promoting All Target Types
This section covers the following:
в– Discovering and Promoting All Target Types Using the Autodiscovery Process
в– Discovering and Adding All Target Types Using the Guided Discovery Process
в– Discovering and Adding All Target Types By Specifying Target Monitoring
Properties
Discovering, Promoting, and Adding Targets 2-5
Discovering and Promoting All Target Types
в– Retrieving Deleted Targets
2.2.1 Discovering and Promoting All Target Types Using the Autodiscovery Process
This section covers the following:
в– Meeting the Prerequisites
в– Discovering and Promoting All Target Types
2.2.1.1 Meeting the Prerequisites
Before you discover targets using the autodiscovery process, meet these prerequisites:
в– To run automatic host discovery (Nmap binary) on a Management Agent installed
on a Solaris system, follow these steps:
1.
Install Cloud Control 3.4.3 or higher on the Solaris system.
2.
Stop the Management Agent from the running.
3.
Execute the following commands on the terminal session which you will use
to start the Management Agent:
bash-2.03$ LD_LIBRARY_PATH=<directory path of Cloud ControlCloud
Control libraries>:$LD_LIBRARY_PATH
bash-2.03$ export LD_LIBRARY_PATH
These steps ensure that Nmap refers to the Cloud Control
libraries while the Management Agent is running.
Note:
4.
Start the Management Agent using the terminal session used for the previous
step.
Add LD_LIBRARY_PATH to your start up scripts so that this
setting is retained after you reboot your system.
Note:
в– To discover hosts from a Management Agent running on the following systems,
follow the prerequisites stated in Table 2–1.
Table 2–1
Prerequisites for discovering hosts
System
Prerequisites
Solaris 11 cluster (SPARC or
x86-64)
Create a static IPv4 address on interface net0:
ipadm create-addr -T static -a local=X.X.X.X/YY net0/ZZ
Here, X.X.X.X is a static IPv4 address, YY is the sub network
mask, and ZZ is the specific identity for the net0 interface. The
created static address will subsequently be identified by
net0/ZZ.
For example:
ipadm create-addr -T static -a local=10.134.108.101/24
net0/hh
2-6 OracleВ® Enterprise Manager Administration
Discovering and Promoting All Target Types
Table 2–1 (Cont.) Prerequisites for discovering hosts
System
Prerequisites
SUSE Linux for System zв„ў
1.
Run the QETH_OPTIONS='fake_ll=1' option by adding it to
the configuration file for the NIC present in the
/etc/sysconfig/hardware directory.
The name of the configuration file changes according to the
NIC used. Contact your system administration for the name
of the configuration file that your system uses.
RedHat Linux for System z
2.
Restart your system for the changes to take effect.
1.
Run the OPTIONS='fake_ll=1' option by adding it to the
configuration file for the NIC present in the
/etc/sysconfig/network-scripts directory.
The name of the configuration file changes according to the
NIC used. Contact your system administration for the name
of the configuration file that your system uses.
2.
Verify that the alias in the /etc/modprobe.conf file includes
the following command:
alias eth0 geth
3.
Restart the system for the changes to take effect.
2.2.1.2 Discovering and Promoting All Target Types
To discover and promote all target types using autodiscovery process, follow these
steps:
в– Step 1: Discovering Unmanaged Hosts Using Network Scan
в– Step 2: Converting Unmanaged Hosts to Managed Hosts
в– Step 3: Discovering Targets on Managed Hosts
в– Step 4: Promoting Targets to Managed Status
2.2.1.2.1
Step 1: Discovering Unmanaged Hosts Using Network Scan
Skip this step if host is already managed in Enterprise
Manager.
Note:
To discover and configure hosts using IP scan, follow these steps:
If automatic host discovery is not available for your platform,
deploy the Management Agent manually. To deploy the Management
Agent, refer to following URL:
Note:
http://docs.oracle.com/cd/E24628_
01/install.121/e24089/install_agent_usng_rsp.htm
1.
From the Setup menu, select Add Target, then select Configure Auto Discovery.
Discovering, Promoting, and Adding Targets 2-7
Discovering and Promoting All Target Types
2.
On the Configure Auto Discovery page, click the Configure icon against Host and
Oracle VM Manager in the Network Scan-based Auto Discovery table.
3.
On the Network Scan Discovery page, click Create. You will now create the
discover job. By default, the Name field will be populated with a title including
that date and time the job was created. Note that you can edit the discovery jobs
and schedule discovery to run immediately or later.
4.
On the Network Scan Discovery: Create page, click Add. You will now select the
Management Agent that will perform the network scan. You can select the
Management Agent that is installed by default on the Oracle Management Service
host, or can select another Agent if desired.
2-8 OracleВ® Enterprise Manager Administration
Discovering and Promoting All Target Types
Note that because the entire network will be scanned, the Sudo Privilege
Delegation must be set on the Management Agent host that will perform the scan.
5.
Select the agent in the IP Ranges for scan table, and enter the IP ranges to scan.
You can specify any or even all of the following:
в– в– в– в– One or more absolute hostnames, each separated by a space; for example:
host1.example.com host3.example.com
One or more IP addresses, each separated by a space
A range of addresses; for example: 10.0.0-255.1-250. Note that IP addresses
and IP ranges must be separated by a comma; for example: 10.0.0-255.1-250
Classless Inter-Domain Routing (CIDR) notations; for example:
128.16.10.0/24
Separate each value with a space; for example:
host1.example.com 192.168.0.1 128.16.10.0/24 10.0.0-255.1-250,254
6.
A default list of ports to scan within the IP ranges you specified is listed in the
Configure Ports table. These are default ports typically used by the listed Oracle
components.
To modify the port values for a component, select the component in the table and
change the values accordingly. Up to 10 ports and/or port ranges can be specified.
7.
If you want to add more component ports to the list, click Add. Enter the name of
the service to include, and specify the port(s) or port range to scan.
8.
Specify the following:
в– в– The schedule at which the discovery job will run. Note that you can start the
job immediately.
The credentials set on the Management Agent that will perform the scan.
As noted, the Sudo Privilege Delegation must be set on the Management
Agent host that will perform the scan. The named credential that will be used
must be configured to run as root.
Click Save and Submit Scan.
9.
After the discovery job executes, you can check for discovered hosts that may
contain potential targets. You can do this two ways:
в– Select the job in the Host Discovery page, then click View Discovered Targets;
or:
в– 2.2.1.2.2
From the Setup menu, select Add Target, then select Auto Discovery Results.
Step 2: Converting Unmanaged Hosts to Managed Hosts
Skip this step if host is already managed in Enterprise
Manager.
Note:
To convert unmanaged hosts to managed hosts, follow these steps:
1.
From the Setup menu, select Add Target, and then select Auto Discovery Results.
2.
On the Auto Discovery Results page, select the Network-scanned Targets tab. All
discovered hosts are listed, with the open ports and identifiable service names
Discovering, Promoting, and Adding Targets 2-9
Discovering and Promoting All Target Types
shown. Based on your understanding of the Oracle components deployed on your
network, you should be able to determine the types of potential targets that have
been discovered.
3.
Select a host from the table, then click Promote to promote the host to managed
target status. The Add Host Targets wizard opens. You will use this wizard to
install a Management Agent on the host.
Installing a Management Agent on an unmanaged host promotes the unmanaged
host to managed target status, thereby converting the host to a managed host.
2.2.1.2.3
Step 3: Discovering Targets on Managed Hosts
To discover targets on managed hosts, follow these steps:
1.
From the Setup menu, select Add Target, and then select Configure Auto
Discovery.
2.
On the Configure Auto Discovery page, click the Configure icon in the All
Discovery Modules row in the Configure Auto Discovery table.
3.
On the Target Discovery (Agent-based) page, expand Search, then enter the
hostname for the host you want to check for targets in the Agent Host Name field.
The host must have a Management Agent installed on it.
4.
To search for a specific Management Agent, click Search. The table lists all the
Management Agents and filters the list based on what you search for.
5.
Select the host in the table and click Configure Discovery Modules.
2-10 OracleВ® Enterprise Manager Administration
Discovering and Promoting All Target Types
6.
On the Configure Target Discovery page, set the schedule at which discovery will
be run, in days. This schedule will be applied to all selected hosts. By default the
discovery will run every 24 hours.
7.
Select the target types you want to discover on the host. Note that you must
supply search parameters for some target types. To specify a parameter, select the
target type in the Discovery Module column and click Edit Parameters.
в– в– Oracle Cluster and High Availability Service: No parameters required.
Oracle Database, Listener and Automatic Storage Management: Specify the
path to the Clusterware Home.
в– Oracle Home Discovery: No parameters required.
в– Oracle Secure Backup Domain: No parameters required.
в– Oracle Fusion Middleware: Specify * (the "star" character) to search all
Middleware Homes, or specify the path to one or more Middleware Homes on
the host, each separated by a comma.
8.
Click OK when finished. Target discovery has been configured on this host.
9.
Repeat these steps for each additional host on which you want to configure
discovery.
10. Click Run Discovery Now to discover targets immediately. The discovery will also
run at the scheduled interval.
2.2.1.2.4
Step 4: Promoting Targets to Managed Status
To promote discovered targets to managed status, follow these steps:
1.
After the discovery job executes, you can check for discovered hosts that may
contain potential targets. You can do this two ways:
в– в– 2.
Select the job in the Host Discovery page, then click View Discovered Targets;
or
From the Setup menu, select Add Target, then select Auto Discovery Results.
Select a target to promote, then click Promote. A wizard specific to the target type
you are promoting opens. Supply the required values.
Discovering, Promoting, and Adding Targets
2-11
Discovering and Promoting All Target Types
3.
Click the Agent-based Targets tab.You can choose one or several targets to
promote.
4.
Note that you can optionally click Ignore for a discovered target. Ignoring a target
puts it into a list of targets that you do not want to manage.
Ignored targets will be displayed in the Ignored Targets tab, and will remain in
Cloud Control as un-managed targets until you decide to either promote or
remove them. If you delete a target, it would be rediscovered the next time
discovery runs.
5.
Check the target type home page to verify that the target is promoted as an Cloud
Control target. Once a target is successfully promoted, the Management Agent
installed on the target host will begin collecting metric data on the target.
2-12 OracleВ® Enterprise Manager Administration
Discovering and Promoting All Target Types
Note:
в– When you promote a discovered target to managed status, the
plug-in required for the target is automatically deployed to the
Management Agent, which monitors the host where the target has
been discovered. For the plug-in to be deployed, the Management
Agent must be secure. Therefore, before promoting the discovered
targets to managed status, ensure that the Management Agent is
secure. You can always unsecure it after the discovered target is
promoted to managed status, that is, after the required plug-in is
deployed.
To verify the secure status of a Management Agent, and to secure
it if required, use any one of the following methods:
в– в– From the Setup menu, select Manage Cloud
Control, and then click Agents. Click the
required Management Agent. Verify
whether the Management Agent is secure. If
it is not secure, from the Agent menu, click
Secure to secure it.
Run the following command to verify if the
Management Agent is secure:
<EMSTATE>/bin/emctl status agent
If the Management Agent is secure, the
Management Agent URL displayed in the
output of the previous command is an
HTTPS URL. However, if the Management
Agent URL displayed is an HTTP URL,
secure the Management Agent by running
the following command:
<EMSTATE>/bin/emctl secure agent
в– Cloud Control supports simultaneous promotion of multiple
targets only for some target types. Additionally, multiple selection
of database targets has been disabled to avoid a user selecting
RAC databases across clusters. This is similar to the user-guided
discovery feature where a user cannot discover targets across a
cluster in the same session.
2.2.2 Discovering and Adding All Target Types Using the Guided Discovery Process
To discover and add all target types using the guided process, follow these steps:
в– Step 1: Identifying Unmanaged Hosts
в– Step 2: Converting Unmanaged Hosts to Managed Hosts
в– Step 3: Adding Targets
2.2.2.1 Step 1: Identifying Unmanaged Hosts
Skip this step if host is already managed in Enterprise
Manager.
Note:
Discovering, Promoting, and Adding Targets
2-13
Discovering and Promoting All Target Types
Identify the unmanaged hosts by getting the list of hosts from your administrator or
by checking the Auto Discovery Results page by selecting the Setup menu, Add
Targets, and then Auto Discovery Results.
2.2.2.2 Step 2: Converting Unmanaged Hosts to Managed Hosts
Skip this step if host is already managed in Enterprise
Manager.
Note:
To convert unmanaged hosts to managed hosts manually, you should manually install
a Management Agent on each host. You can install a Management Agent in graphical
or in silent mode.
For instructions to install in graphical mode, see Oracle Enterprise Manager Cloud
Control Basic Installation Guide. For instructions to install in silent mode, see Enterprise
Manager Cloud Control Advanced Installation and Configuration Guide.
2.2.2.3 Step 3: Adding Targets
To add targets using the guided process, follow these steps:
When you add a target using the guided process, some scripts
and automated processes are run that are particular for the target type
that you select. You may have to input credentials in order to run the
guided process.
Note:
1.
From the Setup menu, select Add Target, then select Add Targets Manually.
Cloud Control displays the Add Targets Manually page.
2.
On the Add Targets Manually page, select Add Targets Using Guided Process
(Also Adds Related Targets).
3.
Choose one of the target types to add from the Target Types list, such as Exalogic
Elastic Cloud, Oracle Cluster and High Availability Service, or Oracle WebLogic
Domain. Click Add Using Guided Process...
4.
After you select the target type, a wizard specific to the target type guides you
through the process of manually adding the target.
2-14 OracleВ® Enterprise Manager Administration
Discovering and Promoting All Target Types
Upon confirmation, the target becomes a managed target in Cloud Control. Cloud
Control simply accepts the information, performs validation of the supplied data
where possible and starts monitoring the target.
When you manually add a non-host target to Cloud Control,
the plug-in required for the target is automatically deployed to the
Management Agent, which monitors the host where the non-host
target exists. For the plug-in to be deployed, the Management Agent
must be secure. Therefore, before manually adding a non-host target
to Cloud Control, ensure that the Management Agent is secure. You
can always unsecure it after the target is added to Cloud Control, that
is, after the required plug-in is deployed.
Note:
To verify the secure status of a Management Agent, and to secure it if
required, use any one of the following methods:
в– в– From the Setup menu, select Manage Cloud Control and then,
click Agents. Click the required Management Agent. Verify
whether the Management Agent is secure. If it is not secure, from
the Agent menu, click Secure to secure it.
Run the following command to verify if the Management Agent is
secure:
<EMSTATE>/bin/emctl status agent
If the Management Agent is secure, the Management Agent URL
displayed in the output of the previous command is an HTTPS
URL. However, if the Management Agent URL displayed is an
HTTP URL, secure the Management Agent by running the
following command:
<EMSTATE>/bin/emctl secure agent
2.2.3 Discovering and Adding All Target Types By Specifying Target Monitoring
Properties
To discover and add all target types by specifying target monitoring properties, follow
these steps:
в– Step 1: Indentifying Unmanaged Hosts
в– Step 2: Converting Unmanaged Hosts to Managed Hosts
в– Step 3: Adding Targets
2.2.3.1 Step 1: Indentifying Unmanaged Hosts
Skip this step if host is already managed in Enterprise
Manager.
Note:
Identify the unmanaged hosts by getting the list of hosts from your administrator or
by checking the Auto Discovery Results page by selecting the Setup menu, Add
Targets, and then Auto Discovery Results.
Discovering, Promoting, and Adding Targets
2-15
Discovering and Promoting All Target Types
2.2.3.2 Step 2: Converting Unmanaged Hosts to Managed Hosts
Skip this step if host is already managed in Enterprise
Manager.
Note:
To convert unmanaged hosts to managed hosts manually, you should manually install
a Management Agent on each host. You can install a Management Agent in graphical
or in silent mode.
For instructions to install in graphical mode, see Oracle Enterprise Manager Cloud
Control Basic Installation Guide. For instructions to install in silent mode, see Enterprise
Manager Cloud Control Advanced Installation and Configuration Guide.
2.2.3.3 Step 3: Adding Targets
To add a target on a managed host by specifying the target monitoring properties,
follow these steps:
1.
From the Setup menu, select Add Target, then select Add Targets Manually.
Cloud Control displays the Add Targets Manually page.
2.
On the Add Targets Manually page, select Add Targets Declaratively by
Specifying Target Monitoring Properties.
3.
Choose one of the target types to add from the Target Types list, such as ADF
Business Components for Java, Cluster Database, or Oracle HTTP Server.
4.
Specify the Management Agent that will be used to monitor the target, or click on
the Search icon to search for and select the Management Agent. Click Add
Manually...
5.
After you select the target type, a wizard specific to the target type guides you
through the process of manually adding the target.
Upon confirmation, the target becomes a managed target in Cloud Control. Cloud
Control simply accepts the information, performs validation of the supplied data
where possible and starts monitoring the target.
2-16 OracleВ® Enterprise Manager Administration
Discovering and Promoting All Target Types
When you manually add a non-host target to Cloud Control,
the plug-in required for the target is automatically deployed to the
Management Agent, which monitors the host where the non-host
target exists. For the plug-in to be deployed, the Management Agent
must be secure. Therefore, before manually adding a non-host target
to Cloud Control, ensure that the Management Agent is secure. You
can always unsecure it after the target is added to Cloud Control, that
is, after the required plug-in is deployed.
Note:
To verify the secure status of a Management Agent, and to secure it if
required, use any one of the following methods:
в– в– From the Setup menu, select Manage Cloud Control, and then,
click Agents. Click the required Management Agent. Verify
whether the Management Agent is secure. If it is not secure, from
the Agent menu, click Secure to secure it.
Run the following command to verify if the Management Agent is
secure:
<EMSTATE>/bin/emctl status agent
If the Management Agent is secure, the Management Agent URL
displayed in the output of the previous command is an HTTPS
URL. However, if the Management Agent URL displayed is an
HTTP URL, secure the Management Agent by running the
following command:
<EMSTATE>/bin/emctl secure agent
2.2.4 Retrieving Deleted Targets
This sections covers the following:
в– Retrieving Deleted Target Types
в– Retrieving Deleted Host and Corresponding Management Agent Targets
2.2.4.1 Retrieving Deleted Target Types
If you have deleted one or more targets (such as a database target or a weblogic
domain, or any othe target), you can retrieve them and add them back to the
Enterprise Manager Cloud Control Console. If autodiscovery is configured on the host
where the targets were present, the targets are automatically discovered during the
next scheduled autodiscovery operation. Once they are autodiscovered, you can
promote them and add them to the console. If autodiscovery is not configured on the
host where the targets were present, you have to discover the targets using one of the
following methods:
в– в– в– в– By enabling autodiscovery as described in Section 2.2.1.2.3 to automatically
discover the targets in the next scheduled autodiscovery operation. Once
discovered, promote them as described in Section 2.2.1.2.4 and add them to the
console.
By using the guided discovery process as described in Section 2.2.2.3 to manually
discover and add the discovered targets to the console.
By specifying the target monitoring properties for each target as described in
Section 2.2.3.3 to manually discover and add the discovered targets to the console.
By using the following EM CLI verb:
Discovering, Promoting, and Adding Targets
2-17
Discovering and Promoting All Target Types
$ emcli add_target
-name="name"
-type="type"
-host="hostname"
[-properties="pname1:pval1;pname2:pval2;..."]
[-separator=properties="sep_string"]
[-subseparator=properties="subsep_string"]
[-credentials="userpropname:username;pwdpropname:password;..."]
[-input_file="parameter_tag:file_path"]
[-display_name="display_name"]
[-groups="groupname1:grouptype1;groupname2:grouptype2;..."]
[-timezone_region="gmt_offset"]
[-monitor_mode="monitor_mode"]
[-instances="rac_database_instance_target_name1:target_type1;..."]
[-force]
[-timeout="time_in_seconds"]
[ ] indicates that the parameter is optional
For more information, access the following URL:
http://docs.oracle.com/cd/E24628_01/em.121/e17786/cli_verb_
ref.htm#CACHFHCA
2.2.4.2 Retrieving Deleted Host and Corresponding Management Agent Targets
If you have deleted a host target and the corresponding Management Agent target,
you can retrieve both of them. To do so, follow these steps:
Discover and add the host and the Management Agent by running the following
command from the agent instance home of the corresponding host:
$ emctl config agent addInternalTargets
Once the host and the Management Agent are discovered and added to the console,
add each target on that host as targets to be monitored in the console, by running the
following EM CLI verb:
$ emcli add_target
-name="name"
-type="type"
-host="hostname"
[-properties="pname1:pval1;pname2:pval2;..."]
[-separator=properties="sep_string"]
[-subseparator=properties="subsep_string"]
[-credentials="userpropname:username;pwdpropname:password;..."]
[-input_file="parameter_tag:file_path"]
[-display_name="display_name"]
[-groups="groupname1:grouptype1;groupname2:grouptype2;..."]
[-timezone_region="gmt_offset"]
[-monitor_mode="monitor_mode"]
[-instances="rac_database_instance_target_name1:target_type1;..."]
[-force]
[-timeout="time_in_seconds"]
[ ] indicates that the parameter is optional
For more information, access the following URL:
http://docs.oracle.com/cd/E24628_01/em.121/e17786/cli_verb_
ref.htm#CACHFHCA
2-18 OracleВ® Enterprise Manager Administration
Discovering and Promoting Oracle Homes
2.3 Discovering and Promoting Oracle Homes
When you deploy an Oracle software component outside of the deployment
procedures provided by Enterprise Manager, the Oracle home is not automatically
discovered and promoted as targets. You will have to manually discover and promote
the Oracle home target.
To discover and promote an Oracle home target, follow these steps:
1.
From the Enterprise menu, select Job, and then select Activity.
2.
On the Job Activity page, from the drop-down list in the table, select Discover
Promote Oracle Home Target.
Click Go.
3.
On the ’Create Discover Promote Oracle Home Target’ Job page, in the General
tab, specify the name of the discovery.
For example: OHDiscovery
You can optionally add a description for the discovery.
Click Add.
Discovering, Promoting, and Adding Targets
2-19
Discovering and Promoting Oracle Homes
4.
In the Search and Select: Target dialog box, select Target Type as Host, and then
select all the host targets listed by clicking Select All.
Click Select.
5.
On the’Create Discover Promote Oracle Home Target’ Job page, the host targets
that you selected are displayed in the table.
6.
Select the Parameters tab, and then do one of the following:
в– в– в– To discover a single Oracle Home, specify the path to the home, and then
select Oracle Home as the manage entity.
To discover all Homes in an inventory, specify the path to the inventory, and
then select Inventory as the manage entity.
To discover all Homes in a Middleware Home, specify the path to the
Middleware Home and select Middleware Home as the manage entity.
2-20 OracleВ® Enterprise Manager Administration
Discovering, Promoting, and Adding Database Targets
7.
To save the job for later, click Save to Library. To submit it, click Submit. When the
discovery is successful, a confirmation is displayed on the Job Activity page.
If you submit the discovery job without specifying a path, a
discovery of the whole host will be performed. In order for a Home to
be discoverable by the Management Agent, it needs to be registered in
an inventory that the Management Agent recognizes. The default
inventory is the central inventory, which in Unix systems is found in
/etc/oraInst.loc. Any Home registered here will automatically be
discovered.
Note:
If there are other inventories in the host, they need to be added to the
inventory list of the Management Agent. A line must be added to
$EMSTATE/sysman/config/OUIinventories.add.
If the inventory is not found here, the Management Agent will not
know of its existence, and hence any Home registered there will not be
discovered.
2.4 Discovering, Promoting, and Adding Database Targets
This section describes how you can discover, promote, and add database targets to be
managed by Cloud Control. In particular, this section covers the following:
в– Discovering Container Database and Pluggable Database Targets
в– Discovering Cluster Database Targets
в– Discovering Single Instance Database Targets
в– Discovering Cluster Targets
в– Discovering Single Instance High Availability Service Targets
в– Discovering Cluster Automatic Storage Management Targets
в– Enabling Autodiscovery of Database Targets
2.4.1 Discovering Container Database and Pluggable Database Targets
This section describes the different methods in which you can discover, promote, and
add container database (CDB) and pluggable database (PDB) targets in Cloud Control.
In particular, this section covers the following:
в– в– Discovering and Promoting CDB and PDB Targets Using Autodiscovery
Discovering and Adding CDB and PDB Targets Using the Guided Discovery
Process
Discovering, Promoting, and Adding Targets
2-21
Discovering, Promoting, and Adding Database Targets
в– Discovering and Adding CDB and PDB Targets By Specifying Target Monitoring
Properties
2.4.1.1 Discovering and Promoting CDB and PDB Targets Using Autodiscovery
To discover and promote a CDB target and its associated PDB targets using automatic
discovery, follow these steps:
By default, promoting a CDB target also promotes all its
associated discovered PDB targets. Also, by default, Enterprise
Manager runs a background job (every 24 hours) to automatically
discover and promote newly created PDB targets present on managed
hosts. Hence, to discover and promote PDB targets, you only need to
discover and promote the associated CDB target, as described in this
section.
Note:
в– Step 1: Discovering Unmanaged Hosts
в– Step 2: Converting Unmanaged Hosts to Managed Hosts
в– Step 3: Discovering CDB and PDB Targets on the Managed Hosts
в– Step 4: Promoting CDB and PDB Targets to Managed Status
2.4.1.1.1
Step 1: Discovering Unmanaged Hosts
Skip this step if host is already managed in Enterprise
Manager.
Note:
Discover the unmanaged hosts, as described in Section 2.2.1.2.1.
2.4.1.1.2
Step 2: Converting Unmanaged Hosts to Managed Hosts
Skip this step if host is already managed in Enterprise
Manager.
Note:
Convert the discovered unmanaged hosts to managed hosts, as described in
Section 2.2.1.2.2.
2.4.1.1.3
Step 3: Discovering CDB and PDB Targets on the Managed Hosts
Discover the CDB targets on the managed hosts, as described in Section 2.2.1.2.3.
Autodiscovery of databases is enabled by default and this
step is only required if it was disabled previously.
Note:
2.4.1.1.4
Step 4: Promoting CDB and PDB Targets to Managed Status
To promote CDB and PDB targets, follow these steps:
1.
From the Setup menu, select Add Target, then select Auto discovery Results.
Click Agent-based Targets.
2-22 OracleВ® Enterprise Manager Administration
Discovering, Promoting, and Adding Database Targets
2.
Search for and select the Database Instance target that you want to promote, then
click Promote.
3.
On the Promote Target: Results page, under the Databases section, select the CDB
target.
By default, selecting a CDB target for promotion also selects all its associated
discovered PDB targets for promotion. If you want to add or remove a PDB target
from the ones selected for promotion, select the CDB target, then click Configure.
Select the Pluggable Databases tab, then click Add or Remove. Click Save.
Discovering, Promoting, and Adding Targets
2-23
Discovering, Promoting, and Adding Database Targets
Enterprise Manager runs a background job (every 24 hours) to automatically
discover and promote newly created PDB targets present on managed hosts. If you
do not want Enterprise Manager to automatically promote the PDB targets
associated with a particular CDB target, and instead want to promote them
manually, select the CDB target on the Promote Target: Results page, then click
Configure. Select the Pluggable Databases tab, then select Manual for Pluggable
Database Discovery Mode. Click Save.
4.
Specify the monitoring credentials for the selected CDB target, that is, the user
name, password, and role. Also, if you want the selected target to be added to a
group, specify a value for Group.
If you specify Normal for Role, then the user name must be dbsnmp; you cannot
provide a different user name. However, if you specify SYSDBA for Role, then you
can provide any SYSDBA user. Only SYSDBA users and the dbsnmp user have the
privileges required for target monitoring.
5.
Click Test Connection to test the connection made to the CDB target using the
specified monitoring credentials.
6.
To specify global target properties for all the targets you have selected on the
Promote Target: Results page, click Set Global Target Properties, specify the
required properties, then click OK.
To specify a common group for all the targets you have selected on the Promote
Target: Results page, click Specify Group for Targets, select a group, then click
Select.
7.
Click Next.
8.
Review the displayed information, then click Submit.
2.4.1.2 Discovering and Adding CDB and PDB Targets Using the Guided Discovery
Process
To discover and add a CDB target and its associated PDB targets using a guided
discovery process, follow these steps:
By default, promoting a CDB target also promotes all its
associated discovered PDB targets. Also, by default, Enterprise
Manager runs a background job (every 24 hours) to automatically
discover and promote newly created PDB targets present on managed
hosts. Hence, to discover and promote PDB targets, you only need to
discover and promote the associated CDB target, as described in this
section.
Note:
в– Step 1: Discovering Unmanaged Hosts
в– Step 2: Converting Unmanaged Hosts to Managed Hosts
в– Step 3: Adding CDB and PDB Targets
2.4.1.2.1
Step 1: Discovering Unmanaged Hosts
Skip this step if host is already managed in Enterprise
Manager.
Note:
2-24 OracleВ® Enterprise Manager Administration
Discovering, Promoting, and Adding Database Targets
Discover the unmanaged hosts, as described in Section 2.2.2.1.
2.4.1.2.2
Step 2: Converting Unmanaged Hosts to Managed Hosts
Skip this step if host is already managed in Enterprise
Manager.
Note:
Convert the discovered unmanaged hosts to managed hosts, as described in
Section 2.2.2.2.
2.4.1.2.3
Step 3: Adding CDB and PDB Targets
To add CDB and PDB targets, follow these steps:
1.
From the Setup menu, select Add Target, then select Add Targets Manually.
2.
Select Add Targets Using Guided Process (Also Adds Related Targets). For
Target Types, select Oracle Database, Listener, and Automatic Storage
Management. Click Add Using Guided Process.
3.
On the Database Discovery: Search Criteria page, for Specify Host or Cluster,
specify the CDB host.
Discovering, Promoting, and Adding Targets
2-25
Discovering, Promoting, and Adding Database Targets
If the specified host belongs to a cluster, then you are prompted to choose if you
want to discover and promote all the database targets present on only on the
specified host, or all the database targets present on all the hosts that belong to the
cluster.
4.
Click Next.
5.
On the Database Discovery: Results page, under the Databases section, select the
CDB target.
By default, selecting a CDB target for promotion also selects all its associated
discovered PDB targets for promotion. If you want to add or remove a PDB target
from the ones selected for promotion, select the CDB target, then click Configure.
Select the Pluggable Databases tab, then click Add or Remove. Click Save.
Enterprise Manager runs a background job (every 24 hours) to automatically
discover and promote newly created PDB targets present on managed hosts. If you
do not want Enterprise Manager to automatically promote the PDB targets
associated with a particular CDB target, and instead want to promote them
manually, select the CDB target on the Promote Target: Results page, then click
Configure. Select the Pluggable Databases tab, then select Manual for Pluggable
Database Discovery Mode. Click Save.
6.
Specify the monitoring credentials for the selected CDB target, that is, the user
name, password, and role. Also, if you want the selected target to be added to a
group, specify a value for Group.
If you specify Normal for Role, then the user name must be dbsnmp; you cannot
provide a different user name. However, if you specify SYSDBA for Role, then you
can provide any SYSDBA user. Only SYSDBA users and the dbsnmp user have the
privileges required for target monitoring.
7.
Click Test Connection to test the connection made to the CDB target using the
specified monitoring credentials.
8.
To specify global target properties for all the targets you have selected on the
Promote Target: Results page, click Set Global Target Properties, specify the
required properties, then click OK.
To specify a common group for all the targets you have selected on the Promote
Target: Results page, click Specify Group for Targets, select a group, then click
Select.
2-26 OracleВ® Enterprise Manager Administration
Discovering, Promoting, and Adding Database Targets
If you have selected multiple databases and you want to set the same monitoring
properties for all of them, select Specify Common Monitoring Credentials. Enter
the monitoring credentials, monitoring password, and role. Click Apply.
9.
Click Next.
10. Review the displayed information, then click Submit.
2.4.1.3 Discovering and Adding CDB and PDB Targets By Specifying Target
Monitoring Properties
To discover and add a CDB target and its associated PDB targets by specifying target
monitoring properties, follow these steps:
By default, promoting a CDB target also promotes all its
associated discovered PDB targets. Also, by default, Enterprise
Manager runs a background job (every 24 hours) to automatically
discover and promote newly created PDB targets present on managed
hosts. Hence, to discover and promote PDB targets, you only need to
discover and promote the associated CDB target, as described in this
section.
Note:
в– Step 1: Discovering Unmanaged Hosts
в– Step 2: Converting Unmanaged Hosts to Managed Hosts
в– Step 3: Adding CDB and PDB Targets
2.4.1.3.1
Step 1: Discovering Unmanaged Hosts
Skip this step if host is already managed in Enterprise
Manager.
Note:
Discover the unmanaged hosts, as described in Section 2.2.3.1.
2.4.1.3.2
Step 2: Converting Unmanaged Hosts to Managed Hosts
Skip this step if host is already managed in Enterprise
Manager.
Note:
Convert the discovered unmanaged hosts to managed hosts, as described in
Section 2.2.3.2.
2.4.1.3.3
Step 3: Adding CDB and PDB Targets
To add a CDB or PDB target, follow these steps:
1.
From the Setup menu, select Add Target, then select Add Targets Manually.
2.
Select Add Targets Declaratively by Specifying Target Monitoring Properties.
For Target Type, select Pluggable Database.
For Monitoring Agent, specify the Management Agent present on the CDB host.
Click Add Manually.
Discovering, Promoting, and Adding Targets
2-27
Discovering, Promoting, and Adding Database Targets
3.
On the Add Pluggable Database: Specify Container Database page, specify the
CDB or click the search icon to select the CDB to which the target will be added.
Click Continue.
4.
On the Add Pluggable Database: Properties page, specify a unique name for the
target, the name, the default service name, and the preferred string connection.
Expand the Container Database section to verify the properties of the CDB.
Click Test Connection to test the connection made to the CDB target using the
specified monitoring credentials.
Click Continue.
5.
Specify the required information on each page and then click Next, until you reach
the Review page.
6.
Review the displayed information, and then click Submit.
2.4.2 Discovering Cluster Database Targets
This section describes the different methods in which you can discover, promote, and
add cluster database targets. In particular, this section cover the following:
2-28 OracleВ® Enterprise Manager Administration
Discovering, Promoting, and Adding Database Targets
в– в– в– Discovering and Promoting Cluster Database Targets Using Autodiscovery
Discovering and Adding Cluster Database Targets Using the Guided Discovery
Process
Discovering and Adding Cluster Database Targets By Specifying Target
Monitoring Properties
2.4.2.1 Discovering and Promoting Cluster Database Targets Using Autodiscovery
To discover and promote cluster database targets using autodiscovery, follow these
steps:
в– Step 1: Discovering Unmanaged Hosts
в– Step 2: Converting Unmanaged Hosts to Managed Hosts
в– Step 3: Promoting Cluster Database Targets to Managed Status
2.4.2.1.1
Step 1: Discovering Unmanaged Hosts
Skip this step if host is already managed in Enterprise
Manager.
Note:
Discover unmanaged hosts, as described Section 2.2.1.2.1.
2.4.2.1.2
Step 2: Converting Unmanaged Hosts to Managed Hosts
Skip this step if host is already managed in Enterprise
Manager.
Note:
Convert discovered unmanaged hosts to managed hosts, as described Section 2.2.1.2.2.
2.4.2.1.3
Step 3: Promoting Cluster Database Targets to Managed Status
To promote cluster database targets, follow these steps:
1.
From the Setup menu, select Add Target, and then select Auto Discovery Results.
If you do not see any results, then autodiscovery of targets has
been disabled. To enable autodiscovery refer to Enabling
Autodiscovery of Database Targets.
Note:
From the results table, from the Agent-based targets tab, select the discovered
cluster database target that you want to add for monitoring, and click Promote.
Discovering, Promoting, and Adding Targets
2-29
Discovering, Promoting, and Adding Database Targets
2.
The Promote Targets:Result page displays the databases discovered on the cluster.
Select the database
On the Promote Target: Results page, under the Databases section, in the Cluster
Databases section, select the cluster database target that you want to promote.
By default, selecting the cluster database target for promotion also selects all its
associated discovered database instance targets for promotion. If you want to add
or remove a database instance target from the ones selected for promotion, select
the cluster database target, then click Configure. Select the Instances tab, then
click Add or Remove. Click Save.
3.
In the Cluster Databases section, specify the monitoring credentials for the selected
cluster database target, that is, the Monitor user name, Monitor password, and
role. Also, if you want the selected target to be added to a group, specify a value
for Group.
If you specify Normal for Role, then the user name must be dbsnmp; you cannot
provide a different user name. However, if you specify SYSDBA for Role, then you
can provide any SYSDBA user. Only SYSDBA users and the dbsnmp user have the
privileges required for target monitoring.
4.
Click Test Connection to test the connection made to the cluster database target
using the specified monitoring credentials.
5.
To specify global target properties for all the targets you have selected on the
Promote Target: Results page, click Set Global Target Properties, specify the
required properties, then click OK.
2-30 OracleВ® Enterprise Manager Administration
Discovering, Promoting, and Adding Database Targets
To specify a common group for all the targets you have selected on the Promote
Target: Results page, click Specify Group for Targets, select a group, then click
Select.
6.
If you have selected multiple databases and you want to set the same monitoring
properties for all of them, select Specify Common Monitoring Credentials. Enter
the monitoring credentials, monitoring password, and role. Click Apply.
7.
Click Next.
8.
Review the displayed information, then click Submit.
2.4.2.2 Discovering and Adding Cluster Database Targets Using the Guided
Discovery Process
To discover and add cluster database targets using the guided process, follow these
steps:
в– Step 1: Discovering Unmanaged Hosts
в– Step 2: Converting Unmanaged Hosts to Managed Hosts
в– Step 3: Adding Cluster Database Targets
2.4.2.2.1
Step 1: Discovering Unmanaged Hosts
Skip this step if host is already managed in Enterprise
Manager.
Note:
Discover the unmanaged hosts, as described in Section 2.2.2.1.
2.4.2.2.2
Step 2: Converting Unmanaged Hosts to Managed Hosts
Skip this step if host is already managed in Enterprise
Manager.
Note:
Convert the discovered unmanaged hosts to managed hosts, as described in
Section 2.2.2.2.
2.4.2.2.3
Step 3: Adding Cluster Database Targets
To add a cluster database target, follow these steps:
1.
From the Setup menu, select Add Target, then select Add Targets Manually.
2.
On the Add Targets Add Manually page, select Add Targets Using Guided
Process (Also Adds Related Targets).
3.
From the Target Types list, select Oracle Database, Listener, and Automatic
Storage Management.
4.
Click Add Using Guided Process.
Discovering, Promoting, and Adding Targets
2-31
Discovering, Promoting, and Adding Database Targets
5.
On the Database Discovery: Search Criteria page, specify the cluster database host,
or click on the Specify Host or Cluster search icon to select the cluster.
If the specified host belongs to a cluster, then you are prompted to choose if you
want to discover and promote all the database targets present on only on the
specified host, or all the database targets present on all the hosts that belong to the
cluster.
Click Next.
6.
On the Database Discovery: Results page, under the Databases section, in the
Cluster Databases section, select the cluster database target that you want to add.
2-32 OracleВ® Enterprise Manager Administration
Discovering, Promoting, and Adding Database Targets
By default, selecting a cluster database target for promotion also selects all its
associated discovered database instance targets for promotion. If you want to add
or remove a database instance target from the ones selected for promotion, select
the cluster database target, then click Configure. Select the Instances tab, then
click Add or Remove. Click Save.
7.
Specify the monitoring credentials for the selected cluster database target, that is,
the user name, password, and role. Also, if you want the selected target to be
added to a group, specify a value for Group.
If you specify Normal for Role, then the user name must be dbsnmp; you cannot
provide a different user name. However, if you specify SYSDBA for Role, then you
can provide any SYSDBA user. Only SYSDBA users and the dbsnmp user have the
privileges required for target monitoring.
8.
Click Test Connection to test the connection made to the cluster database target
using the specified monitoring credentials.
9.
To specify global target properties for all the targets you have selected on the
Promote Target: Results page, click Set Global Target Properties, specify the
required properties, then click OK.
To specify a common group for all the targets you have selected on the Promote
Target: Results page, click Specify Group for Targets, select a group, then click
Select.
If you have selected multiple databases and you want to set the same monitoring
properties for all of them, select Specify Common Monitoring Credentials. Enter
the monitoring credentials, monitoring password, and role. Click Apply.
10. Click Next.
11. Review the displayed information, then click Submit.
2.4.2.3 Discovering and Adding Cluster Database Targets By Specifying Target
Monitoring Properties
To discover and add a cluster database target declaratively by specifying target
monitoring properties, follow these steps:
в– Step 1: Discovering Unmanaged Hosts
Discovering, Promoting, and Adding Targets
2-33
Discovering, Promoting, and Adding Database Targets
в– Step 2: Converting Unmanaged Hosts to Managed Hosts
в– Step 3: Adding Cluster Database Targets
2.4.2.3.1
Step 1: Discovering Unmanaged Hosts
Skip this step if host is already managed in Enterprise
Manager.
Note:
Discover the unmanaged hosts, as described in Section 2.2.3.1.
2.4.2.3.2
Step 2: Converting Unmanaged Hosts to Managed Hosts
Skip this step if host is already managed in Enterprise
Manager.
Note:
Convert the discovered unmanaged hosts to managed hosts, as described in
Section 2.2.3.2.
2.4.2.3.3
Step 3: Adding Cluster Database Targets
To add a cluster database target, follow these steps:
1.
From the Setup menu, select Add Target, then select Add Targets Manually.
2.
On the Add Targets Add Manually page, select Add Targets Declaratively by
Specifying Target Monitoring Properties.
3.
From the Target Type list, select Cluster Database.
4.
In the Monitoring Agent field, select the Management Agent monitoring the
database.
5.
Click Add Manually.
6.
On the Configure Cluster Database: Properties, specify a name and a database
system name for the cluster database target. Next, specify all the properties of the
target, that is, Oracle Home path, monitoring username and password, role,
Listener machine name, port, database SID, and preferred connect string.
2-34 OracleВ® Enterprise Manager Administration
Discovering, Promoting, and Adding Database Targets
Click Next.
7.
Specify all the details required on the Install Packages, Credentials, and
Parameters pages. Click Next after each page until you reach the Review page.
8.
Review the displayed information, then click Submit.
2.4.3 Discovering Single Instance Database Targets
This section describes the different methods in which you can discover, promote, and
add single instance database targets. In particular, this section covers the following:
в– в– в– Discovering and Promoting Single Instance Database Targets Using Autodiscovery
Discovering and Adding Single Instance Database Targets Using Guided
Discovery Process
Discovering and Adding Single Instance Database Targets By Specifying Target
Monitoring Properties
2.4.3.1 Discovering and Promoting Single Instance Database Targets Using
Autodiscovery
To discover and promote single instance database targets using Auto Discovery, follow
these steps:
в– Step 1: Discovering Unmanaged Hosts
в– Step 2: Converting Unmanaged Hosts to Managed Hosts
в– Step 3: Promoting Single Instance Database Targets to Managed Status
2.4.3.1.1
Step 1: Discovering Unmanaged Hosts
Skip this step if host is already managed in Enterprise
Manager.
Note:
Discover the unmanaged hosts, as described to Section 2.2.1.2.1.
2.4.3.1.2
Step 2: Converting Unmanaged Hosts to Managed Hosts
Discovering, Promoting, and Adding Targets
2-35
Discovering, Promoting, and Adding Database Targets
Skip this step if host is already managed in Enterprise
Manager.
Note:
Convert the discovered unmanaged hosts to managed hosts, as described in
Section 2.2.1.2.2.
2.4.3.1.3
Step 3: Promoting Single Instance Database Targets to Managed Status
To promote single instance database targets, follow these steps:
1.
From the Setup menu, select Add Target, and then select Auto Discovery Results.
If you do not see any results, then autodiscovery of targets has
been disabled. To enable autodiscovery refer to Enabling
Autodiscovery of Database Targets.
Note:
From the results table, from the Agent-based targets tab, select the discovered
database instance target that you want to add for monitoring, and click Promote.
2.
The Promote Targets:Result page displays the databases Select the database
instance.
On the Promote Target: Results page, under the Databases section, in the Single
Instance Databases section, select the database instance target that you want to
promote.
2-36 OracleВ® Enterprise Manager Administration
Discovering, Promoting, and Adding Database Targets
3.
Specify the monitoring credentials for the selected database instance target, that is,
the Monitor user name, Monitor password, and role. Also, if you want the selected
target to be added to a group, specify a value for Group.
If you specify Normal for Role, then the user name must be dbsnmp; you cannot
provide a different user name. However, if you specify SYSDBA for Role, then you
can provide any SYSDBA user. Only SYSDBA users and the dbsnmp user have the
privileges required for target monitoring.
4.
Click Test Connection to test the connection made to the database instance target
using the specified monitoring credentials.
5.
To specify global target properties for all the targets you have selected on the
Promote Target: Results page, click Set Global Target Properties, specify the
required properties, then click OK.
To specify a common group for all the targets you have selected on the Promote
Target: Results page, click Specify Group for Targets, select a group, then click
Select.
6.
If you have selected multiple databases and you want to set the same monitoring
properties for all of them, select Specify Common Monitoring Credentials. Enter
the monitoring credentials, monitoring password, and role. Click Apply.
7.
Click Next.
8.
Review the displayed information, then click Submit.
2.4.3.2 Discovering and Adding Single Instance Database Targets Using Guided
Discovery Process
To discover and add single instance database targets using the guided process, follow
these steps:
в– Step 1: Discovering Unmanaged Hosts
в– Step 2: Converting Unmanaged Hosts to Managed Hosts
в– Step 3: Adding Single Instance Database Targets
2.4.3.2.1
Step 1: Discovering Unmanaged Hosts
Skip this step if host is already managed in Enterprise
Manager.
Note:
Discover the unmanaged hosts, as described in Section 2.2.2.1.
2.4.3.2.2
Step 2: Converting Unmanaged Hosts to Managed Hosts
Skip this step if host is already managed in Enterprise
Manager.
Note:
Convert the discovered unmanaged hosts to managed hosts, as described in
Section 2.2.2.2.
2.4.3.2.3
Step 3: Adding Single Instance Database Targets
To add single instance database targets, follow these steps:
Discovering, Promoting, and Adding Targets
2-37
Discovering, Promoting, and Adding Database Targets
1.
From the Setup menu, select Add Target, then select Add Targets Manually.
2.
On the Add Targets Add Manually page, select Add Targets Using Guided
Process (Also Adds Related Targets).
3.
From the Target Types list, select Oracle Database, Listener, and Automatic
Storage Management.
4.
Click Add Using Guided Process.
5.
On the Database Discovery: Search Criteria page, specify the database instance
host, or click on the Specify Host or Cluster search icon to select the host.
If the specified host belongs to a cluster, then you are prompted to choose if you
want to discover and promote all the database targets present on only on the
specified host, or all the database targets present on all the hosts that belong to the
cluster.
Click Next.
6.
The Database Discovery:Result page displays the databases discovered on the
cluster. Select the database
On the Database Discovery: Results page, under the Databases section, in the
Single Instance Databases section, select the database instance target that you want
to promote.
2-38 OracleВ® Enterprise Manager Administration
Discovering, Promoting, and Adding Database Targets
7.
Specify the monitoring credentials for the selected database instance target, that is,
the Monitor user name, Monitor password, and role. Also, if you want the selected
target to be added to a group, specify a value for Group.
If you specify Normal for Role, then the user name must be dbsnmp; you cannot
provide a different user name. However, if you specify SYSDBA for Role, then you
can provide any SYSDBA user. Only SYSDBA users and the dbsnmp user have the
privileges required for target monitoring.
8.
Click Test Connection to test the connection made to the database instance target
using the specified monitoring credentials.
9.
To specify global target properties for all the targets you have selected on the
Promote Target: Results page, click Set Global Target Properties, specify the
required properties, then click OK.
To specify a common group for all the targets you have selected on the Promote
Target: Results page, click Specify Group for Targets, select a group, then click
Select.
10. If you have selected multiple databases and you want to set the same monitoring
properties for all of them, select Specify Common Monitoring Credentials. Enter
the monitoring credentials, monitoring password, and role. Click Apply.
11. Click Next.
12. Review the displayed information, then click Submit.
2.4.3.3 Discovering and Adding Single Instance Database Targets By Specifying
Target Monitoring Properties
To discover and add a single instance database target declaratively by specifying target
monitoring properties, follow these steps:
в– Step 1: Discovering Unmanaged Hosts
в– Step 2: Converting Unmanaged Hosts to Managed Hosts
в– Step 3: Adding Single Instance Database Targets
2.4.3.3.1
Step 1: Discovering Unmanaged Hosts
Skip this step if host is already managed in Enterprise
Manager.
Note:
Discover the unmanaged hosts, as described in Section 2.2.3.1.
2.4.3.3.2
Step 2: Converting Unmanaged Hosts to Managed Hosts
Discovering, Promoting, and Adding Targets
2-39
Discovering, Promoting, and Adding Database Targets
Skip this step if host is already managed in Enterprise
Manager.
Note:
Convert the discovered unmanaged hosts to managed hosts, as described in
Section 2.2.3.2.
2.4.3.3.3
Step 3: Adding Single Instance Database Targets
To add single instance database targets, follow these steps:
1.
From the Setup menu, select Add Target, then select Add Targets Manually.
2.
On the Add Targets Add Manually page, select Add Targets Declaratively by
Specifying Target Monitoring Properties.
3.
From the Target Type list, select Database Instance.
4.
In the Monitoring Agent field, select the Management Agent monitoring the
database.
5.
Click Add Manually.
6.
On the Configure Database Instance: Properties, specify a name and a database
system name for the database instance target. Next, specify all the properties of the
target, that is, Oracle Home path, monitoring username and password, role,
Listener machine name, port, database SID, and preferred connect string.
Click Next.
7.
Specify all the details required on the Install Packages, Credentials, and
Parameters pages. Click Next after each page until you reach the Review page.
8.
Review the displayed information, then click Submit.
2-40 OracleВ® Enterprise Manager Administration
Discovering, Promoting, and Adding Database Targets
2.4.4 Discovering Cluster Targets
This section describes the different methods in which you can discover, promote, add
cluster targets. In particular, this section covers the following:
в– Discovering and Promoting Cluster Targets Using Autodiscovery
в– Discovering and Adding Cluster Targets Using the Guided Discovery Process
в– Discovering and Adding Cluster Targets By Specifying Target Monitoring
Properties
2.4.4.1 Discovering and Promoting Cluster Targets Using Autodiscovery
To automatically discover and promote cluster targets using Auto Discovery, follow
these steps:
в– Step 1: Discovering Unmanaged Hosts
в– Step 2: Converting Unmanaged Hosts to Managed Hosts
в– Step 3: Promoting Cluster Targets to Managed Status
2.4.4.1.1
Step 1: Discovering Unmanaged Hosts
Skip this step if host is already managed in Enterprise
Manager.
Note:
Discover the unmanaged hosts, as described in Section 2.2.1.2.1.
2.4.4.1.2
Step 2: Converting Unmanaged Hosts to Managed Hosts
Skip this step if host is already managed in Enterprise
Manager.
Note:
Convert the discovered unmanaged hosts to managed hosts, as described in
Section 2.2.1.2.2.
2.4.4.1.3
Step 3: Promoting Cluster Targets to Managed Status
To promote cluster targets, follow these steps:
1.
From the Setup menu, select Add Target, and then select Auto Discovery Results.
If you do not see any results, then autodiscovery of targets has
been disabled. To enable autodiscovery refer to Enabling
Autodiscovery of Database Targets.
Note:
From the results table, from the Agent-based targets tab, select the discovered
cluster target that you want to add for monitoring, and click Promote.
Discovering, Promoting, and Adding Targets
2-41
Discovering, Promoting, and Adding Database Targets
2.
The Promote Targets:Result page displays the hosts discovered on the cluster.
Select the host that you want to promote.
Note: A host can belong to only one cluster. If a particular host is not
displayed, it can mean that the host belongs to another cluster.
On the Promote Target: Results page, in the Clusters section, select the cluster
target that you want to promote.
By default, selecting the cluster target for promotion also selects all its associated
discovered hosts for promotion. If you want to add or remove a host from the ones
selected for promotion, select the cluster database target, then click Configure.
Select the Hosts tab, then click Add or Remove. Click Save.
3.
In the Clusters section, specify the monitoring credentials for the selected cluster
target, that is, the Monitor user name, Monitor password, and role. Also, if you
want the selected target to be added to a group, specify a value for Group.
If you specify Normal for Role, then the user name must be dbsnmp; you cannot
provide a different user name. However, if you specify SYSDBA for Role, then you
can provide any SYSDBA user. Only SYSDBA users and the dbsnmp user have the
privileges required for target monitoring.
4.
Click Test Connection to test the connection made to the cluster target using the
specified monitoring credentials.
5.
To specify global target properties for all the targets you have selected on the
Promote Target: Results page, click Set Global Target Properties, specify the
required properties, then click OK.
To specify a common group for all the targets you have selected on the Promote
Target: Results page, click Specify Group for Targets, select a group, then click
Select.
6.
If you have selected multiple databases and you want to set the same monitoring
properties for all of them, select Specify Common Monitoring Credentials. Enter
the monitoring credentials, monitoring password, and role. Click Apply.
7.
Click Next.
8.
Review the displayed information, then click Submit.
2-42 OracleВ® Enterprise Manager Administration
Discovering, Promoting, and Adding Database Targets
2.4.4.2 Discovering and Adding Cluster Targets Using the Guided Discovery
Process
To discover and add cluster targets using the guided process, follow these steps:
в– Step 1: Discovering Unmanaged Hosts
в– Step 2: Converting Unmanaged Hosts to Managed Hosts
в– Step 3: Adding Cluster Targets
2.4.4.2.1
Step 1: Discovering Unmanaged Hosts
Skip this step if host is already managed in Enterprise
Manager.
Note:
Discover the unmanaged hosts, as described in Section 2.2.2.1.
2.4.4.2.2
Step 2: Converting Unmanaged Hosts to Managed Hosts
Skip this step if host is already managed in Enterprise
Manager.
Note:
Convert the discovered unmanaged hosts to managed hosts, as described in
Section 2.2.2.2.
2.4.4.2.3
Step 3: Adding Cluster Targets
To add cluster targets, follow these steps:
1.
From the Setup menu, select Add Target, then select Add Targets Manually.
2.
On the Add Targets Add Manually page, select Add Targets Using Guided
Process (Also Adds Related Targets).
3.
From the Target Types list, select Oracle Database, Listener, and Automatic
Storage Management.
4.
Click Add Using Guided Process.
5.
On the Cluster Discovery: Specify Host page, specify the cluster host, or click on
the Specify Host or Cluster search icon to select the cluster.
If the specified host belongs to a cluster, then you are prompted to choose if you
want to discover and promote all the database targets present on only on the
specified host, or all the database targets present on all the hosts that belong to the
cluster.
Click Next.
Discovering, Promoting, and Adding Targets
2-43
Discovering, Promoting, and Adding Database Targets
6.
The Cluster Discovery:Result page displays the hosts discovered on the cluster.
Select the host that you want to add.
Note: A host can belong to only one cluster. If a particular host is not
displayed, it can mean that the host belongs to another cluster.
On the Cluster Discovery: Result page, in the Clusters Target Properties section,
verify the properties of the cluster.
If you want to add or remove a host, select a host from the Cluster Host and High
Availability Service Targets section. Click Add or Remove.
7.
Click Next.
8.
Review the displayed information, then click Submit.
2.4.4.3 Discovering and Adding Cluster Targets By Specifying Target Monitoring
Properties
To discover and add a cluster target declaratively by specifying target monitoring
properties, follow these steps:
в– Step 1: Discovering Unmanaged Hosts
в– Step 2: Converting Unmanaged Hosts to Managed Hosts
в– Step 3: Adding Cluster Targets
2.4.4.3.1
Step 1: Discovering Unmanaged Hosts
Skip this step if host is already managed in Enterprise
Manager.
Note:
Discover the unmanaged hosts, as described in Section 2.2.3.1.
2.4.4.3.2
Step 2: Converting Unmanaged Hosts to Managed Hosts
Skip this step if host is already managed in Enterprise
Manager.
Note:
2-44 OracleВ® Enterprise Manager Administration
Discovering, Promoting, and Adding Database Targets
Convert the discovered unmanaged hosts to managed hosts, as described in
Section 2.2.3.2.
2.4.4.3.3
Step 3: Adding Cluster Targets
To add cluster targets, follow these steps:
1.
From the Setup menu, select Add Target, then select Add Targets Manually.
2.
On the Add Targets Add Manually page, select Add Targets Declaratively by
Specifying Target Monitoring Properties.
3.
From the Target Type list, select Cluster.
4.
In the Monitoring Agent field, select the Management Agent monitoring the
database.
5.
Click Add Manually.
6.
On the Cluster Discovery:Result page, specify the target name, Oracle Home,
SCAN name, SCAN port, and ONS port.
The SCAN name, SCAN port, and ONS port properties are
applicable only for Clusterware versions 11.2 and higher.
Note:
7.
You can add more hosts and high availability service targets to the cluster. If a
particular host is not displayed when you click Add, it is possible that the host
already belongs to another cluster.
To specify global target properties for all the targets you have selected on the
Promote Target: Results page, click Set Global Target Properties, specify the
required properties, then click OK.
Click Next.
8.
Review the displayed information, then click Submit.
Discovering, Promoting, and Adding Targets
2-45
Discovering, Promoting, and Adding Database Targets
2.4.5 Discovering Single Instance High Availability Service Targets
This section describes the different methods in which you can discover, promote, and
add single instance high availability service targets. In particular, this section covers
the following:
в– в– в– Discovering and Promoting Single Instance High Availability Service Targets
Using Autodiscovery
Discovering and Adding Single Instance High Availability Service Targets Using
the Guided Discovery Process
Discovering and Adding Single Instance High Availability Service Targets By
Specifying Target Monitoring Properties
2.4.5.1 Discovering and Promoting Single Instance High Availability Service
Targets Using Autodiscovery
To automatically discover and promote single instance high availability service targets
using Auto Discovery, follow these steps:
в– Step 1: Discovering Unmanaged Hosts
в– Step 2: Converting Unmanaged Hosts to Managed Hosts
в– Step 3: Promoting Single Instance High Availability Targets to Managed Status
2.4.5.1.1
Step 1: Discovering Unmanaged Hosts
Skip this step if host is already managed in Enterprise
Manager.
Note:
Discover the unmanaged hosts, as described in Section 2.2.1.2.1.
2.4.5.1.2
Step 2: Converting Unmanaged Hosts to Managed Hosts
Skip this step if host is already managed in Enterprise
Manager.
Note:
Convert the unmanaged hosts to managed hosts, as described in Section 2.2.1.2.2.
2.4.5.1.3
Step 3: Promoting Single Instance High Availability Targets to Managed Status
To promote single instance high availability targets, follow these steps:
1.
From the Setup menu, select Add Target, and then select Auto Discovery Results.
If you do not see any results, then autodiscovery of targets has
been disabled. To enable autodiscovery refer to Enabling
Autodiscovery of Database Targets.
Note:
From the results table, from the Agent-based targets tab, select the discovered
High Availability instance target that you want to add for monitoring, and click
Promote.
2-46 OracleВ® Enterprise Manager Administration
Discovering, Promoting, and Adding Database Targets
2.
The Promote Targets:Result page displays the High Availability instances
discovered. Select the database
On the Promote Target: Results page, under the High Availability Services section,
select the SIHA target that you want to promote.
3.
Specify the monitoring credentials for the selected SIHA target, that is, the Monitor
user name, Monitor password, and role. Also, if you want the selected target to be
added to a group, specify a value for Group.
If you specify Normal for Role, then the user name must be dbsnmp; you cannot
provide a different user name. However, if you specify SYSDBA for Role, then you
can provide any SYSDBA user. Only SYSDBA users and the dbsnmp user have the
privileges required for target monitoring.
4.
Click Test Connection to test the connection made to the SIHA target using the
specified monitoring credentials.
5.
To specify global target properties for all the targets you have selected on the
Promote Target: Results page, click Set Global Target Properties, specify the
required properties, then click OK.
To specify a common group for all the targets you have selected on the Promote
Target: Results page, click Specify Group for Targets, select a group, then click
Select.
6.
If you have selected multiple databases and you want to set the same monitoring
properties for all of them, select Specify Common Monitoring Credentials. Enter
the monitoring credentials, monitoring password, and role. Click Apply.
7.
Click Next.
8.
Review the displayed information, then click Submit.
2.4.5.2 Discovering and Adding Single Instance High Availability Service Targets
Using the Guided Discovery Process
To discover and add single instance high availability service targets using the guided
process, follow these steps:
в– Step 1: Discovering Unmanaged Hosts
в– Step 2: Converting Unmanaged Hosts to Managed Hosts
в– Step 3: Adding Single Instance High Availability Service Targets
2.4.5.2.1
Step 1: Discovering Unmanaged Hosts
Discovering, Promoting, and Adding Targets
2-47
Discovering, Promoting, and Adding Database Targets
Skip this step if host is already managed in Enterprise
Manager.
Note:
Discover the unmanaged hosts, as described in Section 2.2.2.1.
2.4.5.2.2
Step 2: Converting Unmanaged Hosts to Managed Hosts
Skip this step if host is already managed in Enterprise
Manager.
Note:
Convert the discovered unmanaged hosts to managed hosts, as described in
Section 2.2.2.2.
2.4.5.2.3
Step 3: Adding Single Instance High Availability Service Targets
To add single instance high availability service targets, follow these steps:
1.
From the Setup menu, select Add Target, then select Add Targets Manually.
2.
On the Add Targets Add Manually page, select Add Targets Using Guided
Process (Also Adds Related Targets).
3.
From the Target Types list, select Oracle Cluster and High Availability Service.
4.
Click Add Using Guided Process.
5.
On the Cluster Discovery: Specify Host page, specify the single instance high
availability service host, or click on the Specify Host or Cluster search icon to
select the cluster.
Click Next.
6.
The Cluster Discovery:Result page displays the high availability service instances
discovered.
On the Cluster Discovery: Result page, under the High Availability Services
section, select the high availability service instance target that you want to
promote.
2-48 OracleВ® Enterprise Manager Administration
Discovering, Promoting, and Adding Database Targets
7.
Click Next.
8.
Review the displayed information, then click Submit.
2.4.5.3 Discovering and Adding Single Instance High Availability Service Targets
By Specifying Target Monitoring Properties
To discover and add a single instance High Availability Service target declaratively by
specifying target monitoring properties, follow these steps:
в– Step 1: Discovering Unmanaged Hosts.
в– Step 2: Converting Unmanaged Hosts to Managed Hosts.
в– Step 3: Adding Single Instance High Availability Service Targets.
2.4.5.3.1
Step 1: Discovering Unmanaged Hosts
Skip this step if host is already managed in Enterprise
Manager.
Note:
Discover the unmanaged hosts, as described in Section 2.2.3.1.
2.4.5.3.2
Step 2: Converting Unmanaged Hosts to Managed Hosts
Skip this step if host is already managed in Enterprise
Manager.
Note:
Convert the discovered unmanaged hosts to managed hosts, as described in
Section 2.2.3.2.
2.4.5.3.3
Step 3: Adding Single Instance High Availability Service Targets
To add single instance high availability service targets, follow these steps:
1.
From the Setup menu, select Add Target, then select Add Targets Manually.
2.
On the Add Targets Add Manually page, select Add Targets Declaratively by
Specifying Target Monitoring Properties.
3.
From the Target Type list, select Oracle High Availability Service.
4.
In the Monitoring Agent field, select the Management Agent monitoring the
database.
Discovering, Promoting, and Adding Targets
2-49
Discovering, Promoting, and Adding Database Targets
5.
Click Add Manually.
6.
On the High Availability Service: Result page, specify a name for the target, the
Oracle home, and the ONS port.
To specify global target properties for all the targets you have selected on the
Promote Target: Results page, click Set Global Target Properties, specify the
required properties, then click OK.
Click Next.
7.
Review the displayed information, and then click Submit.
2.4.6 Discovering Cluster Automatic Storage Management Targets
This section describes the different methods in which you can discover, promote, and
add ASM cluster targets. In particular, this section covers the following:
в– в– в– Discovering and Promoting Cluster ASM Targets Using Autodiscovery
Discovering and Adding Cluster ASM Targets Using the Guided Discovery
Process
Discovering and Adding Cluster ASM Targets By Specifying Target Monitoring
Properties
2.4.6.1 Discovering and Promoting Cluster ASM Targets Using Autodiscovery
To discover and promote Cluster Automatic Storage Management targets using
autodiscovery, follow these steps:
в– Step 1: Discovering Unmanaged Hosts
в– Step 2: Converting Unmanaged Hosts to Managed Hosts
в– Step 3: Promoting Cluster ASM Targets to Managed Status
2.4.6.1.1
Step 1: Discovering Unmanaged Hosts
2-50 OracleВ® Enterprise Manager Administration
Discovering, Promoting, and Adding Database Targets
Skip this step if host is already managed in Enterprise
Manager.
Note:
Discover the unmanaged hosts, as described in Section 2.2.1.2.1.
2.4.6.1.2
Step 2: Converting Unmanaged Hosts to Managed Hosts
Skip this step if host is already managed in Enterprise
Manager.
Note:
Convert the unmanaged hosts to managed hosts, as described in Section 2.2.1.2.2.
2.4.6.1.3
Step 3: Promoting Cluster ASM Targets to Managed Status
To promote cluster ASM targets, follow these steps:
1.
From the Setup menu, select Add Target, and then select Auto Discovery Results.
If you do not see any results, then autodiscovery of targets has
been disabled. To enable autodiscovery refer to Enabling
Autodiscovery of Database Targets.
Note:
From the results table, from the Agent-based targets tab, select the discovered
cluster ASM target that you want to add for monitoring, and click Promote.
2.
The Promote Targets:Result page displays the targets discovered on the cluster
ASM.
On the Promote Target: Results page, in the Cluster ASM section, select the target
that you want to promote.
By default, selecting the cluster ASM target for promotion also selects all its
associated discovered targets for promotion. If you want to add or remove a target
from the ones selected for promotion, select the cluster ASM target, then click
Configure. Select the Instances tab, then click Add or Remove. Click Save.
3.
In the cluster ASM section, specify the monitoring credentials for the selected
cluster ASM target, that is, the Monitor user name, Monitor password, and role.
Discovering, Promoting, and Adding Targets
2-51
Discovering, Promoting, and Adding Database Targets
Also, if you want the selected target to be added to a group, specify a value for
Group.
If you specify Normal for Role, then the user name must be dbsnmp; you cannot
provide a different user name. However, if you specify SYSDBA for Role, then you
can provide any SYSDBA user. Only SYSDBA users and the dbsnmp user have the
privileges required for target monitoring.
4.
Click Test Connection to test the connection made to the cluster ASM target using
the specified monitoring credentials.
5.
To specify global target properties for all the targets you have selected on the
Promote Target: Results page, click Set Global Target Properties, specify the
required properties, then click OK.
To specify a common group for all the targets you have selected on the Promote
Target: Results page, click Specify Group for Targets, select a group, then click
Select.
6.
If you have selected multiple targets and you want to set the same monitoring
properties for all of them, select Specify Common Monitoring Credentials. Enter
the monitoring credentials, monitoring password, and role. Click Apply.
7.
Click Next.
8.
Review the displayed information, then click Submit.
2.4.6.2 Discovering and Adding Cluster ASM Targets Using the Guided Discovery
Process
To discover and add cluster ASM targets using the guided process, follow these steps:
в– Step 1: Discovering Unmanaged Hosts.
в– Step 2: Converting Unmanaged Hosts to Managed Hosts.
в– Step 3: Adding Cluster ASM Targets.
2.4.6.2.1
Step 1: Discovering Unmanaged Hosts
Skip this step if host is already managed in Enterprise
Manager.
Note:
Discover the unmanaged hosts, as described in Section 2.2.2.1.
2.4.6.2.2
Step 2: Converting Unmanaged Hosts to Managed Hosts
Skip this step if host is already managed in Enterprise
Manager.
Note:
Convert the discovered unmanaged hosts to managed hosts, as described in
Section 2.2.2.2.
2.4.6.2.3
Step 3: Adding Cluster ASM Targets
To add cluster ASM targets, follow these steps:
1.
From the Setup menu, select Add Target, then select Add Targets Manually.
2-52 OracleВ® Enterprise Manager Administration
Discovering, Promoting, and Adding Database Targets
2.
On the Add Targets Add Manually page, select Add Targets Using Guided
Process (Also Adds Related Targets).
3.
From the Target Types list, select Oracle Database, Listener, and Automatic
Storage Management.
4.
Click Add Using Guided Process.
5.
On the Database Discovery: Search Criteria page, specify the cluster ASM host, or
click on the Specify Host or Cluster search icon to select the cluster.
If the specified host belongs to a cluster, then you are prompted to choose if you
want to discover and promote all the database targets present on only on the
specified host, or all the database targets present on all the hosts that belong to the
cluster.
Click Next.
6.
The Database Discovery:Result page displays the targets discovered on the cluster
ASM target.
On the Database Discovery: Results page, in the Cluster ASM section, select the
target that you want to promote.
By default, selecting the cluster ASM target for promotion also selects all its
associated discovered targets for promotion. If you want to add or remove a target
Discovering, Promoting, and Adding Targets
2-53
Discovering, Promoting, and Adding Database Targets
from the ones selected for promotion, select the cluster ASM target, then click
Configure. Select the Instances tab, then click Add or Remove. Click Save.
7.
In the cluster ASM section, specify the monitoring credentials for the selected
cluster ASM target, that is, the Monitor user name, Monitor password, and role.
Also, if you want the selected target to be added to a group, specify a value for
Group.
If you specify Normal for Role, then the user name must be dbsnmp; you cannot
provide a different user name. However, if you specify SYSDBA for Role, then you
can provide any SYSDBA user. Only SYSDBA users and the dbsnmp user have the
privileges required for target monitoring.
8.
Click Test Connection to test the connection made to the cluster ASM target using
the specified monitoring credentials.
9.
To specify global target properties for all the targets you have selected on the
Promote Target: Results page, click Set Global Target Properties, specify the
required properties, then click OK.
To specify a common group for all the targets you have selected on the Promote
Target: Results page, click Specify Group for Targets, select a group, then click
Select.
10. If you have selected multiple targets and you want to set the same monitoring
properties for all of them, select Specify Common Monitoring Credentials. Enter
the monitoring credentials, monitoring password, and role. Click Apply.
11. Click Next.
12. Review the displayed information, then click Submit.
2.4.6.3 Discovering and Adding Cluster ASM Targets By Specifying Target
Monitoring Properties
To discover and add a cluster ASM target declaratively by specifying target
monitoring properties, follow these steps:
в– Step 1: Discovering Unmanaged Hosts
в– Step 2: Converting Unmanaged Hosts to Managed Hosts
в– Step 3: Adding Cluster ASM Targets
2.4.6.3.1
Step 1: Discovering Unmanaged Hosts
Skip this step if host is already managed in Enterprise
Manager.
Note:
Discover the unmanaged hosts, as described in Section 2.2.3.1.
2.4.6.3.2
Step 2: Converting Unmanaged Hosts to Managed Hosts
Skip this step if host is already managed in Enterprise
Manager.
Note:
Convert the discovered unmanaged hosts to managed hosts, as described in
Section 2.2.3.2.
2-54 OracleВ® Enterprise Manager Administration
Discovering, Promoting, and Adding Database Targets
2.4.6.3.3
Step 3: Adding Cluster ASM Targets
To add a cluster ASM target, follow these steps:
1.
From the Setup menu, select Add Target, then select Add Targets Manually.
2.
On the Add Targets Add Manually page, select Add Targets Declaratively by
Specifying Target Monitoring Properties.
3.
From the Target Type list, select Cluster ASM.
4.
In the Monitoring Agent field, select the Management Agent monitoring the
database.
5.
Click Add Manually.
6.
On the Configure Cluster ASM: Properties page, specify a name for the Cluster
ASM, the Oracle home path, username and password, role, cluster name, and
service name.
The Service Name is used to establish the cluster ASM
connection. It should be one of the service names the cluster ASM
registers with the listeners.
Note:
7.
You can add instances, ASM IO server instances, and ASM proxy instances by
clicking Add in the respective sections.
Click Test Connection to test the connection made to the cluster ASM target using
the specified monitoring credentials.
OK.
2.4.7 Enabling Autodiscovery of Database Targets
Autodiscovery of database targets is enabled by default. If autodiscovery has been
disabled, you can enable it, by following these steps:
1.
From the Setup menu, select Add Target, then select Configure Auto Discovery.
2.
On the Configure Auto Discovery page, in the Agent-based Auto Discovery table,
select Oracle Database, Listener, and Automatic Storage Management.
Discovering, Promoting, and Adding Targets
2-55
Discovering, Promoting, and Adding Middleware Targets
3.
On the Configure Target Discovery page, click Add Host.
4.
From the Search and Select Targets dialog box, select a target that you want to be
configured, and click Select.
5.
You can click Edit Parameters to edit the parameters of the target.
Click OK.
2.5 Discovering, Promoting, and Adding Middleware Targets
This section describes how you can discover, promote, and add fusion middleware
targets to be managed by Cloud Control. In particular, this section covers the
following:
в– Discovering Weblogic 9.x or 10.x Domains
в– Discovering New or Modified Domain Members
в– Discovering a Standalone Oracle HTTP Server
в– Discovering Exalytics Targets
в– Removing Middleware Targets
2.5.1 Discovering Weblogic 9.x or 10.x Domains
This section describes the different methods in which you can discover, promote, and
add weblogic domain targets in Cloud Control. In particular, this section covers the
following:
2-56 OracleВ® Enterprise Manager Administration
Discovering, Promoting, and Adding Middleware Targets
в– в– в– Discovering and Promoting Weblogic Domains Using Autodiscovery
Discovering and Adding WebLogic 9.x or 10.x Domains Using the Guided
Discovery Process
Adding Multiple WebLogic Domains Using EM CLI
2.5.1.1 Discovering and Promoting Weblogic Domains Using Autodiscovery
To discover and promote Weblogic domains, follow these steps:
Note: The automatic discovery feature is not supported for Oracle WebLogic Server
version 8.x.
в– Step 1: Discovering Unmanaged Hosts
в– Step 2: Converting Unmanaged Hosts to Managed Hosts
в– Step 3: Discovering Weblogic Domains on Managed Hosts
в– Step 4: Promoting Weblogic Domains to Managed Status
2.5.1.1.1
Step 1: Discovering Unmanaged Hosts
Skip this step if host is already managed in Enterprise
Manager.
Note:
Discover the unmanaged hosts, as described in Section 2.2.1.2.1.
2.5.1.1.2
Step 2: Converting Unmanaged Hosts to Managed Hosts
Skip this step if host is already managed in Enterprise
Manager.
Note:
Convert the discovered unmanaged hosts to managed hosts, as described in
Section 2.2.1.2.2.
2.5.1.1.3
Step 3: Discovering Weblogic Domains on Managed Hosts
To discover weblogic domains on managed hosts, follow these steps:
1.
From the Setup menu, select Add Target, then select Configure Auto Discovery.
Figure 2–5 Configure Auto Discovery Option
Discovering, Promoting, and Adding Targets
2-57
Discovering, Promoting, and Adding Middleware Targets
2.
On the Configure Auto Discovery page, select the Oracle Fusion Middleware link
in the table to configure auto discovery for Oracle Fusion Middleware or click the
icon in the Configure Host Discovery column to configure that Oracle Fusion
Middleware row.
Figure 2–6 Configure Auto Discovery Page
3.
Set the schedule at which the discovery job will be run, in days. This schedule will
be applied to all selected hosts. By default the job will run every 24 hours.
Figure 2–7 Schedule for Configuring Target Discovery
4.
Click Add Host. Select the host machines you want to include in the discovery.
2-58 OracleВ® Enterprise Manager Administration
Discovering, Promoting, and Adding Middleware Targets
Figure 2–8 Host Machines Available for Discovery
5.
Select a host in the table, and then click Edit Parameters to specify the Middleware
Homes to search for targets. The Middleware Home is the top-level directory for
all Oracle Fusion Middleware products, created when Oracle WebLogic Server is
installed.
Enter * to search all Middleware Homes, or specify the path to one or more
Middleware Homes on the host, each separated by a comma.
Figure 2–9 Edit Parameters
6.
Click the OK button located at the right of the screen. At this point, automatic
discovery has been enabled, and the discovery job will run at the scheduled
frequency.
2.5.1.1.4
Step 4: Promoting Weblogic Domains to Managed Status
To promote weblogic domains, follow these steps:
1.
From the Setup menu, select Add Target, then select Auto Discovery Results.
Discovering, Promoting, and Adding Targets
2-59
Discovering, Promoting, and Adding Middleware Targets
Figure 2–10 Auto Discovery Results Option
2.
Click the Non-Host Targets tab to view the discovered Oracle Fusion Middleware
targets.
Figure 2–11 Auto Discovery Results Page
3.
Select a target, then click Promote.
If multiple targets of various types are listed, you can expand Search, then select
the Target Type you are looking for (such as Oracle WebLogic Domain). Click
Search to display the selected discovered target types.
2-60 OracleВ® Enterprise Manager Administration
Discovering, Promoting, and Adding Middleware Targets
Figure 2–12 Promote Tab
4.
Supply or accept values for the following parameters:
Figure 2–13 shows the parameters that need to be provided.
Figure 2–13 Parameters
в– Administration Server Host
Enter the host name on which the Administration Server is installed and
running for the Oracle WebLogic Domain that you want to promote to a
managed target, for example: myhost06.example.com
в– Port
Discovering, Promoting, and Adding Targets
2-61
Discovering, Promoting, and Adding Middleware Targets
Enter the WebLogic Administration Server port. The default is 7001.
If the WebLogic Administration Server is secured using the Secure Sockets
Layer (SSL) protocol, specify the location of the trusted keystore file. The
keystore is a protected database that holds keys and certificates for an
enterprise. See Advanced Parameters.
в– Enter the WebLogic Administration Server user name and password.
If you want to discover the target only for monitoring purposes, then it is
sufficient to provide a user name that has a monitoring role. If you want to
monitor the target and also perform start/stop operations, then ensure that
you provide a user name that has either an operator role or an administrator
role.
Note: There is the potential of account locking issues if you enter the default
WebLogic user name, and the account password is changed without updating
the Enterprise Manager monitoring credentials for the Domain and Farm.
в– Unique Domain Identifier.
Specify a Unique Domain Identifier. This value is used as a prefix to ensure
farm names are unique in environments with the same domain name. By
default, Enterprise Manager will pre-pend the name with "Farm", followed by
a two-digit number, such as "Farm01".
в– Agent
Host name for a Management Agent that will be used to discover the Fusion
Middleware targets.
If a Management Agent exists on the WebLogic Administration Server host,
the host name for this Management Agent will be supplied by default.
However, you can specify any Management Agent on any host that is
managed by Cloud Control to perform the discovery.
Note: To access all supported features, Oracle recommends that you have a
Management Agent local to each managed server in the domain and to use
that Management Agent to monitor the WebLogic Servers on that local host
machine. Though remote Management Agents can manage WebLogic Server
targets, the local Management Agent is recommended.
Some features that are not supported when there is no local Management
Agent:
–
To patch a WebLogic Server, you need a local Management Agent on each
WebLogic Server machine.
–
If you want to use Oracle Support Workbench for a WebLogic Server
target, then the target requires a local Management Agent.
–
Cloning requires a local Management Agent at the source administration
server machine and a local Management Agent at all destination
machines.
Advanced Parameters
If the target domain is secured, expand the Advanced node to specify the protocol
to use in the Protocol field. The default value is t3.
For additional details on discovering a domain secured using the Secure Sockets
Layer (SSL) protocol, see section "C" in My Oracle Support Note 1093655.1. You
can access My Oracle Support at the following URL:
2-62 OracleВ® Enterprise Manager Administration
Discovering, Promoting, and Adding Middleware Targets
https://support.oracle.com/CSP/ui/flash.html
в– JMX Protocol
Used to make a JMX connections to the Administration Server. For Secure
domain JMX protocol - use t3s. If WebLogic domain is using a demo
certificate, this certificate is automatically updated to monitoring and
discovery agent. If a custom certificate is used, refer to Monitoring Weblogic
Domains for information on how to import a certificate.
в– Discover Down Servers
Use this check box to discover servers that are down. While adding Oracle
Fusion Middleware WebLogic Domain targets to Cloud Control, you can now
choose whether to add WebLogic Server targets that are discovered in a down
state. This gives you more control in determining what to automatically add to
Cloud Control for centralized management and monitoring.
To monitor down servers, their Listener Address must be set. Otherwise, these
servers will have’temp-host’ as host value and the local agent cannot be
determined for monitoring. Therefore, the servers will always be shown as
down.
When servers come up, this WebLogic domain needs to be refreshed for
populating the correct host name and monitoring.
в– JMX Service URL
Optionally supply the Java Management Extensions (JMX) Service URL that
will be used to establish a JMX connection to the WebLogic Administration
Server. For example:
service:jmx:t3://server.example.com:5555/jndi/weblogic.management.m
beanservers.domainruntime
If you do not specify this URL, Enterprise Manager will provide the Service
URL based on the host port and protocol. If this is specified, the
Administration server host and port information still must be provided in the
input parameters.
в– Discover Application Versions
By default, each version of a deployed Java EE application will be discovered
as a target. Therefore, with every new version, a new target will be discovered.
Deselect this check box if you want only the active version of the application
to be discovered.
в– Enable Automatic Refresh
This option refreshes the weblogic domain every 24 hours.
в– Use Host Name in Service URL
You can use host name in service URL instead of JMX. It is recommended to
use this option if you are using a private network and there are many hosts
using the same IP address.
в– Create Incident for Discovery Failure
This option creates an OMS incident if discovery fails. You can view the
incident from the Support workbench page.
в– External Parameters
Discovering, Promoting, and Adding Targets
2-63
Discovering, Promoting, and Adding Middleware Targets
Optionally enter any system properties to be used by the Java process to
connect to the WebLogic Administration Server in the Parameters field.
Supply space-separated name/value pairs. Preface each parameter with -D.
For example:
-Dparam1=xxx -Dparam2=yyy -Dparam3=zzz
в– Discovery Debug File Name
If problems occur while adding Middleware-related targets or refreshing
domain membership, you can enable additional debugging information to
quickly diagnose and resolve the issue. This file will be created in the
discovery Agent's log directory.
5.
Click Continue. Enterprise Manager will discover all Fusion Middleware targets
within the domain.
6.
Click Close in the Finding Targets dialog to automatically assign Management
Agents to the discovered targets.
The Assign Agents page lists each Fusion Middleware target discovered and the
Management Agent assigned to each. Agents are automatically assigned as
follows:
в– в– If a local Management Agent is installed on the discovered target host, that
Agent will be assigned.
If a local Management Agent cannot be found on the host, the Agent specified
in the Targets page will be assigned.
Note that you can also manually assign Management Agents to specific targets, if
desired.
7.
Click Add Targets to assign Management Agents as listed in the Assign Agents
page.
The Saving Target to Agent processing window appears, indicating how many
total targets have been added and successfully saved. It will also indicate the
number of targets that were unsuccessfully added.
8.
Click Close in the processing window when finished. The Results page displays
the targets and Agent assignments.
9.
Click OK when finished. There may be a delay before these targets are visible and
monitored. All the agents used for monitoring the targets must be up.
After you discover a middleware target for the first time, it is
recommended that you learn the best practices for monitoring and
managing the discovered target. To navigate to the Target
Management Best Practices page, from the target home page, select
the Weblogic Domain menu, and then select Target Management
Best Practices. The page lists the best practices items for a Fusion
Middleware Domain.
Note:
2.5.1.2 Discovering and Adding WebLogic 9.x or 10.x Domains Using the Guided
Discovery Process
Oracle WebLogic Server release 9.x and 10.x domains and their respective components
can be discovered using Cloud Control. A wizard guides you through the discovery
process.
2-64 OracleВ® Enterprise Manager Administration
Discovering, Promoting, and Adding Middleware Targets
Fusion Middleware 12c discovery also discovers dynamic
servers, CAM (Common Administration Model) managed
components, as well as a new target type called Domain Application.
Note:
Domain application represents the deployment of a target to a
domain.
For Fusion Middleware 12c discovery, when Cloud Control discovers a Fusion
Middleware WebLogic domain, it creates a Domain Application target for each Java EE
application deployed in a Weblogic domain. For any Java EE application deployed on
any number of servers, only one Domain Application target will be discovered.
To discover a WebLogic Server domain, the Administration
Server must be up because the Management Agent must make a JMX
connection to it. If the Administration Server is down, discovery
cannot occur.
Note:
Thereafter, to monitor the WebLogic Server domain, the
Administration Server need not be up.
To discover and add a weblogic domain using the guided process, follow these steps:
в– Step 1: Discovering Unmanaged Hosts
в– Step 2: Converting Unmanaged Hosts to Managed Hosts
в– Step 3: Adding Weblogic Domains
2.5.1.2.1
Step 1: Discovering Unmanaged Hosts
Skip this step if host is already managed in Enterprise
Manager.
Note:
Discover the unmanaged host, as described in Section 2.2.2.1.
2.5.1.2.2
Step 2: Converting Unmanaged Hosts to Managed Hosts
Skip this step if host is already managed in Enterprise
Manager.
Note:
Convert the discovered unmanaged hosts to managed hosts, as described in
Section 2.2.2.2.
2.5.1.2.3
Step 3: Adding Weblogic Domains
To add a weblogic domain, follow these steps:
1.
From the Setup menu, select Add Target, then select Add Targets Manually.
2.
Select the Add Non-Host Targets Using Guided Process (Also Adds Related
Targets) option.
3.
Select Oracle Fusion Middleware/Weblogic Domain as the target type.
4.
Click Add Using Guided Discovery.
Discovering, Promoting, and Adding Targets
2-65
Discovering, Promoting, and Adding Middleware Targets
5.
Supply or accept values for the following parameters:
в– Administration Server Host
Enter the host name on which the Administration Server is installed and
running for the Oracle WebLogic Domain that you want to promote to a
managed target, for example: myhost06.example.com
в– Port
Enter the WebLogic Administration Server port. The default value is 7001.
If the WebLogic Administration Server is secured using the Secure Sockets
Layer (SSL) protocol, specify the location of the trusted keystore file. The
keystore is a protected database that holds keys and certificates for an
enterprise. See Advanced Parameters.
в– Enter the WebLogic Administration Server user name and password.
If you want to discover the target only for monitoring purposes, then it is
sufficient to provide a user name that has a monitoring role. If you want to
monitor the target and also perform start/stop operations, then ensure that
you provide a user name that has either an operator role or an administrator
role.
в– Unique Domain Identifier.
2-66 OracleВ® Enterprise Manager Administration
Discovering, Promoting, and Adding Middleware Targets
Specify a Unique Domain Identifier. This value is used as a prefix to ensure
farm names are unique in environments with the same domain name. By
default, Enterprise Manager will pre-pend the name with "Farm", followed by
a two-digit number, such as, "Farm01".
в– Agent
The host name for a Management Agent that will be used to discover the
Fusion Middleware targets.
If a Management Agent exists on the WebLogic Administration Server host,
the host name for this Management Agent will be supplied by default.
However, you can specify any Management Agent on any host that is
managed by Cloud Control to perform the discovery.
Note: To access all supported features, Oracle recommends that you have a
Management Agent local to each managed server in the domain and to use
that Management Agent to monitor the WebLogic Servers on that local host
machine. Though remote Management Agents can manage WebLogic Server
targets, the local Management Agent is recommended.
Some features that are not supported when there is no local Management
Agent:
–
To patch a WebLogic Server, you need a local Management Agent on each
WebLogic Server machine.
–
If you want to use Oracle Support Workbench for a WebLogic Server
target, then the target requires a local Management Agent.
–
Cloning requires a local Management Agent at the source administration
server machine and a local Management Agent at all destination
machines.
Advanced Parameters
If the target domain is secured, expand the Advanced node to specify the protocol
to use in the Protocol field. The default value is t3.
For additional details on discovering a domain secured using the Secure Sockets
Layer (SSL) protocol, see section "C" in My Oracle Support Note 1093655.1. You
can access My Oracle Support at the following URL:
https://support.oracle.com/CSP/ui/flash.html
в– JMX Protocol
Used to make a JMX connections to the Administration Server. For Secure
domain JMX protocol - use t3s. If WebLogic domain is using a demo
certificate, this certificate is automatically updated to monitoring and
discovery Agent.
в– Discover Down Servers
Use this check box to discover servers that are down. While adding Oracle
Fusion Middleware WebLogic Domain targets to Cloud Control, you can now
choose whether to add WebLogic Server targets that are discovered in a down
state. This gives you more control in determining what to automatically add to
Cloud Control for centralized management and monitoring.
To monitor down servers, their Listener Address must be set. Otherwise, these
servers will have’temp-host’ as host value and the local agent cannot be
determined for monitoring. Therefore, the servers will always be shown as
down.
Discovering, Promoting, and Adding Targets
2-67
Discovering, Promoting, and Adding Middleware Targets
When servers come up, this WebLogic domain needs to be refreshed for
populating the correct host name and monitoring.
в– Discover Application Versions
By default, each version of a deployed Java EE application will be discovered
as a target. Therefore, with every new version, a new target will be discovered.
Deselect this check box if you want only the active version of the application
to be discovered.
в– JMX Service URL
Optionally supply the Java Management Extensions (JMX) Service URL that
will be used to establish a JMX connection to the WebLogic Administration
Server. For example:
service:jmx:t3://server.example.com:5555/jndi/weblogic.management.m
beanservers.domainruntime
If you do not supply a value, Enterprise Manager will provide the Service
URL based on the host port and protocol. If this is specified, the
Administration server host and port information still must be provided in the
input parameters.
в– Discover Application Versions
By default, each version of a deployed Java EE application will be discovered
as a target. Therefore, with every new version, a new target will be discovered.
Deselect this check box if you want only the active version of the application
to be discovered.
в– Enable Automatic Refresh
This option refreshes the weblogic domain every 24 hours.
в– Use Host Name in Service URL
You can use host name in service URL instead of JMX. It is recommended to
use this option if you are using a private network and there are many hosts
using the same IP address.
в– Create Incident for Discovery Failure
This option creates an OMS incident if discovery fails. You can view the
incident from the Support workbench page.
в– External Parameters
Optionally enter any system properties to be used by the Java process to
connect to the WebLogic Administration Server in the Parameters field.
Supply space-separated name/value pairs. Preface each parameter with -D.
For example:
-Dparam1=xxx -Dparam2=yyy -Dparam3=zzz
в– Discovery Debug File Name
If problems occur while adding Middleware-related targets or refreshing
domain membership, you can enable additional debugging information to
quickly diagnose and resolve the issue.
6.
Click Continue. Enterprise Manager will discover all Fusion Middleware targets
within the domain.
2-68 OracleВ® Enterprise Manager Administration
Discovering, Promoting, and Adding Middleware Targets
7.
Click Close in the Finding Targets dialog to automatically assign Management
Agents to the discovered targets.
The Assign Agents page lists each Fusion Middleware target discovered and the
Management Agent assigned to each. Agents are automatically assigned as
follows:
в– в– If a local Management Agent is installed on the discovered target host, that
Agent will be assigned.
If a local Management Agent cannot be found on the host, the Agent specified
in the Targets page will be assigned.
Note that you can also manually assign Management Agents to specific targets, if
desired.
8.
Click Add Targets to assign Management Agents as listed in the Assign Agents
page.
The Saving Target to Agent processing window appears, indicating how many
total targets have been added and successfully saved. It will also indicate the
number of targets that were unsuccessfully added.
9.
Click Close in the processing window when finished. The Results page displays
the targets and Agent assignments.
10. Click OK when finished. There may be a delay before these targets are visible and
monitored. All the agents used for monitoring the targets must be up.
After you discover a middleware target for the first time, it is
recommended that you learn the best practices for monitoring and
managing the discovered target. To navigate to the Target
Management Best Practices page, from the target home page, select
the Weblogic Domain menu, and then select Target Management
Best Practices. The page lists the best practices items for a Fusion
Middleware Domain.
Note:
2.5.1.3 Adding Multiple WebLogic Domains Using EM CLI
If you have multiple WebLogic domains that you want to manage through Cloud
Control, you can use the Enterprise Manager Command Line Interface (EM CLI)
discover_wls verb to discover them all at once, rather than discovering them one at a
time using the discovery wizards.
The discover_wls verb can be used to discover WebLogic Server versions 7.x, 8.x, 9.x,
and 10.x domains. The verb reads a file named domain_discovery_file that contains
the information required to discover each domain.
See the Enterprise Manager Command Line Interface book for instructions on using the
discover_wls verb.
2.5.2 Discovering New or Modified Domain Members
In the typical enterprise, Oracle WebLogic domains do not remain static. Instead,
membership in the domain changes regularly: New Java EE applications are deployed,
WebLogic Server instances are created or removed, clusters are added, and so on.
By default, Cloud Control is not automatically aware of changes made to Oracle
WebLogic domains that have been configured as managed targets. However, the
Discovering, Promoting, and Adding Targets
2-69
Discovering, Promoting, and Adding Middleware Targets
application does provide the ability to discover and uptake new or modified domain
members.
This section covers the following:
в– Enabling Automatic Discovery of New Domain Members
в– Manually Checking for New or Modified Domain Members
2.5.2.1 Enabling Automatic Discovery of New Domain Members
You can enable a pre-defined Cloud Control job named "WebLogic Domain Refresh" to
automatically discover new domain members and add them as managed targets.
Whenever you perform the Refresh operation, the
Administration Server must be up and the Discovery Agent must be
able to connect to it using JMX.
Note:
1.
From the Targets menu, select Middleware.
2.
Click on the WebLogic Domain you want to enable the job for in the Middleware
home page.
3.
In the General region of the page, click the timestamp link next to the WebLogic
Domain Refreshed property. The Refresh WebLogic Domain dialog opens.
4.
Check the Enable Automatic Refresh box in the Refresh WebLogic Domain
dialog, then click OK.
Once enabled, the job will check for new domain members once every 24 hours by
default. To change the job settings, including the frequency at which it is run:
1.
Click the Jobs tab.
2.
Click the job title in the Job Activity page.
3.
Click Edit.
2.5.2.2 Manually Checking for New or Modified Domain Members
You can use Cloud Control to check a domain for new or modified members on a
periodic basis.
1.
From the Targets menu, select Middleware.
2-70 OracleВ® Enterprise Manager Administration
Discovering, Promoting, and Adding Middleware Targets
Figure 2–14 Targets Menu
2.
Click the WebLogic Domain you want to (enable the job for in the Middleware
home page) refresh.
Figure 2–15 WebLogic Domain to Enable
3.
From either the Farm or WebLogic Domain menu, select Refresh WebLogic
Domain. The Refresh WebLogic Domain dialog opens.
Discovering, Promoting, and Adding Targets
2-71
Discovering, Promoting, and Adding Middleware Targets
Figure 2–16 Refresh WebLogic Domain Menu Option
4.
Click Add/Update Targets. The Management Agent refreshes by connecting to the
Administration Server. The Administration Server must be up for the refresh to
occur. Click Close on the Confirmation page. Cloud Control will search the
domain for new and modified targets.
When any entity in a domain is removed from a Weblogic domain such as
Weblogic j2eeaserver, j2eeapp, and the like, they are still displayed in Enterprise
Manager. Click Remove Targets if you do not need the historical data of these
targets. The obsolete targets which can be removed from the domain are then
displayed.
Figure 2–17 Add/Update Option on Refresh WebLogic Domain Page
2-72 OracleВ® Enterprise Manager Administration
Discovering, Promoting, and Adding Middleware Targets
Figure 2–18 Confirmation of Finding Targets
5.
Discovery Debug File Name option
If problems occur while adding Middleware-related targets or refreshing domain
membership, you can enable additional debugging information to quickly
diagnose and resolve the issue. This file will be created in the discovery Agent's
log directory.
6.
The Assign Agents page displays the Fusion Middleware targets discovered and
the Management Agent assigned to each. Click Add Targets to assign
Management Agents as listed in the Assign Agents page.
Agents are automatically assigned as follows:
в– в– If a local Agent can be found on the target host, that Agent will be assigned.
If a local Agent cannot be found on the host, the Agent specified in the Targets
page will be assigned.
Note that you can also manually assign Agents to specific targets, if desired.
This page also provides the option of Selective Discovery. Using this option, you
can disable the discovery of only new target types.
You can also modify the Domain Global Properties, for example, Contact, Cost
Center, Lifecycle Status, and so on).
Discovering, Promoting, and Adding Targets
2-73
Discovering, Promoting, and Adding Middleware Targets
Figure 2–19 Assign Agents Page
7.
The Saving Targets to Agent processing window appears, indicating how many
total targets have been added and successfully saved. It will also indicate the
number of targets that were unsuccessfully added.
Figure 2–20 Confirmation Saving Targets to Agent
8.
Click Close in the processing window when finished. The Results page displays
the following options: Show Targets Details and Show Weblogic Domain Global
Properties. The Show Targets Details page shows the targets and Agent
assignments.
Note: If there were targets that were removed, you can go back to the Refresh
WebLogic Domain page and click Remove Targets to remove the targets and any
historical information in the Management Repository. See Removing Middleware
Targets.
2-74 OracleВ® Enterprise Manager Administration
Discovering, Promoting, and Adding Middleware Targets
Figure 2–21 Refresh WebLogic Domain Results Page
9.
Click OK when finished. There may be a delay before these targets are visible and
monitored. All the agents used for monitoring the targets must be up.
Once discovery is done, you will need to set up the monitoring credentials of the
domain, before you can monitor it. To do this, on the Weblogic domain homepage,
from the Weblogic domain menu, select Target setup, and then click Monitoring
credentials.
2.5.3 Discovering a Standalone Oracle HTTP Server
To discover standalone Oracle HTTP Servers, follow these steps:
в– в– Meeting the Prerequisites
Discovering a Standalone Oracle HTTP Server Using the Guided Discovery
Process
To view a visual demonstration on how to discover and
manage standalone Oracle HTTP Servers, access the following URL
and click Begin Video. The discovery described in this visual
demonstration is based Enterprise Manager Cloud Control 12c Release
2 (12.1.0.3) and the Oracle Fusion Middleware Plug-in 12.1.0.5.
Note:
http://apex.oracle.com/pls/apex/f?p=44785:24:0::::P24_
CONTENT_ID,P24_PREV_PAGE:8529,1
2.5.3.1 Meeting the Prerequisites
Before you discover a standalone Oracle HTTP server, meet the following:
в– в– Ensure that an Oracle Management Agent (Management Agent) is installed on the
host where the standalone Oracle HTTP Server is running. For instructions to
install a Management Agent, see the Oracle Enterprise Manager Cloud Control Basic
Installation Guide available in the Enterprise Manager documentation library.
Ensure that the standalone Oracle HTTP Server you are about to discover is of one
of the following releases: 11.1.1.1, 11.1.1.2, 11.1.1.3, 11.1.1.4, 11.1.1.5, 11.1.1.6,
11.1.1.7, 12.1.2.
Discovering, Promoting, and Adding Targets
2-75
Discovering, Promoting, and Adding Middleware Targets
2.5.3.2 Discovering a Standalone Oracle HTTP Server Using the Guided Discovery
Process
To discover and add a standalone Oracle HTTP server, follow these steps:
в– Step 1: Discovering Unmanaged Hosts
в– Step 2: Converting Unmanaged Hosts to Managed Hosts
в– Step 3: Adding Standalone Oracle HTTP Server Targets
2.5.3.2.1
Step 1: Discovering Unmanaged Hosts
Skip this step if host is already managed in Enterprise
Manager.
Note:
Discover unmanaged hosts, as described in Section 2.2.2.1.
2.5.3.2.2
Step 2: Converting Unmanaged Hosts to Managed Hosts
Skip this step if host is already managed in Enterprise
Manager.
Note:
Convert unmanaged hosts to managed hosts, as described in Section 2.2.2.2.
2.5.3.2.3
Step 3: Adding Standalone Oracle HTTP Server Targets
Add standalone Oracle HTTP server targets, by following these steps:
1.
From the Setup menu, select Add Target, then select Add Targets Manually.
2.
On the Add Targets Manually page, select Add Targets Using Guided Process
(Also Adds Related Targets).
3.
From the Target Types list, select Standalone Oracle HTTP Server.
4.
Click Add Using Guided Process.
5.
On the Add Standalone Oracle HTTP Server page, provide the following details,
and click Add Target.
Element
Description
Oracle HTTP Server Host
Click the search icon to search and select the host where the
standalone Oracle HTTP Server is running. In the Search and
Select: Targets dialog, search the host, select it, and click Select.
For example, example.com
Agent URL
URL of the Management Agent that is installed on the host
where the standalone Oracle HTTP Server is running. Appears
automatically by default when you select the standalone Oracle
HTTP Server host.
For example, https://example.com:1838/emd/main/
2-76 OracleВ® Enterprise Manager Administration
Discovering, Promoting, and Adding Middleware Targets
Element
Description
Target Name
Enter a unique name with which you want to monitor the
standalone Oracle HTTP Server target in Enterprise Manager
Cloud Control.
For example, Standalone_OHS1
The name must be unique, and no other standalone Oracle
HTTP Server target already monitored in Enterprise Manager
Cloud Control must have this name. The name cannot contain
colons ( : ), semi-colons ( ; ), or any leading or trailing blanks.
Once the standalone Oracle HTTP Server is discovered and
added to Enterprise Manager Cloud Control, if you want to
search for this particular target, follow these steps:
Oracle Home
1.
From the Targets menu, select All Targets.
2.
On the All Targets page, in the Search Target Name field,
enter the target name with which you discovered and
added the standalone Oracle HTTP Server target, and click
the search icon.
Click the search icon to log in to the standalone Oracle HTTP
Server host, and select the Oracle home where the standalone
Oracle HTTP Server is running.
For example, /u01/software/oracle/Standalone_OHS1
Configuration Path
Click the search icon to log in to the standalone Oracle HTTP
Server host, and select the httpd.conf file. The value for this
field must ideally be the absolute directory path to the
httpd.conf file.
For example, /u01/software/oracle/Standalone_
OHS1/Oracle_WT1/instances/instance1/config/OHS/ohs1/
Node Manager User Name
(This field is only for discovering standalone Oracle HTTP Server 12c
(12.1.2))
Enter the Node Manager user name, which you provided while
configuring the standalone domain. For more information, see
the task on configuring the node manager in Oracle Fusion
Middleware Installing and Configuring Oracle HTTP Server.
Node Manager Password
(This field is only for discovering standalone Oracle HTTP Server 12c
(12.1.2))
Enter the Node Manager password, which you provided while
configuring the standalone domain. For more information, see
the task on configuring the node manager in Oracle Fusion
Middleware Installing and Configuring Oracle HTTP Server.
2.5.4 Discovering Exalytics Targets
In order to manage and monitor Oracle Fusion Middleware components running on
Exalytics, such as WebLogic domain, Oracle BI Foundation 11g, and Oracle TimesTen
(in-memory database), Cloud Control must first discover the Exalytics machine
containing these components.
An Exalytics system consists of one or more Exalytics machines. The machine(s) can be
physical, virtualized, or a mix of both.
Once discovered, the Exalytics machine and the components within it can be promoted
to "managed target" status, enabling Cloud Control to collect the data needed to
monitor the target.
Related targets running on the Exalytics machine such as OBIEE, Times Ten and
WebLogic, need to be discovered following the respective flows for those components.
Discovering, Promoting, and Adding Targets
2-77
Discovering, Promoting, and Adding Middleware Targets
If the Exalytics target has been discovered, the related targets will be associated with
the Exalytics target when they are discovered. Else, the association will happen when
the Exalytics target is discovered.
To discover and add an exalytics target, follow these steps:
в– в– Meeting the Prerequisites
Discovering and Adding Exalytics System Targets Using the Guided Discovery
Process
2.5.4.1 Meeting the Prerequisites
Before discovering an Exalytics machine target, you must meet the following
prerequisites:
в– A Management Agent for discovering targets must be deployed onto the physical
Exalytics machine (host).
For discovering a virtual machine, make sure that at least one of the virtual
Exalytics has a Management Agent running on it. The Management Agent should
be on one of the VM Guest part of the virtual Exalytics Machine deployment.
However, all virtual machines that contain components that need to be monitored
require a Management Agent to be deployed on.
в– в– To identify the Exalytics machine, ensure that you have the context info file which
contains the Exalytics machine ID.
To discover and monitor ILOM for a virtual Exalytics machine, you must install
IMPITOOL on the VM guest. To install IMPITOOL, follow these steps:
1.
Download the latest Hardware Management Pack compatible with the
operating system on your VM guest. Instructions for downloading the
Hardware Management Pack are available here:
http://www.oracle.com/technetwork/server-storage/servermgmt/downloa
ds/index.html
2.
Extract the IMPITOOL package from the downloaded zip file and install it on
the operating system on the VM guest.
2.5.4.2 Discovering and Adding Exalytics System Targets Using the Guided
Discovery Process
To discover and add Exalytics system target in an Exalytics system, follow these steps:
в– Step 1: Discovering Unmanaged Hosts
в– Step 2: Converting Unmanaged Hosts to Managed Hosts
в– Step 3: Adding Exalytics Targets
2.5.4.2.1
Step 1: Discovering Unmanaged Hosts
Skip this step if host is already managed in Enterprise
Manager.
Note:
Discover unmanaged hosts, as described in Section 2.2.2.1.
2.5.4.2.2
Step 2: Converting Unmanaged Hosts to Managed Hosts
2-78 OracleВ® Enterprise Manager Administration
Discovering, Promoting, and Adding Middleware Targets
Skip this step if host is already managed in Enterprise
Manager.
Note:
Convert unmanaged hosts to managed hosts, as described in Section 2.2.2.2.
2.5.4.2.3
Step 3: Adding Exalytics Targets
Add Exalytics targets, by following these steps:
1.
From the Setup menu, select Add Target, and then select Add Targets Manually.
2.
On the Add Targets Manually page, select Add Targets Using Guided Process
(Also Adds Related Targets).
3.
From the Target Types drop-down list, select Exalytics System, and then click Add
Using Guided Process...
4.
On the Discover Exalytics System page, provide a name for the Exalytics System,
and then click Add Machine.
5.
Provide the following details:
в– Machine Name
Provide a unique for the Oracle Exalytics machine target.
в– Agent
Specify or select the Management Agent to use for the Oracle Exalytics
Machine discovery process.
в– Deployment Type
Select Physical or Virtual depending on if you want to discover a physical
target or a virtual target.
в– Host Name (only for Virtual target)
Enter the host name or IP Address where Oracle Virtual Server is running
в– User Name and Password
Discovering, Promoting, and Adding Targets
2-79
Discovering, Promoting, and Adding Middleware Targets
For a physical Exalytics machine, provide credentials of the root user which
has privileges to run the imageinfo command.
For a virtual Exalytics machine, provide credentials to log in to Oracle Virtual
Server (OVS). You should have privileges to run the imageinfo command.
в– ILOM Credentials
Specify the ILOM IP address or hostname, ILOM username, and ILOM
password of the root user.
The ILOM Credentials are optional. If you do not add the
ILOM Credentials, the ILOM target will not be discovered.
Note:
You can add the ILOM target later by using the Refresh Exalytics
System option.
Click Next.
6.
A confirmation box appears. Click OK.
Note:
Click Add Targets to save the targets as a Manageable Entity.
2.5.5 Removing Middleware Targets
Removing Middleware targets from the Management Repository:
в– в– в– в– в– Identifies targets that are deleted from the Weblogic Domain, for example,
WebLogic Servers, Clusters, Applications (both generic and custom), and any
other System Components.
Shows the list of targets which might have been deleted from the product, but
Enterprise Manager cannot determine if they were deleted or not. For these
targets, decide whether these targets should be deleted and mark them as such.
Shows duplicate targets. For example, if for the same application deployment
there is a custom and a generic target, the will shows the generic target which can
be deleted.
Shows the older versioned application deployments which can be deleted if a
newer version of the same application is present.
Lists all the down servers. You can decide to either blackout or delete these
servers.
2-80 OracleВ® Enterprise Manager Administration
3
Using Incident Management
3
Incident management allows you to monitor and resolve service disruptions quickly
and efficiently by allowing you to focus on what is important from a broader
management perspective (incidents) rather than isolated, discrete events that may
point to the same underlying issue.
In this chapter:
You will learn:
Management Concepts
Fundamental approaches to managing your
monitored environment.
Setting Up Your Incident Management
Environment
в– Event Management
в– Incident Management
в– Problem Management
How to set up and configure key Enterprise
Manager components used for incident
management.
в– Working with Incidents
Setting Up Your Monitoring
Infrastructure
в– Setting Up Notifications
в– Setting Up Administrators and Privileges
How to use incident management to track and
resolve IT operation issues.
в– Finding What Needs to be Worked On
в– Searching for Incidents
в– Setting Up Custom Views
в– в– Responding and Working on a Simple
Incident
Responding to and Managing Multiple
Incidents, Events and Problems in Bulk
в– Searching My Oracle Support Knowledge
в– Open Service Request (Problems-only)
в– Suppressing Incidents and Problems
в– в– Managing Workload Distribution of
Incidents
Reviewing Events on a Periodic Basis
Using Incident Management
3-1
In this chapter:
You will learn:
Common Tasks
Step-by-step examples illustrating how to
perform common incident management tasks..
в– Sending Email for Metric Alerts
в– Sending SNMP Traps for Metric Alerts
в– Sending Events to an Event Connector
в– Advanced Topics
Sending Email to Different Email
Addresses for Different Periods of the
Day
How to perform specialized incident
management operations.
в– в– Clearing Stateless Alerts for Metric Alert
Event Types
в– User-reported Events
в– Additional Rule Applications
в– Event Prioritization
в– Moving from Enterprise Manager 10/11g to
12c
Defining Custom Incident Statuses
Root Cause Analysis (RCA) and Target
Down Events
Migrating notification rules to incident rules.
To supplement this chapter, Oracle has created instructional videos that provide you
with a fast way to learn the basics of incident management to monitor your
environment.
Instructional Videos:
For video tutorials on incident management,
see:
Incident Management Overview
https://apex.oracle.com/pls/apex/f?p=44785:24:29612679875
20:::24:P24_CONTENT_ID%2CP24_PREV_PAGE:5738%2C24
Incident Management: Create Views in Incident Manager
https://apex.oracle.com/pls/apex/f?p=44785:24:60910521152
37:::24:P24_CONTENT_ID%2CP24_PREV_PAGE:5739%2C24
Incident Management: View Incident Details
https://apex.oracle.com/pls/apex/f?p=44785:24:10766488894
5874:::24:P24_CONTENT_ID%2CP24_PREV_PAGE:5740%2C24
Incident Management: Use Incident Rule Sets Part 1
https://apex.oracle.com/pls/apex/f?p=44785:24:11471687942
8375:::24:P24_CONTENT_ID%2CP24_PREV_PAGE:5758%2C24
Incident Management: Use Incident Rule Sets Part 2
https://apex.oracle.com/pls/apex/f?p=44785:24:10217270776
0983:::24:P24_CONTENT_ID%2CP24_PREV_PAGE:5759%2C24
3-2 OracleВ® Enterprise Manager Administration
Management Concepts
3.1 Management Concepts
Enterprise Manager exposes three levels of management granularity that, when
combined, provide complete monitoring/management coverage of your environment.
These management levels are:
в– Event Management
в– Incident Management
в– Problem Management
3.1.1 Event Management
Intuitively, you monitor for specific events in your monitored environment. An event
is a significant occurrence on a managed target that typically indicates something has
occurred outside normal operating conditions--they provide a uniform way to indicate
that something of interest has occurred in an environment managed by Enterprise
Manager. Examples of events are:
в– Metric Alerts
в– Compliance Violations
в– Job Events
в– Availability Alerts
Existing Enterprise Manager customers may be familiar with metric alerts and metric
collection errors. For Enterprise Manager 12c, metric alerts are a type of event, one of
many different event types. The notion of an event unifies the different exception
conditions that are detected by Enterprise Manager, such as monitoring issues or
compliance issues, into a common concept. It is backed by a consistent and uniform set
of event management capabilities that can indicate something of interest has occurred
in a datacenter managed by Enterprise Manager.
All events have the following attributes:
Table 3–1
Event Attributes
Attribute
Description
Type
Type of event that is being reported. All events of a specific type
share the same set of attributes that describe the exact nature of
the problem. For example, Metric Alert, Compliance Standard
Score Violation, or Job Status Change.
Severity
Event severity. For example, Fatal, Warning, or Critical.
Internal Name
An internal name that describes the nature of the event and can
be used to search for events. For example, you can search for all
tablespacePctUsed events.
Entity on which the event is
raised.
An event can be raised on a target, a non-target source object
(such as a job) or be related to a target and a non-target source
object. Note: This attribute is important when determining what
privileges are required to manage the event.
Message
Informational text associated with the event.
Reported Date
Time the event was reported.
Using Incident Management
3-3
Management Concepts
Table 3–1 (Cont.) Event Attributes
Attribute
Description
Category
Functional or operational classification for an event.
Available Categories:
Causal Analysis Update
в– Availability
в– Business
в– Capacity
в– Configuration
в– Diagnostics
в– Error
в– Fault
в– Jobs
в– Load
в– Performance
в– Security
Used for Root Cause Analysis of target down events.
Possible Values: Root Cause or Symptom
Event Types
The type of an event defines the structure and payload of an event and provides the
details of the condition it is describing. For example, a metric alert raised by threshold
violation has a specific payload whereas a job state change has a different structure. As
shown in the following table, the range of events types greatly expands Enterprise
Manager’s monitoring flexibility.
Event Type
Description
Target Availability
The Target Availability Event represents a
target's availability status (Example: Up,
Down, Agent Unreachable, or Blackout).
Metric Alert
A metric alert event is generated when an
alert occurs for a metric on a specific target
(Example: CPU utilization for a host target)
or metric on a target and object combination
Example: Space usage on a specific
tablespace of a database target.
Metric Evaluation Error
A metric evaluation error is generated when
the collection for a specific metric group fails
for a target.
Job Status Change
All changes to the status of an Enterprise
Manager job are treated as events, and these
events are made available via the Job Status
Change event class.
Note: A prerequisite to creating Incident
Rules, is to enable the relevant job status and
add required targets to job event generation
criteria. To change this criteria, from the
Setup menu, select Incidents, and then Job
Events.
3-4 OracleВ® Enterprise Manager Administration
Management Concepts
Event Type
Description
Compliance Standard Rule Violation
Events are generated for compliance
standard rule violations. Each event
corresponds to a violation of a compliance
rule on a specific target.
Compliance Standard Score Violation
Events are generated for compliance
standard score violations. An event is
generated when the compliance score for a
compliance standard on a specific target falls
below predefined thresholds.
High Availability
High Availability events are generated for
database availability operations (shutdown
and startup), database backups and Data
Guard operations (switchover, failover, and
other state changes).
Service Level Agreement Alert
These events are generated when a service
level or service level objective is violated for
a service. occurs for a Service Level
Agreement or a Service Level Objective.
User-reported
These events are created by end-users.
Application Dependency and Performance Alert Alerts are raised by the Application
Dependency and Performance (ADP)
monitoring when metrics related to a J2EE
application or component have crossed some
thresholds.
Application Performance Management KPI
Alert
An Application Performance Management
(APM) Key Performance Indicator (KPI) alert
event is generated when a KPI violation alert
occurs for a metric on an APM managed
entity associated with a Business Application
target.
JVM Diagnostics Threshold Violation
A JVMD Diagnostics event is raised when a
JVMD metric exceeds its threshold value on
a Java Virtual Machine target.
Event Severity
The severity of an event indicates the criticality of a specific issue. The following table
shows the various event severity levels along with the associated icon.
Icon
Severity
Description
Fatal
Corresponding service is no longer available. For example, a
monitored target is down (target down event). A Fatal severity is
the highest level severity and only applies to the Target Availability
event type.
Critical
Immediate action is required in a particular area. The area is either
not functional or indicative of imminent problems.
Warning
Attention is required in a particular area, but the area is still
functional.
Advisory
While the particular area does not require immediate attention,
caution is recommended regarding the area's current state. This
severity can be used, for example, to report Oracle best practice
violations.
Using Incident Management
3-5
Management Concepts
Icon
Severity
Description
Clear
Conditions that raised the event have been resolved.
Informational A specific condition has just occurred but does not require any
remedial action.
Events with an informational severity:
в– do not appear in the incident management UI.
в– cannot create incidents.
в– are not stored within Enterprise Manager.
3.1.2 Incident Management
You monitor and manage your Enterprise Manager environment via incidents and not
discrete events (even though an incident can conceivably consist of a single event). Of
all events raised within your managed environment, there is likely only a subset that
you need to act on because they impact your business applications (such as a target
down event). However, managing by incident also allows you to address more
complex situations where the subset of events you are interested in are related and
may indicate a higher level issue needs to be addressed as a single issue and not as
individual events: A cluster of events by themselves may indicate a minor
administrative issue, but when viewed together may signify a larger problem that can
potentially consist of events from multiple domains/layers of your monitored
infrastructure.
For example, you are monitoring a host. If you want to monitor 'load' being placed on
one or more hosts you might be interested in events such as CPU utilization, memory
utilization, and swap utilization exceeding acceptable metric thresholds. Individually,
these events may or may not indicate an issue with the host, but together, these events
form an incident indicating extreme load is being placed on a monitored host.
Incidents represent the larger service disruptions that may impact your business
instead of discrete events. Managing by incidents, therefore, allows you to monitor for
complex operational issues that may affect multiple domains that may impact your
business. These incidents typically need to be tracked, assigned to appropriate
personnel, and resolved as quickly as possible. You can effectively implement a
centralized monitoring that consolidates monitoring information and more effectively
allocate resource across your ecosystem to resolve or prevent issues from occurring.
The end result is better implementation of your business processes that in turn lead to
better performance of your IT resources.
While events indicate issues requiring attention in your managed environment, it is
more efficient to work on a collective subset of related events as a single unit of work-you can work on different events representing the same issue or you can work on one
incident containing multiple space-related events. For example, you have multiple
space events from various targets that indicate you are running low on space. Instead
of managing numerous discrete events, you can more efficiently manage a smaller set
of incidents.
An incident is a significant event or set of related significant events that need to be
managed because it can potentially impact your business applications. These incidents
typically need to be tracked, assigned to appropriate personnel, and resolved as
quickly as possible. You perform these incident management operations through
Incident Manager, an intuitive UI within Enterprise Manager.
3-6 OracleВ® Enterprise Manager Administration
Management Concepts
Incident Manger provides you with a central location from which to view, manage,
diagnose and resolve incidents as well as identify, resolve and eliminate the root cause
of disruptions. See Section 3.1.5, "Incident Manager" for more information about this
UI.
3.1.2.1 Working with Incidents
When an incident is created, Enterprise Manager makes available a rich set of incident
management workflow features that let you to manage and track the incident through
its complete lifecycle.
в– Assign incident ownership.
в– Track the incident resolution status.
в– Set incident priority.
в– Set incident escalation level.
в– Ability to provide a manual summary.
в– Ability to add user comments.
в– Ability to suppress/unsuppress
в– Ability to manually clear the incident.
в– Ability to create a ticket manually.
All incident management/tracking operations are carried out from Incident Manager.
Creation of incidents for events, assignment of incidents to administrators, setting
priority, sending notifications and other actions can be automated using (incident)
rules.
Incident Status
The lifecycle of an incident within an organization is typically determined by two
pieces of information: The current resolution state of the incident (Incident Status) and
how important it is to resolve the incident relative to other incidents (Priority). As key
incident attributes, the following options are available:
в– New
в– Work in Progress
в– Closed
в– Resolved
You can define additional statuses if the default options are not adequate. In addition,
you can change labels using the Enterprise Manager Command Line Interface (EM
CLI). See Advanced Topics for more information.
Priority
By changing the priority, you can escalate the incident and perform operations such as
assigning it to a specific IT operator or notifying upper-management. The following
priority options are available:
в– None
в– Low
в– Medium
в– High
Using Incident Management
3-7
Management Concepts
в– Very High
в– Urgent
Priority is often based on simple business rules determined by the business impact and
the urgency of resolution.
Incident Attributes
Every incident possesses attributes that provide information as identification, status
for tracking, and ownership. The following table lists available incident attributes.
Incident Attribute
Definition
Escalated
An escalation level signifying a escalation to raise the level of
attention on the incident from your organization’s IT or
management hierarchy.
Available escalation levels:
Category
в– None (Not escalated)
в– Level 1 through Level 5
Operational or organizational classification for an incident.
Incidents (and events) can have multiple categories.
Categories for all events within an incident are aggregated.
Available Categories:
в– Availability
в– Business
в– Capacity
в– Configuration
в– Diagnostics
в– Error
в– Fault
в– Jobs
в– Load
в– Performance
в– Security
Summary
An intuitive message indicating what the incident is about. By
default, the incident summary is pulled from the message of the
last event of the incident, however, this message can be changed
to a fixed summary by any administrator working on the
incident.
Incident Created
Date and time the incident was created.
Last Updated
Date and time the incident was last updated or when the
incident was closed.
Severity
Severity is based on the worst severity of the events in the
incident. For example, Fatal, Warning, or Critical.
Source
Source entities of the incident.
3-8 OracleВ® Enterprise Manager Administration
Management Concepts
Incident Attribute
Definition
Priority
Priority Values
Status
в– None (Default)
в– Low
в– Medium
в– High
в– Very High
в– Urgent
Incident Status.
в– New (Default)
в– Work in Progress
в– в– Closed (Terminal state when the incident is closed. See
below for more information.)
Resolved
You can define additional statuses if the default options are not
adequate. In addition, you can change labels using the
Enterprise Manager Command Line Interface (EM CLI).
Closed Status: Enterprise Manager automatically sets the status
to closed when an incident severity is cleared--administrators do
not manually select the Closed status. The incident severity is set
to Clear when all of the events contained within the incident
have been cleared. Typically the Agent sets the Clear severity, as
would be the case when a metric alert value falls below a
severity threshold. If an event or incident supports manual
clearing, then the Clear option will be shown in the Incident
Manager UI. Once an incident has been cleared by an
administrator or by Enterprise Manager, only then will
Enterprise Manager set the status to Closed.
If you do not see the option to clear the incident in the UI, this
means Enterprise Manager will automatically set the status to
Clear if it detects the monitored condition no longer holds true.
For example, you want to indicate that an incident has been
fixed. You can set the status to Resolved and Enterprise Manager
will set the status to Closed when it clears the severity.
Comment
Annotations added by an administrator to communicate
analysis information or actions taken to resolve the incident.
Owner
Administrator/user currently working on the incident.
Acknowledged
Indicates that a user has accepted ownership of an incident or
problem. Available options: Yes or No.
When an incident is acknowledged, it will be implicitly assigned
to the user who acknowledged it. When a user assigns an
incident to himself, it is considered ’acknowledged’. Once
acknowledged, an incident cannot be unacknowledged, but can
be assigned to another user. Acknowledging an incident stops
any repeat notifications for that incident.
Causal Analysis Update
Used for Root Cause Analysis of target down incidents.
Possible Values: Root Cause or Symptom
3.1.2.2 Incident Composed of a Single Event
The simplest incident is composed of a single event. In the following example, you are
concerned whenever any production target is down. You can create an incident for the
target down event which is raised by Enterprise Manager if it detects the monitored
Using Incident Management
3-9
Management Concepts
target is down. Once the incident is created, you will have all incident management
functionality required to track and manage its resolution.
Figure 3–1 Incident with a Single Event
The figure shows how both the incident and event attributes are used to help you
manage the incident. From the figure, we see that the database DB1 has gone down
and an event of Fatal severity has been raised. When the event is newly generated,
there is no ownership or status. An incident is opened that can be updated manually
or by automated rules to set owners, status, as well as other attributes. In the example,
the owner/administrator Scott is currently working to resolve the issue.
The incident severity is currently Fatal as the incident inherits the worst severity of all
the events within incident. In this case there is only one event associated with the
incident so the severity is Fatal.
3.1.2.3 Incident Composed of Multiple Events
Situations of interest may involve more than a single event. It is an incident’s ability to
contain multiple events that allows you to monitor and manage complex and more
meaningful issues.
Multi-event incidents are not automatically generated. An
administrator must manually create them.
Note:
For example, if a monitored system is running out of space, separate multiple events
such as tablespace full and filesystem full may be raised. Both, however, are related to
running out of space. Another machine resource monitoring example might be the
simultaneous raising of CPU utilization, memory utilization, and swap utilization
events. See "Creating an Incident Manually" on page 3-51 for more information.
Together, these events form an incident indicating extreme load is being placed on a
monitored host. The following figure illustrates this example.
3-10 OracleВ® Enterprise Manager Administration
Management Concepts
Figure 3–2 Incident with Multiple Events
Incidents inherit the worst severity of all the events within incident. The incident
summary indicates why this incident should be of interest, in this case, "Machine Load
is high". This message is an intuitive indicator for all administrators looking at this
incident. By default, the incident summary is pulled from the message of the last event
of the incident, however, this message can be changed by any administrator working
on the incident.
Because administrators are interested in overall machine load, administrator Sam has
manually created an incident for these two metric events because they are
related—together these events represent a host overload situation. An administrator
needs to take action because memory is filling up and consumed CPU resource is too
high. In its current state, this condition will impact any applications running on the
host.
3.1.2.4 How are Incidents Created?
Incidents are most commonly created automatically through rules and rule sets
(user-defined instructions that tell Incident Manager how to handle specific events
when they occur). As shown in the preceding examples, incidents can also be created
manually. Once an incident is raised, its severity is inherited from the worst severity of
all events within the incident. The latest event Message, by default, becomes the
Incident Summary. Incidents can also be created manually. See "Creating an Incident
Manually" on page 3-51 for more information.
3.1.3 Problem Management
Problem management involves the functionality that helps track the underlying root
causes of incidents. Once the immediate service disruptions represented by incidents
are resolved, you can then progress to understanding and resolving the underlying
root cause of the issue.
For Enterprise Manager 12c, problems focus on the diagnostic incidents and problem
diagnostic incidents/problems stored in Advanced Diagnostic Repository (ADR),
which are automatically raised by Oracle software when it encounters critical errors in
the software. A problem, therefore, represents the root cause of all the Oracle software
incidents. For these diagnostic incidents, in order to address root cause, a problem is
created that represents the root cause of these diagnostic incidents. A problem is
Using Incident Management
3-11
Management Concepts
identified by a problem key which uniquely identifies the particular error in software.
Each occurrence of this error results in a diagnostic incident which is then associated
with the problem object.
When a problem is raised for Oracle software, Oracle has determined that the
recommended recourse is to open a service request (SR), send support the diagnostic
logs, and eventually provide a solution from Oracle. As an incident, Enterprise
Manager makes available all tracking, diagnostic, and reporting functions for problem
management. Whenever you view all open incidents and problems, whether you are
using Incident Manager, or in context of a target/group home page, you can easily
determine what issues are actually affecting your monitored target.
To manage problems, you can use Support Workbench to package the diagnostic
details gathered in ADR and open SR. Users should then manage the problems in
Incident Manager. Access to Support Workbench functionality is available through
Incident Manager (Guided Resolution area) in context of the problem.
3.1.4 Rule Sets
Incident rules and rule sets automate actions related to events, incidents and problems.
They can automate the creation of incidents based on important events, perform
notification actions such as sending e-mail or opening helpdesk tickets, or perform
operations to manage the incident workflow lifecycle such as changing incident
ownership, priority, or escalation level.
With previous versions of Enterprise Manager, you used notification rules to choose
the individual targets and conditions for which you want to perform actions or receive
notifications (send e-mail, page, open a helpdesk ticket) from Enterprise Manager. For
Enterprise Manager 12c, the concept and function of notification rules has been
replaced with incident rules and rule sets.
в– в– Rules: A rule instructs Enterprise Manager to take specific actions when incidents,
events, or problems occur, such as performing notifications. Beyond notifications,
rules can also instruct Enterprise Manager to perform specific actions, such as
creating incidents, updating incidents and problems. The actions can also be
conditional in nature. For example, a rule action can be defined to page a user
when an incident severity is critical or just send e-mail if it is warning.
Rule Set: An incident rule set is a collection of rules that apply to a common set of
objects such as targets (hosts, databases, groups), jobs, metric extensions, or self
updates and take appropriate actions to automate the business processes
underlying event, incident and problem management.
Operationally, individual rules within a rule set are executed in a specified order as
are the rule sets themselves. Rule sets are executed in a specified order. By default, the
execution order for both rules and rule sets is the order in which they are created, but
they can be reordered from the Incident Rules UI.
The following figure shows typical rule set structure and how the individual rules are
applied to a heterogeneous group of targets.
3-12 OracleВ® Enterprise Manager Administration
Management Concepts
Figure 3–3 Rule Set Application
The graphic illustrates a situation where all rules pertaining to a group of targets can
be put into a single rule set (this is also a best practice). In the above example, a group
named PROD-GROUP consists of hosts, databases, and WebLogic servers exists as
part of a company’s managed environment. A single rule set is created to manage the
group.
In addition to the actual rules contained within a rule set, a rule set possesses the
following attributes:
в– Name: A descriptive name for the rule set.
в– Description: Brief description stating the purpose of the rule set.
в– в– Applies To: Object to which all rules in the rule set apply: Valid rule set objects are
targets, jobs, metric extensions, and self update.
Owner: The Enterprise Manager user who created the rule set. Rule set owners
have the ability to update or delete the rule set and the rules in the rule set.
в– Enabled: Whether or not the rule set is actively being applied.
в– Type: Enterprise or Private. See "Rule Set Types" on page 3-14
3.1.4.1 Out-of-Box Rule Sets
Enterprise Manager provides out-of-box rule sets for incident creation and event
clearing based on typical scenarios. Out-of-box rule sets cannot be edited or deleted,
however, they can be disabled. As a best practice, you should create your own copies
of out-of-box rule sets and then subscribe to the rule set copies rather than subscribing
directly to the out-of-box rule sets. Effectively, you are making a copy of the rule set
and changing the target criteria to fit your enterprise needs by selecting an appropriate
group of targets (preferably an administration group).
Please note that out-of-box rule set definitions and actions they perform can be
changed by Oracle at any time and will be applied during patching or software
upgrade.
Regular Enterprise Manager administrators are allowed to perform the following
operations on rule sets:
в– Subscribe
в– Subscribe for E-Mail notifications
в– Unsubscribe
в– Unsubscribe from E-Mail notifications
Using Incident Management
3-13
Management Concepts
в– Enable
в– Disable
Even though administrators can subscribe to a rule set, they
will only receive notification from the targets for which they have at
least the View Target privilege.
Note:
Enterprise Manager Super Administrators have the added ability to reorder the rule
sets.
Enterprise rule sets are evaluated sequentially and may go through multiple passes as
needed. When there is a change to the entity being processed - such as an incident
being created for an event or an incident priority changing due to a rule - we rerun
through all the rules from the beginning again until there are no matches. Any rule
that is matched in a prior pass will not match again (to prevent infinite loops).
For example, when a new event, incident, or problem arises, the first rule set in the list
is checked to see if any of its member rules apply and appropriate actions specified in
those rules are taken. The second rule is then checked to see if its rules apply and so
on. Private rule sets are only evaluated once all enterprise rule set evaluations are
complete and in no particular order.
Use caution when reordering rule sets as their order
defines the event, incident, and problem handling workflow.
Reordering rule sets without fully understanding the impact on your
system can result in unintended actions being taken on incoming
events, incidents, and problems.
Important:
3.1.4.2 Rule Set Types
There are two types of Rule Sets:
в– Enterprise: Used to implement all operational practices within your IT
organization. All supported actions are available for this type of rule set. However,
because this type of rule set can perform all actions, there are restrictions as to who
can create an enterprise rule set.
In order to create or edit an enterprise rule set, an administrator must have been
granted the Create Enterprise Rule Set privilege on the Enterprise Rule Set resource.
However, if the rule set owner loses the Create Enterprise Rule Set system privilege
at some future time, he can still edit or delete the rule set. Super Administrators
can edit or delete any rule set.
If the originator of the rule set wants other administrators to edit the rule set, he
will need to share access in order to work collaboratively by adding co-authors.
Enterprise rule sets are visible to all administrators.
в– Private: Used when an administrator wants to be notified about something he is
monitoring but not as a standard business practice. The only action a private rule
set can perform is to send e-mail to the rule set owner. Any administrator can
create a private rule set regardless of whether they have been granted the Create
Enterprise Rule Set resource privilege. Oracle recommends that private rule sets be
used only in rare or exceptional situations.
When a rule set performs actions, the privileges of the rule set creator are used. For
example, a rule set owner/creator must have at least View Target privilege in order to
3-14 OracleВ® Enterprise Manager Administration
Management Concepts
receive notifications and at least Manage Target Events privilege in order to update the
incident. The exception is when a rule set sends a notification. In this case, the
privileges of the user it is sent to is used.
3.1.4.3 Rules
Rules are instructions within a rule set that automate actions on incoming events or
incidents or problems. Because rules operate on incoming incidents/events/problems,
if you create a new rule, it will not act retroactively on incidents/events/problems that
have already occurred.
Every rule is composed of two parts:
в– в– Criteria: The events/incidents/problems on which the rule applies.
Action(s): The ordered set of one or more operations on the specified events,
incidents, or problems. Each action can be executed based on additional
conditions.
The following table shows how rule criteria and actions determine rule application. In
this rule operation example there are three rules which take actions on selected events
and incidents. Within a rule set, rules are executed in a specified order. The rule
execution order can be changed at any time. By default, rules are executed in the order
they are created.
Table 3–2
Rule Operation
Rule Execution
Name Order
Criteria
Action
Condition
Rule 1 First
CPU Util(%), Tablespace
Used(%) metric alert
events of warning or
critical severity
Rule 2 Second
Incidents of warning or
critical severity
Rule 3 Third
Actions
Create incident.
If severity = critical
Notify by page
If severity =warning Notify by e-mail
Incidents are
unacknowledged for
more than six hours
Set escalation level to 1
In the rule operation example, Rule 1 applies to two metric alert events: CPU Utilization
and Tablespace Used. Whenever these events reach either Warning or Critical severity
threshold levels, an incident is created.
When the incident severity level (the incident severity is inherited from the worst
event severity) reaches Warning, Rule 2 is applied according to its first condition and
Enterprise Manager sends an e-mail to the administrator. If the incident severity level
reaches Critical, Rule 2’s second condition is applied and Enterprise Manager sends a
page to the administrator.
If the incident remains open for more than six hours, Rule 3 applies and the incident
escalation level is increased from None to Level 1. At this point, Enterprise Manager
runs through all the rule sets and their rules from the beginning again.
3.1.4.3.1 Rule Application Each rule within a rule set applies to an event, incident OR
problem. For each of these, you can choose rule application criteria such as:
в– Apply the rule to incoming events or updated events only
Using Incident Management
3-15
Management Concepts
в– Apply the rule to critical events only.
Rules are applied to events, incidents, and problems according to criteria selected at
the time of rule creation (or update). The following situations illustrate the
methodology used to apply rules.
в– в– в– If one of the rules creates a new incident in response to an incoming event,
Enterprise Manager finishes matching the event to any further rules/rule sets.
Once completed, Enterprise Manager then matches the newly created incident to
all the rule sets from the beginning to see if any incident-specific rules match.
If an incoming event is already associated with an incident (for example, a
Warning event creates an incident and then a Critical event is generated for the
same issue), Enterprise Manager applies all the matching rules to the event and
then matches all rules to the incident.
If, while applying a rule to an incident, changes are made to the incident (change
priority. for example), Enterprise Manager stops rule application at that point and
then re-applies the rules to the incident from the beginning. The conditional action
that updated the incident will not be matched again in the same rule application
cycle.
3.1.4.3.2
Rule Criteria
The following tables list selectable criteria for each type.
Table 3–3
Rule Criteria: Events
Criteria
Description
Type
Rule applies to a specific event type.
Severity
Rule applies to a specific event severity.
Category
Rule applies to a specific event category.
Target type
Rule applies to a specific target type.
Target Lifecycle Status
Rule applies to a specific lifecycle status for a target. Lifecycle
status is a target property that specifies a target’s operational
status.
Associated with incident
Typically, events are associated with incidents through rules.
Specify Yes or No.
Event name
Rule applies to events with a specific name. The specified name
can either be an exact match or a pattern match.
Causal analysis update
Upon completion of Root Cause Analysis (RCA) event, the rule
applies to the event that is marked either as root cause or
symptom. Alternatively, the rule can act on an RCA event when
it is no longer a symptom.
Associated incident
acknowledged
Rule applies to an event that is associated with a specific
incident when that incident is acknowledged by an
administrator. Specify Yes or No.
Total occurrence count
For duplicated events, the rule is applies when the total number
of event occurrences reaches a specified number.
Comment added
Rule applies to events where an administrator adds a comment.
For incidents, a rule can apply to all new and/or updated incidents, or newly created
incidents that match specific criteria shown in the following table.
3-16 OracleВ® Enterprise Manager Administration
Management Concepts
Table 3–4
Rule Criteria: Incidents
Criteria
Description
Rules that created the
incident
Rule applies to incidents raised by a specific rule.
Category
Rule applies to a specific incident category.
Target Type
Rule applies to a specific target type.
Target Lifecycle Status
Rule applies to a specific lifecycle status for a target. Lifecycle
status is a target property that specifies a target’s operational
status.
Severity
Rule applies to a specific incident severity.
Acknowledged
Rule applies if the incident has been acknowledged by an
administrator. Specify Yes or No.
Owner
Rule applies for a specified incident owner.
Priority
Rule applies when incident priority matches a selected priority.
Status
Rule applies when the incident status matches a selected
incident status.
Escalation Level
Rule applies when the incident escalation level matches the
selected level. Available escalation levels: None, Level 1, Level 2,
Level 3, Level 4, Level 5
Associated with Ticket
Rule applies when the incident is associated with a helpdesk
ticket. Specify Yes or No.
Associated with Service
Request
Rule applies when the incident is associated with a service
request. Specify Yes or No.
Diagnostic Incident
Rule applies when the incident is a diagnostic incident. Specify
Yes or No.
Unassigned
Rule applies if the newly raised incident does not have an
owner.
Comment Added
Rule applies if an administrator adds a comment to the incident.
For problems, a rule can apply to all new and/or updated problems, or newly created
problems that match specific criteria shown in the following table.
Table 3–5
Rule Criteria: Problems
Criteria
Description
Problem key
Each problem has a problem key, which is a text string that
describes the problem. It includes an error code (such as ORA
600) and in some cases, one or more error parameters.
Rule can apply to a specific problem key or a key matching a
specific pattern (using a wildcard character).
Category
Rule applies to a specific problem category.
Target Type
Rule applies to a specific target type.
Target Lifecycle Status
Rule applies to a specific lifecycle status for a target. Lifecycle
status is a target property that specifies a target’s operational
status.
Acknowledged
Rule applies when the problem is acknowledged.
Owner
Rule applies for a specified problem owner.
Priority
Rule applies when problem priority matches a selected priority.
Using Incident Management
3-17
Management Concepts
Table 3–5 (Cont.) Rule Criteria: Problems
Criteria
Description
Status
Rule applies when the problems matches a specific status.
Escalation Level
Rule applies when the problem escalation level matches the
selected level. Available escalation levels: None, Level 1, Level 2,
Level 3, Level 4, Level 5
Incident Count
Rule applies when the number of incidents related to the
problem reaches the specified count limit. The problem owner
and the Operations manager are notified via e-mail.
Associated with Service
Request
Rule applies if the incoming problem is has an associated Service
Request. Specify Yes or No.
Associated with Bug
Rule applies if the incoming problem is has an associated bug.
Specify Yes or No.
Unassigned
Rule applies if the newly raised incident does not have an
owner.
Comment Added
Rule applies if an administrator adds a comment to the problem.
3.1.4.3.3 Rule Actions For each rule, Enterprise Manager allows you to define specific
actions.
Some examples of the types of actions that a rule set can perform are:
в– в– в– Create an incident based on an event.
Perform notification actions such as sending an email or generating a helpdesk
ticket.
Perform actions to manage incident workflow notification via e-mail/PL/SQL
methods/ SNMP traps. For example, if a target down event occurs, create an
incident and e-mail administrator Joe about the incident. If the incident is still
open after two days, set the escalation level to one and e-mail Joe’s manager.
The following table summarizes available actions for each rule application.
Table 3–6
Available Rule Actions
Action
Event
Incident Problem
E-mail
Yes
Yes
Yes
Page
Yes
Yes
Yes
Send SNMP Trap
Yes
No
No
Run OS Command
Yes
Yes
Yes
Run PL/SQL Procedure
Yes
Yes
Yes
Create an Incident
Yes
No
No
Set Workflow Attributes
Yes
Yes
Yes
Advanced Notifications
Note: Within an event
rule, the workflow
attributes of the
associated incident can
also be updated.
3-18 OracleВ® Enterprise Manager Administration
Management Concepts
Table 3–6 (Cont.) Available Rule Actions
Action
Event
Incident Problem
Create a Helpdesk Ticket
Yes
Yes
No
Note: Action performed
indirectly by first
creating an incident
and then creating a
ticket for the incident.
3.1.5 Incident Manager
Incident Manager provides, in one location, the ability to search, view, manage, and
resolve incidents and problems impacting your environment. Use Incident Manager to
perform the following tasks:
в– в– в– в– в– в– Filter incidents, problems, and events by using custom views
Search for specific incidents by properties such as target name, summary, status, or
target lifecycle status
Respond and work on an incident
Manage incident lifecycle including assigning, acknowledging, tracking its status,
prioritization, and escalation
Access (in context) My Oracle Support knowledge base articles and other Oracle
documentation to help resolve the incident.
Access direct in-context diagnostic/action links to relevant Enterprise Manager
functionality allowing you to quickly diagnose or resolve the incident.
Figure 3–4 incident Manager
For example, you have an open incident. You can use Incident Manager to track its
ownership, its resolution status, set the priority and, if necessary, add annotations to
the incident to share information with others when working in a collaborative
environment. In addition, you have direct access to pertinent information from MOS
and links to other areas of Enterprise Manager that will help you resolve issues
quickly. By drilling down on an open incident, you can access this information and
modify it accordingly.
Using Incident Management
3-19
Management Concepts
Displaying Target Information in the Context of an Incident
You can directly view information about a target for which an incident or event has
been raised. The type of information shown varies depending on the target type.
To display in-context target information:
1.
From the Enterprise menu, select Monitoring and then Incident Manager.
2.
From the Incident Manager UI, choose an incident. Information pertaining to the
incident displays.
3.
From the Incident Details area of the General tab, click on the information icon "i"
next to the target. Target information as it pertains to the incident displays. See
Figure 3–5
Figure 3–5 Target Information in Context of an Incident
Being able to display target information in this way provides you with more
operational context about the targets on which the events and incidents are raised.
This in turn helps you manage the lifecycle of the incident more efficiently.
Cloud Control Mobile
Also available is the mobile application Cloud Control Mobile, which lets you manage
incidents and problems on the go using any iDevice to remotely connect to Enterprise
Manager.
3-20 OracleВ® Enterprise Manager Administration
Management Concepts
Figure 3–6 Cloud Control Mobile
For more information about this mobile application, see Chapter 28, "Remote Access
To Enterprise Manager"
3.1.5.1 Views
Views let you work efficiently with incidents by allowing you to categorize and focus
on only those incidents of interest. A view is a set of search criteria for filtering
incidents and problems in the system. Incident Manager provides a set of predefined
standard views that cover the most common event, incident, and problem search
scenarios. In addition, Incident Manager also allows you to create your own custom
views. Custom views can be shared with other users. For instructions on creating
custom views, see "Setting Up Custom Views" on page 3-45. For instructions on
sharing a custom view, see "Sharing/Unsharing Custom Views" on page 3-46.
3.1.6 Summing Up
в– Event: A significant occurrence of interest on a target that has been detected by
Enterprise Manager.
Goal: Ensure that your environment is monitored.
в– Incident: A set of significant events or combination of related events that pertain
to the same issue.
Goal: Ensure that service disruptions are either avoided or resolved quickly.
Using Incident Management
3-21
Setting Up Your Incident Management Environment
в– Problems: The underlying root cause of incidents. Currently, this represents
critical errors in Oracle software that represents the underlying root cause of
diagnostic incidents.
Goal: Ensure underlying root causes of issues are resolved to avoid future
occurrence of issues.
Events, incidents, and problems work in concert to allow you to manage your
complete IT ecosystem both effectively and efficiently. The following illustration
summarizes how they work within your managed environment.
Figure 3–7 Event/Incident/Problem Flow
The following sections delve into events, incidents, and problems in more detail.
3.2 Setting Up Your Incident Management Environment
Before you can monitor and manage your environment using incidents, you must
ensure that your monitoring environment is properly configured. Proper configuration
consists of the following:
в– Setting Up Your Monitoring Infrastructure
в– Setting Up Notifications
в– Setting Up Administrators and Privileges
3-22 OracleВ® Enterprise Manager Administration
Setting Up Your Incident Management Environment
3.2.1 Setting Up Your Monitoring Infrastructure
The first step in setting up your monitoring infrastructure is to determine which
conditions need to be monitored and hence are the source of events. To prevent an
inordinate number of extraneous events from being generated, thus reducing system
and administrator overhead, you need to determine what is of interest to you and
enable monitoring based on your requirements. You can leverage Enterprise Manager
features such as Administrations Groups to automatically apply management settings
such as monitoring settings or compliance standards when new targets are added to
your monitored environment. This greatly simplifies the task of ensuring that events
are raised only for those conditions in which you are interested. For more information,
see Chapter 7, "Using Administration Groups".
Example: You want to ensure that the database containing your human resource
information is available round the clock. One condition you are monitoring for is
whether that database target is up or down. If it goes down, you want the appropriate
person to be notified and have them resolve the problem as quickly as possible. Other
conditions that you may want to monitor include performance threshold violations,
any changes in application configuration files, or job failures. Working with events,
you are monitoring and managing individual targets and issues directly related to
those targets. For example, you monitor for individual database availability, individual
host threshold violations such as CPU and I/O load, or perhaps the performance of a
Web service.
In general, if you are primarily interested in availability and some key performance
related metrics, you should use default monitoring templates and other template
features to ensure the only those specific metrics are collected and events are raised
only for those metrics.
Job Events: The status of a job can change throughout its lifecycle - from the time it is
submitted to the time it has executed. For each of these job statuses, events can be
raised to notify administrators of the status of the job.
As a general rule, events should be generated only for job status values that require
administration attention. These job status values include Action Required and Problem
status values such as Failed or Stopped. However, in order to avoid overloading the
system with unnecessary events, job events are not enabled for any target by default.
Hence, if you would like to generate events for jobs, you must:
1.
Set the appropriate job status. You can use the default settings or modify them as
required.
2.
Specify the set of targets for which you would like job-related events to be
generated.
You can perform these operations from the Job Event Generation Criteria page. From
the Setup menu, choose Incidents and then Job Events.
3.2.1.1 Rule Set Development
Before creating incident rules/rule sets, the first step is to strategically determine when
incidents should be created based on the business requirements of your organization.
Important questions to consider are:
1.
What events should create incidents? Which service disruptions need to be tracked
and resolved by IT administrators?
2.
Which administrators should be notified for incoming events or incidents?
3.
Are any of the events or incidents being forwarded to external systems (such as a
helpdesk ticketing system)?
Using Incident Management
3-23
Setting Up Your Incident Management Environment
Once the exact business requirements are understood, you translate those into
enterprise rule sets. Adhering to the following guidelines will result in efficient use of
system resource as well as operational efficiency.
в– в– в– For rule sets that operate on targets (for example, hosts and databases), use groups
to consolidate targets into a smaller number of monitoring entities for the rule set.
Groups should be composed of targets that have similar monitoring requirements
including incident management and response.
All the rules that apply to the same groups of targets should be consolidated into
one rule set. You can create multiple rules that apply to the targets in the rule set.
You can create rules for events specific to an event class, rules that apply to events
of a specific event class and target type, or rules that apply to incidents on these
targets.
Leverage the execution order of rules within the rule set. Rule sets and rules
within a rule set are executed in sequential order. Therefore, ensure that rules and
rule sets are sequenced with that in mind.
Event rulesWhen creating a new rule, you are given a choice as to what object the rule will
that createapply— events, incidents or problems. Use the following rule usage guidelines to help
incidents guide your selection.
Table 3–7
Rule Usage Guidelines
Incident Rule Usage
notificaionRules on Event
Incident
escalatin
Application
To create incidents for the events managed in Enterprise
Manager.
To send notifications on events.
To create tickets for incidents managed by helpdesk analysts,
you want to create an incident for an event, then create a ticket
for the incident.
Send events to third-party management systems.
Rules on Incidents
Automate management of incident workflow operations (assign
owner, set priority, escalation levels..) and send notifications
Create tickets based on incident conditions. For example, create
a ticket if the incident is escalated to level 2.
Rules on Problems
Automate management of problem workflow operations (assign
owner, set priority, escalation levels..) and send notifications
Rule Set Example
The following example illustrates many of the implementation guidelines just
discussed. All targets have been consolidated into a single group, all rules that apply
to group members are part of the same rule set, and the execution order of the rules
has been set. In this example, the rule set applies to a group (Production Group G) that
consists of the following targets:
в– DB1 (database)
в– Host1 (host)
в– WLS1 (WebLogic Server)
All rules in the rule set perform three types of actions: incident creation, notification,
and escalation.
3-24 OracleВ® Enterprise Manager Administration
Setting Up Your Incident Management Environment
Example 3–1 Example Rule Set
в– Rule Set applies to target: Group Target G
в– Rules in the Rule Set:
1.
Rule(s) to create incidents for specified events
2.
Rule(s) that send notifications on incidents
3.
Rule(s) that escalate incidents based on some condition. For example, the
length of time an incident is open.
In a more detailed view of the rule set, we can see how the guidelines have been
followed.
Example 3–2 Example Rule Set in Greater Detail
в– Rule Set for Production Group G
–
Target: Production Group G
–
Rule 1: Create an incident for all target down events.
–
Rule 2: Create an incident for specific database, host, and WebLogic Server
metric alert event of critical or warning severity.
–
Rule 3: Create an incident for any problem job events.
–
Rule 4: For all critical incidents, sent a page. For all warning incidents, send
email.
–
Rule 5: If a Fatal incident is open for more than 12 hours, set the excalation
level to 1 and email a manager.
In this detailed view, there are five rules that apply to all group members. The
execution sequence of the rules (rule 1 - rule 5) has been leveraged to correspond to the
three types of rule actions in the rule set: Rules 1-3
в– Rules 1-3: Incident Creation
в– Rule 4: Notification
в– Rule 5: Escalation
By synchronizing rule execution order with the progression of rule action categories,
execution efficiency is achieved. As shown in this example, by using conditional
actions that take different actions for the same set of events based on severity, it is
easier to change the event selection criteria in the future without having to change
multiple rules. Note: This assumes that the action requirements for all incidents (from
rules 1 - 3) are the same.
The following table illustrates explicit rule set operation for this example.
Table 3–8
Example Rule Set for Production Group G
Rule Execution
Name Order
Criteria
Action
Condition
Actions
Rule Set: Targets within Production Group G
Rule 1 First
DB1 goes down .
Create incident.
Host1 goes down.
WLS1 goes down.
Using Incident Management
3-25
Setting Up Your Incident Management Environment
Table 3–8 (Cont.) Example Rule Set for Production Group G
Rule Execution
Name Order
Criteria
Action
Condition
Rule 2 Second
Actions
DB1
If severity=Warning Create incident.
Tablespace Full (%)
If severity=Critical
Note: The warning and
critical thresholds are
defined in Metric and
Policy settings, not from
the rules UI.
Host1
CPU Utilization (%)
WLS1
Heap Usage (%)
Rule 3 Third
Event generated for
problem job status
changes for DB1, Host1,
and WLS1.
Rule 4 Fourth
All incidents for
Production Group G
Severity=Warning
Send e-mail
Severity=Critical
Send page
Incident remains open
for more than 12 days.
Status=Fatal
Increase escalation
level to 1.
Rule 5 Fifth
Create incident.
3.2.1.1.1 Before Using Rules Before you use rules, ensure the following prerequisites
have been set up:
■■■■User’s Enterprise Manager account has notification preferences (e-mail and
schedule). This is required not just for the administrator who is creating/editing a
rule, but also for any user who is being notified as a result of the rule action.
If you decide to use connectors, tickets, or advanced notifications, you need to
configure them before using them in the actions page.
Ensure that the SMTP gateway has been properly configured to send e-mail
notifications.
User’s Enterprise Manager account has been granted the appropriate privileges to
manage incidents from his managed system.
3.2.1.1.2 Setting Up Notifications After determining which events should be raised for
your monitoring environment, you need to establish a comprehensive notification
infrastructure for your enterprise by configuring Enterprise Manager to send out
e-mail and or pages, setting up e-mail addresses for administrators and tagging them
as e-mail/paging. In addition, depending on the needs of your organization,
notification setup may involve configuring advanced notification methods such as OS
scripts, PL/SQL procedures, or SNMP traps. For detailed information and setup
instructions for Enterprise Manager notifications, see Chapter 4, "Using Notifications".
3.2.2 Setting Up Administrators and Privileges
This step involves defining the appropriate administrators (which includes assigning
the proper privileges for security) and then setting up notification assignments based
on their defined roles and domain ownership within your organization.
3-26 OracleВ® Enterprise Manager Administration
Setting Up Your Incident Management Environment
To perform user account administration, click Setup on the Enterprise Manager home
page, select Security, then select Administrators to access the Administrators page.
There are two types of administrators typically involved in incident management.
в– Business Rules Architect/Analyst: Administrator who has a deep understanding of
how the business works and translates this knowledge to operational rules. Once
these rules have been deployed, the business architect uses their knowledge of the
dynamic organization to keep these rules up-to-date.
In order to create or edit an enterprise rule set, the business architect/analyst must
have been granted the Create Enterprise Rule Set privilege on the Enterprise Rule Set
resource. The architect/analyst can share ownership of the rule sets with other
administrators who may or may not have the Create Enterprise Rule Set privilege
but are responsible for managing a specific rule set.
в– IT Operator/Manager: The IT manager is responsible for day-to-day management of
incident assignment. The IT operator is assigned the incidents and is responsible
for their resolution.
Privileges Required for Enterprise Rule Sets
As the owner of the rule set, an administrator can perform the following:
в– в– в– в– Update or delete the rule set, and add, modify, or delete the rules in the rule set.
Assign co-authors of the rule set. Co-authors can edit the rule set the same as the
author. However, they cannot delete rule sets nor can they add additional
co-authors.
When a rule action is to update an event, incident, or problem (for example,
change priority or clear an event), the action succeeds only if the owner has the
privilege to take that action on the respective event, incident, or problem.
Additionally, user must be granted privilege to create an enterprise rule set.
If an incident or problem rule has an update action (for example, change priority), it
will take the action only if the owner of the respective rule set has manage privilege on
the matching incident or problem.
Using Incident Management
3-27
Setting Up Your Incident Management Environment
To grant privileges, from the Setup menu on the Enterprise Manager home page, select
Security, then select Administrators to access the Administrators page. Select an
administrator from the list, then click Edit to access the Administrator properties
wizard as shown in the following graphic.
Granting User Privileges for Events, Incidents and Problems
In order to work with incidents, all relevant Enterprise Manager administrator
accounts must be granted the appropriate privileges to manage incidents. Privileges
for events, incidents, and problems are determined according to the following rules:
в– в– в– Privileges on events are calculated based on the privilege on the underlying source
objects. For example, the user will have VIEW privilege on an event if he can view
the target for the event.
Privileges on an incident are calculated based on the privileges on the events in the
incident.
Similarly, problem privileges are calculated based on privileges on underlying
incidents.
Users are granted privileges for events, incidents, and problems in the following
situations.
For events, two privileges are defined in the system:
в– в– The View Event privilege allows you to view an event and add comments to the
event.
The Manage Event privilege allows you to take update actions on an event such as
closing an event, creating an incident for an event, and creating a ticket for an
event. You can also associate an event with an incident.
Important:
Incident privilege is inherited from the underlying
events.
3-28 OracleВ® Enterprise Manager Administration
Setting Up Your Incident Management Environment
If an event is raised on a target alone (the majority of event types are raised on targets
such as metric alerts, availability events or service level agreement), you will need the
following privileges:
в– View on target to view the event.
в– Manage Target Events to manage the event.
Note: This is a sub-privilege of Operator.
If an event is raised on both a target and a job, you will need the following privileges:
в– View on target and View on the job to view the event.
в– View on target and Full on the job to manage the event.
If the event is raised on a job alone, you will need the following privileges:
в– View on the job to view the event.
в– Full on the job to manage the event.
If an event is raised on a metric extension, you will need View privilege on the metric
extension to view the event. Because events raised on metric extensions are
informational (and do not appear in Incident Manager) event management privileges
do not apply in this situation.
If an event is raised on a Self-update, only system privilege is required. Self-update
events are strictly informational.
For incidents, two privileges are defined in the system:
в– в– The View Incident privilege allows you to view an incident, and add comments to
the incident.
The Manage Incident privilege allows you to take update actions on an incident.
The update actions supported for an incident includes incident assignment and
prioritization, resolution management, manually closing events, and creating
tickets for incidents.
If an incident consists of a single event, you can view the incident if you can view the
event and manage the incident if you can manage the event.
If an incident consists of more than one event, you can view the incident if you can
view at least one event and manage incident if you can manage at least one of the
events.
For problems, two privileges are defined:
в– в– The View Problem privilege allows you to view a problem and add comments to
the problem.
The Manage Problem privilege allows you to take update actions on the problem.
The update actions supported for a problem include problem assignment and
prioritization, resolution management, and manually closing the problem.
In Enterprise Manager 12c, problems are always related to a single target. So the View
Problem privilege, if an administrator has View privilege on the target, and the
Manage Problem privilege, if an administrator has manage_target_events privilege on
the target, implicitly grants management privileges on the associated event. This, in
turn, grants management privileges on the incident within the problem.
Using Incident Management
3-29
Setting Up Your Incident Management Environment
3.2.3 Monitoring Privileges
The monitoring functions that an administrator can perform within the Enterprise
Manager environment depend on privileges that have been granted to that user. To
maintain the integrity and security of a monitored infrastructure, only the required
privileges for a specific role should be granted. The following guidelines can be used
to grant proper privilege levels based on user roles.
Administrators who set up monitoring
Create a role with privileges and grant it to administrators:
в– Recommend using individual user accounts instead of shared account
в– If using super administrator, do not use sysman
в– If privilege is based on targets, create privilege-propagating group containing the
targets (or use administration group if it meets requirements) and grant privilege
on the group to the role
Administrators who respond to events / incidents
в– Create a role and grant it to administrators
в– Create privilege-propagating group (or use administration group if it meets
requirements) containing relevant targets and grant appropriate privilege on the
group to the role
Example: You create the role DB_Admins and grant Manage Target Events on a the
privilege-propagating group named DB-group containing relevant databases. You
then grant role DB_Admins to the DBAs.
Monitoring Actions and Required Privileges
Enterprise Manager supports fine-grained privileges to enable more granular control
over actions performed in Enterprise Manager.
The table below shows a (non-exhaustive) list of various job responsibilities and the
corresponding privilege in Enterprise Manager required to support these
The following tables summarize the privilege levels required to perform specific
monitoring responsibilities.
Table 3–9
Monitoring Operations and Required Privileges
Monitoring Operation
Required Privilege(s)
Monitoring Setup
Configure SMTP gateway (email)
Super Administrator
Create Advanced Notification Methods (e.g. SNMP traps) Super Administrator
Configure event or ticketing connector
Super Administrator
Creating Roles
Super Administrator
Create Administration Group Hierarchy
Full Any Target
Create Privilege Propagating
Group
3-30 OracleВ® Enterprise Manager Administration
Setting Up Your Incident Management Environment
Table 3–9 (Cont.) Monitoring Operations and Required Privileges
Monitoring Operation
Required Privilege(s)
Edit Administration Group Hierarchy
Full Any Target
Create Privilege Propagating
Group (if adding new target
property values as group criteria
within a level of the
administration group hierarchy)
Delete Administration Group Hierarchy
Full Any Target
View entire Administration Group hierarchy in Group
Administration pages
View Any Target
Use Monitoring Templates
No privileges required to create
new monitoring templates.
However if the monitoring
template contains a corrective
action, then Create on Job System
privilege is required
Note: Administrators who have
privileges to only a subset of the
groups can view these groups in
the Groups list page accessible via
Targets-->Groups
View on specific monitoring
template to use the template
created by another user (e.g. to
add the monitoring template to a
Template Collection
Use Template Collections
Create Template Collection (to
create new Template Collections)
View Template Collection on
specific Template Collection to
view/associate the Template
Collection created by another user
View Any Template Collection to
view/associate any Template
Collection
Full Template Collection on
specific Template Collection to
edit/delete the Template
Collection created by another user
Associate a Template Collection with an Administration
Group
Manage Template Collection
Operations on the group (this
includes Manage Target
Compliance and Manage Target
Metrics privileges)
View Template Collection on the
Template Collection
Operations on the Administration Group
Manage privileges on the group (for example, grant to
other users)
Group Administration on the
group
Add a target to an Administration Group by setting its
target properties
Configure Target (on the target to
be added to the Administration
Group)
Perform a manual sync of the group with the associated
Template Collection
Manage Template Collection
Operations on the group
Using Incident Management
3-31
Setting Up Your Incident Management Environment
Table 3–9 (Cont.) Monitoring Operations and Required Privileges
Monitoring Operation
Required Privilege(s)
Operations on the members of the Administration
Group
Delete the target from Enterprise Manager
Full on the target (Full also
contains the privileges
enumerated below
Operator on the target also
contains the following privileges:
Set blackout for planned downtime
Change monitoring settings
Change monitoring configuration
Manage events and incidents on the target
в– в– в– в– Blackout Target on the target
Manage Target Metrics on the
target
Configure Target on the target
Manage Target Events on the
target
View on the target
View target, receive notifications for events or incidents
в– Create Incident Rule Sets
Create Enterprise Rule Set
Manage Target Events on target if
rule is creating incidents for the
target
Granting privileges on administration group to roles
No extra privilege required if
creator of the administration
group
Set a target’s property values
Configure Target
Edit Monitoring Template that is part of Template
Collection
Full on the Monitoring Template
Change monitoring settings on specific target
Manage Target Metrics
Receive email for events, incidents
View on Target and/or
Manage Target Metrics on
administration group
View on source object (for
example, view on job for job
events)
Create incident for event
Manage Target Events
Incident management actions (for example, acknowledge, Manage Target Events
assign incident, prioritize, set escalation level)
SYSMAN is a system account intended for Enterprise Manager
infrastructure installation and maintenance. It should never be used
for administrator access to Enterprise Manager as a Super
Administrator.
Note:
3-32 OracleВ® Enterprise Manager Administration
Setting Up Your Incident Management Environment
3.2.4 Setting Up Rule Sets
Rule sets automate actions in response to incoming events, incidents and problems or
updates to them. This section covers the most common tasks and examples.
в– Creating a Rule Set
в– Creating a Rule to Create an Incident
в– Creating a Rule to Manage Escalation of Incidents
в– Creating a Rule to Escalate a Problem
в– Testing Rule Sets
в– Subscribing to Receive Email from a Rule
в– Receiving E-mail for Private Rules
3.2.4.1 Creating a Rule Set
In general, to create a rule set, perform the following steps:
1.
From the Setup menu, select Incidents then select Incident Rules.
2.
On the Incident Rules - All Enterprise Rules page, edit the existing rule set or
create a new rule set. For new rule sets, you will need to first select the targets to
which the rules apply. Rules are created in the context of a rule set.
In the case where there is no existing rule set, create a rule set
by clicking Create Rule Set... You then create the rule as part of
creating the rule set.
Note:
Narrowing Rule Set Scope Based on Target Lifecycle Status
When creating a new rule set, you can choose to have the rule set apply to a
narrower set of targets based on the target’s Lifecycle Status value. For example,
you can create one rule set that only applies only to targets that have a Lifecycle
Status of Staging and Production. As shown in the following graphic, you
determine rule set scope by setting the Lifecycle Status filter.
Using this filter allows you to create rules for targets based on their Lifecycle Status
without having to first create a group containing only such targets.
3.
In the Rules tab of the Edit Rule Set page, click Create... and select the type of rule
to create (Event, Incident, Problem) on the Select Type of Rule to Create pop-up
dialog. Click Continue.
4.
In the Create New Rule wizard, provide the required information.
Using Incident Management
3-33
Setting Up Your Incident Management Environment
5.
Once you have finished defining the rule, click Continue to add the rule to the
rule set. Click Save to save the changes made to the rule set.
3.2.4.2 Creating a Rule to Create an Incident
To create a rule that creates an incident, perform the following steps:
1.
From the Setup menu, select Incidents, then select Incident Rules.
2.
Determine whether there is an existing rule set that contains a rule that manages
the event. In the Incident Rules page, use the Search option to find the rule/rule
set name, description, target name, or target type for the target and the associated
rule set. You can search by target name or the group target name to which this
target belongs to locate the rule sets that manage the targets.
Note: In the case where there is no existing rule set, create a rule set by clicking
Create Rule Set... You then create the rule as part of creating the rule set.
3.
Select the rule set that will contain the new rule. Click Edit... In the Rules tab of the
Edit Rule Set page,
1.
Click Create ...
2.
Select "Incoming events and updates to events"
3.
Click Continue.
Provide the rule details using the Create New Rule wizard.
a.
Select the Event Type the rule will apply to, for example, Metric Alert. (Metric
Alert is available for rule sets of the type Targets.) Note: Only one event type
can be selected in a single rule and, once selected, it cannot be changed when
editing a rule.
You can then specify metric alerts by selecting Specific Metrics. The table for
selecting metric alerts displays. Click the +Add button to launch the metric
selector. On the Select Specific Metric Alert page, select the target type, for
example, Database Instance. A list of relevant metrics display. Select the ones
in which you are interested. Click OK.
You also have the option to select the severity and corrective action status.
b.
Once you have provided the initial information, click Next. Click +Add to add
the actions to occur when the event is triggered. One of the actions is to Create
Incident.
As part of creating an incident, you can assign the incident to a particular user,
set the priority, and create a ticket. Once you have added all the conditional
actions, click Continue.
4.
c.
After you have provided all the information on the Add Actions page, click
Next to specify the name and description for the rule. Once on the Review
page, verify that all the information is correct. Click Back to make corrections;
click Continue to return to the Edit (Create) Rule Set page.
d.
Click Save to ensure that the changes to the rule set and rules are saved to the
database.
Test the rule by generating a metric alert event on the metrics chosen in the
previous steps.
3.2.4.3 Creating a Rule to Manage Escalation of Incidents
To create a rule to manage incident escalation, perform the following steps:
3-34 OracleВ® Enterprise Manager Administration
Setting Up Your Incident Management Environment
1.
From the Setup menu, select Incidents, then select Incident Rules.
2.
Determine whether there is an existing rule set that contains a rule that manages
the incident. You can add it to any of your existing rule sets on incidents.
Note: In the case where there is no existing rule set, create a rule set by clicking
Create Rule Set... You then create the rule as part of creating the rule set.
3.
4.
Select the rule set that will contain the new rule. Click Edit... in the Rules tab of the
Edit Rule Set page, and then:
1.
Click Create ...
2.
Select "Newly created incidents or updates to incidents"
3.
Click Continue.
For demonstration purposes, the escalation is in regards to a production database.
As per the organization's policy, the DBA manager is notified for escalation level 1
incidents where a fatal incident is open for 48 hours. Similarly, the DBA director is
paged if the incident has been escalated to level 2, the severity is fatal and it has
been open for 72 hours. If the fatal incident is still open after 96 hours, then it is
escalated to level 3 and the operations VP is notified.
Provide the rule details using the Create New Rule wizard.
a.
To set up the rule to apply to all newly created incidents or when the incident
is updated with fatal severity, select the Specific Incidents option and add the
condition Severity is Fatal .
b.
In the Conditions for Actions region located on the Add Actions page, select
Only execute the actions if specified conditions match.
Select Incident has been open for some time and is in a particular state
(select time and optional expressions).
Select the time to be 48 hours and Status is not resolved or closed.
c.
In the Notification region, type the name of the administrator to be notified by
e-mail or page. Click Continue to save the current set of conditions and
actions.
Using Incident Management
3-35
Setting Up Your Incident Management Environment
5.
d.
Repeat steps b and c to page the DBA director (Time in this state is 72 hours,
Status is Not Resolved or Closed). If open for more than 96 hours, set
escalation level to 3, page Operations VP.
e.
After reviewing added actions sets, click Next.
f.
Click Next to go to the Summary screen. Review the summary information
and click Continue to save the rule.
Review the sequence of existing enterprise rules and position the newly created
rule in the sequence.
In Edit Rule Set page, click on the desired rule from the Rules table and select
Reorder Rules from the Actions menu to reorder rules within the rule set, then
click Save to save the rule sequence changes.
Example Scenario
To facilitate the incident escalation process, the administration manager creates a rule
to escalate unresolved incidents based on their age:
в– To level 1 if the incident is open for 30 minutes
в– To level 2 if the incident is open for 1 hour
в– To level 3 if the incident is open for 90 minutes
As per the organization's policy, the DBA manager is notified for escalation level 1.
Similarly, the DBA director and operations VP are paged for incidents escalated to
levels "2" and "3" respectively.
Accordingly, the administration manager inputs the above logic and the respective
Enterprise Manager administrator IDs in a separate rule to achieve the above
notification requirement. Enterprise Manager administrator IDs represents the
respective users with required target privileges and notification preferences (that is,
e-mail addresses and schedule).
3.2.4.4 Creating a Rule to Escalate a Problem
In an organization, whenever an unresolved problem has more than 20 occurrences of
associated incidents, the problem should be auto-assigned to the appropriate
administrator based on target type of the target on which the problem has been raised.
Accordingly, a problem rule is created to observe the count of incidents attached to the
problem and notify the appropriate administrator handling that specific target type.
The problem owner and the Operations manager are notified by e-mail.
To create a rule to escalate a problem, perform the following steps:
1.
Navigate to the Incident Rules page.
From the Setup menu, select Incidents, then select Incident Rules.
2.
On the Incident Rules - All Enterprise Rules page, either create a new rule set
(click Create Rule Set...) or edit an existing rule set (highlight the rule set and click
Edit...). Rules are created in the context of a rule set.
Note: In the case where there is no existing rule set, create a rule set by clicking
Create Rule Set... You then create the rule as part of creating the rule set.
3.
In the Rules section of the Edit Rule Set page, select Create...
4.
From the Select Type of Rule to Create dialog, select Newly created problems or
updates to problems and click Continue.
3-36 OracleВ® Enterprise Manager Administration
Setting Up Your Incident Management Environment
5.
On the Create New Rule page, select Specific problems and add the following
criteria:
The Attribute Name is Incident Count, the Operator is Greater than or equals and
the Values is 20.
Click Next.
6.
In the Conditions for Actions region on the Add Actions page select Always
execute the action. As the actions to take when the rule matches the condition:
в– в– In the Notifications region, send e-mail to the owner of the problem and to the
Operations Manager.
In the Update Problem region, enter the e-mail address of the appropriate
administrator in the Assign to field.
Click Continue.
7.
Review the rules summary. Make corrections as needed. Click Continue to return
to Edit Rule Set page and then click Save to save the rule set.
3.2.4.5 Testing Rule Sets
When developing a rule set, it can be difficult to develop rule criteria to match all
possible event conditions. Previously, the only way to test rules was to trigger an event
within your monitored environment and seeing which rules match the event and what
actions the rules perform. Beginning with Enterprise Manager Release 12.1.0.4, you
can simulate existing events, thus allowing you to test rule actions during the rule set
development phase and not waiting for specific event conditions to occur. The rule
simulation feature lets you see how the rules will perform given a specific event. You
immediately see which rules match for a given event and then see what actions are
taken.
Note:
The simulate rule feature can only be used with event rules
Incident rules cannot be tested with this feature.
To simulate rules:
This procedure assumes you have already created rule sets. See "Creating a Rule Set"
on page 3-33 for instructions on creating a rule set. Ensure that the rule type is
Incoming events and updates to events.
1.
From the Setup menu, select Incidents, and then Incident Rules. The Incident Rules
- All Enterprise Rules page displays.
2.
Click Simulate Rules. The Simulate Rules dialog displays.
Using Incident Management
3-37
Setting Up Your Incident Management Environment
3.
Enter the requisite search parameters to find matching events and click Search.
4.
Select an event from the list of results.
5.
Click Start Simulation. The event will be passed through the rules as if the event
had newly occurred. Rules will be simulated based on the current notification
configuration (such as email address, schedule for the assigned administrator, or
repeat notification setting).
Changing the Target Name: Under certain circumstances, an event matching rule
criteria may occur on a target that is not a rule target. For testing purposes, you are
only interested in the event. To use the alternate target for the simulation, click
Alter Target Name and Start Simulation.
Results are displayed.
6.
If the rule actions are not what you intended, edit the rules and repeat the rule
simulation process until the rules perform the desired actions.
3-38 OracleВ® Enterprise Manager Administration
Setting Up Your Incident Management Environment
3.2.4.6 Subscribing to Receive Email from a Rule
A DBA is aware that incidents owned by him will be escalated when not resolved in 48
hours. The DBA wants to be notified when the rule escalates the Incident. The DBA
can subscribe to the Rule, which escalates the Incident and will be notified whenever
the rule escalates the Incident.
Before you set up a notification subscription, ensure there exists a rule that escalates
High Priority Incidents for databases that have not been resolved in 48 hours
Perform the following steps:
1.
From the Setup menu, select Incidents, and then select Incident Rules.
2.
On the Incident Rules - All Enterprise Rules page, click on the rule set containing
incident escalation rule in question and click Edit... Rules are created in the context
of a rule set.
Note: In the case where there is no existing rule set, create a rule set by clicking
Create Rule Set... You then create the rule as part of creating the rule set.
3.
In the Rules section of the Edit Rule Set page, highlight the escalation rule and
click Edit....
4.
Navigate to the Add Actions page.
5.
Select the action that escalates the incident and click Edit...
6.
In the Notifications section, add the DBA to the E-mail cc list.
7.
Click Continue and then navigate back to the Edit Rule Set page and click Save.
As a result of the edit to the enterprise rule, when an incident stays unresolved for 48
hours, the rule marks it to escalation level 1. An e-mail is sent out to the DBA notifying
him about the escalation of the incident.
Alternate Rule Set Subscription Method: From the Incident Rules - All Enterprise
Rules page, select the rule in incident rules table. From the Actions menu, select
E-mail and then Subscribe me (or Subscribe administrator....).
3.2.4.7 Receiving E-mail for Private Rules
A DBA has setup a backup job on the database that he is administering. As part of the
job, the DBA has subscribed to e-mail notification for "completed" job status. Before
you create the rule, ensure that the DBA has the requisite privileges to create jobs. See
Chapter 11, "Utilizing the Job System and Corrective Actions" for job privilege
requirements.
Perform the following steps:
1.
Navigate to the Rules page.
From the Setup menu, select Incidents, then select Incident Rules.
2.
On the Incident Rules - All Enterprise Rules page, either edit an existing rule set
(highlight the rule set and click Edit...) or create a new rule set.
Note: The rule set must be defined as a Private rule set.
3.
In the Rules tab of the Edit Rule Set page, select Create... and select Incoming
events and updates to events. Click Continue.
4.
On the Select Events page, select Job Status Change as the Event Type. Select the
job in which you are interested either by selecting a specific job or selecting a job
by providing a pattern, for example, Backup Management.
Using Incident Management
3-39
Working with Incidents
Add additional criteria by adding an attribute: Target Type as Database Instance.
5.
Add conditional actions: Event matches the following criteria (Severity is
Informational) and E-mail Me for notifications.
6.
Review the rules summary. Make corrections as needed. Click Save.
7.
Create a database backup job and subscribe for e-mail notification when the job
completes.
When the job completes, Enterprise Manager publishes the informational event for
"Job Complete" state of the job. The newly created rule is considered ’matching’
against the incoming job events and e-mail will be sent to the DBA.
The DBA receives the e-mail and clicks the link to access the details section in
Enterprise Manager console for the event.
3.3 Working with Incidents
Data centers follow operational practices that enable them to manage events and
incidents by business priority and in a collaborative manner. Enterprise Manager
provides the following features to enable this management and automation:
в– Send notifications to the appropriate administrators.
в– Create incidents and rules.
в– Assigning initial ownership of an incident and perhaps transferring ownership
based on shift assignments or expertise.
в– Tracking its resolution status.
в– Assigning priorities based on the component affected and nature of the incident.
в– Escalating incidents.
в– Accessing My Oracle Support knowledge articles.
в– Opening Oracle Service Requests to request assistance with issues with Oracle
software (Problems).
You can update resolution information for an incident by performing the following:
1.
In the All Open Incidents view, select the incident.
2.
In the resulting Details page, click the General tab, then click Manage. The
Manage dialog displays.
3-40 OracleВ® Enterprise Manager Administration
Working with Incidents
You can then adjust the priority, escalate the incident, and assign it to a specific IT
operator.
Working with incidents involves the following stages:
1.
Finding What Needs to be Worked On
2.
Searching for Incidents
3.
Setting Up Custom Views
4.
Responding and Working on a Simple Incident
5.
Responding to and Managing Multiple Incidents, Events and Problems in Bulk
6.
Managing Workload Distribution of Incidents
7.
Creating an Incident Manually
3.3.1 Finding What Needs to be Worked On
Enterprise Manager provides multiple access points that allow you to find out what
needs to be worked on. The primary focal point for incident management is the
Incident Manager console, however Enterprise Manager also provides other methods
of notification. The most common way to be notified that you have an issue that needs
to be addressed is by e-mail. However, incident information can also be found in the
following areas:
Custom Views (See "Setting Up Custom Views")
Using Incident Management
3-41
Working with Incidents
Group or System Homepages (See Chapter 6, "Managing Groups")
Target Homepages
3-42 OracleВ® Enterprise Manager Administration
Working with Incidents
Incident Manager (in context of a system or target)
Enterprise Manager Console
Using Incident Management
3-43
Working with Incidents
3.3.2 Searching for Incidents
You can search for incidents based on a variety of incident attributes such as the time
incidents were last updated, target name, target type, or incident status.
1.
Navigate to the Incident Manager page.
From the Enterprise menu on the Enterprise Manager home page, select
Monitoring, then select Incident Manager.
2.
In the Views region located on the left, click Search.
a.
In the Search region, search for Incidents using the Type list and select
Incidents.
b.
In the Criteria region, choose all the criteria that are appropriate. To add fields
to the criteria, click Add Fields... and select the appropriate fields.
c.
After you have provided the appropriate criteria, click Get Results.
Validate that the list of incidents match what you are looking for. If not,
change the search criteria as needed.
d.
To view all the columns associated with this table, in the View menu, select
Columns, then select Show All.
Searching for Incidents by Target Lifecycle Status
In addition to searching for incidents using high-level incident attributes, you can also
perform more granular searches based on individual target lifecycle status. Briefly,
lifecycle status is a target property that specifies a target’s operational status. Status
options for which you can search are:
в– All
в– Mission Critical
в– Production
3-44 OracleВ® Enterprise Manager Administration
Working with Incidents
в– Staging
в– Test
в– Development
For more discussion on lifecycle status, see Section 3.4.7, "Event Prioritization."
To search for incidents by target lifecycle status:
1.
Navigate to the Incident Manager page.
From the Enterprise menu on the Enterprise Manager home page, select
Monitoring, then select Incident Manager.
2.
In the Views region located on the left, click Search.
3.
In the Search region, click Add Fields. A pop-up menu appears showing the
available lifecycle statuses.
4.
Choose on one or more of the lifecycle status options.
5.
Enter any additional search criteria.
6.
Click Get Results.
3.3.3 Setting Up Custom Views
Incident Manager also allows you to define custom views to help you gain quick
access to the incidents and problems on which you need to focus. For example, you
may define a view to display all critical database incidents that you own. By specifying
and saving view preferences to display only those incident attributes that you are
interested in Enterprise Manager will show only the list of matching incidents.
You can then search the incidents for only the ones with specific attributes, such as
priority 1. The view allows easy access to pertinent incidents for daily triage.
Accordingly, you can save the search criteria as a filter named "All priority 1 incidents
for my targets". The view becomes available in the UI for immediate use and will be
available anytime you log in to access the specific incidents. The last view you used
will be the default view used on your next login.
Perform the following steps:
1.
Navigate to the Incident Manager page.
From the Enterprise menu on the Enterprise Manager home page, select
Monitoring, then select Incident Manager.
2.
In the MyViews region located on the left, click the create "+" icon.
a.
In the Search region, search for Incidents using the Type list and select
Incidents.
b.
In the Criteria region, choose all the criteria that are appropriate. To add fields
to the criteria, click Add Fields... and select the appropriate fields.
c.
After you have provided the appropriate criteria, click Get Results.
Validate that the list of incidents match what you are looking for. If not,
change the search criteria as needed.
d.
To view all the columns associated with this table, in the View menu, select
Columns, then select Show All.
Using Incident Management
3-45
Working with Incidents
To select a subset of columns to display and also the order in which to display
them, from the View menu, select Columns, then Manage Columns. A dialog
displays showing a list of columns available to be added in the table.
e.
Click the Create View... button.
f.
Enter the view name. If you want other administrators to use this view, check
the Share option.
g.
Click OK to save the view.
Note: From the View creation dialog, you can also mark the view as
shared. See Section 3.3.4, "Sharing/Unsharing Custom Views" for
more information.
3.3.4 Sharing/Unsharing Custom Views
When you create your own views, they are private (only you can see them). Beginning
with Enterprise Manager Release 12.1.0.4, you can share your private views with other
administrators. When you share a view, all Enterprise Manager users will be able to
use the view.
As mentioned previously, you are given the opportunity to share a view during the
view creation process. If you have already created custom views, you can share them
at any time.
1.
Navigate to Incident Manager.
From the Enterprise menu on the Enterprise Manager home page, select
Monitoring, then select Incident Manager.
2.
From the My Views region, click the Manage icon.
3.
From the Manage Custom Views dialog, choose a custom view.
4.
Click Share (or Unshare if the view is already shared and you want to unshare it.)
5.
Click Yes to confirm the share/unshare operation.
3.3.5 Responding and Working on a Simple Incident
The following steps take you through one possible incident management scenario.
1.
Navigate to Incident Manager.
3-46 OracleВ® Enterprise Manager Administration
Working with Incidents
From the Enterprise menu on the Enterprise Manager home page, select
Monitoring, then select Incident Manager.
2.
Use a view to filter the list of incidents. For example, you should use My Open
Incidents and Problems view to see incidents and problems assigned to you. You
can then sort the list by priority.
3.
To work on an incident, select the incident. In the General tab, click Acknowledge
to indicate that you are working on this incident, and to stop receiving repeat
notifications for the incident.
In addition to the acknowledging the incident, you can perform other incident
management operations such as:
в– в– Adding a comment.
Managing the incident. See Section 3.3.6, "Responding to and Managing
Multiple Incidents, Events and Problems in Bulk" for more information on
incident management options.
в– Editing the summary.
в– Manually creating a ticket.
в– Suppressing/unsuppressing the incident.
в– Clearing the incident.
Be aware that as you are working on an individual incident, new incidents might
be coming in. Update the list of incidents by clicking the Refresh icon.
4.
If the solution for the incident is unknown, use one or all of the following methods
made available in the Incident page:
в– в– в– Use the Guided Resolution region and access any recommendations,
diagnostic and resolution links available.
Check My Oracle Support Knowledge base for known solutions for the
incident.
Study related incidents available through the Related Events and Incidents
tab.
5.
Once the solution is known and can be resolved right away, resolve the incident by
using tools provided by the system, if possible.
6.
In most cases, once the underlying cause has been fixed, the incident is cleared in
the next evaluation cycle. However, in cases like log-based incidents, clear the
incident.
Alternatively, you can work with incidents for a specific target from that target’s home
page. From the target menu, select Monitoring and then select Incident Manager to
access incidents for that target (or group).
3.3.6 Responding to and Managing Multiple Incidents, Events and Problems in Bulk
There may be situations where you want to respond to multiple incidents in the same
way. For example, you find that a cluster of incidents that are assigned to you are due
to insufficient tablespace issues on several production databases. Your manager
suggests that these tablespaces be transferred to a storage system being procured by
another administrator. In this situation, you want to set all of the tablespace incidents
to a customized resolution state "Waiting for Hardware." You also want to assign the
incidents to the other administrator and add a comment to explain the scenario. In this
situation, you want to update all of these incidents in bulk rather than individually.
Using Incident Management
3-47
Working with Incidents
To respond to incidents in bulk:
1.
Navigate to Incident Manager.
From the Enterprise menu on the Enterprise Manager home page, select
Monitoring, then select Incident Manager.
2.
Use a view to filter the list of incidents to the subset of incidents you want to work
on. For example, you can use My Open Incidents and Problems view to see
incidents and problems assigned to you. You can then sort the list by priority.
3.
Select the incidents to which you want to respond. You can select multiple
incidents by holding down the Control key and selecting individual incidents or
you can hold down the Shift key and select the first and last incidents to select a
contiguous block of incidents.
4.
From the Action menu, choose the desired response action.
в– в– в– Acknowledge: Indicate that you have viewed the incidents. This option also
stops any repeat notifications sent out for the incidents. This sets the
Acknowledged flag to Yes and also makes you the owner of the incident
Manage: Allows you to perform a multi-action response to the incidents.
–
Acknowledge: If an incident is acknowledged, it will be implicitly assigned
to the user who acknowledged it. When a user assigns an incident to
himself, it is considered acknowledged. Once acknowledged, an incident
cannot be unacknowledged. Acknowledgement also stops any repeat
notifications for that incident
–
Assign to: Assign the incident(s) to the administrator who will take
ownership of the incident.
–
Prioritization: The priority level of an incident can be set by selecting one of
the out-of-the-box priority values: None, Urgent, Very High, High,
Medium, Low
–
Incident Status: The resolution state for the incident can be set by selecting
either Work in Progress or Resolved or to any custom status defined.
–
Escalation Level: Administrators can update incidents to set an escalation
level: Level 1 through 5, in addition to the default value of None. An
escalated issue can be de-escalated by setting the escalation to None. The
appropriate Escalation Level depends on the IT procedures you have in
place.
–
Comment: You can enter comments such as those you want to pass to the
owner of the incident.
Suppress: Suppressing an incident stops corresponding notifications, and
removes it from out-of-the-box views and default totals (such as those
presented in the summary region). Suppression is typically performed when
you want to defer action on the incident until a future time and in the
meantime want to visually hide them from appearing in the console.
Administrators can see suppressed incidents by explicitly searching for them
such as performing a search on incidents where the search criteria includes the
Suppressed search field
Incidents can be suppressed until any of the following conditions are met:
–
Until the suppression is manually removed
–
Until specified date in the future
3-48 OracleВ® Enterprise Manager Administration
Working with Incidents
■■–
Until the severity state changes (incidents only)
–
Until it is closed
Clear: Administrators can clear incidents or problems manually. For incidents,
this applies only to incidents containing incidents that can be manually
cleared.
Add Comment: Users can add comments on incidents and events. Comments
may be used for sharing information with other users or to provide tracking
information on any actions being taken. Comments can be added even on
closed issues.
Note: The single action Acknowledge and Clear buttons are enabled
for open incidents and can be used for multiple incident selection.
If any of the above actions applies only to a subset of selected incidents (for
example, if an administrator tries to acknowledge multiple incidents, of which
some are already acknowledged), the action will be performed only where
applicable. The administrator will be informed of the success or failure of the
action.
When an administrator selects any of these actions, a corresponding annotation is
added to the incident for future reference.
5.
Click OK. Enterprise Manager displays a process summary and confirmation
dialogs.
6.
Continue working with the incidents as required.
3.3.7 Searching My Oracle Support Knowledge
To access My Oracle Support Knowledge base entries from within Incident Manager,
perform the following steps:
1.
Navigate to Incident Manager.
From the Enterprise menu on the Enterprise Manager home page, select
Monitoring, then select Incident Manager.
2.
Select one of the standard views. Choose the appropriate incident or problem in
the View table.
3.
In the resulting details region, click My Oracle Support Knowledge.
If your My Oracle Support (MOS) login credentials have been saved as MOS
Preferred Credentials, you do not need to log in manually. If not, you will need to
sign in to My Oracle Support. To save your MOS login information as Preferred
Credentials.
Setting MOS Preferred Credentials: From the Setup menu, select Security and
then Preferred Credentials. From the My Oracle Support Preferred Credentials
region, click Set MOS Credentials.
4.
On the My Oracle Support page, click the Knowledge tab to browse the
knowledge base.
From this page, in addition to accessing formal Oracle documentation, you can
also change the search string in to look for additional knowledge base entries.
Using Incident Management
3-49
Working with Incidents
3.3.8 Open Service Request (Problems-only)
There are times when you may need assistance from Oracle Support to resolve a
problem. This procedure is not relevant for incidents or events.
To submit a service request (SR), perform the following steps:
1.
Navigate to Incident Manager.
From the Enterprise menu on the Enterprise Manager home page, select
Monitoring, then select Incident Manager.
2.
Use one of the views to find the problem or search for it or use one of your custom
views. Select the appropriate problem from table.
3.
Click on the Support Workbench: Package Diagnostic link.
4.
Complete the workflow for opening an SR. Upon completing the workflow, a draft
SR will have been created.
5.
Sign in to My Oracle Support if you are not already signed in.
6.
On the My Oracle Support page, click the Service Requests tab.
7.
Click Create SR button.
3.3.9 Suppressing Incidents and Problems
There are times when it is convenient to hide an incident or problem from the list in
the All Open Incidents page or the All Open Problems page. For example, you need to
defer work on the incident until a future date (for example, until maintenance
window). In order to avoid having it appear in the UI, you want to temporarily hide or
suppress the incident until a future date. In order to find a suppressed incident, you
must explicitly search for the incident using either the Show all or the Only show
suppressed search option. In order to unhide a suppressed incident or problem, it must
be manually unsuppressed.
To suppress an incident or problem:
1.
Navigate to Incident Manager.
From the Enterprise menu on the Enterprise Manager home page, select
Monitoring, then select Incident Manager.
2.
Select either the All Open Incidents view or the All Open Problems view.
Choose the appropriate incident or problem. Click the General tab.
3.
In the resulting details region, click More, then select Suppress.
4.
On the resulting Suppress pop-up, choose the appropriate suppression type.
Add a comment if desired.
5.
Click OK.
3.3.10 Managing Workload Distribution of Incidents
Incident Manager enables you to manage incidents and problems to be addressed by
your team.
Perform the following tasks:
1.
Navigate to Incident Manager.
3-50 OracleВ® Enterprise Manager Administration
Working with Incidents
From the Enterprise menu on the Enterprise Manager home page, select
Monitoring, then select Incident Manager.
2.
Use the standard or custom views to identify the incidents for which your team is
responsible. You may want to focus on unassigned and unacknowledged incidents
and problems.
3.
Review the list of incidents. This includes: determining person assigned to the
incident, checking its status, progress made, and actions taken by the incident
owner.
4.
Add comments, change priority, reassign the incident as needed by clicking on the
Manage button in the Incident Details region.
Example Scenario
The DBA manager uses Incident Manager to view all the incidents owned by his team.
He ensures all of them are correctly assigned; if not, he reassigns and prioritizes them
appropriately. He monitors the escalated events for their status and progress, adds
comments as needed for the owner of the incident. In the console, he can view how
long each of the incidents has been open. He also reviews the list of unassigned
incidents and assigns them appropriately.
3.3.11 Reviewing Events on a Periodic Basis
Oracle recommends managing via incidents in order to focus on important events or
groups of related events. Due to the variety and sheer number of events that can be
generated, it is possible that not all important events will be covered by incidents. To
help you find these important yet untreated events, Enterprise Manager provides the
Events without incidents standard view.
Perform the following steps:
1.
From the Enterprise menu, select Monitoring, then select Incident Manager.
2.
In the Views region, click Events without incidents.
3.
Select the desired event in the table. The event details display.
4.
In the details area, choose More and then either Create Incident or Add Event to
Incident.
Example Scenario
During the initial phase of Enterprise Manager uptake, every day the DBA manager
reviews the events for the databases his team is responsible for and filters them to
view only the ones which are not tracked by ticket or incident. He browses such events
to ensure that none of them requires incidents to track the issue. If he feels that one
such event requires an incident to track the issue, he creates an incident directly for
this event.
3.3.11.1 Creating an Incident Manually
If an event of interest occurs that is not covered by any rule and you want to convert
that event to an incident, perform the following:
1.
Using an available view, find the event of interest.
2.
Select the event in the table.
3.
From the More... drop-down menu, choose Create Incident...
4.
Enter the incident details and click OK.
Using Incident Management
3-51
Advanced Topics
5.
Should you decide to work on the incident, set yourself as owner of the incident
and update status to Work in Progress.
Example Scenario
As per the operations policy, the DBA manager has setup rules to create incidents for
all critical issues for his databases. The remainder of the issues are triaged at the event
level by one of the DBAs.
One of the DBA receives e-mail for an "SQL Response" event (not associated with an
incident) on the production database. He accesses the details of the event by clicking
on the link in the e-mail. He reviews the details of the event. This is an issue that needs
to be tracked and resolved, so he opens an incident to track the resolution of the issue.
He marks the status of the incident as "Work in progress".
3.4 Advanced Topics
The following sections discuss incident/event management features relating advanced
applications or operational areas.
3.4.1 Automatic Diagnostic Repository (ADR): Incident Flood Control
ADR is a file-based repository that stores database diagnostic data such as traces,
dumps, the alert log, and health monitor reports. ADR's unified directory structure
and a unified set of tools enable customers and Oracle Support to correlate and
analyze diagnostic data across multiple instances and Oracle products.
Like Enterprise Manager, ADR creates and tracks incidents and problems to allow you
to resolve issues.
в– в– A problem is a critical error in the database. Critical errors manifest as internal
errors, such as ORA-00600, or other severe errors, such as ORA-07445 (operating
system exception) or ORA-04031 (out of memory in the shared pool).
An incident is a single occurrence of a problem. When a problem (critical error)
occurs multiple times, an incident is created for each occurrence. Incidents are
timestamped and tracked in ADR. When an incident occurs, ADR sends a
diagnostic incident alert to Enterprise Manager.
3.4.1.1 Working with ADR Diagnostic Incidents Using Incident Manager
Each diagnostic incident recorded in the ADR is also recorded as an incident in
Enterprise Manager, thus providing you with a unified view of ADR/Enterprise
Manager incidents and problems from within Incident Manager. For the ADR
diagnostic incidents, you can access Enterprise Manager Support Workbench to take
further action, such as packaging a problem or raising a service request with Oracle
Support.
3.4.1.2 Incident Flood Control
Prior to Enterprise Manager Release 12.1.0.4, there was no limit to the number of
diagnostic incidents recorded for a single problem in Incident Manager. It is
conceivable that a problem could generate dozens or perhaps hundreds of incidents in
a short period of time. While incidents generated during the early stages of a problem
may be useful, after a certain point the excess diagnostic data would provide little
value and possibly slow down your efforts to diagnose and resolve the problem.
Because diagnostic problems typically tend to be long-lived, a significant number of
3-52 OracleВ® Enterprise Manager Administration
Advanced Topics
incidents could be generated over time. Also, depending on the size of your monitored
environment, the diagnostic data may consume considerable system resources.
For these reasons, the Enterprise Manager applies flood control limits on the number
of diagnostic incidents that can be raised for a given problem in Incident Manager.
Flood-controlled incidents provide a way of informing you that a critical error is
ongoing, without overloading the system with diagnostic data.
Beginning with Enterprise Manager Release 12.1.0.4, two limits are placed on the
number of diagnostic incidents that can be raised for a given problem in Incident
Manager. A problem is identified by a unique problem signature called a problem key
and is associated with a single target.
Enterprise Manager Limits on Diagnostic Incidents
Enterprise Manager enforces two limits for diagnostic incidents:
в– в– For any given hour, Enterprise Manager only records up to five (default value)
diagnostic incidents for a given target and problem key combination.
On any given day, Enterprise Manager only records up to 25 (default value)
diagnostic incidents for a given problem key and target combination.
When either of these limits is reached, any diagnostic incidents for the same
target/problem key combination will not be recorded until the corresponding hour or
day is over. Diagnostic incident recording will commence once a new hour or day
begins.
Note:
Hour and day calculations are based on UTC (or GMT).
These diagnostic incident limits only apply to Incident Manager and not to the
underlying ADR. All incidents continue to be recorded in the ADR repository. Using
Enterprise Manager Support Workbench, users can view all the incidents for a given
problem at any time and take appropriate actions.
Enterprise Manager diagnostic incident limits are configurable. As mentioned earlier,
the defaults for these two limits are set to 5 incidents per hour and 25 incidents per
day. These defaults should not be changed unless there is a clear business reason to
track all diagnostic incidents.
Changing Enterprise Manager Diagnostic Incident Limits
To update the diagnostic limits, execute the following SQL against the Enterprise
Manager repository as the SYSMAN user using the appropriate limit values as shown
in the following example.
Example 3–3 SQL Used to Change Diagnostic Incident Limits
exec
EM_EVENT_UTIL.SET_ADR_INC_LIMITS(5,25);
The PL/SQL shown in the following example prints out the current limits.
Example 3–4 SQL Used to Print Out Current Diagnostic Incident Limits
DECLARE
l_adr_hour_limit NUMBER;
l_adr_day_limit NUMBER;
BEGIN
em_event_util. GET_ADR_INC_LIMITS
(p_hourly_limit => l_adr_hour_limit,
Using Incident Management
3-53
Advanced Topics
p_daily_limit => l_adr_day_limit);
dbms_output.put_line(l_adr_hour_limit || '-' || l_adr_day_limit);
END;
The Enterprise Manager incident limits are in addition to
any diagnostic incident limits imposed by underlying applications
such as Oracle database, Middleware and Fusion Applications. These
limits are specific to each application. See the respective application
documentation for more information.
Important:
3.4.2 Defining Custom Incident Statuses
As discussed in "Working with Incidents" on page 3-40, one of the primary incident
workflow attributes is status. For most conditions, these predefined status attributes
will suffice. However, the uniqueness of your monitoring and management
environment may require an incident workflow requiring specialized incident states.
To address this need, you can define custom states using the create_resolution_state EM
CLI verb.
3.4.2.1 Creating a New Resolution State
emcli create_resolution_state
-label="Label for display"
-position="Display position"
[-applies_to="INC|PBLM"]
This verb creates a new resolution state for describing the state of incidents or
problems.
This command can only be executed by Enterprise
Manager Super Administrators.
Important:
The new state is always added between the New and Closed states. You must specify
the exact position of this state in the overall list of states by using the -position
option. The position can be between 2 and 98.
By default, the new state is applicable to both incidents and problems. The
-applies_to option can be used to indicate that the state is applicable only to
incidents or problems.
A success message is reported if the command is successful. An error message is
reported if the change fails.
Examples
The following example adds a resolution state that applies to both incidents and
problems at position 25.
emcli create_resolution_state
-position=25
-label="Waiting for Ticket"
The following example adds a resolution state that applies to problems only at
position 35.
emcli create_resolution_state
-position=35 -applies_to=PBLM
3-54 OracleВ® Enterprise Manager Administration
-label="Waiting for SR"
Advanced Topics
3.4.2.2 Modifying an Existing Resolution State
You can chance the both the display label and the position of an existing state by using
the modify_resolution_state verb.
emcli modify_resolution_state
-label="old label of the state to be changed"
-new_label="New label for display"
-position="New display position"
[-applies_to=BOTH]
This verb modifies an existing resolution state that describes the state of incidents or
problems. As with the create_resolution_state verb, this command can only be
executed by Super Administrators.
You can optionally indicate that the state should apply to both incidents and problems
using the -applies_to option.
Examples
The following example updates the resolution state with old label "Waiting for TT"
with a new label "Waiting for Ticket" and if necessary, changes the position to 25.
emcli modify_resolution_state -label="Waiting for TT" -new_
label="Waiting for Ticket" -position=25
The following example updates the resolution state with the old label "SR Waiting"
with a new label "Waiting for SR" and if necessary, changes the position to 35. It also
makes the state applicable to incidents and problems.
emcli modify_resolution_state -label="SR Waiting" -new_
label="Waiting for SR" -position=35 -applies_to=BOTH
3.4.3 Clearing Stateless Alerts for Metric Alert Event Types
For metric alert event types, an event (metric alert) is raised based on the metric
threshold values. These metric alert events are called stateful alerts. For those metric
alert events that are not tied to the state of a monitored system (for example, snapshot
too old, or resumable session suspended ), these alerts are called stateless alerts. Because
stateless alerts are not cleared automatically, they need to be cleared manually. You can
perform a bulk purge of stateless alerts using the clear_stateless_alerts EM CLI verb.
For large numbers of incidents, you can manually clear
incidents in bulk. See "Responding to and Managing Multiple
Incidents, Events and Problems in Bulk".
Note:
clear_stateless_alerts clears the stateless alerts associated with the specified target. The
clearing must be manually performed as the Management Agent does not
automatically clear stateless alerts. To find the metric internal name associated with a
stateless alert, use the EM CLI get_metrics_for_stateless_alerts verb.
Format
emcli clear_stateless_alerts -older_than=number_in_days -target_type=target_type
-target_name=target_name [-include_members][-metric_internal_name=target_type_
metric:metric_name:metric_column] [-unacknowledged_only][-ignore_notifications]
[-preview]
[ ] indicates that the parameter is optional
Using Incident Management
3-55
Advanced Topics
Options
в– older_than
Specify the age of the alert in days. (Specify 0 for currently open stateless alerts.)
в– target_type
Internal target type identifier, such as host, oracle_database, and emrep.
в– target_name
Name of the target.
в– include_members
Applicable for composite targets to examine alerts belonging to members as well.
в– metric_internal_name
Metric to be cleaned up. Use the get_metrics_for_stateless_alerts verb to see a
complete list of supported metrics for a given target type.
в– unacknowledged_only
Only clear alerts if they are not acknowledged.
в– ignore_notifications
Use this option if you do not want to send notifications for the cleared alerts. This
may reduce the notification sub-system load.
в– ignore_notifications
Use this option if you do not want to send notifications for the cleared alerts. This
may reduce the notification sub-system load.
в– preview
Shows the number of alerts to be cleared on the target(s).
Example
The following example clears alerts generated from the database alert log over a week
old. In this example, no notifications are sent when the alerts are cleared.
emcli clear_stateless_alerts -older_than=7 -target_type=oracle_database -tar
name=database -metric_internal_name=oracle_database:alertLog:genericErrStack
-ignore_notifications
get_
3.4.4 Automatically Clearing "Manually Clearable" Events
There are those events that clear automatically, such as CPU Utilization and those
events that must be manually cleared, either through the Incident Manager UI or
automatically via rule (such as Job Failure, or Log Metric events). Auto-clear events, as
the term implies, are cleared automatically by Enterprise Manager once the underlying
issue is resolved. In the case of CPU Utilization, the event CPU Utilization clears
automatically once the percent utilization falls below the warning threshold. However,
for those events that must be cleared manually, a user must intervene and clear the
event using Incident Manager either by selecting the incident/event and clicking
Clear, or creating an event rule to do the job (recommended method).
3-56 OracleВ® Enterprise Manager Administration
Advanced Topics
As mentioned previously, an event rule automates the clearing of manually clearable
events. Enterprise Manager provides a limited number of out-of-box rules that
automatically clear manually clearable events, such as job failures or ADP events that
remain open for seven days. However, to more accurately meet the needs of your
monitoring environment, Oracle recommends creating your own event rules to
automatically clear those manually clearable events that are most prevalent in your
environment.
During the rule creation process, you can specify that an event be automatically
cleared by selecting the Clear Event option while you are adding conditional actions.
Getting Notified when the Event Clears
The event clearing action is an asynchronous operation, which means that when the
rule action (clear) is initiated, the manually clearable event will be enqueued for
clearing, but not actually cleared. Hence, an email notification sent upon rule
execution will indicate that the event has not been cleared. Asynchronous clearing is
by design as it reduces overall rule engine processing load and processing time.
Subscribing to this event clearing rule with the intent to be notified when the event
clears will be of little value. If you want to be notified when the event clears, you must
create a new event rule and explicitly specify a Clear severity. In doing so, you will be
notified once the event is actually cleared.
3.4.5 User-reported Events
Users may create (publish) events manually using the EM CLI verb publsh_event. A
User-reported event is published as an event of the "User-reported event" class. Only
users with Manage Target privilege can publish these events for a target. An error
message is reported if the publish fails.
Using Incident Management
3-57
Advanced Topics
After an event is published with a severity other than CLEAR (see below), end-users
with appropriate privileges can manually clear the event from the UI, or they can
publish a new event using a severity level of CLEAR and the same details to report
clearing of the underlying situation.
3.4.5.1 Format
emcli publish_event
-target_name="Target name"
-target_type="Target type internal name"
-message="Message for the event"
-severity="Severity level"
-name="event name"
[-key="sub component name"
-context="name1=value1;name2=value2;.."
-separator=context="alt. pair separator"
-subseparator=context="alt. name-value separator"]
[ ] indicates that the parameter is optional
3.4.5.2 Options
в– target_name
Target name.
в– target_type
Target type name.
в– message
Message to associate for the event. The message cannot exceed 4000 characters.
в– severity
Numeric severity level to associate for the event. The supported values for severity
level are as follows:
"CLEAR"
"MINOR_WARNING"
"WARNING"
"CRITICAL"
"FATAL"
в– name
Name of the event to publish. The event name cannot exceed 128 characters.
This is indicative of the nature of the event. Examples include "Disk Used
Percentage," "Process Down," "Number of Queues," and so on. The name must be
repeated and identical when reporting different severities for the same sequence of
events. This should not have any identifying information about a specific event;
for example, "Process xyz is down." To identify any specific components within a
target that the event is about, see the key option below.
в– key
Name of the sub-component within a target this event is related to. Examples
include a disk name on a host, name of a tablespace, and so forth. The key cannot
exceed 256 characters.
в– context
3-58 OracleВ® Enterprise Manager Administration
Advanced Topics
Additional context that can be published for a given event. This is a series of
strings of format name:value separated by a semi-colon. For example, it might be
useful to report the percentage size of a disk when reporting space issues on the
disk. You can override the default separator ":" by using the sub-separator option,
and the pair separator ";" by using the separator option.
The context names cannot exceed 256 characters, and the values cannot exceed
4000 characters.
в– separator
Set to override the default ";" separator. You typically use this option when the
name or the value contains ";". Using "=" is not supported for this option.
в– subseparator
Set to override the default ":" separator between the name-value pairs. You
typically use this option when the name or value contains ":". Using "=" is not
supported for this option.
3.4.5.3 Examples
Example 1
The following example publishes a warning event for "my acme target" indicating that
a HDD restore failed, and the failure related to a component called the "Finance DB
machine" on this target.
emcli publish_event -target_name="my acme target" -target_type="oracle_acme"
-name="HDD restore failed" -key="Finance DB machine" -message="HDD restoration
failed due to corrupt disk" -severity=WARNING
Example 2
The following example publishes a minor warning event for "my acme target"
indicating that a HDD restore failed, and the failure related to a component called the
"Finance DB machine" on this target. It specifies additional context indicating the
related disk size and name using the default separators. Note the escaping of the \ in
the disk name using an additional "\".
emcli publish_event -target_name="my acme target" -target_type="oracle_acme"
-name="HDD restore failed" -key="Finance DB machine" -message="HDD restoration
failed due to corrupt disk" -severity=MINOR_WARNING -context="disk
size":800GB\;"disk name":\\uddo0111245
Example 3
The following example publishes a critical event for "my acme target" indicating that a
HDD restore failed, and the failure related to a component called the "Finance DB
machine" on this target. It specifies additional context indicating the related disk size
and name. It uses alternate separators, because the name of the disk includes the ":"
default separator.
emcli publish_event -target_name="my acme target" -target_type="oracle_acme"
-name="HDD restore failed" -key="Finance DB machine" -message="HDD restoration
failed due to corrupt disk" -severity=CRITICAL -context="disk size"^800GB\;"disk
name"^\\sdd1245:2 -subseparator=context=^
Using Incident Management
3-59
Advanced Topics
3.4.6 Additional Rule Applications
Rules can be set up to perform more complicated tasks beyond straightforward
notifications. The following tasks illustrate additional rule capabilities.
в– Setting Up a Rule to Send Different Notifications for Different Severity States of an
Event
в– Creating a Rule to Notify Different Administrators Based on the Event Type
в– Creating a Rule to Create a Ticket for Incidents
в– Creating a Rule to Send SNMP Traps to Third Party Systems
3.4.6.1 Setting Up a Rule to Send Different Notifications for Different Severity
States of an Event
Before you perform this task, ensure the DBA has set appropriate thresholds for the
metric so that a critical metric alert is generated as expected.
Consider the following example:
The Administration Manager sets up a rule to page the specific DBA when a critical
metric alert event occurs for a database in a production database group and to e-mail
the DBA when a warning metric alert event occurs for the same targets. This task
occurs when a new group of databases is deployed and DBAs request to create
appropriate rules to manage such databases.
Perform the following tasks to set appropriate thresholds:
1.
From the Setup menu, select Incidents, then select Incident Rules.
2.
On the Incident Rules - All Enterprise Rules page, highlight a rule set and click
Edit.... (Rules are created in the context of a rule set. If there is no existing rule set
to manage the newly added target, create a rule set.)
3.
In the Edit Rule Set page, locate the Rules section. Click Create...
4.
From the Select Type of Rule to Create dialog, choose Incoming events and
updates to events. Click Continue.
5.
Provide the rule details as follows:
a.
For Type, select Metric Alerts as the Type.
b.
In the criteria section, select Severity. From the drop-down list, check and
Critical and Warning as the selected values. Click Next.
c.
On the Add Actions page, click +Add.
In the Create Incident section, check the Create Incident option. Click
Continue. The Add Action page displays with the new rule. Click Next.
d.
Specify a name for the rule and a description. Click Next.
e.
On the Review page, ensure your settings are correct and click Continue. A
message appears informing you that the rule has been successfully created.
Click OK to dismiss the message.
Next, you need to create a rule to perform the notification actions.
6.
From the Rules section on the Edit Rules page, click Create.
7.
Select Newly created incidents or updates to incidents as the rule type and click
Continue.
8.
Check Specific Incidents.
3-60 OracleВ® Enterprise Manager Administration
Advanced Topics
9.
Check Severity and from the drop-down option selector, check Critical and
Warning. Click Next.
10. On the Add Actions page, click Add. The Conditional Actions page displays.
11. In the Conditions for actions section, choose Only execute the actions if specified
conditions match.
12. From the Incident matches the following criteria list, choose Severity and then
Critical from the drop-down option selector.
13. In the Notifications section, enter the DBA in the Page field. Click Continue. The
Add Actions page displays.
14. Click Add to create a new action for the Warning severity.
15. In the Conditions for actions section, choose Only execute the actions if specified
conditions match.
16. From the Incident matches the following criteria list, choose Severity and then
Warning from the drop-down option selector.
17. In the Notifications section, enter the DBA in the E-mail to field. Click Continue.
The Add Actions page displays with the two conditional actions. Click Next.
18. Specify a rule name and description. Click Next.
19. On the Review page, ensure your rules have been defined correctly and click
Continue. The Edit Rule Set page displays.
20. Click Save to save your newly defined rules.
3.4.6.2 Creating a Rule to Notify Different Administrators Based on the Event Type
As per operations policy for production databases, the incidents that relate to
application issues should go to the application DBAs and the incidents that relate to
system parameters should go to the system DBAs. Accordingly, the respective
incidents will be assigned to the appropriate DBAs and they should be notified by way
of e-mail.
Before you set up rules, ensure the following prerequisites are met:
в– в– в– DBA has setup appropriate thresholds for the metric so that critical metric alert is
generated as expected.
Rule has been setup to create incident for all such events.
Respective notification setup is complete, for example, global SMTP gateway,
e-mail address, and schedule for individual DBAs.
Perform the following steps:
1.
Navigate to the Incident Rules page.
From the Setup menu, select Incidents, then select Incident Rules.
2.
Search the list of enterprise rules matching the events from the production
database.
3.
On the Incident Rules - All Enterprise Rules page, highlight a rule set and click
Edit....
Rules are created in the context of a rule set. If there is no existing rule set, create a
rule set.
4.
From the Edit Rule Set page (Rules tab), select the rule which creates the incidents
for the metric alert events for the database. Click Edit
Using Incident Management
3-61
Advanced Topics
5.
From the Select Events page, click Next.
6.
From the Add Actions page, click +Add. The Add Conditional Actions page
displays.
7.
In the Notifications area, enter the e-mail address of the DBA you want to be
notified for this specific event type and click Continue to add the action.
Enterprise Manager returns you to the Add Actions page.
8.
Click Next.
9.
On the Specify Name and Description page, enter an intuitive rule name and a
brief description.
10. Click Next.
11. On the Review page, review the Applies to, Actions and General information for
correctness .
12. Click Continue to create the rule.
13. Create/Edit additional rules to handle alternate additional administrator
notifications according to event type.
14. Review the rules summary and make corrections as needed. Click Save to save
your rule set changes.
3.4.6.3 Creating a Rule to Create a Ticket for Incidents
If your IT process requires a helpdesk ticket be created to resolve incidents, then you
can use the helpdesk connector to associate the incident with a helpdesk ticket and
have Enterprise Manager automatically open a ticket when the incident is created.
Communication between Incident Manager and your helpdesk system is bidirectional,
thus allowing you to check the changing status of the ticket from within Incident
Manager. Enterprise Manager also allows you to link out to a Web-based third-part
console directly from the ticket so that you can launch the console in context directly
from the ticket.
For example, according to the operations policy of an organization, all critical
incidents from a production database should be tracked by way of Remedy tickets. A
rule is set up to create a Remedy ticket when a critical incident occurs for the database.
When such an incident occurs, the ticket is generated by the rule, the incident is
associated with the ticket, and the operation is logged for future reference to the
updates of the incident. While viewing the details of the incident, the DBA can view
the ticket ID and, using the attached URL link, access the Remedy to get the details
about the ticket.
Before you perform this task, ensure the following prerequisites are met:
в– Monitoring support has been set up.
в– Remedy ticketing connector has been configured.
Perform the following steps:
1.
From the Setup menu, select Incidents, then select Incident Rules.
2.
On the Incident Rules - All Enterprise Rules page, select the appropriate rule set
and click Edit.... (Rules are created in the context of a rule set. If there is no
applicable rule set , create a new rule set.)
3.
Select the appropriate rule that covers the incident conditions for which tickets
should be generated and click Edit...
4.
Click Next to proceed to the Add Actions page.
3-62 OracleВ® Enterprise Manager Administration
Advanced Topics
5.
Click +Add to access the Add Conditional Actions page.
a.
Specify that a ticket should be generated for incidents covered by the rule.
b.
Specify the ticket template to be used.
6.
Click Continue to return to the Add actions page.
7.
On the Add Actions page, click Next.
8.
On the Review page, click Continue.
9.
On the Specify Name and Description page, click Next.
10. On the Review page, click Continue. A message displays indicating that the rule
has been successfully modified. Click OK to close the message.
11. Repeat steps 3 through 10 until all appropriate rules have been edited.
12. Click Save to save your changes to the rule set.
3.4.6.4 Creating a Rule to Send SNMP Traps to Third Party Systems
As mentioned in Chapter 4, "Using Notifications," Enterprise Manager supports
integration with third-party management tools through the SNMP. Sending SNMP
traps to third party systems is a two-step process:
Step 1: Create an advanced notification method based on an SNMP trap.
Step 2: Create an incident rule that invokes the SNMP trap notification method.
The following procedure assumes you have already created the SNMP trap
notification method. For instruction on creating a notification method based on an
SNMP trap, see "Sending SNMP Traps to Third Party Systems" on page 4-38.
1.
From the Setup menu, select Incidents, then select Incident Rules.
2.
On the Incident Rules - All Enterprise Rules page, click Create Rule Set...
3.
Enter the rule set Name, a brief Description, and select the type of source object
the rule Applies to (Targets).
4.
Click on the Rules tab and then click Create...
5.
On the Select Type of Rule to Create dialog, select Incoming events and updates
to events and then click Continue.
6.
On the Create New Rule : Select Events page, specify the criteria for the events for
which you want to send SNMP traps and then click Next.
You must create one rule per event type. For example, if you
want to send SNMP traps for Target Availability events and Metric
Alert events, you must specify two rules.
Note:
7.
On the Create New Rule : Add Actions page, click Add. The Add Conditional
Actions page displays.
8.
In the Notifications section, under Advanced Notifications, select an existing
SNMP trap notification method as shown in the following graphic.
Using Incident Management
3-63
Advanced Topics
For information on creating SNMP trap notification methods, see "Sending SNMP
Traps to Third Party Systems" on page 4-38.
9.
Click Continue to return to the Create New Rule : Add Actions page.
10. Click Next to go to the Create New Rule : Specify Name and Description page.
11. Specify a rule name and a concise description and then click Next.
12. Review the rule definition and then click Continue add the rule to the rule set. A
message displays indicating the rule has been added to the rule set but has not yet
been saved. Click OK to close the message.
13. Click Save to save the rule set. A confirmation is displayed. Click OK to close the
message.
3.4.7 Event Prioritization
When working in a large enterprise, it is conceivable that when systems are under
heavy load, a large number of incidents and events may be generated. All of these
need to be processed in a timely and efficient manner in accordance with your
business priorities. An effective prioritization scheme is needed to determine which
events/incidents should be resolved first.
In order to determine which event/incidents are high priority, Enterprise Manager
uses a prioritization protocol based on two incident/event attributes: Lifecycle Status
of the target and the Incident/Event Type. Lifecycle Status is a target property that
specifies a target’s operational status. You can set/view a target’s Lifecycle Status from
the UI (from a target’s Target Setup menu, select Properties). You can set target
Lifecycle Status properties across multiple targets simultaneously by using the
Enterprise Manager Command Line Interface (EM CLI) set_target_property_value
verb.
3-64 OracleВ® Enterprise Manager Administration
Advanced Topics
A target’s Lifecycle Status is set when it is added to Enterprise Manager for
monitoring. At that time, you determine where in the prioritization hierarchy that
target belongs—the highest level being "mission critical" and the lowest being
"development."
Target Lifecycle Status
в– Mission Critical (highest priority)
в– Production
в– Stage
в– Test
в– Development (lowest priority)
Incident/Event Type
в– Availability events (highest priority)
в– Non-informational events.
в– Informational events
3.4.8 Root Cause Analysis (RCA) and Target Down Events
Root Cause Analysis (RCA) tries to identify the root causes of issues that cause
operational events. Beginning with Enterprise Manager Could Control 12.1.0.3,
Incident Manager automatically performs RCA over target down events, thus actively
identifying whether the target down event is the cause or symptom of other target
down events.
The term target down event specifically pertains to Target Availability events that are
raised when the targets are detected to be down.
3.4.8.1 How RCA Works
RCA is an ongoing process that identifies whether a target down event is root cause or
symptom. It uses the Causal Analysis Update attribute of the event to store the results
of its analysis, i.e. identifying whether or not the target down event is root cause or
symptom. Whenever a new target availability event comes in, RCA is automatically
performed on the incoming event and existing target down events that are related to it.
Based on the analysis, it updates the Causal Analysis Update attribute value if the
incoming event is a target down event. It also updates the Causal Analysis Update
attribute for the related target down events if there is a change.
Two types of target relationships are used for identifying the related targets:
dependency and containment.
When one target depends on another target for its availability, dependency
relationship exists between them. For example, J2EE application target depends on the
WebLogic Server target over which it is deployed.
When a target has a containment relationship over its member targets from
operational point of view, i.e., the member target does not exist outside of the
container, a containment relationship exists between the container target and each
member. For example, Real Application Cluster (RAC) database target has
containment relationship with each of its RAC database instance targets.
The causal analysis update attribute is used only for target down events (such as a
Target Availability event for target down) and can have be assigned any one of the
following values by the RCA process:
Using Incident Management
3-65
Advanced Topics
в– в– в– в– в– Symptom -- this means the target down event has been caused by another target
down event. In the case of containment relationships, target down events of
targets that are part of a container target are set as symptoms to de-emphasize the
events in favor of the target down event at the target container level.
Root Cause - this means the target down event has caused another target down
event and it is not the symptom of any other target down event.
Composite Down - this applies to a container target that is down and it is not a
symptom of any other container target down event. It is marked as composite
down to emphasize its importance over the target down events of targets that are
part of the container target.
N/A - root cause analysis is not applicable to this event .
Not a cause and not a symptom - the target down event is not root cause and not a
symptom of other target down events This is shown in Incident Manager as a "-".
The following rules describe the RCA process:
в– Rule 1: Down event on a non-container target (a target that does not have
members) is marked as the cause if a dependent target is down and it is not
symptom of other target down events.
Examples:
■–
You have J2EE applications deployed on a standalone WebLogic Server. If both
J2EE application and WebLogic Server targets are down, the WebLogic Server
down event is the cause for the J2EE applications deployed on it.
–
You have a J2EE application deployed on couple of WebLogic Servers, which
are part of a WebLogic Cluster. If one WebLogic Server is down along with its
J2EE application, then the WebLogic Server down event is the cause of the
J2EE application target down. This assumes the WebLogic Cluster is not
down.
Rule 2: Down event on a non-container target (a target that does not have
members) is marked as a symptom if a target it depends on is down or if the target
containing it is down.
Examples:
–
You have a J2EE application deployed on a standalone WebLogic Server. If
both J2EE application and WebLogic Server targets are down, J2EE application
down event is the symptom of WebLogic Server being down.
–
You have a couple of WebLogic Servers which are part of a WebLogic Cluster.
Each WebLogic Server has a J2EE application deployed on it. If the WebLogic
Cluster is down, this means both WebLogic Servers are down. This in turn
means the J2EE applications that are deployed on them are also down. The
WebLogic Server down events would be marked as the symptoms of
WebLogic Cluster being down. Logically, WebLogic Servers down events are
both a cause (for the J2EE application down events) and a symptom (of the
WebLogic Cluster down event). However, the WebLogic Server down is
marked as symptom instead of root cause because root cause is meant to
represent the main issue that administrators should focus on. In this scenario,
the main focus is on resolving the WebLogic Cluster down event (i.e. restoring
the WebLogic Cluster). The WebLogic Cluster down event is marked as
composite down. See Rule 3 for details.
–
You have a couple of RAC database instance targets that are part of cluster
database target. If the cluster database is down, then means all RAC instances
3-66 OracleВ® Enterprise Manager Administration
Advanced Topics
are also down. The RAC instance down events would be marked as the
symptoms of cluster database being down. The RAC instance down is
marked as a symptom because the focus should be on resolving the cluster
database down event. The cluster database down event is marked as
composite down. See Rule 3 for details.
в– Rule 3: Down event on a container target is marked as composite down if all
member targets are down and anytarget containing it is not down.
Examples:
■–
You have a couple of WebLogic Servers, which are part of a standalone
WebLogic Cluster. A WebLogic Cluster down event would be marked as
composite down, if both the WebLogic Servers are down. The focus should be
on the WebLogic Cluster down event (restoring the WebLogic Cluster) so that
is why it is marked as composite down.
–
You have a couple of RAC database instance targets that are part of a cluster
database target. The cluster database target down event would be marked as
composite down, if both database instances are down. The focus should be on
the cluster database down event (i.e. restoring the cluster database) so that is
why it is marked as composite down.
Rule 4: Down event on a container target is marked as symptom if the target
containing it is down.
Example:
You have a couple of WebLogic Clusters that are part of a WebLogic Domain
target. If the WebLogic domain is down, this means the WebLogic Clusters are also
down. The WebLogic Cluster target down events would be the symptoms of
WebLogic Domain being down. The WebLogic Domain down event would be
marked as composite down because that should be the focus,
3.4.8.2 Leveraging RCA Results in Incident Rule Sets
As described above, RCA is an ongoing process which results in marking target down
events as cause, symptom, composite down or neither as new target down events come in
and are processed. So a target down event may be marked as a cause or symptom or
composite down as it comes in or after some time when RCA has analyzed additional
event information.
Most datacenters automatically create incidents for target down events since these are
important events that need to be resolved right away. This is recommended best
practice and also implemented by the out-of-the-box rule sets. However, in terms of
notifying response teams or creating trouble tickets, it is not desirable to do so for
symptom incidents. Some datacenters may also choose to not create incidents for
symptom events.
So the RCA results can be leveraged to do the following:
1.
Notify or create tickets only for non-symptom events:
This can be achieved in 2 ways:
в– Create two separate event rules , one event rule to create incidents for all
relevant events, but take no further action (no notification or ticket creation)
and another one to create incidents for non-symptom events only and also
send notifications and create tickets. See "Creating Incidents On
Non-symptom Events" on page 3-71 for instructions.
Using Incident Management
3-67
Advanced Topics
в– 2.
Create an event rule that creates incidents for all target down events. Create
another rule to update the incident priority, send notifications and create
tickets only for incidents stemming from non-symptom events. Once the
incident priority is set to say "Urgent", customer can also create additional
incident rules to take additional actions on the Urgent priority incidents. See
"Creating a Rule to Update Incident Priority for Non-symptom Events" on
page 3-71.
Only create incidents after a suitable wait for events that are not initially marked
as neither a cause nor a symptom:
As mentioned previously, RCA is an iterative process whereby incoming target
down events are continually being evaluated, resulting in updates to causal
analysis state of existing events. Over a period of time (minutes), a target down
event that was initially marked as a root cause may or may not remain a root cause
depending on other incoming target down events. The original target down event
may later be classified as a symptom.
To avoid prematurely creating an incident and opening a ticket for an event which
may later turn out to be a symptom event, you can set up your rules as follows:
в– In addition to the rules already defined in the previous step, create an
additional event rule to act upon RCA updates to events and when the RCA
update indicates that the event is marked as a symptom, lower the priority of
the incident to "Low". This will also send an update to the ticket automatically.
This is recommended. See "Introducing a Time Delay" on page 3-73 for
instructions.
OR
в– 3.
To allow time for target down events to be reported, analyzed, and then acted
upon (such as creating an incident or updating an incident), you can add a
delay in the rule actions. This is useful when customer have some tolerance to
take action after some minimum delay (typically 5 minutes).
Only create incidents for non-symptom events.
Some datacenters may choose not to create any incidents for symptom events. This
can be achieved by changing the rules to only create incidents for events marked
as cause or neither a cause nor symptom. See "Creating Incidents On
Non-symptom Events" on page 3-71 for instructions.
Please note that, even in this approach, it is possible that an event that was
originally marked as cause or neither a cause nor symptom, may be marked as a
symptom when more information is received. Customers can use an approach
similar to that of the second option in step 2 to build some delay in creating the
incidents.
Even with this, it is still feasible but a bit unlikely, that newer information shows
up after the pre-set delay and ends up marking the event as symptom. So it is
recommended to use the approach of setting incident priority and using that as a
way to manage workflow.
3.4.8.3 Leveraging RCA Results in Incident Manager
You can use the RCA results to focus on the non-symptom incidents in Incident
Manager. This involves using the Causal Analysis Update incident attribute when
creating custom views.
1.
From the Enterprise menu, select Monitoring, and then Incidents. The Indicent
Manager page displays.
3-68 OracleВ® Enterprise Manager Administration
Advanced Topics
2.
From the Views region, click Create. The Search page displays.
3.
Click Add Fields... and then choose Causal analysis update. The Causal analysis
update displays as additional search criteria.
4.
Choose 'Do Not Show Symptoms' from the list of available criteria. This will
automatically exclude incidents that have been marked as 'symptom'. Incidents
that are not marked as symptom or root cause will be included as long as it
matches any other criteria you may have specified.
5.
Click Create View, enter a View Name when prompted, and then click OK.
Showing RCA Results in an Incident Detail
An incident that is a root cause or symptom will be identified prominently as part of
the details of the incident in Incident Manager. In addition, in case the incident is a
symptom, a Causes section will be added to identify the root cause(s) of the incident.
In case the incident has, in turn, caused other target down incidents, an Impacted
Targets section will also be added to show the targets that have been affected, that is.
other targets that are down as a result of the original target down. The following figure
shows the incident detail.
Using Incident Management
3-69
Advanced Topics
Figure 3–8 Incident that is a Root Cause
3.4.8.4 Leveraging RCA Results in the System Dashboard
In the System Dashboard, you can use the RCA results to exclude symptom incidents
from the Incidents table so administrators can focus their attention on incidents that
are root cause or have not been caused by other target down events.
To exclude Symptom Incidents:
1.
In the System Dashboard, click on the View option that is accessible from the
upper left hand corner of the Incidents and Problems table.
2.
Choose the option to 'Exclude symptoms'. Alternatively, you can also choose the
option 'Cause only' show only shows target down incidents that have been
identified as cause of other target down incidents. Regardless of the option
chosen, incidents that have not been marked as symptom or root cause will
continue to be displayed.
3-70 OracleВ® Enterprise Manager Administration
Advanced Topics
3.4.8.5 Creating a Rule to Update Incident Priority for Non-symptom Events
1.
Create an event rule to select only non-symptom events.
2.
When adding an action, select the priority to be set for incidents associated with
the non-symptom events selected above.
3.4.8.6 Creating Incidents On Non-symptom Events
You can leverage Incident Manager’s RCA capability creating rule sets that generate
incidents. For monitoring situations where a high number of symptom target down
Using Incident Management
3-71
Advanced Topics
events are generated, but only a few non-symptom target down events, you can create
rule sets that generate incidents and send notifications only for non-symptom events.
To create a rule set that creates incidents for non-symptom target down events:
1.
From the Setup menu, select Incidents, then select Incident Rules.
2.
Click Create Rule Set... You then create the rule as part of creating the rule set.
3.
Select the rule set that will contain the new rule. Click Edit... in the Rules tab of the
Edit Rule Set page, and then:
1.
Click Create ...
2.
Select "Incoming events and updates to events."
3.
Click Continue. The Create New Rule : Select Events dialog displays.
4.
In the Advanced Selection Options region, choose Causal analysis update. Three
causal event options display.
event is marked as cause or composite down: A target down is considered a
cause if other targets depending on it are down or if its a composite.
event is marked as a symptom: A target down is considered a symptom if a
target it depends on is also down, or if the target is part of a composite which
is down.
event is not a cause and not a symptom: A target down is neither a cause or
symptom.
3-72 OracleВ® Enterprise Manager Administration
Advanced Topics
By selecting one or more options, you can filter out extraneous target down
events and focus on those target availability events that pertain to targets with
interdependencies. To create an incident for only non-symptom events, choose
event is marked as cause or composite down and event is not a cause and not a
symptom.
Click Next.
5.
On the Create New Rule : Add Actions page, click Add. The Add Conditional
Actions page displays.
6.
In the Create Incident or Update Incident region, choose Create Incident.
7.
Specify the remaining assignment and notification details and click Continue.
8.
Complete the remaining Create Rule Set wizard pages. See "Creating a Rule
Set" on page 3-33 for more information on creating rule sets.
3.4.8.7 Introducing a Time Delay
As mentioned previously, Incident Manager RCA is an iterative process whereby
incoming target down events are continually being evaluated, resulting in updates to
causal analysis states. Over a period of time (minutes), a root cause may or may not
remain a root cause depending on incoming target down events. The original target
down event may later be classified as a symptom. To allow time for target down
events to be reported, analyzed, and then acted upon (such as creating an incident),
you can define an event evaluation time delay when creating a rule set.
In the previous example, where incidents are created for non-symptom events,
without a time delay in the rule, there could potentially be an incident created for a
non-symptom event that eventually becomes a symptom.
To add a time delay to the rule:
1.
From the Create Rule Set wizard Add Actions page, click Add or Edit (modify an
existing rule). The Add Conditional Actions page displays.
2.
In the Conditions for Actions region, choose Only execute the actions if specified
conditions match. A list of conditions displays.
3.
Choose Event has been open for specified duration.
4.
Specify the desired time delay.
Using Incident Management
3-73
Moving from Enterprise Manager 10/11g to 12c
5.
Click Continue and complete the remaining steps in the wizard.
3.5 Moving from Enterprise Manager 10/11g to 12c
Enterprise Manager 12c incident management functionality leverages your existing
pre-12c monitoring setup out-of-box. Migration is seamless and transparent. For
example, if your Enterprise Manager 10/11g monitoring system sends you e-mails
based on specific monitoring conditions, you will continue to receive those e-mails
without interruption. To take advantage of 12c features, however, you may need to
perform additional migration tasks.
Alerts that were generated pre-12c will still be available.
For example, critical metric alerts will be available as critical incidents.
Important:
Rules
When you migrate to Enterprise Manger 12c, all of your existing notification rules are
automatically converted to rules. Technically, they are converted to event rules first
with incidents automatically being created for each event rule.
In general, event rules allow you to define which events should become incidents.
However, they also allow you to take advantage of the Enterprise Manager’s increased
monitoring flexibility.
For more information on rule migration, see the following documents:
в– в– Appendix A, " Overview of Notification in Enterprise Manager Cloud Control"
section "Migrating Notification Rules to Rule Sets" in the Enterprise Manager Cloud
Control Upgrade Guide.
Chapter 29 "Updating Rules" in the Enterprise Manager Cloud Control Upgrade
Guide.
Privilege Requirements
The Create Enterprise Rule Set resource privilege is now required in order to edit/create
enterprise rule sets and rules contained within. The exception to this is migrated
notification rules. When pre-12c notification rules are migrated to event rules, the
original notification rule owners will still be able to edit their own rules without
having been granted the Create Enterprise Rule Set resource privilege. However, they
must be granted the Create Enterprise Rule Set resource privilege if they wish to create
new rules. Enterprise Manager Super Administrators, by default, can edit and create
rule sets.
3-74 OracleВ® Enterprise Manager Administration
Monitoring: Common Tasks
Monitoring: Common Tasks
The following sections provide "how-to" examples illustrating common tasks for
incident/monitoring setup and usage.
в– "Setting Up an Email Gateway" on page 3-76
в– "Sending Email for Metric Alerts" on page 3-78
в– "Sending SNMP Traps for Metric Alerts" on page 3-82
в– "Sending Events to an Event Connector" on page 3-87
в– "Sending Email to Different Email Addresses for Different Periods of the Day" on
page 3-92
Using Incident Management
3-75
Setting Up an Email Gateway
Setting Up an Email Gateway
Task
In order for Enterprise Manager to send email notifications to administrators, it must
access an available email gateway within your organization. The instructions below
step you through the process of configuring Enterprise Manager to use a designated
email gateway.
User Roles
в– Enterprise Manager Administrator
Prerequisites
в– User must have Super Administrator privileges.
For more information, see "Setting Up a Mail Server for Notifications" on page 4-2.
How to do it:
From the Setup menu, select Notifications, then select Notification Methods.
1.
The Notification Methods page displays.
2.
Enter the requisite parameters. The following examples illustrate valid parameter
values.
3-76 OracleВ® Enterprise Manager Administration
Monitoring: Common Tasks
в– Outgoing Mail (SMTP) Server - smtp01.example.com:587,
smtp02.example.com
в– User Name - myadmin
в– Password - ******
в– Confirm Password - ******
в– Identify Sender As - Enterprise Manager
в– Sender’s E-mail Address - [email protected]
в– Use Secure Connection - No: Email is not encrypted. SSL: E-mail is encrypted
using the Secure Sockets Layer protocol. TLS, if available: E-mail is encrypted
using the Transport Layer Security protocol if the mail server supports TLS. If
the server does not support TLS, the e-mail is automatically sent as plain text.
3.
Ensure Enterprise Manager can connect to the specified email gateway. Click Test
Mail Servers. Enterprise Manager displays a success/failure message. Click OK to
return to the Notification Methods page.
4.
Once Enterprise Manager verifies that it can successfully connect to your email
gateway, click Apply.
What you have accomplished:
At this point, you have configured Enterprise Manager to use your corporate email
gateway. Enterprise Manager can now notify registered users while monitoring
conditions within your managed environment.
What’s next?
"Defining E-mail Addresses" on page 4-5
"Setting Up a Notification Schedule" on page 4-6
"Setting Up E-mail for Yourself" on page 4-5
"Setting Up E-mail for Other Administrators" on page 4-9
Using Incident Management
3-77
Sending Email for Metric Alerts
Sending Email for Metric Alerts
Task
Configure Enterprise Manager to send email to administrators when a metric alert
threshold is reached. In this example, you want to send an email notification when a
metric alert is raised when CPU Utilization reaches Critical severity.
User Roles
в– IT Operator/Manager
в– Enterprise Manager Administrator
Prerequisites
Set up an Email Gateway that allows Enterprise Manager to send email to
administrators.
в– For more information, see "Setting Up a Mail Server for Notifications" on page 4-2.
в– в– Metric thresholds have been set for CPU Utilization.
User’s Enterprise Manager account has been granted the appropriate privileges to
manage incidents from his managed system.
For information, see "Setting Up Administrators and Privileges" on page 3-26.
■User’s Enterprise Manager account has notification preferences (e-mail and
schedule). This is required not just for the administrator who is creating/editing a
rule, but also for any user who is being notified as a result of the rule action.
For more information, see "Setting Up a Notification Schedule" on page 4-6.
How to do it:
1. From the Setup menu, select Incidents, then select Incident Rules.
2.
Click Create Rule Set.
3.
Enter a name and description for the rule set.
3-78 OracleВ® Enterprise Manager Administration
Monitoring: Common Tasks
4.
In the Targets tab, select All targets that the rule set owner can view.
Alternative:
Having the rule set apply to specific targets/group.
Although we have chosen to have the rule set apply to all targets in
this example, alternatively, you can have a rule set apply only to
specific targets or groups.
To do this:
1.
From the Targets tab, select Specific targets.
2.
From the Add drop-down menu, choose Groups or Targets
3.
Click Add. The Target selector dialog displays.
4.
Either search for a target/group name or select one from the table.
5.
Click Select once you have chosen the targets/groups of interest. The
dialog closes and the targets appear in the Specific Targets list.
5.
In the Rules tab, click Create. The Select Type of Rule to Create dialog appears.
6.
Select Incoming events and updates to events, and click Continue.
7.
On the Select Events page, set the criteria for events based on which the rule
should act. In this case, choose Metric Alert from the drop down list.
Using Incident Management
3-79
Sending Email for Metric Alerts
Click Next.
8.
Select the Specific events of type Metric Alert option. A metric selection area
displays:
In this example, we only want to send notifications for CPU % Utilization greater
reaches the defined Critical threshold.
9.
Choose Severity Critical from the drop down menu.
Click OK.
10. Click Next.
11. On the Add Actions page, click Add and add actions to be taken by the rule. In the
Notifications section, enter the e-mail addresses where the notifications must be
send. Click Next.
Multiple conditional actions can be specified and evaluated sequentially (top
down) in the order you add them.
Alternative:
Sending email notifications to mailing list.
In addition to specifying email addresses, you may also specify
defined Enterprise Manager administrators. Mailing distribution lists
can also be specified to notify entire categories of users. Using mailing
lists allows you to change who gets notified without having to update
individual rule sets.
12. On the Specify Name and Description page, enter a name and description for the
rule. Click Next.
13. On the review page, review the details, and click Continue.
3-80 OracleВ® Enterprise Manager Administration
Monitoring: Common Tasks
14. On the Create Rule Set page, click Save.
What you have accomplished:
At this point, you have created a new rule set that will send an administrator email a
notification whenever the CPU Utilization reaches the Critical metric threshold. To
subscribe to this rule set, see "Subscribing to Receive Email from a Rule" on page 3-39
for further instructions.
What’s Next?
в– How Do I Set Up Email Notifications for Other Administrators
в– Add/Update/Delete Email Addresses and Define a Notification Schedule
в– "Responding and Working on a Simple Incident" on page 3-46
Using Incident Management
3-81
Sending SNMP Traps for Metric Alerts
Sending SNMP Traps for Metric Alerts
Task
You want to configure Enterprise Manager to send event information (for example, a
metric alert) via SNMP trap to an HP Openview console. This is done in two phases
1.
Create a notification method to send the SNMP Trap
2.
Create an incident rule to send an SNMP trap when a metric alert is raised.
User Roles
в– Enterprise Manager Administrator
Prerequisites
в– User must have Super Administrator privileges.
For more information, see "Setting Up a Mail Server for Notifications" on page 4-2.
How to do it:
Create a notification method based on an SNMP Trap.
1.
From the Setup menu, select Notifications, then select Notification Methods.
The Notification Methods page displays.
3-82 OracleВ® Enterprise Manager Administration
Monitoring: Common Tasks
2.
From the Add drop-down menu, choose SNMP Trap and then click Go. The Add
SNMP Trap page displays.
You must provide the name of the host (machine) on which the SNMP master
agent is running and other details as shown in the following graphic.
The following examples illustrate valid parameter values.
3.
Click Test SNMP Trap to validate the SNMP trap settings. Enterprise Manager
displays a success/failure message. Click OK to return to the Add SNMP Trap
page.
4.
Click OK to return to the Notification Methods page.
5.
Click OK to add the new SNMP Trap-based notification method.
Create an incident rule to send an SNMP trap when a metric alert is raised.
1.
From the Setup menu, select Incidents, then select Incident Rules.
The Incident Rules - All Enterprise Rules page displays.
Using Incident Management
3-83
Sending SNMP Traps for Metric Alerts
2.
On the Incident Rules - All Enterprise Rules page, click Create Rule Set... The
Create Rule Set page displays.
3.
Enter the rule set Name, a brief Description, and select the type of source object
the rule Applies to (Targets).
4.
Click on the Rules tab and then click Create...
5.
On the Select Type of Rule to Create dialog, select Incoming events and updates
to events and then click Continue.
6.
On the Select Events page, set the criteria for events based on which the rule
should act. In this case, choose Metric Alert from the drop down list.
3-84 OracleВ® Enterprise Manager Administration
Monitoring: Common Tasks
Click Next.
7.
Select the Specific events of type Metric Alert option. A metric selection area
displays:
In this example, we only want to send notifications for CPU % Utilization greater
reaches the defined Critical threshold.
8.
Choose Severity Critical from the drop down menu.
Click OK.
9.
Click Next.
10. On the Create New Rule : Add Actions page, click Add. The Add Conditional
Actions page displays.
11. In the Notifications section, under Advanced Notifications, select an existing
SNMP trap notification method as shown in the following graphic.
Using Incident Management
3-85
Sending SNMP Traps for Metric Alerts
For information on creating SNMP trap notification methods, see "Sending SNMP
Traps to Third Party Systems" on page 4-38.
12. Click Continue to return to the Create New Rule : Add Actions page.
13. Click Next to go to the Create New Rule : Specify Name and Description page.
14. Specify a rule name and a concise description and then click Next.
15. Review the rule definition and then click Continue add the rule to the rule set. A
message displays indicating the rule has been added to the rule set but has not yet
been saved. Click OK to close the message.
16. Click Save to save the rule set. A confirmation is displayed. Click OK to close the
message.
What you have accomplished:
At this point, you have created an incident rule set that instructs Enterprise Manager
to send an SNMP trap to a third-party system whenever a metric alert is raised (%CPU
Utilization).
What’s next?
"Subscribing to Receive Email from a Rule" on page 3-39
в– в– "Searching for Incidents" on page 3-44
3-86 OracleВ® Enterprise Manager Administration
Monitoring: Common Tasks
Sending Events to an Event Connector
Task
You want to send event information from Enterprise Manager to IBM Tivoli
Netcool/OMNIbus using a connector. To do so, you must create an incident rule that
invokes the IBM Tivoli Netcool/OMNIbus Connector connector.
User Roles
в– System Administrator
в– IT Operator
Prerequisites
User must have user must have the Create Enterprise Rule Set resource privilege
and at least View privileges on the targets where events are to be forward to
Netcool/OMNIbus.
в– For more information, see "Setting Up a Mail Server for Notifications" on page 4-2.
в– The IBM Tivoli Netcool/OMNIbus connector must be installed and configured.
For more information, see the OracleВ® Enterprise Manager IBM Tivoli
Netcool/OMNIbus Connector Installation and Configuration Guide.
How to do it:
1. From the Setup menu, select Incidents, then select Incident Rules.
The Incident Rules - All Enterprise Rules page displays.
Using Incident Management
3-87
Sending Events to an Event Connector
2.
Click Create Rule Set.
3.
Enter a name and description for the rule set.
4.
In the Targets tab, select All targets that the rule set owner can view.
Having the rule set apply to specific targets/groups: Although we
have chosen to have the rule set apply to all targets in this example,
you can alternatively have a rule set apply only to specific targets or
groups.
To do this:
1.
From the Targets tab, select Specific targets.
2.
From the Add drop-down menu, choose Groups or Targets
3.
Click Add. The Target selector dialog displays.
4.
Either search for a target/group name or select one from the table.
5.
Click Select once you have chosen the targets/groups of interest. The
dialog closes and the targets appear in the Specific Targets list.
5.
In the Rules tab, click Create. The Select Type of Rule to Create dialog appears.
6.
Select Incoming events and updates to events, and click Continue.
7.
On the Select Events page, set the criteria for events based on which the rule
should act. In this case, choose Metric Alert from the drop down list.
3-88 OracleВ® Enterprise Manager Administration
Monitoring: Common Tasks
Click Next.
8.
Select the Specific events of type Metric Alert option. A metric selection area
displays:
In this example, we only want to send notifications for CPU % Utilization greater
reaches the defined Critical threshold.
9.
Choose Severity Critical from the drop down menu.
Click OK.
10. Click Next. The Add Actions page displays.
Using Incident Management
3-89
Sending Events to an Event Connector
11. Click Add. The Add Conditional Actions page displays.
12. Select one or more connector instances listed in the Forward to Event Connectors
section and, click > button to add the connector to the Selected Connectors list
and then click Continue.
13. The Add Actions page appears again and lists the new action.
14. Click Next. The Specify Name and Description page displays.
15. Enter a name and description for the rule, then click Next. The Review page
displays.
16. Click Continue if everything appears correct.
An information pop-up appears that states, "Rule has been successfully added to
the current rule set. Newly added rules are not saved until the Save button is
clicked."
You can click Back and make corrections to the rule if necessary.
3-90 OracleВ® Enterprise Manager Administration
Monitoring: Common Tasks
What you have accomplished:
At this point, you have created a rule that invokes the IBM Tivoli Netcool/OMNIbus
Connector connector when a metric alert is raised.
What’s next?
"Subscribing to Receive Email from a Rule" on page 3-39
Using Incident Management
3-91
Sending Email to Different Email Addresses for Different Periods of the Day
Sending Email to Different Email Addresses for Different Periods of the Day
Task
Your worldwide IT department operates 24/7. Support responsibility rotates to
different data centers across the globe depending on the time of day. When Enterprise
Manager sends an email notification, you want it sent to the administrator currently
on duty (normal work day), which in this situation changes depending on the time of
day.
There are four adminstrators to handle Enterprise Manager notification:
в– ADMIN_ASIA
в– ADMIN_EU
в– ADMIN_UK
в– ADMIN_US
You want the following notifications to be sent to specific administrators during their
normal work hours.
User Roles
в– System Administrator
в– IT Operator
Prerequisites
Email addresses have been defined for all administrators you want to send email
nofifications.
в– For more information, see "Defining E-mail Addresses" on page 4-5.
в– You must have Super Administrator privileges.
в– All administrators who are to receive email notifications have been defined.
For more information, see
How to do it:
From the Setup menu, select Notifications, then select Notification Schedule.
1.
From the vertical navigation bar, click Schedules (under Notification). The
Notification Schedule page appears.
2.
Specify the administrator who’s notification schedule you wish to edit and click
Change.
3.
Click Edit Schedule Definition. The Edit Schedule Definition: Time Period page
appears. If necessary, modify the rotation schedule.
4.
Click Continue. The Edit Schedule Definition: E-mail Addresses page appears.
5.
Follow the directions on the Edit Schedule Definition: E-mail Addresses page to
modify the notification schedule.
6.
Click Finish when you are done.
7.
Repeat steps three through seven for each administrator until
3-92 OracleВ® Enterprise Manager Administration
Monitoring: Common Tasks
What you have accomplished:
At this point, you have created a rule that invokes the IBM Tivoli Netcool/OMNIbus
Connector connector when a metric alert is raised.
What’s next?
"Subscribing to Receive Email from a Rule" on page 3-39
Using Incident Management
3-93
Disabling Out-of-Box Rulesets
Disabling Out-of-Box Rulesets
3-94 OracleВ® Enterprise Manager Administration
4
Using Notifications
4
The notification system allows you to notify Enterprise Manager administrators when
specific incidents, events, or problems arise.
This chapter assumes that you are familiar with incident
management. For information about monitoring and managing your
IT infrastructure via incident management, see Chapter 3, "Using
Incident Management".
Note:
As an integral part of the management framework, notifications can also perform
actions such as executing operating system commands (including scripts) and PL/SQL
procedures when specific incidents, events, or problems occur. This capability allows
you to automate IT practices. For example, if an incident (such as monitoring of the
operational (up/down) status of a database) arises, you may want the notification
system to automatically open an in-house trouble-ticket using an OS script so that the
appropriate IT staff can respond in a timely manner.
By using Simple Network Management Protocol (SNMP) traps, the Enterprise
Manager notification system also allows you to send traps to SNMP-enabled
third-party applications such as HP OpenView for events published in Enterprise
Manager. Some administrators may want to send third-party applications a
notification when a certain metric has exceeded a threshold.
This chapter covers the following:
в– Setting Up Notifications
в– Extending Notification Beyond E-mail
в– Sending Notifications Using OS Commands and Scripts
в– Sending Notifications Using PL/SQL Procedures
в– Sending SNMP Traps to Third Party Systems
в– Management Information Base (MIB)
в– Passing Corrective Action Status Change Information
в– Passing Job Execution Status Information
в– Passing User-Defined Target Properties to Notification Methods
в– Troubleshooting Notifications
в– EMOMS Properties
в– Passing Event, Incident, Problem Information to an OS Command or Script
Using Notifications 4-1
Setting Up Notifications
в– Passing Information to a PL/SQL Procedure
4.1 Setting Up Notifications
All Enterprise Manager administrators can set up e-mail notifications for themselves.
Super Administrators also have the ability to set up notifications for other Enterprise
Manager administrators.
4.1.1 Setting Up a Mail Server for Notifications
Before Enterprise Manager can send e-mail notifications, you must first specify the
Outgoing Mail (SMTP) servers to be used by the notification system. Once set, you can
then define e-mail notifications for yourself or, if you have Super Administrator
privileges, you can also define notifications for other Enterprise Manager
administrators.
You specify the Outgoing Mail (SMTP) server on the Notification Methods page
(Figure 4–1). To display the Notification Methods page, from the Setup menu, select
Notifications, then select Notification Methods.
You must have Super Administrator privileges in order to
configure the Enterprise Manager notifications system. This includes:
Note:
в– Setting up the SMTP server
в– Defining notification methods
в– Customizing notification email formats
Specify one or more outgoing mail server names, the mail server authentication
credentials (User Name, Password, and Confirm Password), if required, the name you
want to appear as the sender of the notification messages, and the e-mail address you
want to use to send your e-mail notifications. This address, called the Sender’s Mail
Address, must be a valid address on each mail server that you specify. A message will
be sent to this e-mail address if any problem is encountered during the sending of an
e-mail notification. Example 4–1 shows sample notification method entries.
Example 4–1 Mail Server Settings
в– Outgoing Mail (SMTP) Server - smtp01.example.com:587, smtp02.example.com
в– User Name - myadmin
в– Password - ******
в– Confirm Password - ******
в– Identify Sender As - Enterprise Manager
в– Sender’s E-mail Address - [email protected]
в– Use Secure Connection - No: E-mail is not encrypted. SSL: E-mail is encrypted
using the Secure Sockets Layer protocol. TLS, if available: E-mail is encrypted using
the Transport Layer Security protocol if the mail server supports TLS. If the server
does not support TLS, the e-mail is automatically sent as plain text.
4-2 OracleВ® Enterprise Manager Administration
Setting Up Notifications
Figure 4–1 Defining a Mail Server
The e-mail address you specify on this page is not the
e-mail address to which the notification is sent. You will have to
specify the e-mail address (where notifications will be sent) from
the Password and E-mail page. From the Setup menu, choose
MyPreferences and then Enterprise Manager Password & E-mail.
Note:
As standard practice, each user should have their own e-mail
address.
After configuring the e-mail server, click Test Mail Servers to verify your e-mail setup.
You should verify that an e-mail message was received by the e-mail account specified
in the Sender’s E-mail Address field.
Defining multiple mail servers will improve the reliability of e-mail notification
delivery. E-mail notifications will be delivered if at least one e-mail server is up. The
notification load is balanced across multiple e-mail servers by the OMS, which
switches through them (servers are allocated according to availability) after 20 e-mails
have been sent. Switching is controlled by the oracle.sysman.core.notification.emails_per_
connection emoms property.
Using Notifications 4-3
Setting Up Notifications
Setting the Cloud Control Console URL when Using an SLB
If you have a multi-OMS environment with a Server Load Balancer (SLB) configured
for the OMS instances, you should update the console URL to ensure that any emails
from Enterprise Manager direct you to the Enterprise Manager console through the
SLB URL and not the specific OMS URL from which the email may have originated.
To change the console URL:
1.
From the Setup menu, select Manage Cloud Control, and then Health Overview.
The Management Services and Repository page displays.
2.
On the Management Services and Repository page, in the Overview section, click
Add/Edit against the Console URL label.
The Console URL page displays.
3.
Modify the Console URL to the SLB URL.
Examples:
http://www.example.com
https://www.example.com:4443.
Note that path, typically /em, should not be specified.
4.
Click OK.
4-4 OracleВ® Enterprise Manager Administration
Setting Up Notifications
4.1.2 Setting Up E-mail for Yourself
If you want to receive notifications by e-mail, you will need to specify your e-mail
address(s) in the Password & E-mail page (from the Setup menu, select
MyPreferences, then select Enterprise Manager Password & E-mail). In addition to
defining notification e-mail addresses, you associate the notification message format
(long, short, pager) to be used for your e-mail address.
Setting up e-mail involves three steps:
Step 1: Define an e-mail addresses.
Step 2: Set up a Notification Schedule.
Step 3: Subscribe to incident rules in order to receive e-mails.
4.1.2.1 Defining E-mail Addresses
An e-mail address can have up to 128 characters. There is no upper limit with the
number of e-mail addresses.
To add an e-mail address:
1.
From username drop-down menu, select Enterprise Manager Password & E-mail.
2.
Click Add Another Row to create a new e-mail entry field in the E-mail
Addresses table.
3.
Specify the e-mail associated with your Enterprise Manager account. All e-mail
notifications you receive from Enterprise Manager will be sent to the e-mail
addresses you specify.
For example, [email protected]
Select the E-mail Type (message format) for your e-mail address. E-mail (Long)
sends a HTML formatted e-mail that contains detailed information. Example 4–2
shows a typical notification that uses the long format.
E-mail (Short) and Pager(Short) (Example 4–3) send a concise, text e-mail that is
limited to a configurable number of characters, thereby allowing the e-mail be
received as an SMS message or page. The content of the message can be sent
entirely in the subject, entirely in the body or split across the subject and body.
For example, in the last case, the subject could contain the severity type (for
example, Critical) and the target name. The body could contain the time the
severity occurred and the severity message. Since the message length is limited,
some of this information may be truncated. If truncation has occurred there will be
an ellipsis end of the message. Pager(Short) addresses are used for supporting the
paging feature in incident rules. Note that the incident rules allow the rule author
to designate some users to receive a page for critical issues.
4.
Click Apply to save your e-mail address.
Example 4–2 Long E-mail Notification for Metric Alerts
Target type=Host
Target name=machine6140830.example.com
Message=Filesystem / has 54.39% available space, fallen below warning (60) or
critical (30) threshold.
Severity=Warning
Event reported time=Apr 28, 2011 2:33:55 PM PDT
Event Type=Metric Alert
Event name=Filesystems:Filesystem Space Available (%)
Metric Group=Filesystems
Using Notifications 4-5
Setting Up Notifications
Metric=Filesystem Space Available (%)
Metric value=54.39
Key Value=/
Key Column 1=Mount Point
Rule Name=NotifRuleSet1,Event rule1
Rule Owner=SYSMAN
Example 4–3 Short E-mail Notification for Alerts
Subject is :
EM:Unreachable Start:myhost
Body is :
Nov 16, 2006 2:02:19 PM EST:Agent is Unreachable (REASON = Connection refused)
but the host is UP
More about E-mail(Short) and Pager(Short) Formats
Enterprise Manager does not directly support message services such as paging or
SMS, but instead relies on external gateways to, for example, perform the conversion
from e-mail to page. Beginning with Enterprise Manager 12c, the notification system
allows you to tag e-mail addresses explicitly as 'page' or 'e-mail'. Explicit system
differentiation between these two notification methods allows you to take advantage
of the multiple action capability of incident rules. For example, the e-mail versus page
distinction is required in order to send you an e-mail if an event severity is 'warning'
or page you if the severity is 'critical'. To support this capability, a Pager format has
been made available that sends an abbreviated version of the short format e-mail.
To receive a Page, an administrator should be added to the
Page Notification option in the Incident Rule.
Note:
4.1.2.2 Setting Up a Notification Schedule
Once you have defined your e-mail notification addresses, you will need to define a
notification schedule. For example, if your e-mail addresses are [email protected],
[email protected], [email protected], you can choose to use one or more of these
e-mail addresses for each time period in your notification schedule. Only e-mail
addresses that have been specified with your user preferences (Enterprise Manager
Password and Email page) can be used in the notification schedule.
When you enter e-mail addresses for the first time, a 24x7
weekly notification schedule is set automatically. You can then review
and modify the schedule to suit your monitoring needs.
Note:
A notification schedule is a repeating schedule used to specify your on-call
schedule—the days and time periods and e-mail addresses that should be used by
Enterprise Manager to send notifications to you. Each administrator has exactly one
notification schedule. When a notification needs to be sent to an administrator,
Enterprise Manager consults that administrator's notification schedule to determine
the e-mail address to be used. Depending on whether you are Super Administrator or
a regular Enterprise Manager administrator, the process of defining a notification
schedule differs slightly.
4-6 OracleВ® Enterprise Manager Administration
Setting Up Notifications
Figure 4–2 Notification Schedule Page
If you are a regular Enterprise Manager administrator and are defining your own
notification schedule:
1. From Setup menu, select Notifications, then select My Notification Schedule.
2.
Follow the directions on the Notification Schedule page to specify when you want
to receive e-mails.
4.1.2.3 Subscribe to Receive E-mail for Incident Rules
An incident rule is a user-defined rule that specifies the criteria by which notifications
should be sent for specific events that make up the incident. An incident rule set, as
the name implies, consists of one or more rules associated with the same incident.
When creating an incident rule, you specify criteria such as the targets you are
interested in, the types of events to which you want the rule to apply. Specifically, for a
given rule, you can specify the criteria you are interested in and the notification
methods (such as e-mail) that should be used for sending these notifications. For
example, you can set up a rule that when any database goes down or any database
backup job fails, e-mail should be sent and the "log trouble ticket" notification method
should be called. Or you can define another rule such that when the CPU or Memory
Utilization of any host reach critical severities, SNMP traps should be sent to another
management console.
Notification flexibility is further enhanced by the fact that with a single rule, you can
perform multiple actions based on specific conditions. Example: When monitoring a
condition such as machine memory utilization, for an incident severity of 'warning'
(memory utilization at 80%), send the administrator an e-mail, if the severity is
'critical' (memory utilization at 99%), page the administrator immediately.
You can subscribe to a rule you have already created.
1.
From the Setup menu, select Incidents, then select Incident Rules.
2.
On the Incident Rules - All Enterprise Rules page, click on the rule set containing
incident escalation rule in question and click Edit... Rules are created in the context
of a rule set.
Note: In the case where there is no existing rule set, create a rule set by clicking
Create Rule Set... You then create the rule as part of creating the rule set.
Using Notifications 4-7
Setting Up Notifications
3.
In the Rules section of the Edit Rule Set page, highlight the escalation rule and
click Edit....
4.
Navigate to the Add Actions page.
5.
Select the action that escalates the incident and click Edit...
6.
In the Notifications section, add the DBA to the E-mail cc list.
7.
Click Continue and then navigate back to the Edit Rule Set page and click Save.
Out-of-Box Incident Rules
Enterprise Manager comes with two incident rule sets that cover the most common
monitoring conditions, they are:
в– Incident Management Ruleset for All Targets
в– Event Management Ruleset for Self Update
If the conditions defined in the out-of-box incident rules meet your requirements, you
can simply subscribe to receive e-mail notifications for the conditions defined in the
rule using the subscribe procedure shown in the previous section.
The out-of-box incident rule set for all targets does not generate emails for warning
alerts by default.
Creating Your Own Incident Rules
You can define your own custom rules. The following procedure documents the
process of incident rule creation for non-Super Administrators.
To create your own incident rule:
1.
From the Setup menu, select Incidents, then select Incident Rules.
The Incident Rules page displays. From this page you can create a new rule set, to
which you can add new rules. Alternatively, if you have the requisite permissions,
you can add new rules to existing
2.
Click Create Rule Set...
The create rule set page displays.
3.
Specify the Name, Description, and the Targets to which the rules set should
apply.
4.
Click the Rules tab, then click Create.
5.
Choose the incoming incident, event or problem to which you want the rule to
apply. See "Setting Up Rule Sets" for more information.
6.
Click Continue.
Enterprise Manager displays the Create Incident Rule pages. Enter the requisite
information on each page to create your incident rule.
7.
Follow the wizard instructions to create your rule.
Once you have completed defining your rule, the wizard returns you to the create
rule set page.
8.
Click Save to save the incident rule set.
4-8 OracleВ® Enterprise Manager Administration
Setting Up Notifications
4.1.3 Setting Up E-mail for Other Administrators
If you have Super Administrator privileges, you can set up e-mail notifications for
other Enterprise Manager administrators. To set up e-mail notifications for other
Enterprise Manager administrators, you need to:
Step 1: Ensure Each Administrator Account has an Associated E-mail Address
Each administrator to which you want to send e-mail notifications must have a valid
e-mail address.
1.
From the Setup menu, select Security and then Administrators.
2.
For each administrator, define an e-mail address. This sets up a 24x7 notification
schedule for this user that uses all the e-mail addresses specified. By default, this
adds the Email ID with type set to Email Long. It is not possible to specify the Email
Type option here.
Enterprise Manager also allows you to specify an administrator address when editing
an administrator’s notification schedule.
Step 2: Define Administrators’ Notification Schedules
Once you have defined e-mail notification addresses for each administrator, you will
need to define their respective notification schedules. Although a default 24x7
notification schedule is created when you specify an e-mail address for the first time,
you should review and edit the notification schedule as needed.
1.
From the Setup menu, select Notifications, then select Notification Schedule.
From the vertical navigation bar, click Schedules (under Notification). The
Notification Schedule page appears.
2.
Specify the administrator who’s notification schedule you wish to edit and click
Change.
3.
Click Edit Schedule Definition. The Edit Schedule Definition: Time Period page
appears. If necessary, modify the rotation schedule.
4.
Click Continue. The Edit Schedule Definition: E-mail Addresses page appears.
5.
Follow the directions on the Edit Schedule Definition: E-mail Addresses page to
modify the notification schedule.
6.
Click Finish when you are done.
7.
Repeat steps three through seven for each administrator.
Step 3: Assign Incident Rules to Administrators
With the notification schedules set, you now need to assign the appropriate incident
rules for each designated administrator.
1.
From the Setup menu, select Incidents, then select Incident Rules.
2.
Select the desired Ruleset and click Edit.
3.
Click on the Rules tab, select the desired rule and click Edit.
4.
Click Add Actions, select desire action and click Edit.
5.
Enter the Administrator name on either E-mail To or E-mail Cc field in the Basic
Notification region.
6.
Click Continue, click Next, click Next, click Continue, and finally click Save.
Using Notifications 4-9
Setting Up Notifications
4.1.4 E-mail Customization
Enterprise Manager allows Super Administrators to customize global e-mail
notifications for the following types: All events, incidents, problems, and specific event
types installed. You can alter the default behavior for all events by customizing Default
Event Email Template. In addition, you can further customize the behavior for a specific
event type by customizing the template for the event type. For instance, you can
customize the Metric Alert Events template for the metric alert event type. Using
predefined building blocks (called attributes and labels) contained within a simple
script, Super Administrators can customize alert e-mails by selecting from a wide
variety of information content.
To customize an e-mail:
1.
From the Setup menu, select Notifications, then select Customize Email Formats.
2.
Choose the Type and Format.
3.
Click Customize. The Customize Email Template page displays.
From the Customize E-mail Template page, you can modify the content of the e-mail
template Enterprise Manager uses to generate e-mail notifications. Extensive
information on script formatting, syntax, and options is available from the Edit E-mail
Template page via imbedded assistance and online help.
Figure 4–3 E-mail Customization
4-10 OracleВ® Enterprise Manager Administration
Setting Up Notifications
4.1.4.1 E-mail Customization Reference
The following reference summarizes the semantics and component syntax of the
pseudo-language used to define e-mails. The pseudo-language provides you with a
simple, yet flexible way to customize e-mail notifications. The following is a summary
of pseudo-language conventions/limitations:
в– в– в– в– в– в– You can add comments (or any free-form text) using separate lines beginning with
"--" or at end of lines.
You can use attributes.
You can use IF & ELSE & ENDIF control structures. You can also use multiple
conditions using "AND" or "OR". Nested IF statements are not supported.
You can insert spaces for formatting purposes. Spaces at the beginning of a line
will be ignored in the actual e-mail. To insert spaces at the beginning of a line, use
the [SP] attribute.
Use "/" to escape and "[" or "]" if you want to add attribute names, operators, or IF
clauses to the actual e-mail.
HTML is not supported.
Reserved Words and Operators
The following table lists all reserved words and operators used when modifying
e-mail scripts.
Table 4–1
Reserved Words and Operators
Reserved Word/Operator
Description
IF, ELSIF, ENDIF, ELSE
Used in IF-ELSE constructs.
AND, OR
Boolean operators – used in IF-ELSE constructs only.
NULL
To check NULL value for attributes - used in IF-ELSE constructs
only.
|
Pipe operator – used to show the first non-NULL value in a list
of attributes.
For example:
METRIC_NAME|SEVERITY
EQ, NEQ
Equal and Not-Equal operators – applicable to NULL, STRING
and NUMERIC values.
/
Escape character – used to escape reserved words and operators.
Escape characters signify that what follows the escape character
takes an alternative interpretation.
[,]
Delimiters used to demarcate attribute names and IF clauses.
Syntax Elements
Literal Text
You can specify any text as part of the e-mail content. The text will be displayed in the
e-mail and will not be translated if the Oracle Management Services (OMS) language
setting is changed. For example, �my Oracle Home’ appears as �my Oracle Home’ in
the generated e-mail.
Using Notifications 4-11
Setting Up Notifications
Predefined Attributes
Predefined attributes/labels will be substituted with actual values in a specific context.
To specify a predefined attribute/label, use the following syntax:
[PREDEFINED_ATTR]
Attribute names can be in either UPPER or LOWER case. The parsing process is
case-insensitive.
A pair of square brackets is used to distinguish predefined attributes from literal text.
For example, for a job e-mail notification, the actual job name will be substituted for
[EXECUTION_STATUS]. For a metric alert notification, the actual metric column name
will be substituted for [METIRC_COLUMN].
You can use the escape character “/” to specify words and not have them interpreted
as predefined labels/attributes. For example, "/[NEW/]” will not be considered as the
predefined attribute [NEW] when parsed.
Operators
EQ, NEQ – for text and numeric values
NULL- for text and numeric values
GT, LT, GE, LE – for numeric values
Control Structures
The following table lists acceptable script control structures.
Table 4–2
Control Structures
Control Structure
Description
Pipe "|"
Two or more attributes can be separated by �|’ character. For
example,
[METRIC_NAME|SEVERITY]
In this example, only the applicable attribute within the current
alert context will be used (replaced by the actual value) in the
e-mail. If more than one attribute is applicable, only the left-most
attribute is used.
4-12 OracleВ® Enterprise Manager Administration
Setting Up Notifications
Table 4–2 (Cont.) Control Structures
Control Structure
Description
IF
Allows you to make a block of text conditional. Only one level of
IF and ELSIF is supported. Nested IF constructs are not
supported.
All attributes can be used in IF or ELSIF evaluation using
EQ/NEQ operators on NULL values. Other operators are
allowed for “SEVERITY” and “REPEAT_COUNT” only.
Inside the IF block, the values need to be contained within
quotation marks “ ”. Enterprise Manager will extract the
attribute name and its value based on the position of “EQ” and
other key words such as “and”, “or”. For example,
[IF REPEAT_COUNT EQ “1” AND SEVERITY EQ “CRITICAL”
THEN]
The statement above will be true when the attributes of the alert
match the following condition:
в– Attribute Name: REPEAT_COUNT
в– Attribute Value: 1
в– Attribute Name: SEVERITY
в– Attribute Value: CRITICAL
Example IF Block:
[IF EXECUTION_STATUS NEQ NULL]
[JOB_NAME_LABEL]=[EXECUTION_STATUS]
[JOB_OWNER_LABEL]=[JOB_OWNER]
[ENDIF]
[IF SEVERITY_CODE EQ CRITICAL ]
[MTRIC_NAME_LABEL]=[METRIC_GROUP]
[METRIC_VALUE_LABEL]=[METRIC_VALUE]
[TARGET_NAME_LABEL]=[TARGET_NAME]
[KEY_VALUES]
[ENDIF]
Example IF and ELSEIF Block:
[IF SEVERITY_CODE EQ CRITICAL]
statement1
[ELSIF SEVERITY_CODE EQ WARNING]
statement2
[ELSIF SEVERITY_CODE EQ CLEAR]
statement3
[ELSE]
statement4
[ENDIF]
Comments
You can add comments to your script by prefacing a single line of text with two
hyphens "--". For example,
-- Code added on 8/3/2009
[IF REPEAT_COUNT NEQ NULL]
. . .
Using Notifications 4-13
Setting Up Notifications
Comments may also be placed at the end of a line of text.
[IF SEVERITY_SHORT EQ W] -- for Warning alert
HTML Tags in Customization Content
Use of HTML tags is not supported.
When Enterprise Manager parses the e-mail script, it will convert the “<” and “>”
characters of HTML tags into encoded format (&lt; and &gt;). This ensures that the
HTML tag is not treated as HTML by the destination system.
Examples
E-mail customization template scripts support three main operators.
в– Comparison operators: EQ/NEQ/GT/LT/GE/LE
в– Logic operators: AND/OR
в– Pipeline operator: |
4.1.5 Setting Up Repeat Notifications
Repeat notifications allow administrators to be notified repeatedly until an incident is
either acknowledged or the number of Maximum Repeat Notifications has been
reached. Enterprise Manager supports repeat notification for all notification methods
(e-mail, OS command, PL/SQL procedure, and SNMP trap).
Configuring Repeat Notifications Globally
To enable repeat notifications for a notification method (globally), select the Send
Repeat Notifications option on the Notification Methods page . In addition to setting
the maximum number of repeat notifications, you can also set the time interval at
which the notifications are sent.
For Oracle database versions 10 and higher, it is
recommend that no modification be made to aq_tm_processes init.ora
parameter. If, however, this parameter must be modified, its value
should be at least one for repeat notification functionality. If the
Enterprise Manager Repository database version is 9.2, the aq_tm_
processes init.ora parameter must be set to at least one to enable repeat
notification functionality.
Important:
Configuring Repeat Notifications Via Incident Rules
Setting repeat notifications globally at the notification method level may not provide
sufficient flexibility. For example, you may want to have different repeat notification
settings based on event type. Enterprise Manager accomplishes this by allowing you to
set repeat notifications for individual incident rule sets or individual rules within a
rule set. Repeat notifications set at the rule level take precedence over those defined at
the notification method level.
Repeat notifications will only be sent if the Send Repeat
Notifications option is enabled in the Notification Methods page.
Important:
4-14 OracleВ® Enterprise Manager Administration
Extending Notification Beyond E-mail
Non-E-mail Repeat Notifications
For non-e-mail repeat notifications (PL/SQL, OS command, and SNMP trap
notification methods), you must enable each method to support repeat notifications.
You can select Supports Repeat Notifications option when adding a new notification
method or by editing an existing method.
Figure 4–4 Enabling Repeat Notification for an OS Command Notification Method
4.2 Extending Notification Beyond E-mail
Notification Methods are the mechanisms by which notifications are sent. Enterprise
Manager Super Administrators can set up e-mail notifications by configuring the
'e-mail' notification method. Most likely this would already have been set up as part of
the Oracle Management Service installation.
Enterprise Manager Super Administrators can also define other custom notification
methods. For example, event notifications may need to be forwarded to a 3rd party
trouble-ticketing system. Assuming APIs to the third-party trouble-ticketing system
are available, a custom notification method can be created to call a custom OS script
that has the appropriate APIs. The custom notification method can be named in a
user-friendly fashion, for example, "Log trouble ticket". Once the custom method is
defined, whenever an administrator needs to send alerts to the trouble-ticketing
system, he simply needs to invoke the now globally available notification method
called "Log trouble ticket".
Custom notification methods can be defined based on any custom OS script, any
custom PL/SQL procedure, or by sending SNMP traps. A fourth type of notification
method (Java Callback) exists to support Oracle internal functionality and cannot be
created or edited by Enterprise Manager administrators.
Only Super Administrators can define OS Command, PL/SQL, and SNMP Trap
notification methods. However, any Enterprise Manager administrator can add these
notification methods (once defined by the Super Administrator) as actions to their
incident rules.
Using Notifications 4-15
Sending Notifications Using OS Commands and Scripts
Through the Notification Methods page, you can:
в– в– в– Set up the outgoing mail servers if you plan to send e-mail notifications through
incident rules
Create other custom notification methods using OS and PL/SQL scripts and
SNMP traps.
Set global repeat notifications.
4.3 Sending Notifications Using OS Commands and Scripts
Notification system can invoke a custom script when an incident rule matches the OS
Command advanced notification action. A custom script receives notifications for
matching events, incidents and problem through environment variables.
The length of any environment variable's value is limited to 512 characters by default.
Configure emoms property named oracle.sysman.core.notification.oscmd.max_env_var_
length for changing the default limit.
Notification methods based on OS commands must be
configured by an administrator with Super Administrator
privileges.
Important:
Running OS Command Scripts: Running an OS command such as
"sudo" for receiving notifications will fail because the command does
not have read permission of the OMS account. The OMS account must
have read permission over the OS command in order to send
notifications.
To overcome the permissions problem, embed the command in a
wrapper script that is readable by the OMS administrator account.
Once the command is contained within the wrapper script, you then
specify this script in place of the OS command.
Registering a Custom Script
In order to use a custom script, you must first register the script with the notification
system. This is performed in four steps:
1.
Define your OS command or script.
2.
Deploy the script on each Management Service host.
3.
Register your OS Command or Script as a new Notification Method.
4.
Assign the notification method to an incident rule.
Step 1: Define your OS command or script.
You can specify an OS command or script that will be called by the notification system
when an incident rule matches the OS Command advanced notification action. You
can use incident, event, or problem context information, corrective action execution
status and job execution status within the body of the script. Passing this contextual
information to OS commands/scripts allows you to customize automated responses
specific event conditions. For example, if an OS script opens a trouble ticket for an
in-house support trouble ticket system, you will want to pass severity levels (critical,
4-16 OracleВ® Enterprise Manager Administration
Sending Notifications Using OS Commands and Scripts
warning, and so on) to the script to open a trouble ticket with the appropriate details
and escalate the problem. For more information on passing specific types of
information to OS Command or Scripts, see:
в– в– в– "Passing Event, Incident, Problem Information to an OS Command or Script" on
page 4-59
"Passing Corrective Action Execution Status to an OS Command or Script" on
page 4-50
"Passing Job Execution Status to an OS Command or Script" on page 4-54
Step 2: Deploy the script on each Management Service host.
You must deploy the OS Command or Script on each Management Service host
machine that connects to the Management Repository. The OS Command is run as the
user who started the Management Service. The OS Command or Script should be
deployed on the same location on each Management Service host machine.
Important: Both scripts and OS Commands should be specified
using absolute paths. For example, /u1/bin/logSeverity.sh.
The command is run by the user who started the Management Service. If an error is
encountered during the running of the OS Command, the Notification System can be
instructed to retry the sending of the notification to the OS Command by returning an
exit code of 100. The procedure is initially retried after one minute, then two minutes,
then three minutes, eventually progressing to 30 minutes. From here, the procedure is
retried every 30 minutes until the notification is a 24 hours old. The notification will be
then be purged.
Example 4–4 shows the parameter in emoms.properties that controls how long the OS
Command can execute without being killed by the Management Service. This prevents
OS Commands from running for an inordinate length of time and blocks the delivery
of other notifications. By default the command is allowed to run for 30 seconds before
it is killed. The oracle.sysman.core.notification.os_cmd_timeout emoms property can be
configured to change the default timeout value.
Example 4–4 Changing the oracle.sysman.core.notification.os_cmd_timeout emoms
Property
emctl set property -name oracle.sysman.core.notification.os_cmd_timeout value 30
Step 3: Register your OS Command or Script as a new Notification Method.
Add this OS command as a notification method that can be called in incident rules.
Log in as a Super Administrator. From the Setup menu, select Notifications, then
select Notification Methods. From this page, you can define a new notification based
on the ’OS Command’ type. See "Sending Notifications Using OS Commands and
Scripts".
The following information is required for each OS command notification method:
в– Name
в– Description
Both Name and Description should be clear and intuitive so that the function of
the method is clear to other administrators.
Using Notifications 4-17
Sending Notifications Using OS Commands and Scripts
в– OS Command
You must enter the full path of the OS command or script in the OS command field
(for example, /u1/bin/myscript.sh). For environments with multiple
Management Services, the path must be exactly the same on each machine that has a
Management Service. Command line parameters can be included after the full path
(for example, /u1/bin/myscript.sh arg1 arg2).
Example 4–5 shows information required for the notification method.
Example 4–5 OS Command Notification Method
Name Trouble Ticketing
Description Notification method to log trouble ticket for a severity occurrence
OS Command /private/mozart/bin/logTicket.sh
Note:
There can be more than one OS Command configured per
system.
Figure 4–5 Notification Method: Add OS Command
Step 4: Assign the notification method to an incident rule.
You can edit an existing rule (or create a new instance rule), then go to the Methods
page. From the Setup menu, choose Incidents and then Incident Rules. The Incident
Rules page provides access to all available rule sets.
For detailed reference information on passing event, incident, and problem
information to an OS Command or script, see "Passing Event, Incident, Problem
Information to an OS Command or Script" on page 4-59.
4.3.1 Script Examples
The sample OS script shown in Example 4–6 appends environment variable entries to
a log file. In this example, the script logs a severity occurrence to a file server. If the file
server is unreachable then an exit code of 100 is returned to force the Oracle
Management Service Notification System to retry the notification
Example 4–6 Sample OS Command Script
#!/bin/ksh
4-18 OracleВ® Enterprise Manager Administration
Sending Notifications Using OS Commands and Scripts
LOG_FILE=/net/myhost/logs/event.log
if test -f $LOG_FILE
then
echo $TARGET_NAME $MESSAGE $EVENT_REPORTED_TIME >> $LOG_FILE
else
exit 100
fi
Example 4–7 shows an OS script that logs alert information for both incidents and
events to the file 'oscmdNotify.log'. The file is saved to the /net/myhost/logs
directory.
Example 4–7 Alert Logging Scripts
#!/bin/sh
#
LOG_FILE=/net/myhost/logs/oscmdNotify.log
echo '-------------' >> $LOG_FILE
echo
echo
echo
echo
echo
echo
echo
echo
echo
echo
echo
echo
echo
echo
echo
echo
'issue_type=' $ISSUE_TYPE >> $LOG_FILE
'notif_type=' $NOTIF_TYPE >> $LOG_FILE
'message=' $MESSAGE >> $LOG_FILE
'message_url' = $MESSAGE_URL >>$LOG_FILE
'severity=' $SEVERITY >> $LOG_FILE
'severity_code' = $SEVERITY_CODE >>$LOG_FILE
'ruleset_name=' $RULESET_NAME >> $LOG_FILE
'rule_name=' $RULE_NAME >> $LOG_FILE
'rule_owner=' $RULE_OWNER >> $LOG_FILE
'repeat_count=' $REPEAT_COUNT >> $LOG_FILE
'categories_count' = $CATEGORIES_COUNT >>$LOG_FILE
'category_1' = $CATEGORY_1 >>$LOG_FILE
'category_2' = $CATEGORY_2 >>$LOG_FILE
'category_code_1' = $CATEGORY_CODE_1 >>$LOG_FILE
'category_code_2' = $CATEGORY_CODE_2 >>$LOG_FILE
'category_codes_count' = $CATEGORY_CODES_COUNT >>$LOG_FILE
# event
if [ $ISSUE_TYPE -eq 1 ]
then
echo 'host_name=' $HOST_NAME >> $LOG_FILE
echo 'event_type=' $EVENT_TYPE >> $LOG_FILE
echo 'event_name=' $EVENT_NAME >> $LOG_FILE
echo 'event_occurrence_time=' $EVENT_OCCURRENCE_TIME >> $LOG_FILE
echo 'event_reported_time=' $EVENT_REPORTED_TIME >> $LOG_FILE
echo 'sequence_id=' $SEQUENCE_ID >> $LOG_FILE
echo 'event_type_attrs=' $EVENT_TYPE_ATTRS >> $LOG_FILE
echo 'source_obj_name=' $SOURCE_OBJ_NAME >> $LOG_FILE
echo 'source_obj_type=' $SOURCE_OBJ_TYPE >> $LOG_FILE
echo 'source_obj_owner=' $SOURCE_OBJ_OWNER >> $LOG_FILE
echo 'target_name' = $TARGET_NAME >>$LOG_FILE
echo 'target_url' = $TARGET_URL >>$LOG_FILE
echo 'target_owner=' $TARGET_OWNER >> $LOG_FILE
echo 'target_type=' $TARGET_TYPE >> $LOG_FILE
echo 'target_version=' $TARGET_VERSION >> $LOG_FILE
echo 'lifecycle_status=' $TARGET_LIFECYCLE_STATUS >> $LOG_FILE
echo 'assoc_incident_escalation_level' = $ASSOC_INCIDENT_ESCALATION_LEVEL
>>$LOG_FILE
echo 'assoc_incident_id' = $ASSOC_INCIDENT_ID >>$LOG_FILE
echo 'assoc_incident_owner' = $ASSOC_INCIDENT_OWNER >>$LOG_FILE
Using Notifications 4-19
Sending Notifications Using OS Commands and Scripts
echo 'assoc_incident_acknowledged_by_owner' = $ASSOC_INCIDENT_ACKNOWLEDGED_BY_
OWNER >>$LOG_FILE
echo 'assoc_incident_acknowledged_details' = $ASSOC_INCIDENT_ACKNOWLEDGED_
DETAILS >>$LOG_FILE
echo 'assoc_incident_priority' = $ASSOC_INCIDENT_PRIORITY >>$LOG_FILE
echo 'assoc_incident_status' = $ASSOC_INCIDENT_STATUS >>$LOG_FILE
echo 'ca_job_status' = $CA_JOB_STATUS >>$LOG_FILE
echo 'event_context_attrs' = $EVENT_CONTEXT_ATTRS >>$LOG_FILE
echo 'last_updated_time' = $LAST_UPDATED_TIME >>$LOG_FILE
echo 'sequence_id' = $SEQUENCE_ID >>$LOG_FILE
echo 'test_date_attr_noref' = $TEST_DATE_ATTR_NOREF >>$LOG_FILE
echo 'test_raw_attr_noref' = $TEST_RAW_ATTR_NOREF >>$LOG_FILE
echo 'test_str_attr1' = $TEST_STR_ATTR1 >>$LOG_FILE
echo 'test_str_attr2' = $TEST_STR_ATTR2 >>$LOG_FILE
echo 'test_str_attr3' = $TEST_STR_ATTR3 >>$LOG_FILE
echo 'test_str_attr4' = $TEST_STR_ATTR4 >>$LOG_FILE
echo 'test_str_attr5' = $TEST_STR_ATTR5 >>$LOG_FILE
echo 'test_str_attr_ref' = $TEST_STR_ATTR_REF >>$LOG_FILE
echo 'total_occurrence_count' = $TOTAL_OCCURRENCE_COUNT >>$LOG_FILE
fi
# incident
if [ $ISSUE_TYPE -eq 2 ]
then
echo 'action_msg=' $ACTION_MSG >> $LOG_FILE
echo 'incident_id=' $INCIDENT_ID >> $LOG_FILE
echo 'incident_creation_time=' $INCIDENT_CREATION_TIME >> $LOG_FILE
echo 'incident_owner=' $INCIDENT_OWNER >> $LOG_FILE
echo 'incident_acknowledged_by_owner' = $INCIDENT_ACKNOWLEDGED_BY_OWNER >>$LOG_
FILE
echo 'incident_status' = $INCIDENT_STATUS >>$LOG_FILE
echo 'last_modified_by=' $LAST_MODIFIED_BY >> $LOG_FILE
echo 'last_updated_time=' $LAST_UPDATED_TIME >> $LOG_FILE
echo 'assoc_event_count=' $ASSOC_EVENT_COUNT >> $LOG_FILE
echo 'adr_incident_id=' $ADR_INCIDENT_ID >> $LOG_FILE
echo 'occurrence_count=' $OCCURRENCE_COUNT >> $LOG_FILE
echo 'escalated=' $ESCALATED >> $LOG_FILE
echo 'escalated_level=' $ESCALATED_LEVEL >> $LOG_FILE
echo 'priority=' $PRIORITY >> $LOG_FILE
echo 'priority_code' = $PRIORITY_CODE >>$LOG_FILE
echo 'ticket_id=' $TICKET_ID >> $LOG_FILE
echo 'ticket_status=' $TICKET_STATUS >> $LOG_FILE
echo 'ticket_url=' $TICKET_ID_URL >> $LOG_FILE
echo 'total_duplicate_count=' $TOTAL_DUPLICATE_COUNT >> $LOG_FILE
echo 'source_count=' $EVENT_SOURCE_COUNT >> $LOG_FILE
echo 'event_source_1_host_name' = $EVENT_SOURCE_1_HOST_NAME >>$LOG_FILE
echo 'event_source_1_target_guid' = $EVENT_SOURCE_1_TARGET_GUID >>$LOG_FILE
echo 'event_source_1_target_name' = $EVENT_SOURCE_1_TARGET_NAME >>$LOG_FILE
echo 'event_source_1_target_owner' = $EVENT_SOURCE_1_TARGET_OWNER >>$LOG_FILE
echo 'event_source_1_target_type' = $EVENT_SOURCE_1_TARGET_TYPE >>$LOG_FILE
echo 'event_source_1_target_url' = $EVENT_SOURCE_1_TARGET_URL >>$LOG_FILE
echo 'event_source_1_target_lifecycle_status' = $EVENT_SOURCE_1_TARGET_
LIFECYCLE_STATUS >>$LOG_FILE
echo 'event_source_1_target_version' = $EVENT_SOURCE_1_TARGET_VERSION >>$LOG_
FILE
fi
exit 0
Example 4–8 shows a script that sends an alert to an HP OpenView console from
Enterprise Manager Cloud Control. When a metric alert is triggered, the Enterprise
4-20 OracleВ® Enterprise Manager Administration
Sending Notifications Using OS Commands and Scripts
Manager Cloud Control displays the alert. The HP OpenView script is then called,
invoking opcmsg and forwarding the information to the HP OpenView management
server.
Example 4–8 HP OpenView Script
/opt/OV/bin/OpC/opcmsg severity="$SEVERITY" app=OEM msg_grp=Oracle msg_
text="$MESSAGE" object="$TARGET_NAME"
4.3.2 Migrating pre-12c OS Command Scripts
This section describes how to map pre-12c OS Command notification shell
environment variables to 12c OS Command shell environment variables.
Note: Policy Violations are no longer supported beginning with the
Enterprise Manager 12c release.
4.3.2.1 Migrating Metric Alert Event Types
Following table is the mapping for the OS Command shell environment variables
when the event_type is metric_alert.
Table 4–3
Pre-12c/12c metric_alert Environment Variable Mapping
Pre-12c Environment
Variable
Corresponding 12c Environment Variables
ACKNOWLEDGED
ASSOC_INCIDENT_ACKNOWLEDGED_BY_OWNER
ACKNOWLEDGED_BY
ASSOC_INCIDENT_OWNER
CYCLE_GUID
CYCLE_GUID
HOST
HOST_NAME
KEY_VALUE
Note: See detail description below.
KEY_VALUE_NAME
Note: See detail description below
MESSAGE
MESSAGE
METRIC
METRIC_COLUMN
NOTIF_TYPE
NOTIF_TYPE; use the map in section 2.3.5
REPEAT_COUNT
REPEAT_COUNT
RULE_NAME
RULESET_NAME
RULE_OWNER
RULE_OWNER
SEVERITY
SEVERITY
TARGET_NAME
TARGET_NAME
TARGET_TYPE
TARGET_TYPE
TIMESTAMP
EVENT_REPORTED_TIME
METRIC_VALUE
VALUE
VIOLATION_CONTEXT
EVENT_CONTEXT_ATTRS
VIOLATION_GUID
SEVERITY_GUID
POLICY_RULE
No mapping, Obsolete as of Enterprise Manager 12c release.
Using Notifications 4-21
Sending Notifications Using OS Commands and Scripts
To obtain KEY_VALUE_NAME and KEY_VALUE, perform the following steps.
в– в– If $NUM_KEYS variable is null, then $KEY_VALUE_NAME and $KEY_VALUE
are null.
If $NUM_KEYS equals 1
KEY_VALUE_NAME=$KEY_COLUMN_1
KEY_COLUMN_1_VALUE
в– If $NUM_KEYS is greater than 1
KEY_VALUE_NAME="$KEY_COLUMN_1;$KEY_COLUMN_2;..;KEY_
COLUMN_x"
KEY_VALUE="$KEY_COLUMN_1_VALUE;$KEY_COLUMN_2_VALUE;..;KEY_
COLUMN_x_VALUE "
Where x is the value of $NUM_KEYS and ";" is the separator.
4.3.2.2 Migrating Target Availability Event Types
Following table is the mapping for the OS Command shell environment variables
when the event_type is 'target_availability'.
Table 4–4
pre-12c/12c target_availability Environment Variable Mappings
Pre-12c Environment
Variable
Corresponding 12c Environment Variables
TARGET_NAME
TARGET_NAME
TARGET_TYPE
TARGET_TYPE
METRIC
Status
CYCLE_GUID
CYCLE_GUID
VIOLATION_CONTEXT
EVENT_CONTEXT_ATTRS
SEVERITY
TARGET_STATUS
HOST
HOST_NAME
MESSAGE
MESSAGE
NOTIF_TYPE
NOTIF_TYPE; use the map in section 2.3.5
TIMESTAMP
EVENT_REPORTED_TIME
RULE_NAME
RULESET_NAME
RULE_OWNER
RULE_OWNER
REPEAT_COUNT
REPEAT_COUNT
KEY_VALUE
""
KEY_VALUE_NAME
""
4.3.2.3 Migrating Job Status Change Event Types
Following table is the mapping for the OS Command shell environment variables
when the event_type is 'job_status_change'.
4-22 OracleВ® Enterprise Manager Administration
Sending Notifications Using OS Commands and Scripts
Table 4–5
pre-12c/12c job_status_change Environment Variable Mappings
Pre-12c Environment
Variable
Corresponding 12c Environment Variables
JOB_NAME
SOURCE_OBJ_NAME
JOB_OWNER
SOURCE_OBJ_OWNER
JOB_TYPE
SOURCE_OBJ_SUB_TYPE
JOB_STATUS
EXECUTION_STATUS
NUM_TARGETS
1 if $ TARGET_NAME is not null, 0 otherwise
TARGET_NAME1
TARGET_NAME
TARGET_TYPE1
TARGET_TYPE
TIMESTAMP
EVENT_REPORTED_TIME
RULE_NAME
RULESET_NAME
RULE_OWNER
RULE_OWNER
4.3.2.4 Migrating Corrective Action-Related OS Scripts
Refer to section "Migrating Metric Alert Event Types" for mapping the following
environment variables while receiving notifications for corrective actions.
KEY_VALUE
KEY_VALUE_NAME
METRIC
METRIC_VALUE
RULE_NAME
RULE_OWNER
SEVERITY
TIMESTAMP
VIOLATION_CONTEXT
Use the map below for mapping other environment variables.
Table 4–6
pre-12c/12c Corrective Action Environment Variable Mappings
Pre-12c Environment
Variable
Corresponding 12c Environment Variables
NUM_TARGETS
1
TARGET_NAME1
TARGET_NAME
TARGET_TYPE1
TARGET_TYPE
JOB_NAME
CA_JOB_NAME
JOB_OWNER
CA_JOB_OWNER
JOB_STATUS
CA_JOB_STATUS
JOB_TYPE
CA_JOB_TYPE
Using Notifications 4-23
Sending Notifications Using PL/SQL Procedures
4.3.2.5 Notification Type Mapping
Table 4–7
pre-12c/12c notif_type Mappings
notif_type (12c)
notif_type (Pre-12c)
NOTIF_NORMAL
1
NOTIF_REPEAT
4
NOTIF_DURATION
9
NOTIF_RETRY
2
4.4 Sending Notifications Using PL/SQL Procedures
A user-defined PL/SQL procedure can receive notifications for matching events,
incidents and problems.
Note: When upgrading from pre-12c to 12c versions of Enterprise
Manager, existing pre-12c PL/SQL advanced notification methods
will continue to work without modification. You should, however,
update the procedures to use new signatures.
New PL/SQL advanced notification methods created with Enterprise
Manager 12c must use the new signatures documented in the
following sections.
Complete the following four steps to define a notification method based on a PL/SQL
procedure.
4.4.1 Defining a PL/SQL-based Notification Method
Creating a PL/SQL-based notification method consists of four steps:
1.
Define the PL/SQL procedure.
2.
Create the PL/SQL procedure on the Management Repository.
3.
Register your PL/SQL procedure as a new notification method.
4.
Assign the notification method to an incident rule.
Step 1: Define the PL/SQL Procedure
The procedure must have one of the following signatures depending on the type of
notification that will be received.
For Events:
PROCEDURE event_proc(event_msg IN gc$notif_event_msg)
For Incidents:
PROCEDURE incident_proc(incident_msg IN gc$notif_incident_msg)
For Problems:
PROCEDURE problem_proc(problem_msg IN gc$notif_problem_msg)
4-24 OracleВ® Enterprise Manager Administration
Sending Notifications Using PL/SQL Procedures
The notification method based on a PL/SQL procedure
must be configured by an administrator with Super Administrator
privileges before a user can select it while creating/editing a
incident rule.
Note:
For more information on passing specific types of information to scripts or PL/SQL
procedures, see the following sections:
"Passing Information to a PL/SQL Procedure" on page 4-69
"Passing Corrective Action Status Change Information" on page 4-50
"Passing Job Execution Status Information" on page 4-52
Step 2: Create the PL/SQL procedure on the Management Repository.
Create the PL/SQL procedure on the repository database using one of the following
procedure specifications:
PROCEDURE event_proc(event_msg IN gc$notif_event_msg)
PROCEDURE incident_proc(incident_msg IN gc$notif_incident_msg)
PROCEDURE problem_proc(problem_msg IN gc$notif_problem_msg)
The PL/SQL procedure must be created on the repository database using the database
account of the repository owner (such as SYSMAN)
If an error is encountered during the running of the procedure, the Notification System
can be instructed to retry the sending of the notification to the procedure by raising a
user-defined exception that uses the error code -20000. The procedure initially retried
after one minute, then two minutes, then three minutes and so on, until the
notification is a day old, at which point it will be purged.
Step 3: Register your PL/SQL procedure as a new notification method.
Log in as a Super Administrator. From the Setup menu, choose Notifications and then
Notification Methods to access the Notification Methods page. From this page, you
can define a new notification based on ’PL/SQL Procedure’. See Section 4.4, "Sending
Notifications Using PL/SQL Procedures".
Make sure to use a fully qualified name that includes the schema owner, package
name and procedure name. The procedure will be executed by the repository owner
and so the repository owner must have execute permission on the procedure.
Create a notification method based on your PL/SQL procedure. The following
information is required when defining the method:
в– Name
в– Description
в– PL/SQL Procedure
You must enter a fully qualified procedure name (for example,
OWNER.PKGNAME.PROCNAME) and ensure that the owner of the Management
Repository has execute privilege on the procedure.
An example of the required information is shown in Example 4–9.
Example 4–9 PL/SQL Procedure Required Information
Name Open trouble ticket
Using Notifications 4-25
Sending Notifications Using PL/SQL Procedures
Description Notification method to open a trouble ticket in the event
PLSQL Procedure ticket_sys.ticket_ops.open_ticket
Figure 4–6 illustrates how to add a PL/SQL-based notification method from the
Enterprise Manager UI.
Figure 4–6 Adding a PL/SQL Procedure
Step 4: Assign the notification method to an incident rule.
You can edit an existing rule (or create a new incident rule). From the Setup menu,
select Incidents and then select Incident Rules. The Incident Rules page displays.
From here, you can add an action to a rule specifying the new PL/SQL procedure
found under Advanced Notification Method.
There can be more than one PL/SQL-based method configured for your Enterprise
Manager environment.
See "Passing Information to a PL/SQL Procedure" on page 4-69 for more information
about how incident, event, and problem information is passed to the PLSQL
procedure.
Example 4–10
PL/SQL Script
-- Assume log_table is created by following DDL
-- CREATE TABLE log_table (message VARCHAR2(4000)) ;
-- Define PL/SQL notification method for Events
CREATE OR REPLACE PROCEDURE log_table_notif_proc(s IN GC$NOTIF_EVENT_MSG)
IS
l_categories gc$category_string_array;
l_category_codes gc$category_string_array;
l_attrs gc$notif_event_attr_array;
l_ca_obj gc$notif_corrective_action_job;
BEGIN
INSERT INTO log_table VALUES ('notification_type: ' || s.msg_info.notification_
type);
INSERT INTO log_table VALUES ('repeat_count: ' || s.msg_info.repeat_count);
INSERT INTO log_table VALUES ('ruleset_name: ' || s.msg_info.ruleset_name);
INSERT INTO log_table VALUES ('rule_name: ' || s.msg_info.rule_name);
INSERT INTO log_table VALUES ('rule_owner: ' || s.msg_info.rule_owner);
INSERT INTO log_table VALUES ('message: ' || s.msg_info.message);
INSERT INTO log_table VALUES ('message_url: ' || s.msg_info.message_url);
INSERT INTO log_table VALUES ('event_instance_guid: ' || s.event_payload.event_
instance_guid);
4-26 OracleВ® Enterprise Manager Administration
Sending Notifications Using PL/SQL Procedures
INSERT INTO log_table VALUES ('event_type: ' || s.event_payload.event_type);
INSERT INTO log_table VALUES ('event_name: ' || s.event_payload.event_name);
INSERT INTO log_table VALUES ('event_msg: ' || s.event_payload.event_msg);
INSERT INTO log_table VALUES ('source_obj_type: ' || s.event_
payload.source.source_type);
INSERT INTO log_table VALUES ('source_obj_name: ' || s.event_
payload.source.source_name);
INSERT INTO log_table VALUES ('source_obj_url: ' || s.event_
payload.source.source_url);
INSERT INTO log_table VALUES ('target_name: ' || s.event_payload.target.target_
name);
INSERT INTO log_table VALUES ('target_url: ' || s.event_payload.target.target_
url);
INSERT INTO log_table VALUES ('severity: ' || s.event_payload.severity);
INSERT INTO log_table VALUES ('severity_code: ' || s.event_payload.severity_
code);
INSERT INTO log_table VALUES ('event_reported_date: ' || to_char(s.event_
payload.reported_date, 'D MON DD HH24:MI:SS'));
l_categories := s.event_payload.categories;
IF l_categories IS NOT NULL
THEN
FOR c IN 1..l_categories.COUNT
LOOP
INSERT INTO log_table VALUES ('category ' || c || ' - ' || l_categories(c));
END LOOP;
END IF;
l_category_codes := s.event_payload.category_codes;
IF l_categories IS NOT NULL
THEN
FOR c IN 1..l_category_codes.COUNT
LOOP
INSERT INTO log_table VALUES ('category_code ' || c || ' - ' || l_category_
codes(c));
END LOOP;
END IF;
l_attrs := s.event_payload.event_attrs;
IF l_attrs IS NOT NULL
THEN
FOR c IN 1..l_attrs.COUNT
LOOP
INSERT INTO log_table VALUES ('EV.ATTR name=' || l_attrs(c).name || '
value=' || l_attrs(c).value || ' nls_value=' || l_attrs(c).nls_value);
END LOOP;
END IF;
COMMIT ;
END ;
/
Example 4–11
PL/SQL Script to Log Events to a Table
CREATE TABLE event_log (
notification_type
VARCHAR2(32),
repeat_count
NUMBER,
ruleset_name
VARCHAR2(256),
rule_owner
VARCHAR2(256),
Using Notifications 4-27
Sending Notifications Using PL/SQL Procedures
rule_name
message
message_url
event_instance_guid
event_type
event_name
event_msg
categories
source_obj_type
source_obj_name
source_obj_url
severity
severity_code
target_name
target_type
target_url
host_name
timezone
occured
ca_guid
ca_name
ca_owner
ca_type
ca_status
ca_status_code
ca_job_step_output
ca_execution_guid
ca_stage_change_guid
VARCHAR2(256),
VARCHAR2(4000),
VARCHAR2(4000),
RAW(16),
VARCHAR2(20),
VARCHAR2(512),
VARCHAR2(4000),
VARCHAR2(4000),
VARCHAR2(120),
VARCHAR2(256),
VARCHAR2(4000),
VARCHAR2(128),
VARCHAR2(32),
VARCHAR2(256),
VARCHAR2(128),
VARCHAR2(4000),
VARCHAR2(256),
VARCHAR2(64),
DATE,
RAW(16),
VARCHAR2(128),
VARCHAR2(256),
VARCHAR2(256),
VARCHAR2(64),
NUMBER,
VARCHAR2(4000),
RAW(16),
RAW(16)
)
;
CREATE OR REPLACE PROCEDURE log_event(s IN GC$NOTIF_EVENT_MSG)
IS
l_categories gc$category_string_array;
l_ca_obj gc$notif_corrective_action_job;
l_categories_new VARCHAR2(1000);
BEGIN
-- save event categories
l_categories := s.event_payload.categories;
IF l_categories IS NOT NULL
THEN
FOR c IN 1..l_categories.COUNT
LOOP
l_categories_new := (l_categories_new|| c || ' - ' || l_
categories(c)||',');
END LOOP;
END IF;
-- save event message
IF s.msg_info.notification_type = 'NOTIF_CA' AND s.event_payload.corrective_
action IS NOT NULL
THEN
l_ca_obj := s.event_payload.corrective_action;
INSERT INTO event_log (notification_type, repeat_count, ruleset_name, rule_
name, rule_owner, message, message_url, event_instance_guid, event_type, event_
name, event_msg, categories, source_obj_type, source_obj_name, source_obj_url,
severity, severity_code, target_name, target_type, target_url, host_name,
timezone, occured, ca_guid, ca_name, ca_owner, ca_type, ca_status, ca_status_code,
ca_job_step_output, ca_execution_guid, ca_stage_change_guid)
VALUES (s.msg_info.notification_type, s.msg_info.repeat_count, s.msg_
4-28 OracleВ® Enterprise Manager Administration
Sending Notifications Using PL/SQL Procedures
info.ruleset_name, s.msg_info.rule_name,s.msg_info.rule_owner, s.msg_info.message,
s.msg_info.message_url, s.event_payload.event_instance_guid, s.event_
payload.event_type, s.event_payload.event_name, s.event_payload.event_msg, l_
categories_new, s.event_payload.source.source_type, s.event_payload.source.source_
name, s.event_payload.source.source_url, s.event_payload.severity, s.event_
payload.severity_code, s.event_payload.target.target_name, s.event_
payload.target.target_type, s.event_payload.target.target_url, s.event_
payload.target.host_name, s.event_payload.target.target_timezone, s.event_
payload.occurrence_date, l_ca_obj.JOB_GUID, l_ca_obj.JOB_NAME, l_ca_obj.JOB_OWNER,
l_ca_obj.JOB_TYPE, l_ca_obj.JOB_STATUS, l_ca_obj.JOB_STATUS_CODE, l_ca_obj.JOB_
STEP_OUTPUT, l_ca_obj.JOB_EXECUTION_GUID, l_ca_obj.JOB_STATE_CHANGE_GUID);
ELSE
INSERT INTO event_log (notification_type, repeat_count, ruleset_name, rule_
name, rule_owner, message, message_url, event_instance_guid, event_type, event_
name, event_msg, categories, source_obj_type, source_obj_name, source_obj_url,
severity, severity_code, target_name, target_type, target_url, host_name,
timezone, occured, ca_guid, ca_name, ca_owner, ca_type, ca_status, ca_status_code,
ca_job_step_output, ca_execution_guid, ca_stage_change_guid)
VALUES (s.msg_info.notification_type, s.msg_info.repeat_count, s.msg_
info.ruleset_name, s.msg_info.rule_name, s.msg_info.rule_owner, s.msg_
info.message, s.msg_info.message_url, s.event_payload.event_instance_guid,
s.event_payload.event_type, s.event_payload.event_name, s.event_payload.event_msg,
l_categories_new, s.event_payload.source.source_type, s.event_
payload.source.source_name, s.event_payload.source.source_url, s.event_
payload.severity, s.event_payload.severity_code, s.event_payload.target.target_
name, s.event_payload.target.target_type, s.event_payload.target.target_url,
s.event_payload.target.host_name, s.event_payload.target.target_timezone, s.event_
payload.occurrence_date, null,null,null,null,null,null,null,null,null);
END IF;
COMMIT;
END log_event;
/
Example 4–12
PL/SQL Script to Log Incidents to a Table
CREATE TABLE incident_log (
notification_type
VARCHAR2(32),
repeat_count
NUMBER,
ruleset_name
VARCHAR2(256),
rule_owner
VARCHAR2(256),
rule_name
VARCHAR2(256),
message
VARCHAR2(4000),
message_url
VARCHAR2(4000),
incident_id
VARCHAR2(128),
ticket_url
VARCHAR2(4000),
assoc_event_cnt
NUMBER,
severity
VARCHAR2(128),
severity_code
VARCHAR2(32),
priority
VARCHAR2(128),
priority_code
VARCHAR2(32),
status
VARCHAR2(32),
categories
VARCHAR2(1000),
target_name
VARCHAR2(256),
target_type
VARCHAR2(128),
host_name
VARCHAR2(256),
timezone
VARCHAR2(64),
occured
DATE
)
;
Using Notifications 4-29
Sending Notifications Using PL/SQL Procedures
CREATE OR REPLACE PROCEDURE log_incident(s IN GC$NOTIF_INCIDENT_MSG)
IS
l_src_info_array GC$NOTIF_SOURCE_INFO_ARRAY;
l_src_info GC$NOTIF_SOURCE_INFO;
l_categories gc$category_string_array;
l_target_obj GC$NOTIF_TARGET;
l_target_name VARCHAR2(256);
l_target_type VARCHAR2(256);
l_target_timezone VARCHAR2(256);
l_hostname VARCHAR2(256);
l_categories_new VARCHAR2(1000);
BEGIN
-- Save Incident categories
IF l_categories IS NOT NULL
THEN
FOR c IN 1..l_categories.COUNT
LOOP
l_categories_new := (l_categories_new|| c || ' - ' || l_
categories(c)||',');
END LOOP;
END IF;
-- GET target info
l_src_info_array := s.incident_payload.incident_attrs.source_info_arr;
IF l_src_info_array IS NOT NULL
THEN
FOR I IN 1..l_src_info_array.COUNT
LOOP
IF l_src_info_array(I).TARGET IS NOT NULL
THEN
l_target_name := l_src_info_array(I).TARGET.TARGET_NAME;
l_target_type := l_src_info_array(I).TARGET.TARGET_TYPE;
l_target_timezone := l_src_info_array(I).TARGET.TARGET_TIMEZONE;
l_hostname := l_src_info_array(I).TARGET.HOST_NAME;
END IF;
END LOOP;
END IF;
-- save Incident notification message
INSERT INTO incident_log(notification_type, repeat_count, ruleset_name, rule_
owner, rule_name, message, message_url, incident_id, ticket_url, assoc_event_cnt,
severity, severity_code, priority, priority_code, status, categories, target_name,
target_type, host_name, timezone, occured)
VALUES (s.msg_info.notification_type, s.msg_info.repeat_count, s.msg_
info.ruleset_name, s.msg_info.rule_owner, s.msg_info.rule_name, s.msg_
info.message, s.msg_info.message_url, s.incident_payload.incident_attrs.id,
s.incident_payload.ticket_url, s.incident_payload.assoc_event_count, s.incident_
payload.incident_attrs.severity, s.incident_payload.incident_attrs.severity_code,
s.incident_payload.incident_attrs.priority, s.incident_payload.incident_
attrs.priority_code, s.incident_payload.incident_attrs.STATUS, l_categories_new,
l_target_name, l_target_type, l_hostname,l_target_timezone, s.incident_
payload.incident_attrs.creation_date);
COMMIT;
END log_incident;
/
4-30 OracleВ® Enterprise Manager Administration
Sending Notifications Using PL/SQL Procedures
Example 4–13
PL/SQL Script to Log Problems to a Table
CREATE TABLE problem_log (
notification_type
VARCHAR2(32),
repeat_count
NUMBER,
ruleset_name
VARCHAR2(256),
rule_owner
VARCHAR2(256),
rule_name
VARCHAR2(256),
message
VARCHAR2(4000),
message_url
VARCHAR2(4000),
problem_key
VARCHAR2(850),
assoc_incident_cnt
NUMBER,
problem_id
NUMBER,
owner
VARCHAR2(256),
severity
VARCHAR2(128),
severity_code
VARCHAR2(32),
priority
VARCHAR2(128),
priority_code
VARCHAR2(32),
status
VARCHAR2(32),
categories
VARCHAR2(1000),
target_name
VARCHAR2(256),
target_type
VARCHAR2(128),
host_name
VARCHAR2(256),
timezone
VARCHAR2(64),
occured
DATE
)
;
CREATE OR REPLACE PROCEDURE log_problem(s IN GC$NOTIF_PROBLEM_MSG)
IS
l_src_info_array GC$NOTIF_SOURCE_INFO_ARRAY;
l_src_info GC$NOTIF_SOURCE_INFO;
l_categories gc$category_string_array;
l_target_obj GC$NOTIF_TARGET;
l_target_name VARCHAR2(256);
l_target_type VARCHAR2(256);
l_target_timezone VARCHAR2(256);
l_hostname VARCHAR2(256);
l_categories_new VARCHAR2(1000);
BEGIN
-- Save Problem categories
l_categories := s.problem_payload.problem_attrs.categories;
IF l_categories IS NOT NULL
THEN
FOR c IN 1..l_categories.COUNT
LOOP
l_categories_new := (l_categories_new|| c || ' - ' || l_
categories(c)||',');
END LOOP;
END IF;
-- GET target info
l_src_info_array := s.problem_payload.problem_attrs.source_info_arr;
IF l_src_info_array IS NOT NULL
THEN
FOR I IN 1..l_src_info_array.COUNT
LOOP
IF l_src_info_array(I).TARGET IS NOT NULL
THEN
l_target_name := l_src_info_array(I).TARGET.TARGET_NAME;
l_target_type := l_src_info_array(I).TARGET.TARGET_TYPE;
l_target_timezone := l_src_info_array(I).TARGET.TARGET_TIMEZONE;
Using Notifications 4-31
Sending Notifications Using PL/SQL Procedures
l_hostname := l_src_info_array(I).TARGET.HOST_NAME;
END IF;
END LOOP;
END IF;
-- save Problem notification message
INSERT INTO problem_log(notification_type, repeat_count, ruleset_name, rule_
owner, rule_name, message, message_url, problem_key, assoc_incident_cnt, problem_
id, owner, severity, severity_code, priority, priority_code, status, categories,
target_name, target_type, host_name, timezone, occured)
VALUES (s.msg_info.notification_type, s.msg_info.repeat_count, s.msg_
info.ruleset_name, s.msg_info.rule_owner, s.msg_info.rule_name, s.msg_
info.message, s.msg_info.message_url, s.problem_payload.problem_key,
s.problem_payload.ASSOC_INCIDENT_COUNT, s.problem_payload.problem_attrs.id,
s.problem_payload.problem_attrs.owner, s.problem_payload.problem_attrs.severity,
s.problem_payload.problem_attrs.severity_code, s.problem_payload.problem_
attrs.PRIORITY, s.problem_payload.problem_attrs.PRIORITY_CODE, s.problem_
payload.problem_attrs.status, l_categories_new, l_target_name, l_target_type, l_
hostname,l_target_timezone, s.problem_payload.problem_attrs.CREATION_DATE);
COMMIT;
END log_problem;
/
4.4.2 Migrating Pre-12c PL/SQL Advanced Notification Methods
Pre-12c notifications map to event notifications in Enterprise Manager 12c. The event
types metric_alert, target_availability and job_status_alert correspond to the pre-12c
notification functionality.
Policy Violations are no longer available beginning with
Enterprise Manager 12c.
Note:
This section describes the mapping between Enterprise Manager 12c PL/SQL
notification payload to the pre-12c PL/SQL notification payload. You can use this
information for updating the existing pre-12c PL/SQL user callback procedures to use
the 12c PL/SQL notification payload.
Please note that Policy Violations are no longer supported in the 12c release.
4.4.2.1 Mapping for MGMT_NOTIFY_SEVERITY
When event type is metric_alert
Use the following map when gc$notif_event_payload .event_type='metric_alert'.
Table 4–8
Metric Alert Mapping
MGMT_NOTIFY_SEVERITY 12c Notification Payload
TARGET_NAME
gc$notif_target.target_name
TARGET_TYPE
gc$notif_target.target_type
TIMEZONE
gc$notif_target.target_timezone
HOST_NAME
gc$notif_target.host_name
4-32 OracleВ® Enterprise Manager Administration
Sending Notifications Using PL/SQL Procedures
Table 4–8 (Cont.) Metric Alert Mapping
MGMT_NOTIFY_SEVERITY 12c Notification Payload
MERTIC_NAME
gc$notif_event_attr.value where its name='
metric_group' in gc$notif_event_attr_array.
METRIC_DESCRIPTION
gc$notif_event_attr.value where its name='
metric_description' in gc$notif_event_attr_
array.
METRIC_COLUMN
gc$notif_event_attr.value where its name='
metric_column' in gc$notif_event_attr_array.
METRIC_VALUE
gc$notif_event_attr.value where its name='
value' in gc$notif_event_attr_array.
KEY_VALUE
It is applied for multiple keys based metric
when value of gc$notif_event_
attr.name='num_keys' is not null and is
greater than 0 in gc$notif_event_attr_array.
See detail descriptions below.
KEY_VALUE_NAME
It is applied for multiple keys based metric
when value of gc$notif_event_
attr.name='num_keys' is not null and is
greater than 0 in gc$notif_event_attr_array.
See detail descriptions below.
KEY_VALUE_GUID
gc$notif_event_attr.value where its
name='key_ value' in gc$notif_event_attr_
array.
CTXT_LIST
gc$notif_event_context_array
COLLECTION_
TIMESTAMP
gc$notif_event_payload. reported_date
SEVERITY_CODE
Derive from gc$notif_event_
payload.severity_code, see Table 4–9,
" Severity Code Mapping".
MESSAGE
gc$notif_msg_info.message
SEVERITY_GUID
gc$notif_event_attr.value where its name='
severity_guid' in gc$notif_event_attr_array.
METRIC_GUID
gc$notif_event_attr.value where its name='
metric_guid’ in gc$notif_event_attr_array.
TARGET_GUID
gc$notif_target.target_guid
RULE_OWNER
gc$notif_msg_info.rule_owner
RULE_NAME
gc$notif_msg_info.ruleset_name
The following example illustrates how to obtain similar pre-12c KEY_VALUE and
KEY_VALUE_NAME from an Enterprise Manager 12c notification payload.
Example 4–14
Extracting KEY_VALUE and KEY_VALUE_NAME
-- Get the pre-12c KEY_VALUE and KEY_VALUE_NAME from an Enterprise Manager 12c
-- notification payload
-- parameters
-IN Parameters:
-event_msg : The event notification payload
-OUT Parameters
-key_value_name_out : the KEY_VALUE_NAME backward compitable to pre-12c
Using Notifications 4-33
Sending Notifications Using PL/SQL Procedures
-notification payload
-key_value_out
: the KEY_VALUE backward compitable to pre-12c
-notification payload
-CREATE OR REPLACE PROCEDURE get_pre_12c_key_value(
event_msg IN GC$NOTIF_EVENT_MSG,
key_value_name_out OUT VARCHAR2,
key_value_out OUT VARCHAR2)
IS
l_key_columns MGMT_SHORT_STRING_ARRAY := MGMT_SHORT_STRING_ARRAY();
l_key_column_values MGMT_MEDIUM_STRING_ARRAY := MGMT_MEDIUM_STRING_ARRAY();
l_key_value VARCHAR2(1790) := NULL;
l_num_keys NUMBER := 0;
l_attrs gc$notif_event_attr_array;
l_key_value_name VARCHAR2(512);
BEGIN
l_attrs := event_msg.event_payload.event_attrs;
key_value_name_out := NULL;
key_value_out := NULL;
IF l_attrs IS NOT NULL AND
l_attrs.COUNT > 0
THEN
l_key_columns.extend(7);
l_key_column_values.extend(7);
FOR c IN 1..l_attrs.COUNT
LOOP
CASE l_attrs(c).name
WHEN 'num_keys' THEN
BEGIN
l_num_keys := to_number(l_attrs(c).value);
EXCEPTION
WHEN OTHERS THEN
-- should never happen, but guard against it l_num_keys := 0;
END;
WHEN 'key_value' THEN
l_key_value := substr(l_attrs(c).nls_value,1,1290);
WHEN 'key_column_1' THEN
l_key_columns(1) := substr(l_attrs(c).nls_value,1,64);
WHEN 'key_column_2' THEN
l_key_columns(2) := substr(l_attrs(c).nls_value,1,64);
WHEN 'key_column_3' THEN
l_key_columns(3) := substr(l_attrs(c).nls_value,1,64);
WHEN 'key_column_4' THEN
l_key_columns(4) := substr(l_attrs(c).nls_value,1,64);
WHEN 'key_column_5' THEN
l_key_columns(5) := substr(l_attrs(c).nls_value,1,64);
WHEN 'key_column_6' THEN
l_key_columns(6) := substr(l_attrs(c).nls_value,1,64);
WHEN 'key_column_7' THEN
l_key_columns(7) := substr(l_attrs(c).nls_value,1,64);
WHEN 'key_column_1_value' THEN
l_key_column_values(1) := substr(l_attrs(c).nls_value,1,256);
WHEN 'key_column_2_value' THEN
l_key_column_values(2) := substr(l_attrs(c).nls_value,1,256);
WHEN 'key_column_3_value' THEN
l_key_column_values(3) := substr(l_attrs(c).nls_value,1,256);
WHEN 'key_column_4_value' THEN
l_key_column_values(4) := substr(l_attrs(c).nls_value,1,256);
4-34 OracleВ® Enterprise Manager Administration
Sending Notifications Using PL/SQL Procedures
WHEN 'key_column_5_value' THEN
l_key_column_values(5) := substr(l_attrs(c).nls_value,1,256);
WHEN 'key_column_6_value' THEN
l_key_column_values(6) := substr(l_attrs(c).nls_value,1,256);
WHEN 'key_column_7_value' THEN
l_key_column_values(7) := substr(l_attrs(c).nls_value,1,256);
ELSE
NULL;
END CASE;
END LOOP;
-- get key_value and key_value_name when l_num_keys > 0
IF l_num_keys > 0
THEN
-- get key value name
IF l_key_columns IS NULL OR l_key_columns.COUNT = 0
THEN
key_value_name_out := NULL;
ELSE
l_key_value_name := NULL;
FOR i in l_key_columns.FIRST..l_num_keys
LOOP
IF i > 1
THEN
l_key_value_name := l_key_value_name || ';';
END IF;
l_key_value_name := l_key_value_name || l_key_columns(i);
END LOOP;
key_value_name_out := l_key_value_name;
END IF;
-- get key_value
IF l_num_keys = 1
THEN
key_value_out := l_key_value;
ELSE
l_key_value := NULL;
IF l_key_column_values IS NULL OR l_key_column_values.COUNT = 0
THEN
key_value_out := NULL;
ELSE
FOR i in l_key_column_values.FIRST..l_num_keys
LOOP
IF i > 1
THEN
l_key_value := l_key_value || ';';
END IF;
l_key_value := l_key_value || l_key_column_values(i);
END LOOP;
-- max length for key value in pre-12c = 1290
key_value_out := substr(l_key_value,1,1290);
END IF;
END IF;
END IF; -- l_num_keys > 0
END IF; -- l_attrs IS NOT NULL
END get_pre_12c_key_value;
/
Using Notifications 4-35
Sending Notifications Using PL/SQL Procedures
When the event type is metric_alert:
Use the following severity code mapping from 12c to pre-12c when the event type is
metric_alert.
Table 4–9
Severity Code Mapping
12c Severity Code
Pre-12c Severity Code
GC_EVENT_RECEIVER.FATAL 32
MGMT_GLOBAL.G_SEVERITY_CRITICAL
25
GC_EVENT_RECEIVER.CRITICAL 16 MGMT_GLOBAL.G_SEVERITY_CRITICAL
25
GC_EVENT_RECEIVER.WARNING 8 MGMT_GLOBAL.G_SEVERITY_WARNING
20
GC_EVENT_RECEIVER.CLEAR 0
MGMT_GLOBAL.G_SEVERITY_CLEAR 15
When event type is target_availability:
Use the following map when gc$notif_event_payload .event_type='target_availability'.
Table 4–10
Target Availability Mapping
MGMT_NOTIFY_SEVERITY
12c Notification Payload
TARGET_NAME
gc$notif_target.target_name
TARGET_TYPE
gc$notif_target.target_type
TIMEZONE
gc$notif_target.target_timezone
HOST_NAME
gc$notif_target.host_name
MERTIC_NAME
Use fixed value "Response".
METRIC_DESCRIPTION
NULL
METRIC_COLUMN
Use fixed value "Status".
METRIC_VALUE
gc$notif_event_attr.value where its
name='target_status' in gc$notif_event_attr_
array.
KEY_VALUE
NULL
KEY_VALUE_NAME
NULL
KEY_VALUE_GUID
NULL
CTXT_LIST
gc$notif_event_context_array
COLLECTION_TIMESTAMP
gc$notif_event_payload. reported_date
SEVERITY_CODE
gc$notif_event_attr.value where its name='
avail_severity' in gc$notif_event_attr_array.
MESSAGE
gc$notif_msg_info.message
SEVERITY_GUID
gc$notif_event_attr.value where its name='
severity_guid' in gc$notif_event_attr_array.
METRIC_GUID
gc$notif_event_attr.value where its name='
metric_guid_id' in gc$notif_event_attr_array.
TARGET_GUID
gc$notif_target.target_guid
RULE_OWNER
gc$notif_msg_info.rule_owner
RULE_NAME
gc$notif_msg_info.ruleset_name
4-36 OracleВ® Enterprise Manager Administration
Sending Notifications Using PL/SQL Procedures
4.4.2.2 Mapping for MGMT_NOTIFY_JOB
Use the following map when gc$notif_event_payload .event_type=job_status_change'.
Table 4–11
Job Status Change Mapping
MGMT_NOTIFY_JOB
12c Notification Payload
JOB_NAME
gc$notif_source.source_name
JOB_OWNER
gc$notif_source.source_owner
JOB_TYPE
gc$notif_source.source_sub_type
JOB_STATUS
gc$notif_event_attr.value where its name='
execution_status_code' in gc$notif_event_
attr_array.
STATE_CHANGE_GUID
gc$notif_event_attr.value where its name='
state_change_guid' in gc$notif_event_attr_
array.
JOB_GUID
gc$notif_source.source_guid
EXECUTION_ID
gc$notif_event_attr.value where its name='
execution_id' in gc$notif_event_attr_array.
TARGETS
gc$notif_target.target_name, gc$notif_
target.target_type
RULE_OWNER
gc$notif_msg_info.rule_owner
RULE_NAME
gc$notif_msg_info.ruleset_name
OCCURRED_DATE
gc$notif_event_payload. reported_date
4.4.2.3 Mapping for MGMT_NOTIFY_CORRECTIVE_ACTION
Note that corrective action related payload is populated when gc$notif_msg_
info.notification_type is set to NOTIF_CA.
For mapping the following attributes, use the mapping information provided for
MGMT_NOTIFY_SEVERITY object Table 4–8, " Metric Alert Mapping"
MERTIC_NAME
METRIC_COLUMN
METRIC_VALUE
KEY_VALUE
KEY_VALUE_NAME
KEY_VALUE_GUID
CTXT_LIST
RULE_OWNER
RULE_NAME
OCCURRED_DATE
For mapping the job related attributes in MGMT_NOTIFY_CORRECTIVE_ACTION
object, use the following map.
Using Notifications 4-37
Sending SNMP Traps to Third Party Systems
Table 4–12
Corrective Action Mapping
MGMT_NOTIFY_CORRECTIVE_ACTION 12c Notification Payload
JOB_NAME
gc$ notif_corrective_action_job.job_name
JOB_OWNER
gc$ notif_corrective_action_job.job_owner
JOB_TYPE
gc$ notif_corrective_action_job.job_type
JOB_STATUS
gc$ notif_corrective_action_job.status_code
STATE_CHANGE_GUID
gc$ notif_corrective_action_job. job_state_
change_guid
JOB_GUID
gc$ notif_corrective_action_job. job _guid
EXECUTION_ID
gc$ notif_corrective_action_job. job_
execution_guid
OCCURRED_DATE
gc$ notif_corrective_action_job.occurred_
date
TARGETS
There can be at most one target. Use the
values from gc$notif_target.target_name,
gc$notif_target.target_type for the associated
target.
4.5 Sending SNMP Traps to Third Party Systems
Enterprise Manager supports integration with third-party management tools through
the Simple Network Management Protocol (SNMP). For example, you can use SNMP
to notify a third-party application that a selected metric has exceeded its threshold.
In order for a third-party system to interpret traps sent by
the OMS, the omstrap.v1 file must first be loaded into the third-party
SNMP console. For more information about this file and its location,
see "MIB Definition" on page 4-48.
Important:
The Enterprise Manager 12c version of the MIB file incorporates the
10g and 11g MIB content, thus ensuring backward compatibility with
earlier Enterprise Manager releases.
Enterprise Manager supports both SNMP Version 1 and Version 3 traps. The traps are
described by the MIB definition shown in Appendix B, "Enterprise Manager MIB
Definition." See "Management Information Base (MIB)" on page 4-48 for an
explanation of how the MIB works. If you are using Enterprise Manager 12c, see
Appendix A, "Interpreting Variables of the Enterprise Manager MIB" and Appendix B,
"Enterprise Manager MIB Definition." If you are upgrading from a pre-12c version of
Enterprise Manager, see Appendix C, "SNMP Trap Mappings" for specific version
mappings.
For Enterprise Manager 12c, SNMP traps are delivered for event notifications only.
SNMP trap notifications are not supported for incidents or problems.
4-38 OracleВ® Enterprise Manager Administration
Sending SNMP Traps to Third Party Systems
Notification methods based on SNMP traps must be
configured by an administrator with Super Administrator
privileges before any user can then choose to select one or more of
these SNMP trap methods while creating/editing a incident rule.
Note:
4.5.1 SNMP Version 1 Versus SNMP Version 3
SNMP Version 3 shares the same basic architecture of Version 1, but adds numerous
enhancements to SNMP administration and security. The primary enhancement
relevant to Enterprise Manager involves additional security levels that provide both
authentication and privacy as well as authorization and access control.
User-based Security Model (USM)
USM defines the security-related procedures followed by an SNMP engine when
processing SNMP messages. Enterprise Manager SNMP V3 support takes advantage
of this added SNMP message-level security enhancement to provide a secure
messaging environment.
USM protects against two primary security threats:
в– в– Modification of information: The modification threat is the danger that some
unauthorized entity may alter in-transit SNMP messages generated on behalf of an
authorized principal in such a way as to effect unauthorized management
operations, including falsifying the value of an object.
Masquerade: The masquerade threat is the danger that management operations
not authorized for some user may be attempted by assuming the identity of
another user that has the appropriate authorizations.
For both SNMP versions, the basic methodology for setting up Enterprise Manager
advanced notifications using SNMP traps remains the same:
1.
Define the notification method based on an SNMP trap.
2.
Assign the notification method to an incident rule.
4.5.2 Working with SNMP V3 Trap Notification Methods
The procedure for defining an SNMP V3 trap notification method differs slightly from
that of V1. Beginning with Enterprise Manager Release 12.1.0.4, a separate interface
consolidates key information and configuration functionality pertaining to SNMP V3
trap notification methods. The SNMP V3 Trap interface helps guide you through the
process of creating SNMP notification methods, enabling the OMS to send SNMP
traps, and defining user security settings for SNMP trap notifications.
4.5.2.1 Configuring the OMS to Send SNMP Trap Notifications
Before creating an SNMP Trap notification method, you must enable at least one OMS
is your environment to handle SNMP Trap notifications. For SNMP V3, the OMS
serves as an SNMP Agent which sends traps to the SNMP Manager that is monitoring
all SNMP Agents deployed in the network.
1.
From the Setup menu, select Notifications and then SNMP V3 Traps. The Getting
Started page displays. This page documents the high-level workflow for
configuring Enterprise Manager to send traps to third-party SNMP Managers.
Using Notifications 4-39
Sending SNMP Traps to Third Party Systems
2.
Click the Configuration tab. The Configuration page displays.
3.
In the OMS Configuration region, select the OMS you wish to enable.
4.
Check the following for each OMS and make changes, if necessary:
в– в– 5.
OMS requires a port for SNMPv3 traps. Check if the default port can be used
by OMS.
OMS requires a unique Engine ID for sending traps. By default, it is being
generated from the host name and port.
Click Enable.
4-40 OracleВ® Enterprise Manager Administration
Sending SNMP Traps to Third Party Systems
4.5.2.2 Creating/Editing an SNMP V3 Trap Notification Method
Once an OMS has been enabled to send SNMP traps notifications, the next step is to
create a notification method than can be used by an incident rule.
1.
From the Setup menu, select Notifications and then SNMP V3 Traps. The Getting
Started page displays.
Note: If want to edit an existing Notification Method, select the
desired method from the Notification Methods region and click Edit.
2.
Click the Configuration tab. The Configuration page displays.
3.
From the Notification Methods region, click Create. The SNMPv3 Traps: Create
Notification Method page displays.
Using Notifications 4-41
Sending SNMP Traps to Third Party Systems
4.
Enter the requisite Notification Method definition parameters. Note: You can
enable Repeat Notifications at this point.
5.
If you choose to create a new User Security Model entry, from the User Security
Model region, ensure the Create New option is chosen.
1.
Specify a Username that uniquely identifies the credential. SNMP V3 allows
multiple usernames to be set in an SNMP Agent as well as SNMP Manager
applications.
2.
Select a Security Level from the drop-down menu. Available parameters
become available depending on the security level. There are three levels from
which to choose:
AuthPriv (Authentication + Privacy:) The sender’s identity must be confirmed
by the receiver (authentication). SNMP V3 messages are encrypted by the
sender and must be decrypted by the receiver (privacy).
AuthNoPriv (Authentication only): The receiver must authenticate the
sender’s identity before accepting the message.
NoAuthNoPriv (no security): Neither sender identity confirmation nor
message encryption is used.
3.
For AuthPriv and AuthNoPriv security levels, choose a the desired
Authentication Protocol. Two authentication protocols are available:
Secure Hash Algorithm (SHA)
Message Digest algorithm (MD5)
The authentication protocols are used to build the message digest when the
message is authenticated.
Privacy Protocol (used for the AuthPriv security level) is used to
encrypt/decrypt messages. USM uses the Data Encryption Standard (DES).
The Privacy Password is used in conjunction with the Privacy Protocol. the
privacy password on both the SNMP Agent and SNMP Manager must match
in order for encryption/decryption to succeed.
If you have already have predefined User Security Model entries, choose the Use
Existing option and select one of the USM entries from the drop-down menu.
USM entries are listed by username.
Ensure that the USM credentials are identical in OMS and
the external trap receiver. If they do not match, Enterprise Manager
will still send the SNMP trap, but the trap will not be received. If the
USM credentials are invalid, Enterprise Manager will still send the
SNMP trap, however, the trap will not be received as the incorrect
credentials will result in an authentication error at the SNMP receiver.
This type of authentication error will not be apparent from the
Enterprise Manager console.
Important:
6.
Once you have entered the requisite Notification Method and USM parameters,
click Save. The newly created notification method appears in the Notification
Method region of the Configuration page.
4-42 OracleВ® Enterprise Manager Administration
Sending SNMP Traps to Third Party Systems
Once you have defined the SNMP V3 Trap notification
method, you must add it to a rule. See "Creating a Rule to Send SNMP
Traps to Third Party Systems" on page 3-63 for instructions.
Note:
4.5.2.3 Editing a User Security Model Entry
You can add USM entries at any time.
1.
From the Setup menu, select Notifications, and then SNMP V3 Traps.
2.
Click on the Configurations tab.
3.
From the User Security Model Entries region, click Create. The User Security
Model Entries dialog displays.
4.
Specify a Username that uniquely identifies the credential. SNMP V3 allows
multiple usernames to be set in an SNMP Agent as well as SNMP Manager
applications.
5.
Select a Security Level from the drop-down menu. Available parameters become
available depending on the security level. There are three levels from which to
choose:
AuthPriv (Authentication + Privacy:) The sender’s identity must be confirmed by
the receiver (authentication). SNMP V3 messages are encrypted by the sender and
must be decrypted by the receiver (privacy).
AuthNoPriv (Authentication only): The receiver must authenticate the sender’s
identity before accepting the message.
NoAuthNoPriv (no security): Neither sender identity confirmation nor message
encryption is used.
Using Notifications 4-43
Sending SNMP Traps to Third Party Systems
6.
For AuthPriv and AuthNoPriv security levels, choose a the desired
Authentication Protocol. Two authentication protocols are available:
Secure Hash Algorithm (SHA)
Message Digest algorithm (MD5)
The authentication protocols are used to build the message digest when the
message is authenticated.
Privacy Protocol (used for the AuthPriv security level) is used to encrypt/decrypt
messages. USM uses the Data Encryption Standard (DES). The Privacy Password
is used in conjunction with the Privacy Protocol. the privacy password on both the
SNMP Agent and SNMP Manager must match in order for encryption/decryption
to succeed.
7.
Click OK.
The new USM username will appear in the User Security Model Entries table.
When creating new SNMP V3 Trap notification methods, the USM username will
appear as a selectable option from the Existing Entries drop-down menu.
After editing the USM, you should verify the change via the notification methods
that use it.
4.5.2.4 Viewing Available SNMP V3 Trap Notification Methods
To view available SNMP V3 Trap notification methods:
1.
From the Setup menu, select Notifications, and then SNMP V3 Traps.
2.
Click on the Configurations tab.
3.
The Notification Methods region displays existing SNMP V3 Trap notification
methods.
4.5.2.5 Deleting an SNMP V3 Trap Notification Method
To delete available SNMP V3 Trap notification methods:
1.
From the Setup menu, select Notifications, and then SNMP V3 Traps.
2.
Click on the Configurations tab.
3.
From the Notification Methods region, select an existing SNMP V3 Trap
notification method.
4.
Click Delete.
4.5.3 Creating an SNMP V1 Trap
Step 1: Define a new notification method based on an SNMP trap.
Log in to Enterprise Manager as a Super Administrator. From the Setup menu, select
Notifications and then select Notification Method to access the Notification Methods
page. From this page you can add a new method based on an SNMP trap.
You must provide the name of the host (machine) on which the SNMP master agent is
running and other details as shown in the following example. As shown in, the SNMP
host will receive your SNMP traps.
4-44 OracleВ® Enterprise Manager Administration
Sending SNMP Traps to Third Party Systems
Figure 4–7 SNMP Trap Required Information
Note:
A Test SNMP Trap button exists for you to test your setup.
Metric severity information will be passed as a series of variables in the SNMP trap.
Step 2: Assign the notification method to a rule.
You can edit an existing rule (or create a new incident rule), then add an action to the
rule that subscribes to the advanced notification method. For instructions on setting up
incident rules using SNMP traps, see "Creating a Rule to Send SNMP Traps to Third
Party Systems" on page 3-63.
Example SNMP Trap Implementation
In this scenario, you want to identify the unique issues from the SNMP traps that are
sent. Keep in mind that all events that are related to the same issue are part of the same
event sequence. Each event sequence has a unique identification number.
An event sequence is a sequence of related events that represent the life of a specific
issue from the time it is detected and an event is raised to the time it is fixed and a
corresponding clear event is generated. For example, a warning metric alert event is
raised when the CPU utilization of a host crosses 80%. This starts the event sequence
representing the issue CPU Utilization of the host is beyond normal level. Another critical
event is raised for the same issue when the CPU utilization goes above 90% and the
event is added to the same event sequence. After a period of time, the CPU utilization
returns to a normal level and a clear event is raised. At this point, the issue is resolved
and the event sequence is closed.
The SNMP trap sent for this scenario is shown in Example 4–15. Each piece of
information is sent as a variable embedded in the SNMP Trap.
Example 4–15
SNMP Trap
**************V1 TRAP***[1]*****************
Community : public
Enterprise :1.3.6.1.4.1.111.15.2
Generic :6
Specific :3
TimeStamp :67809
Agent adress :10.240.36.109
Using Notifications 4-45
Sending SNMP Traps to Third Party Systems
1.3.6.1.4.1.111.15.3.1.1.2.1: NOTIF_NORMAL
1.3.6.1.4.1.111.15.3.1.1.3.1: CPU Utilization is 92.658%, crossed warning (80) or
critical (90) threshold.
1.3.6.1.4.1.111.15.3.1.1.4.1:
https://sampleserver.oracle.com:5416/em/redirect?pageType=sdk-core-event-console-d
etailEvent&issueID=C77AE9E578F00773E040F00A6D242F90
1.3.6.1.4.1.111.15.3.1.1.5.1: Critical
1.3.6.1.4.1.111.15.3.1.1.6.1: CRITICAL
1.3.6.1.4.1.111.15.3.1.1.7.1: 0
1.3.6.1.4.1.111.15.3.1.1.8.1:
1.3.6.1.4.1.111.15.3.1.1.9.1:
1.3.6.1.4.1.111.15.3.1.1.10.1: Aug 17, 2012 3:26:36 PM PDT
1.3.6.1.4.1.111.15.3.1.1.11.1: Capacity
1.3.6.1.4.1.111.15.3.1.1.12.1: Capacity
1.3.6.1.4.1.111.15.3.1.1.13.1: Metric Alert
1.3.6.1.4.1.111.15.3.1.1.14.1: Load:cpuUtil
1.3.6.1.4.1.111.15.3.1.1.15.1: 281
1.3.6.1.4.1.111.15.3.1.1.16.1:
1.3.6.1.4.1.111.15.3.1.1.17.1: No
1.3.6.1.4.1.111.15.3.1.1.18.1: New
1.3.6.1.4.1.111.15.3.1.1.19.1: None
1.3.6.1.4.1.111.15.3.1.1.20.1: 0
1.3.6.1.4.1.111.15.3.1.1.21.1: sampleserver.oracle.com
1.3.6.1.4.1.111.15.3.1.1.22.1:
https://sampleserver.oracle.com:5416/em/redirect?pageType=TARGET_
HOMEPAGE&targetName=sampleserver.oracle.com&targetType=host
1.3.6.1.4.1.111.15.3.1.1.23.1: Host
1.3.6.1.4.1.111.15.3.1.1.24.1: sampleserver.oracle.com
1.3.6.1.4.1.111.15.3.1.1.25.1: SYSMAN
1.3.6.1.4.1.111.15.3.1.1.26.1:
1.3.6.1.4.1.111.15.3.1.1.27.1: 5.8.0.0.0
1.3.6.1.4.1.111.15.3.1.1.28.1: Operating System=Linux, Platform=x86_64,
1.3.6.1.4.1.111.15.3.1.1.29.1:
1.3.6.1.4.1.111.15.3.1.1.30.1:
1.3.6.1.4.1.111.15.3.1.1.31.1:
1.3.6.1.4.1.111.15.3.1.1.32.1:
1.3.6.1.4.1.111.15.3.1.1.33.1:
1.3.6.1.4.1.111.15.3.1.1.34.1:
1.3.6.1.4.1.111.15.3.1.1.35.1:
1.3.6.1.4.1.111.15.3.1.1.36.1:
1.3.6.1.4.1.111.15.3.1.1.37.1:
1.3.6.1.4.1.111.15.3.1.1.38.1:
1.3.6.1.4.1.111.15.3.1.1.39.1: SnmpNotifRuleset
1.3.6.1.4.1.111.15.3.1.1.40.1: SnmpNotifRuleset,SnmpNotifEvent
1.3.6.1.4.1.111.15.3.1.1.41.1: SYSMAN
1.3.6.1.4.1.111.15.3.1.1.42.1: C77AE9E578F00773E040F00A6D242F90
1.3.6.1.4.1.111.15.3.1.1.43.1:
1.3.6.1.4.1.111.15.3.1.1.44.1:
1.3.6.1.4.1.111.15.3.1.1.45.1:
1.3.6.1.4.1.111.15.3.1.1.46.1: CPU Utilization is 92.658%, crossed warning (80) or
critical (90) threshold., Incident created by rule (Name = Incident management
Ruleset for all targets, Incident creation Rule for metric alerts.; Owner =
<SYSTEM>).
1.3.6.1.4.1.111.15.3.1.1.61.1: Metric GUID=0C71A1AFAC2D7199013837DA35522C08
1.3.6.1.4.1.111.15.3.1.1.62.1: Severity GUID=C77AE9E578EC0773E040F00A6D242F90
1.3.6.1.4.1.111.15.3.1.1.63.1: Cycle GUID=C77AE9E578EC0773E040F00A6D242F90
1.3.6.1.4.1.111.15.3.1.1.64.1: Collection Name=LoadLinux
1.3.6.1.4.1.111.15.3.1.1.65.1: Metric Group=Load
1.3.6.1.4.1.111.15.3.1.1.66.1: Metric=CPU Utilization (%)
1.3.6.1.4.1.111.15.3.1.1.67.1: Metric Description=
4-46 OracleВ® Enterprise Manager Administration
Sending SNMP Traps to Third Party Systems
1.3.6.1.4.1.111.15.3.1.1.68.1: Metric value=92.658
1.3.6.1.4.1.111.15.3.1.1.69.1: Key Value=
1.3.6.1.4.1.111.15.3.1.1.70.1:
1.3.6.1.4.1.111.15.3.1.1.71.1:
1.3.6.1.4.1.111.15.3.1.1.72.1:
1.3.6.1.4.1.111.15.3.1.1.73.1:
1.3.6.1.4.1.111.15.3.1.1.74.1:
1.3.6.1.4.1.111.15.3.1.1.75.1:
1.3.6.1.4.1.111.15.3.1.1.76.1:
1.3.6.1.4.1.111.15.3.1.1.77.1:
1.3.6.1.4.1.111.15.3.1.1.78.1:
1.3.6.1.4.1.111.15.3.1.1.79.1:
1.3.6.1.4.1.111.15.3.1.1.80.1:
1.3.6.1.4.1.111.15.3.1.1.81.1:
1.3.6.1.4.1.111.15.3.1.1.82.1:
1.3.6.1.4.1.111.15.3.1.1.83.1:
1.3.6.1.4.1.111.15.3.1.1.84.1: Number of keys=0
1.3.6.1.4.1.111.15.3.1.1.85.1:
**************END V1 TRAP******************
This following example illustrates how OIDs are used during the lifecycle of an event.
Here, for one event (while the event is open), the event sequence OID remains the
same even though the event severity changes.
The OID for the event sequence is:
1.3.6.1.4.1.111.15.3.1.1.42.1: C77AE9E578F00773E040F00A6D242F90
The OID for the event severity code is:
1.3.6.1.4.1.111.15.3.1.1.6.1: CRITICAL
When the event clears, these OIDs show the same event sequence with a different
severity code:
The OID for the event sequence is:
1.3.6.1.4.1.111.15.3.1.1.42.1: C77AE9E578F00773E040F00A6D242F90
The OID for the event severity code is:
1.3.6.1.4.1.111.15.3.1.1.6.1: CLEAR
The length of the SNMP OID value is limited to 2560 bytes by default. Configure the
emoms property oracle.sysman.core.notification.snmp.max_oid_length to change the
default limit.
4.5.4 SNMP Traps: Moving from Previous Enterprise Manager Releases to 12c
When you upgrade from a pre-Enterprise Manager 12c release
to 12c, SNMP advanced notification methods defined using previous
versions of Enterprise Manager (pre-12c) will continue to function
without modification.
Note:
For Enterprise Manager 11g and earlier, there were two types of SNMP traps:
в– Alerts
в– Job Status
Using Notifications 4-47
Management Information Base (MIB)
For Enterprise Manager 12c there is now a single, comprehensive SNMP trap type that
covers all available event types such as metric alerts, target availability, compliance
standard violations, or job status changes. For more information about pre-12c to 12c
SNMP trap mappings, see Appendix C, "SNMP Trap Mappings." Traps will conform to
the older Enterprise Manager MIB definition. Hence, pre-Enterprise Manager 12c traps
will continue to be sent. See Appendix C, "SNMP Trap Mappings" for more
information.
Also, for Enterprise Manager 12c, size of SNMP trap has increased in order to
accommodate all event types and provide more comprehensive information. By
default, the maximum SNMP packet size is 5120 bytes. If the third party system has a
limit in the size of SNMP trap it can receive, you can change the default size of SNMP
trap that Enterprise Manager sends. To change the default packet size, set this emoms
oracle.sysman.core.notification.snmp_packet_length parameter, and
then bounce the OMS.
When limiting the SNMP trap packet size, Oracle recommends
not setting the oracle.sysman.core.notification.snmp_packet_length
parameter any lower than 3072 bytes (3K).
Note:
The Enterprise Manager 12c MIB includes all pre-Enterprise Manager 12c MIB
definitions. Hence, if you have an Enterprise Manager 12c MIB in your third party
system, you can receive SNMP traps from both pre-Enterprise Manager 12c as well as
Enterprise Manager 12c sites. For detailed information on version mapping, see
Appendix C, "SNMP Trap Mappings."
4.6 Management Information Base (MIB)
Enterprise Manager Cloud Control can send SNMP Traps to third-party,
SNMP-enabled applications. Details of the trap contents can be obtained from the
management information base (MIB) variables. The following sections discuss
Enterprise Manager MIB variables in detail.
4.6.1 About MIBs
A MIB is a text file, written in ASN.1 notation, which describes the variables
containing the information that SNMP can access. The variables described in a MIB,
which are also called MIB objects, are the items that can be monitored using SNMP.
There is one MIB for each element being monitored. Each monolithic or subagent
consults its respective MIB in order to learn the variables it can retrieve and their
characteristics. The encapsulation of this information in the MIB is what enables
master agents to register new subagents dynamically — everything the master agent
needs to know about the subagent is contained in its MIB. The management
framework and management applications also consult these MIBs for the same
purpose. MIBs can be either standard (also called public) or proprietary (also called
private or vendor).
The actual values of the variables are not part of the MIB, but are retrieved through a
platform-dependent process called "instrumentation". The concept of the MIB is very
important because all SNMP communications refer to one or more MIB objects. What
is transmitted to the framework is, essentially, MIB variables and their current values.
4.6.2 MIB Definition
You can find the SNMP MIB file at the following location:
4-48 OracleВ® Enterprise Manager Administration
Management Information Base (MIB)
OMS_HOME/network/doc/omstrap.v1
The omstrap.v1 file is compatible with both SNMP V1 and
SNMP V3.
Note:
The file omstrap.v1 is the OMS MIB.
For more information, see Appendix A, "Interpreting Variables of the Enterprise
Manager MIB."
A hardcopy version of omstrap.v1 can be found in Appendix B, "Enterprise Manager
MIB Definition."
The length of the SNMP OID value is limited to 2560 bytes by default. Configure
emoms property oracle.sysman.core.notification.snmp.max_oid_length to change the
default limit.
For Enterprise Manager 12c, SNMP traps are delivered for event notifications only.
SNMP trap notifications are not supported for incidents or problems.
SNMP advanced notification methods defined using previous
versions of Enterprise Manager (pre-12c) will continue to function
without modification. Traps will conform to the older Enterprise
Manager MIB definition.
Note:
4.6.3 Reading the MIB Variable Descriptions
This section covers the format used to describe MIB variables. Note that the STATUS
element of SNMP MIB definition, Version 1, is not included in these MIB variable
descriptions. Since Oracle has implemented all MIB variables as CURRENT, this value
does not vary.
4.6.3.1 Variable Name
Syntax
Maps to the SYNTAX element of SNMP MIB definition, Version 1.
Max-Access
Maps to the MAX-ACCESS element of SNMP MIB definition, Version 1.
Status
Maps to the STATUS element of SNMP MIB definition, Version 1.
Explanation
Describes the function, use and precise derivation of the variable. (For example, a
variable might be derived from a particular configuration file parameter or
performance table field.) When appropriate, incorporates the DESCRIPTION part of
the MIB definition, Version 1.
Typical Range
Describes the typical, rather than theoretical, range of the variable. For example, while
integer values for many MIB variables can theoretically range up to 4294967295, a
typical range in an actual installation will vary to a lesser extent. On the other hand,
some variable values for a large database can actually exceed this "theoretical" limit (a
"wraparound"). Specifying that a variable value typically ranges from 0 to 1,000 or
Using Notifications 4-49
Passing Corrective Action Status Change Information
1,000 to 3 billion will help the third-party developer to develop the most useful
graphical display for the variable.
Significance
Describes the significance of the variable when monitoring a typical installation.
Alternative ratings are Very Important, Important, Less Important, or Not Normally
Used. Clearly, the DBA will want to monitor some variables more closely than others.
However, which variables fall into this category can vary from installation to
installation, depending on the application, the size of the database, and on the DBA’s
objectives. Nevertheless, assessing a variable’s significance relative to the other
variables in the MIB can help third-party developers focus their efforts on those
variables of most interest to the most DBAs.
Related Variables
Lists other variables in this MIB, or other MIBs implemented by Oracle, that relate in
some way to this variable. For example, the value of this variable might derive from
that of another MIB variable. Or perhaps the value of this variable varies inversely to
that of another variable. Knowing this information, third-party developers can
develop useful graphic displays of related MIB variables.
Suggested Presentation
Suggests how this variable can be presented most usefully to the DBA using the
management application: as a simple value, as a gauge, or as an alarm, for example.
4.7 Passing Corrective Action Status Change Information
Passing corrective action status change attributes (such as new status, job name, job
type, or rule owner) to PL/SQL procedures or OS commands/scripts allows you to
customize automated responses to status changes. For example, you many want to call
an OS script to open a trouble ticket for an in-house support trouble ticket system if a
critical corrective action fails to run. In this case, you will want to pass status (for
example, Problems or Aborted) to the script to open a trouble ticket and escalate the
problem.
4.7.1 Passing Corrective Action Execution Status to an OS Command or Script
The notification system passes information to an OS script or executable via system
environment variables. Conventions used to access environmental variables vary
depending on the operating system:
в– UNIX: $ENV_VARIABLE
в– MS Windows: %ENV_VARIABLE%
The notification system sets the following environment variables before calling the
script. The notification system will set the environment variable $NOTIF_TYPE =
NOTIF_CA for Corrective Action Execution. The script can then use any or all of these
variables within the logic of the script.
Following table lists the environment variables for corrective action, they are
populated when a corrective action is completed for an event.
Table 4–13
Corrective Action Environment Variables
Environment Variable
Description
CA_JOB_STATUS
Corrective action job execution status.
CA_JOB_NAME
Name of the Corrective Action.
4-50 OracleВ® Enterprise Manager Administration
Passing Corrective Action Status Change Information
Table 4–13 (Cont.) Corrective Action Environment Variables
Environment Variable
Description
CA_JOB_OWNER
Owner of Corrective Action.
CA_JOB_STEP_OUTPUT
The value will be the text output from the Corrective Action
execution.
CA_JOB_TYPE
Corrective Action Job type
4.7.2 Passing Corrective Action Execution Status to a PLSQL Procedure
The notification system passes corrective action status change information to PL/SQL
procedure - PROCEDURE p(event_msg IN gc$notif_event_msg). The instance
gc$notif_corrective_action_job object is defined in event_msg.event_payload.
corrective_action if event_msg. msg_info. notification_type is equal to
GC$NOTIFICATIONNOTIF_CA. When a corrective action executes, the notification
system calls the PL/SQL procedure associated with the incident rule and passes the
populated object to the procedure. The procedure is then able to access the fields of the
object that has been passed to it. See Table 4–44, " Corrective Action Job-Specific
Attributes" for details.
The following status codes are possible values for the job_status field of the MGMT_
NOTIFY_CORRECTIVE_ACTION object.
Table 4–14
Corrective Action Status Codes
Name
Datatype
Value
SCHEDULED_STATUS
NUMBER(2)
1
EXECUTING_STATUS
NUMBER(2)
2
ABORTED_STATUS
NUMBER(2)
3
FAILED_STATUS
NUMBER(2)
4
COMPLETED_STATUS
NUMBER(2)
5
SUSPENDED_STATUS
NUMBER(2)
6
AGENTDOWN_STATUS
NUMBER(2)
7
STOPPED_STATUS
NUMBER(2)
8
SUSPENDED_LOCK_STATUS
NUMBER(2)
9
SUSPENDED_EVENT_STATUS
NUMBER(2)
10
SUSPENDED_BLACKOUT_STATUS
NUMBER(2)
11
STOP_PENDING_STATUS
NUMBER(2)
12
SUSPEND_PENDING_STATUS
NUMBER(2)
13
INACTIVE_STATUS
NUMBER(2)
14
QUEUED_STATUS
NUMBER(2)
15
FAILED_RETRIED_STATUS
NUMBER(2)
16
WAITING_STATUS
NUMBER(2)
17
SKIPPED_STATUS
NUMBER(2)
18
REASSIGNED_STATUS
NUMBER(2)
20
Using Notifications 4-51
Passing Job Execution Status Information
4.8 Passing Job Execution Status Information
Passing job status change attributes (such as new status, job name, job type, or rule
owner) to PL/SQL procedures or OS commands/scripts allows you to customize
automated responses to status changes. For example, you many want to call an OS
script to open a trouble ticket for an in-house support trouble ticket system if a critical
job fails to run. In this case you will want to pass status (for example, Problems or
Aborted) to the script to open a trouble ticket and escalate the problem. The job
execution status information is one of event type - job_status_change event, and its
content is in OS command and PL/SQL payload as described in Section 4.3, "Sending
Notifications Using OS Commands and Scripts" and Section 4.4, "Sending
Notifications Using PL/SQL Procedures".
4.8.1 Passing Job Execution Status to a PL/SQL Procedure
The notification system passes job status change information to a PL/SQL procedure
via the event_msg.event_payload object where event_type is equal to job_status_
change. An instance of this object is created for every status change. When a job
changes status, the notification system calls the PL/SQL p(event_msg IN
gc$notif_event_msg) procedure associated with the incident rule and passes the
populated object to the procedure. The procedure is then able to access the fields of the
event_msg.event_payload object that has been passed to it.
Table 4–15 lists all corrective action status change attributes that can be passed:
Table 4–15
Job Status Attributes
Attribute
Datatype
Additional Information
event_msg.event_
payload.source.source_
name
VARCHAR2(128)
The job name.
event_msg.event_
payload.source.source_
owner
VARCHAR2(256)
The owner of the job.
event_msg.event_
payload.source.source_
sub_type
VARCHAR2(32)
The type of the job.
event_msg.event_
payload. event_
attrs(i).value where
event_attrs(i).name='
execution_status'
NUMBER
The new status of the job.
event_msg.event_
payload. event_
attrs(i).value where
event_
attrs(i).name='state_
change_guid'
RAW(16)
The GUID of the state change record.
event_msg.event_
payload.source.source_
guid
RAW(16)
The unique id of the job.
event_msg target.event_ RAW(16)
payload. event_
attrs(i).value where
event_attrs(i).name='
execution_id'
4-52 OracleВ® Enterprise Manager Administration
The unique id of the execution.
Passing Job Execution Status Information
Table 4–15 (Cont.) Job Status Attributes
Attribute
Datatype
Additional Information
event_msg.event_
payload.target
gc$notif_target
Target Information object..
event_msg.msg_
info.rule_owner
VARCHAR2(64)
The name of the notification rule that cause
the notification to be sent.
event_msg.msg_
info.rule_name
VARCHAR2(132)
The owner of the notification rule that cause
the notification to be sent.
event_msg.event_
payload. reported_date
DATE
The time and date when the status change
happened.
When a job status change occurs for the job, the notification system creates an instance
of the event_msg.event_payload. event_attrs(i).value where event_attrs(i).name='
execution_status' object and populates it with values from the status change. The
following status codes have been defined as constants in the MGMT_JOBS package
and can be used to determine the type of status in the job_status field of the event_
msg.event_payload. event_attrs(i).value where event_attrs(i).name=' execution_status'
object.
Table 4–16
Job Status Codes
Name
Datatype
Value
SCHEDULED_STATUS
NUMBER(2)
1
EXECUTING_STATUS
NUMBER(2)
2
ABORTED_STATUS
NUMBER(2)
3
FAILED_STATUS
NUMBER(2)
4
COMPLETED_STATUS
NUMBER(2)
5
SUSPENDED_STATUS
NUMBER(2)
6
AGENTDOWN_STATUS
NUMBER(2)
7
STOPPED_STATUS
NUMBER(2)
8
SUSPENDED_LOCK_STATUS
NUMBER(2)
9
SUSPENDED_EVENT_STATUS
NUMBER(2)
10
SUSPENDED_BLACKOUT_STATUS
NUMBER(2)
11
STOP_PENDING_STATUS
NUMBER(2)
12
SUSPEND_PENDING_STATUS
NUMBER(2)
13
INACTIVE_STATUS
NUMBER(2)
14
QUEUED_STATUS
NUMBER(2)
15
FAILED_RETRIED_STATUS
NUMBER(2)
16
WAITING_STATUS
NUMBER(2)
17
SKIPPED_STATUS
NUMBER(2)
18
REASSIGNED_STATUS
NUMBER(2)
20
Example 4–16
CREATE
PL/SQL Procedure Using a Status Code (Job)
TABLE job_log (jobid RAW(16), status_code NUMBER(2), occured DATE);
CREATE OR REPLACE PROCEDURE LOG_JOB_STATUS_CHANGE(event_msg IN GC$NOTIF_EVENT_MSG)
Using Notifications 4-53
Passing Job Execution Status Information
IS
l_attrs gc$notif_event_attr_array;
exec_status_code NUMBER(2) := NULL;
occured_date DATE := NULL;
job_guid RAW(16) := NULL;
BEGIN
IF event_msg.event_payload.event_type = 'job_status_change'
THEN
l_attrs := event_msg.event_payload.event_attrs;
IF l_attrs IS NOT NULL
THEN
FOR i IN 1..l_attrs.COUNT
LOOP
IF l_attrs(i).name = 'exec_status_code'
THEN
exec_status_code := TO_NUMBER(l_attrs(i).value);
END IF;
END LOOP;
END IF;
occured_date := event_msg.event_payload.reported_date;
job_guid := event_msg.event_payload.source.source_guid;
-- Log all jobs' status
BEGIN
INSERT INTO job_log (jobid, status_code, occured)
VALUES (job_guid, exec_status_code, occured_date);
EXCEPTION
WHEN OTHERS
THEN
-- If there are any problems then get the notification retried
RAISE_APPLICATION_ERROR(-20000, 'Please retry');
END;
COMMIT;
ELSE
null; -- it is not a job_status_change event, ignore
END IF;
END LOG_JOB_STATUS_CHANGE;
/
4.8.2 Passing Job Execution Status to an OS Command or Script
The notification system passes job execution status information to an OS script or
executable via system environment variables. Conventions used to access
environmental variables vary depending on the operating system:
в– UNIX: $ENV_VARIABLE
в– MS Windows: %ENV_VARIABLE%
The notification system sets the following environment variables before calling the
script. The script can then use any or all of these variables within the logic of the script.
Table 4–17
Environment Variables
Environment Variable
Description
SOURCE_OBJ_NAME
The name of the job.
SOURCE_OBJ_OWNE
The owner of the job.
4-54 OracleВ® Enterprise Manager Administration
Notification Reference
Table 4–17 (Cont.) Environment Variables
Environment Variable
Description
SOURCE_OBJ_SUB_TYPE The type of job.
EXEC_STATUS_CODE
The job status.
EVENT_REPORTED_
TIME
Time when the severity occurred.
TARGET_NAME
The name of the target.
TARGET_TYPE
The type of the target.
RULE_NAME
Name of the notification rule that resulted in the severity.
RULE_OWNER
Name of the Enterprise Manager administrator who owns the
notification rule.
4.9 Passing User-Defined Target Properties to Notification Methods
Enterprise Manager allows you to define target properties (accessed from the target
home page) that can be used to store environmental or usage context information
specific to that target. Target property values are passed to custom notification
methods where they can be processed using conditional logic or simply passed as
additional alert information to third-party devices, such as ticketing systems. By
default, Enterprise Manager passes all defined target properties to notification
methods.
Target properties are not passed to notification methods when
short e-mail format is used.
Note:
Figure 4–8 Host Target Properties
4.10 Notification Reference
This section contains the following reference material:
в– EMOMS Properties
в– Passing Event, Incident, Problem Information to an OS Command or Script
Using Notifications 4-55
Notification Reference
в– Passing Information to a PL/SQL Procedure
в– Troubleshooting Notifications
4.10.1 EMOMS Properties
EMOMS properties can be used for controlling the size and format of the short e-mail.
The following table lists emoms properties for Notification System.
Table 4–18
emoms Properties for Notifications
Default
Property Name
Value
Description
oracle.sysman.core.notification.em 250
ails_per_minute
Email delivery limits per minute. The
Notification system uses this value to throttle
number of Email delivery per minutes.
Customer should set the value lower if doesn't
want to over flow the Email server, or set the
value higher if the Email server can handle
high volume of Emails.
oracle.sysman.core.notification.cm 100
ds_per_minute
OS Command delivery limits per minute. The
Notification system uses this value to throttle
number of OS Command delivery per minutes.
oracle.sysman.core.notification.os_ 30
cmd_timeout
OS Command delivery timeout in seconds.
This value indicates how long to allow OS
process to execute the OS Command delivery.
Set this value higher if the OS command script
requires longer time to complete execution.
oracle.sysman.core.notification.pls 250
ql_per_minute
PL/SQL delivery limits per minute. The
Notification system uses this value to throttle
number of PL/SQL delivery per minutes.
em.notification.java_per_minute
500
JAVA delivery limits per minute. The
Notification system uses this value to throttle
number of Java delivery per minutes.
em.notification.ticket_per_minute
250
Ticket delivery limits per minute. The
Notification system uses this value to throttle
number of Ticket delivery per minutes.
oracle.sysman.core.notification.tra 250
ps_per_minute
SNMP delivery limits per minute. The
Notification system uses this value to control
the number of SNMP trap per minutes.
4-56 OracleВ® Enterprise Manager Administration
Notification Reference
Table 4–18 (Cont.) emoms Properties for Notifications
Default
Property Name
Value
oracle.sysman.core.notification.loc OMS
ale.plsql
Locale
Description
This property specifies the Locale delivered by
advanced PL/SQL notification. The customer
can define this property to overwrite the
default Locale where the OMS is installed.
Valid Locales:
oracle.sysman.core.notification.loc OMS
ale.email
Locale
в– en (English)
в– de (German)
в– es (Spanish)
в– fr (French)
в– it (Italian)
в– ja (Japanese)
в– ko (Korean)
в– pt_br (Portuguese, Brazilian)
в– zh_cn (Chinese, simplified)
в– zh_tw (Chinese, traditional)
This property specifies the Locale delivered by
Email. Customer can define this property to
overwrite the default Locale where the OMS is
installed.
Valid Locales:
в– en (English)
в– de (German)
в– es (Spanish)
в– fr (French)
в– it (Italian)
в– ja (Japanese)
в– ko (Korean)
в– pt_br (Portuguese, Brazilian)
в– zh_cn (Chinese, simplified)
в– zh_tw (Chinese, traditional)
Using Notifications 4-57
Notification Reference
Table 4–18 (Cont.) emoms Properties for Notifications
Default
Property Name
Value
oracle.sysman.core.notification.loc OMS
ale.oscmd
Locale
Description
This property specifies the Locale delivered by
OS Command. Customer can define this
property to overwrite the default Locale where
the OMS is installed.
Valid Locales:
oracle.sysman.core.notification.loc OMS
ale.snmp
Locale
в– en (English)
в– de (German)
в– es (Spanish)
в– fr (French)
в– it (Italian)
в– ja (Japanese)
в– ko (Korean)
в– pt_br (Portuguese, Brazilian)
в– zh_cn (Chinese, simplified)
в– zh_tw (Chinese, traditional)
This property specifies the Locale delivered by
SNMP trap. Customer can define this property
to overwrite the default Locale where the OMS
is installed.
Valid Locales:
в– en (English)
в– de (German)
в– es (Spanish)
в– fr (French)
в– it (Italian)
в– ja (Japanese)
в– ko (Korean)
в– pt_br (Portuguese, Brazilian)
в– zh_cn (Chinese, simplified)
в– zh_tw (Chinese, traditional)
oracle.sysman.core.notification.osc 512
md.max_env_var_length
The maximum length of OS Common
environment variable value.
oracle.sysman.core.notification.sn
mp.max_oid_length
2560
The maximum length of SNMP OID value.
oracle.sysman.core.notification.mi
n_delivery_threads
6
The minimum number of active threads in the
thread pool initially and number of active
threads are running when system is in low
activities. Setting the value higher will use
more system resources, but will deliver more
notifications.
oracle.sysman.core.notification.ma 24
x_delivery_threads
4-58 OracleВ® Enterprise Manager Administration
The maximum number of active threads in the
thread pool when the system is in the high
activities. This value should greater than
em.notification.min_delivery_threads. Setting
the value higher will use more system
resources and deliver more notifications.
Notification Reference
Table 4–18 (Cont.) emoms Properties for Notifications
Default
Property Name
Value
Description
oracle.sysman.core.notification.sh
ort_format_length
>=1
(155)
The size limit of the total number of characters
in short email format. The customer should
modify this property value to fit their email or
pager limit content size. The email subject is
restricted to a maximum of 80 characters for
short email format notifications.
oracle.sysman.core.notification.sn
mp_packet_length
<=1
(5120)
The maximum size of SNMP Protocol Data
unit.
oracle.sysman.core.notification.em 8-bit,
ail_content_transfer_encoding
7-bit(Q
P),
7-bit(B
ASE64)
(8-bit)
The character set that can encode the Email.
Oracle supports three character sets : 8-bit,
7-bit(QP), and7-bit(BASE64).
oracle.sysman.core.notification.em >=1 (20) The maximum number of emails delivered to
ails_per_connection
same email gateway before switching to the
next available email gateway (assumes
customers have configured multiple email
gateways). This property is used for email
gateway load balance.
oracle.sysman.core.notification.sh
ort_format
both,
subject,
body
(both)
Use short format on both subject and body,
subject only, or body only..
You must establish the maximum size your device can support and whether the
message is sent in subject, body or both.
You can modify the emoms properties by using the Enterprise Manager command line
control emctl get/set/delete/list property command.
The following commands require an OMS restart in order for
the changes to take place.
Note:
Get Property Command
emctl get [-sysman_pwd "sysman password"]-name
oracle.sysman.core.notification.short_format_length
Set Property Command
emctl set property -name oracle.sysman.core.notification.short_format_length
-value 155
Emoms Properties Entries for a Short E-mail Format
emctl set property -name oracle.sysman.core.notification.short_format_length
-value 155
emctl set property -name oracle.sysman.core.notification.short_format -value both
4.10.2 Passing Event, Incident, Problem Information to an OS Command or Script
The notification system passes information to an OS script or executable using system
environment variables.
Using Notifications 4-59
Notification Reference
Conventions used to access environmental variables vary depending on the operating
system:
в– UNIX: $ENV_VARIABLE
в– Windows:%ENV_VARIABLE%
The notification system sets the following environment variables before calling the
script. The script can then use any or all of these variables within the logic of the script.
4.10.2.1 Environment Variables Common to Event, Incident and Problem
Table 4–19
Generic Environment Variables
Environment Variable
NOTIF_TYPE
Description
Type of notification and possible values
NOTIF_NORMAL,
NOTIF_RETRY,
NOTIF_DURATION,
NOTIF_REPEAT,
NOTIF_CA,
NOTIF_RCA
REPEAT_COUNT
How many times the notification has been sent out
before this notification.
RULESET_NAME
The name of the ruleset that triggered this notification.
RULE_NAME
The name of the rule that triggered this notification.
RULE_OWNER
The owner of the ruleset that triggered this notification.
MESSAGE
The message of the event, incident, or problem.
MESSAGE_URL
EM console URL for this message.
Table 4–20
Category-Related Environment Variables
Environment Variable
Description
CATEGORIES_COUNT
Number of categories in this notification. This value is equal to1 if
one category is associated with event, incident or problem. It is
equal to 0 if no category associated with event, incident or
problem.
CATEGORY_CODES_
COUNT
Number of category codes in this notification.
CATEGORY_n
Category is translated based on locale defined in OMS server.
Valid values for the suffix "_n" are between 1.. $CATEGORIES_
COUNT
CATEGORY_CODE_n
Codes for the categories. Valid values for the suffix "_n" are
between 1..$CATEGORY_CODES_COUNT
Table 4–21 lists the common environment variables for User Defined Target Properties.
They will be populated under the following cases: (a) When an event has a related
target, (b) When an incident or a problem have single event source and have a related
target.
4-60 OracleВ® Enterprise Manager Administration
Notification Reference
Table 4–21
User-Defined Target Property Environment Variables
Environment Variable
Description
ORCL_GTP_COMMENT
Comment
ORCL_GTP_CONTACT
Contact
ORCL_GTP_COST_
CENTER
Cost Center
ORCL_GTP_
DEPARTMENT
Department
ORCL_GTP_
DEPLOYMENT_TYPE
Deployment type
ORCL_GTP_LINE_OF_
BUS
Line of Business
ORCL_GTP_LOCATION
Location
4.10.2.2 Event Notification-Specific Environment Variables
Table 4–22
Event Notification-Specific Environment Variables
Environment Variable
Description
EVENT_NAME
Event Name.
EVENT_REPORTED_
TIME
Event reported date.
EVENT_SOURCE_
COUNT
Number of Sources associated with this event.
EVENT_TYPE
Event type.
EVENT_OCCURRENCE_
TIME
Event occurrence time.
EVENT_TYPE_ATTRS
The list of event type specific attributes.
EVENT_CONTEXT_
ATTRS
Event context data.
LAST_UPDATED_TIME
Last updated time
SEQUENCE_ID
The unique event sequence identifier. An event sequence may
consist of one or more events. All events in this sequence have the
same event sequence ID.
SEVERITY
Severity of event, it is translated.
SEVERITY_CODE
Code for event severity. Possible values are the following.
FATAL, CRITICAL, WARNING, MINOR_WARNING,
INFORMATIONAL, and CLEAR
ACTION_MSG
Message describing the action to take for resolving the event.
TOTAL_OCCURRENCE_
COUNT
Total number of duplicate occurrences
RCA_DETAILS
If RCA is associated with this events.
CURRENT_
OCCURRENCE_COUNT
Total number of occurrences of the event in the current collection
period. This attribute only applies to de-duplicated events.
CURRENT_FIRST_
OCCUR_DATE
Time stamp when the event first occurred in the current collection
period. This attribute only applies to de-duplicated events.
Using Notifications 4-61
Notification Reference
Table 4–22 (Cont.) Event Notification-Specific Environment Variables
Environment Variable
Description
CURRENT_LAST_
OCCUR_DATE_DESC
Time stamp when the e vent last occurred in the current collection
period. This attribute only applies to de-duplicated events.
Table 4–23 lists the environment variables for the incident associated with an event.
They are populated when the event is associated with an incident.
Table 4–23
Associated Incident Environment Variables
Environment Variable
Description
ASSOC_INCIDENT_
ACKNOWLEDGED_BY_
OWNER
Set to yes, if associated incident was acknowledged by owner
ASSOC_INCIDENT_
ACKNOWLEDGED_
DETAILS
The details of associated incident acknowledgement. For example:
No - if not acknowledged
Yes By userName - if acknowledged
ASSOC_INCIDENT_
STATUS
Associated Incident Status
ASSOC_INCIDENT_ID
Associated Incident ID
ASSOC_INCIDENT_
PRIORITY
Associated Incident priority. Supported value are Urgent, Very
High, High, Medium,Low, None.
ASSOC_INCIDENT_
OWNER
Associated Incident Owner if it is existed.
ASSOC_INCIDENT_
ESCALATION_LEVEL
Escalation level of the associated incident has a value between 0
to 5.
Table 4–24 lists the common environment variables related to the Source Object. They
are populated when $SOURCE_OBJ_TYPE is not TARGET.
Table 4–24
Source Object-Related Environment Variables
Environment Variable
Description
SOURCE_OBJ_TYPE
Type of the Source object. For example, JOB, TEMPLATE.
SOURCE_OBJ_NAME
Source Object Name.
SOURCE_OBJ_NAME_
URL
Source's event console URL.
SOURCE_OBJ_SUB_TYPE Sub-type of the Source object. For example, it provides the
underlying job type for job status change events.
SOURCE_OBJ_OWNER
Owner of the Source object.
Table 4–25 lists the common environment variables for the target, associated with the
given issue. They are populated when the issue is related to a target.
Table 4–25
Target-Related Environment Variables
Environment Variable
Description
TARGET_NAME
Name of Target
TARGET_TYPE
Type of Target
4-62 OracleВ® Enterprise Manager Administration
Notification Reference
Table 4–25 (Cont.) Target-Related Environment Variables
Environment Variable
Description
TARGET_OWNER
Owner of Target
HOST_NAME
The name of the host on which the target is deployed upon.
TARGET_URL
Target's Enterprise Manager Console URL.
TARGET_LIFECYCLE_
STATUS
Life Cycle Status of the target.
Possible values: Production, Mission Critical, Stage, Test, and
Development.
It is null if not defined.
TARGET_VERSION
Target Version of the target
4.10.2.3 Environment Variables Specific to Event Types
Events are classified into multiple types. For example, the mertc_alert event type is
used for modeling metric alerts. You can use SQL queries to list the event types in your
deployment as well as their event-specific payload. The following SQL example can be
used to list all internal event type names that are registered in Enterprise Manager.
Select event_class as event_type, upper(name) as env_var_name
from em_event_class_attrs
where notif_order != 0
and event_class is not null
union
select event_class as event_type, upper(name) || '_NLS' as env_var_name
from em_event_class_attrs
where notif_order != 0
and event_class is not null
and is_translated = 1
order by event_type, env_var_name;
The environment variable payload specific to each event type can be accessed via the
OS scripts. The following tables list notification attributes for the most critical event
types.
Table 4–26
Environment Variables Specific to Metric Alert Event Type
Environment Variable
Description
COLL_NAME
The name of the collection collecting the metric.
COLL_NAME_NLS
The translated name of the collection collecting the metric
KEY_COLUMN_X
Internal name of Key Column X where X is a number between 1
and 7.
KEY_COLUMN_X_NLS
Translated name of Key Column X where X is a number between
1 and 7.
KEY_COLUMN_X_
VALUE
Value of Key Column X where X is a number between 1 and 7.
KEY_VALUE
Monitored object for the metric corresponding to the Metric Alert
event.
METRIC_COLUMN
The name of the metric column
METRIC_COLUMN_NLS
The translated name of the metric column.
METRIC_DESCRIPTION
Brief description of the metric.
Using Notifications 4-63
Notification Reference
Table 4–26 (Cont.) Environment Variables Specific to Metric Alert Event Type
Environment Variable
Description
METRIC_DESCRIPTION_
NLS
Translated brief description of the metric.
METRIC_GROUP
The name of the metric.
METRIC_GROUP_NLS
The translated name of the metric
NUM_KEYS
The number of key metric columns in the metric.
SEVERITY_GUID
The GUID of the severity record associated with this metric alert.
CYCLE_GUID
A unique identifier for a metric alert cycle, which starts from the
time the metric alert is initially generated until the time it is clear.
VALUE
Value of the metric when the event triggered.
Table 4–27
Environment variables specific to Target Availability Event Type
Environment Variable
Description
AVAIL_SEVERITY
The transition severity that resulted in the status of the target to
change to the current availability status.
Possible Values for AVAIL_SEVERITY
в– 15 (Target Up)
в– 25 (Target Down)
в– 115 (Agent Unreachable, Cleared)
в– 125 (Agent Unreachable)
в– 215 (Blackout Ended)
в– 225 (Blackout Started)
в– 315 (Collection Error Cleared)
в– 325 (Collection Error)
в– 425 (No Beacons Available)
в– 515 (Status Unknown)
AVAIL_SUB_STATE
The substatus of a target for the current status.
CYCLE_GUID
A unique identifier for a metric alert cycle, which starts from the
time the metric alert is initially generated until the time it is clear.
METRIC_GUID
Metric GUID of response metric.
SEVERITY_GUID
The GUID of the severity record associated with this availability
status.
TARGET_STATUS
The current availability status of the target.
TARGET_STATUS_NLS
The translated current availability status of the target.
Table 4–28
Environment variables specific to Job Status Change event type
Environment Variable
Description
EXECUTION_ID
Unique ID of the job execution..
EXECUTION_LOG
The job output of the last step executed.
4-64 OracleВ® Enterprise Manager Administration
Notification Reference
Table 4–28 (Cont.) Environment variables specific to Job Status Change event type
Environment Variable
Description
EXECUTION_STATUS
The internal status of the job execution.
EXECUTION_STATUS_
NLS
The translated status of the job execution.
EXEC_STATUS_CODE
Execution status code of job execution. For possible values, see
Table 4–16, " Job Status Codes".
STATE_CHANGE_GUID
Unique ID of last status change
You can use SQL queries to list the deployed event types in your deployment and the
payload specific to each one of them. The following SQL can be used to list all internal
event type names which are registered in the Enterprise Manager.
select class_name as event_type_name from em_event_class;
Following SQL lists environment variables specific to metric_alert event type.
select env_var_name
from
( Select event_class as event_type, upper(name) as env_var_name
from em_event_class_attrs
where notif_order != 0
and event_class is not null
union
select event_class as event_type, upper(name) || '_NLS' as env_var_name
from em_event_class_attrs
where notif_order != 0
and event_class is not null
and is_translated = 1)
where event_type = 'metric_alert';
You can also obtain the description of notification attributes specific to an event type
directly from the Enterprise Manager console:
1.
From the Setup menu, select Notifications, then select Customize Email Formats.
2.
Select the event type.
3.
Click Customize.
4.
Click Show Predefined Attributes.
Environment variables, ending with the suffix _NLS, provide the translated value for
given attribute. For example, METRIC_COLUMN_NLS environment variable will
provide the translated value for the metric column attribute. Translated values will be
in the locale of the OMS.
4.10.2.4 Environment Variables Specific to Incident Notifications
Table 4–29
Incident-Specific Environment Variables
Environment Variable
Description
SEVERITY
Incident Severity, it is translated. Possible Values: Fatal, Critical,
Warning, Informational, Clear
Using Notifications 4-65
Notification Reference
Table 4–29 (Cont.) Incident-Specific Environment Variables
Environment Variable
Description
SEVERITY_CODE
Code for Severity.
Possible values are the
FATAL, CRITICAL, WARNING, MINOR_WARNING,
INFORMATIONAL, and CLEAR
INCIDENT_REPORTED_
TIME
Incident reported time
INCIDENT_
ACKNOWLEDGED_BY_
OWNER
Set yes, if incident is acknowledged by owner.
INCIDENT_ID
Incident ID
INCIDENT_OWNER
Incident Owner
ASSOC_EVENT_COUNT
The number events associated with this incident.
INCIDENT_STATUS
Incident status. There are two internal fixed resolution status.
NEW
CLOSED
Users can define additional statuses.
ESCALATED
Is Incident escalated
ESCALATED_LEVEL
The escalated level of incident.
PRIORITY
Incident priority. It is the translated priority name. Possible
Values: Urgent, Very High, High, Medium, Low, None
PRIOTITY_CODE
Incident priority code
It is the internal value defined in EM.
PRIORITY_URGENT
PRIORITY_VERY_HIGH
PRIORITY_HIGH
PRIORITY_MEDIUM
PRIORITY_LOW
PRIORITY_NONE
TICKET_STATUS
Status of external ticket, if it exists.
TICKET_ID
ID of external ticket, if it exists.
LAST_UPDATED_TIME
Incident last update time.
ADR_INCIDENT_ID
Automatic Diagnostic Reposito ry (ADR) Incident ID: A unique
numeric identifier for the ADR Incident. An ADR I ncident is an
occurrence of a Problem.
ADR_IMPACT
Impact of the Automatic Diagnostic Repository (ADR) Incident.
ADR_ECID
Execution Context ID (ECID) associated with the associated
Automatic Diagnostic Repository (ADR) incident. An ECID i s a
globally unique identifier used to tag and track a single call
through the Oracle software stack. It is used to correlate problems
that could occur across multiple tiers of the stack.
ASSOC_PROBLEM_KEY
Problem key associated with the Automatic Diagnostic
Repository (ADR) incident. Problems are critical error s in an
Oracle product. The Problem key is a text string that describes the
prob lem. It includes an error code and in some cases, other
error-specific values.
4-66 OracleВ® Enterprise Manager Administration
Notification Reference
Table 4–30 lists the associated problem's environment variables, when the incident is
associated with a problem.
Table 4–30
Associated Problem Environment Variables for Incidents
Environment Variable
Description
ASSOC_PROBLEM_
ACKNOWLEDGED_BY_
OWNER
Set to yes, if this problem was acknowledged by owner
ASSOC_PROBLEM_
STATUS
Associated Problem Status
ASSOC_PROBLEM_ID
Associated Problem ID
ASSOC_PROBLEM_
PRIORITY
Associated Problem priority
ASSOC_PROBLEM_
OWNER
Associated Problem Owner if it is existed.
ASSOC_PROBLEM_
ESCALATION_LEVEL
Escalation level of the associated Problem has a value between 0
to 5.
4.10.2.5 Environment Variables Specific to Problem Notifications
Table 4–31
Problem-Specific Environment Variables
Environment Variable
Description
SEVERITY
Problem Severity, it is translated.
SEVERITY_CODE
Code for Severity.
Possible values are :
FATAL, CRITICAL, WARNING, MINOR_WARNING,
INFORMATIONAL, and CLEAR
PROBLEM_REPORTED_
TIME
Problem reported time.
PROBLEM_
ACKNOWLEDGED_BY_
OWNER
Set yes, if problem is acknowledged by owner.
PROBLEM_ID
Problem ID
PROBLEM_KEY
Problem Key
PROBLEM_OWNER
Problem Owner
ASSOC_INCIDENT_
COUNT
The number incident associated with this problem..
PROBLEM_STATUS
Incident status. They are
STATUS_NEW
STATUS_CLOSED
Any other user defined status.
ESCALATED
Is Incident escalated. Yes if it is escalated, otherwise No.
ESCALATED_LEVEL
The escalated level of incident.
PRIORITY
Incident priority. It is the translated priority name..
Using Notifications 4-67
Notification Reference
Table 4–31 (Cont.) Problem-Specific Environment Variables
Environment Variable
Description
PRIOTITY_CODE
Incident priority code
It is the internal value defined in Enterprise Manager.
PRIORITY_URGENT
PRIORITY_VERY_HIGH
PRIORITY_HIGH
PRIORITY_MEDIUM
PRIORITY_LOW
PRIORITY_NONE
LAST_UPDATED_TIME
Last updated time
SR_ID
Oracle Service Request Id, if it exists.
BUD_ID
Oracle Bug ID, if an associated bug exists.
4.10.2.6 Environment Variables Common to Incident and Problem Notifications
An incident or problem may be associated with multiple event sources. An event
source can be a Target, a Source Object, or both.
4.10.2.6.1 Environment Variables Related to Event Sources The number of event sources is
set by the EVENT_SOURCE_COUNT environment variable. Using the EVENT_
SOURCE_COUNT information, a script can be written to loop through the relevant
environment variables to fetch the information about multiple event sources.
Environment variables for all event sources are prefixed with EVENT_SOURCE_.
Environment variables for source objects are suffixed with SOURCE_<attribute_
name> . For example, EVENT_SOURCE_1_SOURCE_TYPE provides the source object
type of first event source. Environment variables for a target are suffixed with
TARGET_<attribute_name>. For example, EVENT_SOURCE_1_TARGET_NAME
provides the target name of first event source.
The following table lists the environment variables for source object of x-th Event
Source.
Table 4–32
Source Object of the x-th Event Source
Environment Variable
Description
EVENT_SOURCE_x_
SOURCE_GUID
Source Object GUID.
EVENT_SOURCE_x_
SOURCE_TYPE
Source Object Type
EVENT_SOURCE_x_
SOURCE_NAME
Source Object Name.
EVENT_SOURCE_x_
SOURCE_OWNER
Source Object Owner.
EVENT_SOURCE_x_
SOURCE_SUB_TYPE
Source Object Sub-Type.
EVENT_SOURCE_x_
SOURCE_URL
Source Object URL to EM console.
Table 4–33 lists the environment variables for a target of xth Event Source.
4-68 OracleВ® Enterprise Manager Administration
Notification Reference
Table 4–33
Target of x-th Event Source
Environment Variable
Description
EVENT_SOURCE_x_
TARGET_GUID
Target GUID
EVENT_SOURCE_x_
TARGET_NAME
Target name
EVENT_SOURCE_x_
TARGET_OWNER
Target Owner
EVENT_SOURCE_x_
TARGET_VERSION
Target version
EVENT_SOURCE_x_
TARGET_LIFE_CYCLE_
STATUS
Target life cycle status
EVENT_SOURCE_x_
TARGET_TYPE
Target Type
EVENT_SOURCE_x_
HOST_NAME
Target Host Name
EVENT_SOURCE_x_
TARGET_URL
Target URL to EM Console.
4.10.3 Passing Information to a PL/SQL Procedure
Passing event, incident, and problem information (payload) to PL/SQL procedures
allows you to customize automated responses to these conditions. All three types of
notification payloads have a common element: gc$notif_msg_info. It provides
generic information that applies to all types of notifications. In addition, each of the
three payloads have one specific element that provides the payload specific to the
given issue type.
gc$notif_event_msg (payload for event notifications)
gc$notif_event_msg contains two objects - event payload object and message
information object.
Table 4–34
Event Notification Payload
Attribute
Datatype
Additional Information
EVENT_PAYLOAD
gc$notif_event_
payload
Event notification payload. See gc$notif_
event_payload type definition for detail.
MSG_INFO
gc$notif_msg_info
Notification message. See gc$notif_msg_info
definition for detail.
gc$notif_incident_msg (payload for incident notifications)
gc$notif_incident_msg type contains two objects - incident payload and message
information. This object represents the delivery payload for Incident notification
message, contains all data associated with Incident notification, and can be accessed by
user's custom PL/SQL procedures.
Using Notifications 4-69
Notification Reference
Table 4–35
Incident Notification Payload
Attribute
Datatype
Additional Information
INCIDENT_PAYLOAD
gc$notif_incident_
payload
Incident notification payload. See gc$notif_
incident_payload type definition for detail.
MSG_INFO
gc$notif_msg_info
Envelope level notification information. See
gc$notif_msg_info type definition for detail.
gc$notif_problem_msg (payload for problem notifications)
This object represents the delivery payload for Problem notification message, contains
all data associated with problem notification, and can be accessed by a user's custom
PL/SQL procedures.
Table 4–36
Problem Notification Payload
Attribute
Datatype
Additional Information
PROBLEM_PAYLOAD
gc$notif_problem_
payload
Problem notification payload. See gc$notif_
problem_payload type definition for detail.
MSG_INFO
gc$notif_msg_info
Notification message. See gc$notif_msg_info
type definition for detail.
gc$notif_msg_info (common for event/incident/problem payloads)
This object contains the generic notification information including notification_type,
rule set and rule name, etc. for Event, Incident or Problem delivery payload.
Table 4–37
Event, Incident, Problem Common Payload
Attribute
Datatype
Description
NOTIFICATION_TYPE
VARCHAR2(32)
Type of notification, can be one of the
following values.
GC$NOTIFICATION.NOTIF_NORMAL
GC$NOTIFICATION.NOTIF_RETRY
GC$NOTIFICATION.NOTIF_REPEAT
GC$NOTIFICATION.NOTIF_DURATION
GC$NOTIFICATION.NOTIF_CA
GC$NOTIFICATION.NOTIF_RCA
REPEAT_COUNT
NUMBER
Repeat notification count
RULESET_NAME
VARCHAR2(256)
Name of the rule set that triggered the
notification
RULE_NAME
VARCHAR2(256)
Name of the rule that triggered the
notification
RULE_OWNER
VARCAH2(256)
EM User who owns the rule set
MESSAGE
VARCHAR2(4000)
Message about event/incident/problem.
MESSAGE_URL
VARCHAR2(4000)
Link to the Enterprise Manager console page
that provides the details of the
event/incident/problem.
gc$notif_event_payload (payload specific to event notifications)
4-70 OracleВ® Enterprise Manager Administration
Notification Reference
This object represents the payload specific to event notifications.
Table 4–38
Common Payloads for Events, Incidents, and Problems
Attribute
Datatype
Additional Information
EVENT_INSTANCE_
GUID
RAW(16)
Event instance global unique identifier.
EVENT_SEQUENCE_
GUID
RAW(16)
Event sequence global unique identifier.
TARGET
gc$notif_target
Related Target Information object. See
gc$notif_target type definition for detail.
SOURCE
gc$notif_source
Related Source Information object, that is not
a target. See gc$notif_source type definition
for detail.
EVENT_ATTRS
gc$notif_event_attr_
array
The list of event specified attributes. See
gc$notif_event_attr type definition for detail.
CORRECTIVE_
ACTION
gc$notif_corrective_
action_job
Corrective action information, optionally
populated when corrective action job
execution has completed.
EVENT_TYPE
VARCHAR2(20)
Event type - example: Metric Alert.
EVENT_NAME
VARCHAR2(512)
Event name.
EVENT_MSG
VARCHAR2(4000)
Event message.
REPORTED_DATE
DATE
Event reported date.
OCCURRENCE_DATE
DATE
Event occurrence date.
SEVERITY
VARCHAR2(128)
Event Severity. It is the translated severity
name.
SEVERITY_CODE
VARCHAR2(32)
Event Severity code. It is the internal severity
name used in Enterprise Manager.
ASSOC_INCIDENT
gc$notif_issue_
summary
Summary of associated incident. It is
populated if the event is associated with an
incident. See gc$notif_issue_summary type
definition for detail
ACTION_MSG
VARCHAR2(4000)
Message describing the action to take for
resolving the event.
RCA_DETAIL
VARCHAR2(4000)
Root cause analysis detail. The size of RCA
details output is limited to 4000 characters
long.
EVENT_CONTEXT_
DATA
gc$notif_event_
context_array
Event context data. See gc$notif_event_
context type definition for detail.
CATEGORIES
gc$category_string_
array
List of categories that the event belongs to.
Category is translated based on locale
defined in OMS server. Notification system
sends up to 10 categories.
CATEGORY_CODES
gc$category_string_
array
Codes for the categories. The size of array is
up to 10.
gc$notif_incident_payload (payload specific to incident notifications)
Contains the incident specific attributes, associated problem and ticket information.
Using Notifications 4-71
Notification Reference
Table 4–39
Incident Notification Payloads
Attribute
Datatype
Additional Information
INCIDENT_ATTRS
gc$notif_issue_attrs
Incident specific attributes. See gc$notif_
issue_attrs type definition for detail.
ASSOC_EVENT_
COUNT
NUMBER
The total number of events associated with
this incident.
TICKET_STATUS
VARCHAR2(64)
The status of external Ticket, if it exists.
TICKET_ID
VARCHAR2(128)
The ID of external Ticket, if it exists.
TICKET_URL
VARCHAR2(4000)
The URL for external Ticket, if it exists.
ASSOC_PROBLEM
gc$notif_issue_
summary
Summary of the problem, if it has an
associated problem. See gc$notif_issue_
summary type definition for detail.
gc$notif_problem_payload (payload specific to problems)
Contains problem specific attributes, key, Service Request(SR) and Bug information.
Table 4–40
Problem Payload
Attribute
Datatype
Additional Information
PROBLEM_ATTRS
gc$notif_issue_attrs
Problem specific attributes. See gc$notif_
issue_attrs type definition for detail.
PROBLEM_KEY
VARCHAR2(850)
Problem key if it is generated.
ASSOC_INCIDENT_
COUNT
NUMBER
Number of incidents associated with this
problem.
SR_ID
VARCHAR2(64)
Oracle Service Request Id, if it exists.
SR_URL
VARCHAR2(4000)
URL for Oracle Service Request, if it exists.
BUG_ID
VARCHAR2(64)
Oracle Bug ID, if an associated bug exists.
gc$notif_issue_attrs (payload common to incidents and problems)
Provides common details for incident and problem. It contains details such as id,
severity, priority, status, categories, acknowledged by owner, and source information
with which it is associated.
Table 4–41
Payload Common to Incidents and Problems
Attribute
Datatype
Additional Information
ID
NUMBER(16)
ID of the incident or problem.
SEVERITY
VARCHAR2(128)
Issue Severity. It is the translated.
SEVERITY_CODE
VARCHAR2(32)
Issue Severity Code.The possible values are
defined in descending order of severity:
GC$EVENT.FATAL
GC$EVENT.CRITICAL
GC$EVENT.WARNING
GC$EVENT.MINOR_WARNING
GC$EVENT.INFORMATIONAL
GC$EVENT.CLEAR
4-72 OracleВ® Enterprise Manager Administration
Notification Reference
Table 4–41 (Cont.) Payload Common to Incidents and Problems
Attribute
Datatype
Additional Information
PRIORITY
VARCHAR2(128)
Issue Priority. It is the translated priority
name.
PRIORITY_CODE
VARCHAR2(32)
Issue Priority. It is the internal value defined
in EM. The possible values are defined in
descending order of priority:
GC$EVENT.PRIORITY_URGENT
GC$EVENT.PRIORITY_VERY_HIGH
GC$EVENT.PRIORITY_HIGH
GC$EVENT.PRIORITY_MEDIUM
GC$EVENT.PRIORITY_LOW
GC$EVENT.PRIORITY_NONE
STATUS
VARCHAR2(32)
Status of Issue. The possible values are
GC$EVENT.STATUS_NEW
GC$EVENT.STATUS_CLOSED
Any other user defined status.
ESCALATION_LEVEL
NUMBER(1)
Escalation level of the issue, has a value
between 0 to 5.
OWNER
VARCHAR(256)
Issue Owner. Set to NULL if no owner exists.
ACKNOWLEDGED_
BY_OWNER
NUMBER(1)
Set to 1, if this issue was acknowledged by
owner.
CREATION_DATE
DATE
Issue creation date.
CLOSED_DATE
DATE
Issue closed date, null if not closed.
CATEGORIES
gc$category_string_
array
List of categories that the event belongs to.
Category is translated based on locale
defined in OMS server. Notification system
sends up to 10 categories.
CATEGORY_CODES
gc$category_string_
array
Codes for the categories. Notification system
sends up to 10 category codes.
SOURCE_INFO_ARR
gc$notif_source_
info_array
Array of source information associated with
this issue. See $gcnotif_source_info type
definition for detail.
LAST_MODIFIED_BY
VARCHAR2(256)
Last modified by user.
LAST_UPDATED_
DATE
DATE
Last updated date.
gc$notif_issue_summary (common to incident and problem payloads)
Represents the associated incident summary in the event payload, or associated
problem summary in the incident payload, respectively.
Table 4–42
Payload
Attribute
Datatype
Additional Information
ID
NUMBER
Issue Id, either Incident Id or Problem Id.
SEVERITY
VARCHAR(128)
The severity level of an issue. It is translated
severity name.
Using Notifications 4-73
Notification Reference
Table 4–42 (Cont.) Payload
Attribute
Datatype
Additional Information
SEVERITY_CODE
VARCHAR2(32)
Issue Severity Code, has one of the following
values.
GC$EVENT.FATAL
GC$EVENT.CRITICAL
GC$EVENT.WARNING
GC$EVENT.MINOR_WARNING
GC$EVENT.INFORMATIONAL
GC$EVENT.CLEAR
PRIORITY
VARCHAR2(128)
Current priority. It is the translated priority
name.
PRIORITY_CODE
VARCHAR2(32)
Issue priority code, has one of the following
values. GC$EVENT.PRIORITY_URGENT
GC$EVENT.PRIORITY_VERY_HIGH
GC$EVENT.PRIORITY_HIGH
GC$EVENT.PRIORITY_MEDIUM
GC$EVENT.PRIORITY_LOW
GC$EVENT.PRIORITY_NONE
STATUS
VARCHAR2(64)
Status of issue. The possible values are
GC$EVENT.STATUS_NEW
GC$EVENT.STATUS_CLOSED
GC$EVENT.WIP (work in progress)
GC$EVENT.RESOLVED
any other user defined status
ESCALATION_LEVEL
VARCHAR2(2)
Issue escalation level range from 0 to 5,
default 0.
OWNER
VARCHAR2(256)
Issue Owner. Set to NULL if no owner exists.
ACKNOWLEDGED_
BY_OWNER
NUMBER(1)
Set to 1, if this issue was acknowledged by
owner.
gc$category_string_array
gc$category_string_array is an array of string containing the categories which event,
incident or problem is associated with. The notification system delivers up to 10
categories.
gc$notif_event_context_array
gc$notif_event_context_array provides information about the additional diagnostic data
that was captured at event detection time. Note that notification system delivers up to
200 elements from the captured event context. Each element of this array is of the type
gc$notif_event_context.
gc$notif_event_context: This object represents the detail of event context data which is
additional contextual information captured by the source system at the time of event
generation that may have diagnostic value. The context for an event should consist of
a set of keys and values along with data type (Number or String only).
4-74 OracleВ® Enterprise Manager Administration
Notification Reference
Table 4–43
Event Context Type
Attribute
Datatype
Additional Information
NAME
VARCHAR2(256)
The event context name.
TYPE
NUMBER(1)
The data type of the value, which is stored
(0) - for numeric data
(1) - for string data.
VALUE
NUMBER
The numerical value.
STRING_VALUE
VARCHAR2(4000)
The string value.
gc$notif_corrective_action_job
Provides information about the execution of a corrective action job. Note that the
corrective actions are supported for metric alert and target availability events only.
Table 4–44
Corrective Action Job-Specific Attributes
Attribute
Datatype
Additional Information
JOB_GUID
RAW(16)
Corrective Action Job global unique
identifier.
JOB_NAME
VARCHAR2(128)
The value will be the name of the Corrective
Action. It applies to Metric Alert and Target
Availability Events.
JOB_OWNER
VARCHAR2(256)
Corrective action job owner.
JOB_TYPE
VARCHAR2(256)
Corrective action job type.
JOB_STATUS
VARCHAR2(64)
Corrective action job execution status.
JOB_STATUS_CODE
NUMBER
Corrective action job execution status code. It
is the internal value defined in Enterprise
Manager. For more information on status
codes, see Table 4–14, " Corrective Action
Status Codes".
JOB_STEP_OUTPUT
VARCHAR2(4000)
The value will be the text output from the
Corrective Action execution. This will be
truncated to last 4000 characters.
JOB_EXECUTION_
GUID
RAW(16)
Corrective Action Job execution global
unique identifier.
JOB_STATE_CHANGE_ RAW(16)
GUID
Corrective Action Job change global unique
identifier.
OCCURRED_DATE
Corrective action job occurred date.
DATE
gc$notif_source_info_array
Provides access to the multiple sources to which an incident or a problem could be
related. NOTE: The notification system delivers up to 200 sources associated with an
incident or a problem.
CREATE OR REPLACE TYPE gc$notif_source_info_array AS VARRAY(200)
OF gc$notif_source_info;
gc$notif_source_info
Notification source information which is used for referencing source information
containing either target or source, or both.
Using Notifications 4-75
Notification Reference
Table 4–45
Source Information Type
Attribute
Datatype
Additional Information
TARGET
gc$notif_target
It is populated when the event is related to a
target. See gc$notif_target type definition for
detail.
SOURCE
gc$notif_source
It is populated when the event is related to a
(non-target) source. See gc$notif_source type
definition for detail.
gc$notif_source
Used for referencing source objects other than a job target.
Table 4–46
Payload
Attribute
Datatype
Additional Information
SOURCE_GUID
RAW(16)
Source's global unique identifier.
SOURCE_TYPE
VARCHAR2(120)
Type of the Source object, e.g., TARGET, JOB,
TEMPLATE, etc.
SOURCE_NAME
VARCHAR2(256)
Source Object Name.
SOURCE_OWNER
VARCHAR2(256)
Owner of the Source object.
SOURCE_SUB_TYPE
VARCHAR2(256)
Sub-type of the Source object, for example,
within the TARGET these would be the
target types like Host, Database etc.
SOURCE_URL
VARCHAR2(4000)
Source's event console URL.
gc$notif_target
Target information object is used for providing target information.
Table 4–47
Target Information
Attribute
Datatype
Additional Information
TARGET_GUID
RAW(16)
Target's global unique identifier.
TARGET_NAME
VARCHAR2(256)
Name of target.
TARGET_OWNER
VARCHAR2(256)
Owner of target.
TARGET_LIFECYCLE_
STATUS
VARCHAR2(1024)
Life Cycle Status of the target.
TARGET_VERSION
VARCHAR2(64)
Target Version of the target.
TARGET_TYPE
VARCHAR2(128)
Type of a target.
TARGET_TIMEZONE
VARCHAR2(64)
Target's regional time zone.
HOST_NAME
VARCHAR2(256)
The name of the host on which the target is
deployed upon.
TARGET_URL
VARCHAR2(4000)
Target's EM Console URL.
UDTP_ARRAY
gc$notif_udtp_array
The list of user defined target properties. It is
populated for events that are associated with
a target. It is populated for incidents and
problems, when they are associated with a
single source (gc$notif_source_info).
4-76 OracleВ® Enterprise Manager Administration
Notification Reference
gc$notif_udtp_array
Array of gc$notif_udtp type with a maximum size of 20.
CREATE OR REPLACE TYPE gc$notif_udtp_array AS VARRAY(20) OF
gc$notif_udtp;
gc$notif_udtp
Used for referencing User-defined target properties. UDTP should consist of a set of
property key names and property values.
Table 4–48
Payload
Attribute
Datatype
Additional Information
NAME
VARCHAR2(64),
The name of property.
VALUE
VARCHAR2(1024)
Property value.
LABEL
VARCHAR(256)
Property label.
NLS_ID
VARCHAR(64)
Property nls id
4.10.3.1 Notification Payload Elements Specific to Event Types
gc$notif_event_attr_array
Array of gc$notif_event_attr is used for referencing event-specific attributes. The array
has a maximum size of 25. Each element of the array is of type gc$notif_event_attr (used
for referencing event type-specific attributes).
Table 4–49
Event Attribute Type
Attribute
Datatype
Additional Information
NAME
VARCHAR2(64)
The internal name of event type specific
attribute.
VALUE
VARCHAR2(4000)
value.
NLS_VALUE
VARCHAR2(4000)
Translated value for the attribute.
You can use SQL queries to list the deployed event types in your deployment and the
payload specific to each. The following SQL can be used to list all internal event type
names which are registered in the Enterprise Manager.
Select event_class as event_type, upper(name) as env_var_name
from em_event_class_attrs
where notif_order != 0
and event_class is not null
order by event_type, env_var_name;
You should convert the attribute name to upper case before using the name for
comparison.
There is an attribute variable payload specific to each event type that can be accessed
from a gc$notif_event_attr_array database type. The following tables list notification
attributes for the most critical event types. You should convert the attribute name to
uppercase before using the name for comparison.
Table 4–50
Environment variables specific to Metric Alert Event Type
Environment Variable
Description
COLL_NAME
The name of the collection collecting the metric.
Using Notifications 4-77
Notification Reference
Table 4–50 (Cont.) Environment variables specific to Metric Alert Event Type
Environment Variable
Description
KEY_COLUMN_X
Internal name of Key Column X where X is a number between 1
and 7.
KEY_COLUMN_X_
VALUE
Value of Key Column X where X is a number between 1 and 7.
KEY_VALUE
Monitored object for the metric corresponding to the Metric Alert
event.
METRIC_COLUMN
The name of the metric column
METRIC_DESCRIPTION
Brief description of the metric.
METRIC_GROUP
The name of the metric.
NUM_KEYS
The number of key metric columns in the metric.
SEVERITY_GUID
The GUID of the severity record associated with this metric alert.
VALUE
Value of the metric when the event triggered.
Table 4–51
Environment variables specific to Target Availability Event Type
Environment Variable
Description
AVAIL_SEVERITY
The transition severity (0-6) that resulted in the status of the target
to change to the current availability status.
Possible Values for AVAIL_SEVERITY
в– 0 (Target Down)
в– 1 (Target Up)
в– 2 (Target Status Error)
в– 3 (Agent Down)
в– 4 (Target Unreachable)
в– 5 (Target Blackout)
в– 6 (Target Status Unknown)
AVAIL_SUB_STATE
The substatus of a target for the current status.
CYCLE_GUID
A unique identifier for a metric alert cycle, which starts from the
time the metric alert is initially generated until the time it is clear.
METRIC_GUID
Metric GUID of response metric.
SEVERITY_GUID
The GUID of the severity record associated with this availability
status.
TARGET_STATUS
The current availability status of the target.
Table 4–52
Environment variables specific to Job Status Change event type
Environment Variable
Description
EXECUTION_ID
Unique ID of the job execution..
EXECUTION_LOG
The job output of the last step executed.
EXECUTION_STATUS
The internal status of the job execution.
EXEC_STATUS_CODE
Execution status code of job execution. For possible values, see
Table 4–16, " Job Status Codes".
STATE_CHANGE_GUID
Unique ID of last status change
4-78 OracleВ® Enterprise Manager Administration
Notification Reference
Example 4–17
PL/SQL Script: Event Type Payload Elements
-- log_table table is created by following DDL to demostrate how to access
-- event notification payload GC$NOTIF_EVENT_MSG.
CREATE TABLE log_table (message VARCHAR2(4000)) ;
-- Define PL/SQL notification method for Events
CREATE OR REPLACE PROCEDURE log_table_notif_proc(s IN GC$NOTIF_EVENT_MSG)
IS
l_categories gc$category_string_array;
l_category_codes gc$category_string_array;
l_attrs gc$notif_event_attr_array;
l_ca_obj gc$notif_corrective_action_job;
BEGIN
INSERT INTO log_table VALUES ('notification_type: ' || s.msg_info.notification_
type);
INSERT INTO log_table VALUES ('repeat_count: ' || s.msg_info.repeat_count);
INSERT INTO log_table VALUES ('ruleset_name: ' || s.msg_info.ruleset_name);
INSERT INTO log_table VALUES ('rule_name: ' || s.msg_info.rule_name);
INSERT INTO log_table VALUES ('rule_owner: ' || s.msg_info.rule_owner);
INSERT INTO log_table VALUES ('message: ' || s.msg_info.message);
INSERT INTO log_table VALUES ('message_url: ' || s.msg_info.message_url);
INSERT INTO log_table VALUES ('event_instance_guid: ' || s.event_payload.event_
instance_guid);
INSERT INTO log_table VALUES ('event_type: ' || s.event_payload.event_type);
INSERT INTO log_table VALUES ('event_name: ' || s.event_payload.event_name);
INSERT INTO log_table VALUES ('event_msg: ' || s.event_payload.event_msg);
INSERT INTO log_table VALUES ('source_obj_type: ' || s.event_
payload.source.source_type);
INSERT INTO log_table VALUES ('source_obj_name: ' || s.event_
payload.source.source_name);
INSERT INTO log_table VALUES ('source_obj_url: ' || s.event_
payload.source.source_url);
INSERT INTO log_table VALUES ('target_name: ' || s.event_payload.target.target_
name);
INSERT INTO log_table VALUES ('target_url: ' || s.event_payload.target.target_
url);
INSERT INTO log_table VALUES ('severity: ' || s.event_payload.severity);
INSERT INTO log_table VALUES ('severity_code: ' || s.event_payload.severity_
code);
INSERT INTO log_table VALUES ('event_reported_date: ' || to_char(s.event_
payload.reported_date, 'D MON DD HH24:MI:SS'));
IF s.event_payload.target.TARGET_LIFECYCLE_STATUS IS NOT NULL
THEN
INSERT INTO log_table VALUES ('target lifecycle_status: ' || s.event_
payload.target.TARGET_LIFECYCLE_STATUS);
END IF;
-- Following block illustrates the list of category codes to which the event
-- belongs.
l_category_codes := s.event_payload.category_codes;
IF l_categories IS NOT NULL
THEN
FOR c IN 1..l_category_codes.COUNT
LOOP
INSERT INTO log_table VALUES ('category_code ' || c || ' - ' || l_category_
codes(c));
END LOOP;
Using Notifications 4-79
Notification Reference
END IF;
--- Each event type has a specific set of attributes modeled. Examples of
-- event types include metric_alert, target_availability, job_status_change.
-- Following block illustrates how to access the attributes for job_status
change
-- event type
-IF s.event_payload.event_type = 'job_staus_chage'
THEN
l_attrs := s.event_payload.event_attrs;
IF l_attrs IS NOT NULL
THEN
FOR c IN 1..l_attrs.COUNT
LOOP
INSERT INTO log_table VALUES ('EV.ATTR name=' || l_attrs(c).name || '
value=' || l_attrs(c).value || ' nls_value=' || l_attrs(c).nls_value);
END LOOP;
END IF;
END IF;
-- Following block illustrates how to access corrective action job's attributes
IF s.msg_info.notification_type = GC$NOTIFICATION.NOTIF_CA AND s.event_
payload.corrective_action IS NOT NULL
THEN
l_ca_obj := s.event_payload.corrective_action;
INSERT INTO log_table VALUES ('CA JOB_GUID: ' || l_ca_obj.JOB_GUID);
INSERT INTO log_table VALUES ('CA JOB_NAME: ' || l_ca_obj.JOB_NAME);
INSERT INTO log_table VALUES ('CA JOB_OWNER: ' || l_ca_obj.JOB_OWNER);
INSERT INTO log_table VALUES ('CA JOB_TYPE: ' || l_ca_obj.JOB_TYPE);
INSERT INTO log_table VALUES ('CA JOB_STATUS: ' || l_ca_obj.JOB_STATUS);
INSERT INTO log_table VALUES ('CA JOB_STATUS_CODE: ' || l_ca_obj.JOB_STATUS_
CODE);
INSERT INTO log_table VALUES ('CA JOB_STEP_OUTPUT: ' || l_ca_obj.JOB_STEP_
OUTPUT);
INSERT INTO log_table VALUES ('CA JOB_EXECUTION_GUID: ' || l_ca_obj.JOB_
EXECUTION_GUID);
INSERT INTO log_table VALUES ('CA JOB_STATE_CHANGE_GUID: ' || l_ca_obj.JOB_
STATE_CHANGE_GUID);
INSERT INTO log_table VALUES ('CA OCCURRED_DATE: ' || l_ca_obj.OCCURRED_DATE);
END IF;
COMMIT ;
END ;
/
4.10.4 Troubleshooting Notifications
To function properly, the notification system relies on various components of
Enterprise Manager and your IT infrastructure. For this reason, there can be many
causes of notification failure. The following guidelines and suggestions can help you
isolate potential problems with the notification system.
4.10.4.1 General Setup
The first step in diagnosing notification issues is to ensure that you have properly
configured and defined your notification environment.
4-80 OracleВ® Enterprise Manager Administration
Notification Reference
OS Command, PL/SQL and SNMP Trap Notifications
Make sure all OS Command, PLSQL and SNMP Trap Notification Methods are valid
by clicking the Test button. This will send a test notification and show any problems
the OMS has in contacting the method. Make sure that your method was called, for
example, if the OS Command notification is supposed to write information to a log
file, check that it has written information to its log file.
E-mail Notifications
в– Make sure an e-mail gateway is set up under the Notification Methods page of
Setup. The Sender's e-mail address should be valid. Clicking the Test button will
send an e-mail to the Sender's e-mail address. Make sure this e-mail is received.
Note that the Test button ignores any Notification Schedule.
в– в– в– Make sure an e-mail address is set up. Clicking the Test button will send an e-mail
to specified address and you should make sure this e-mail is received. Note that
the Test button ignores any Notification Schedule.
Make sure an e-mail schedule is defined. No e-mails will be sent unless a
Notification Schedule has been defined.
Make sure a incident rule is defined that matches the states you are interested and
make sure e-mail and notification methods are assigned to the rule.
4.10.4.2 Notification System Errors
For any alerts involving problems with notifications, check the following for
notification errors.
в– в– Any serious errors in the Notification System are logged as system errors in the
MGMT_SYSTEM_ERROR_LOG table. From the Setup menu, select Management
Services and Repository to view these errors.
Check for any delivery errors. You can view them from Incident Manager. From
the Enterprise menu, select Monitoring, then select Incident Manager. The details
will give the reason why the notification was not delivered.
4.10.4.3 Notification System Trace Messages
The Notification System can produce trace messages in sysman/log/emoms.trc file.
Tracing is configured by setting the log4j.category.oracle.sysman.em.notification property
flag using the emctl set property command. You can set the trace level to INFO,
WARN, DEBUG. For example,
emctl set property -name log4j.category.oracle.sysman.em.notification -value
DEBUG -module logging
Note: The system will prompt you for the SYSMAN password.
Trace messages contain the string "em.notification". If you are working in a UNIX
environment, you can search for messages in the emoms.trc and emoms_pbs.trc files
using the grep command. For example,
grep em.notification emoms.trc emoms_pbs.trc
What to look for in the trace file.
The following entries in the emoms.trc file are relevant to notifications.
Normal Startup Messages
Using Notifications 4-81
Notification Reference
When the OMS starts, you should see these types of messages.
2011-08-17 13:50:29,458 [EventInitializer] INFO em.notification init.167 - Short
format maximum length is 155
2011-08-17 13:50:29,460 [EventInitializer] INFO em.notification init.185 - Short
format is set to both subject and body
2011-08-17 13:50:29,460 [EventInitializer] INFO em.notification init.194 Content-Transfer-Encoding is 8-bit
2011-08-17 13:50:29,460 [EventInitializer] DEBUG em.notification
registerAdminMsgCallBack.272 - Registering notification system message call back
2011-08-17 13:50:29,461 [EventInitializer] DEBUG em.notification
registerAdminMsgCallBack.276 - Notification system message callback is registered
successfully
2011-08-17 13:50:29,713 [EventInitializer] DEBUG em.notification
upgradeEmailTemplates.2629 - Enter upgradeEmailTemplates
2011-08-17 13:50:29,735 [EventInitializer] INFO em.notification
upgradeEmailTemplates.2687 - Email template upgrade is not required since no
customized templates exist.
2011-08-17 13:49:28,739 [EventCoordinator] INFO events.EventCoordinator logp.251
- Creating event worker thread pool: min = 4 max = 15
2011-08-17 13:49:28,791 [[STANDBY] ExecuteThread: '2' for queue:
'weblogic.kernel.Default (self-tuning)'] INFO emdrep.pingHBRecorder
initReversePingThreadPool.937 - Creating thread pool for reverse ping : min = 10
max = 50
2011-08-17 13:49:28,797 [[STANDBY] ExecuteThread: '2' for queue:
'weblogic.kernel.Default (self-tuning)'] DEBUG emdrep.HostPingCoordinator logp.251
- Creating thread pool of worker thread for host ping: min = 1 max = 10
2011-08-17 13:49:28,799 [[STANDBY] ExecuteThread: '2' for queue:
'weblogic.kernel.Default (self-tuning)'] DEBUG emdrep.HostPingCoordinator logp.251
- Creating thread pool for output of worker's output for host ping: min = 2 max =
20
2011-08-17 13:49:30,327 [ConnectorCoordinator] INFO
connector.ConnectorPoolManager logp.251 - Creating Event thread pool: min = 3 max
= 10
2011-08-17 13:51:48,152 [NotificationMgrThread] INFO notification.pbs logp.251 Creating thread pool: min = 6 max = 24
2011-08-17 13:51:48,152 [NotificationMgrThread] INFO em.rca logp.251 - Creating
RCA thread pool: min = 3 max = 20
Notification Delivery Messages
2006-11-08 03:18:45,387 [NotificationMgrThread] INFO
Notification ready on EMAIL1
em.notification run.682 -
2006-11-08 03:18:46,006 [DeliveryThread-EMAIL1] INFO
Deliver to SYSMAN/[email protected]
em.notification run.114 -
2006-11-08 03:18:47,006 [DeliveryThread-EMAIL1] INFO
Notification handled for SYSMAN/[email protected]
em.notification run.227 -
Notification System Error Messages
2011-08-17 14:02:23,905 [NotificationMgrThread]
Notification ready on EMAIL1
2011-08-17 14:02:23,911 [NotificationMgrThread]
Notification ready on PLSQL4
2011-08-17 14:02:23,915 [NotificationMgrThread]
Notification ready on OSCMD14
2011-08-17 14:02:19,057 [DeliveryThread-EMAIL1]
Deliver to To: [email protected]; issue type:
4-82 OracleВ® Enterprise Manager Administration
DEBUG notification.pbs logp.251 DEBUG notification.pbs logp.251 DEBUG notification.pbs logp.251 INFO notification.pbs logp.251 1; notification type: 1
Notification Reference
2011-08-17 14:02:19,120 [DeliveryThread-OSCMD14] INFO notification.pbs logp.251 Deliver to SYSMAN, OSCMD, 8; issue type: 1; notification type: 1
2011-08-17 14:02:19,346 [DeliveryThread-PLSQL4] INFO notification.pbs logp.251 Deliver to SYSMAN, LOG_JOB_STATUS_CHANGE, 9; issue type: 1; notification type: 1
2011-08-17 14:02:19,977 [DeliveryThread-PLSQL4] DEBUG notification.pbs logp.251 Notification handled for SYSMAN, LOG_JOB_STATUS_CHANGE, 9
2011-08-17 14:02:20,464 [DeliveryThread-EMAIL1] DEBUG notification.pbs logp.251 Notification handled for To: [email protected]
2011-08-17 14:02:20,921 [DeliveryThread-OSCMD14] DEBUG notification.pbs logp.251 Notification handled for SYSMAN, OSCMD, 8
4.10.4.4 E-mail Errors
The SMTP gateway is not set up correctly:
Failed to send e-mail to [email protected]: For e-mail notifications to be sent,
your Super Administrator must configure an Outgoing Mail (SMTP) Server within
Enterprise Manager. (SYSMAN, myrule)
Invalid host name:
Failed to connect to gateway: badhost.oracle.com: Sending failed;
nested exception is:
javax.mail.MessagingException: Unknown SMTP host: badhost.example.com;
Invalid e-mail address:
Failed to connect to gateway: rgmemeasmtp.oraclecorp.com: Sending failed;
nested exception is:
javax.mail.MessagingException: 550 5.7.1 <[email protected]>... Access
denied
Always use the Test button to make sure the e-mail gateway configuration is valid.
Check that an e-mail is received at the sender's e-mail address
4.10.4.5 OS Command Errors
When attempting to execute an OS command or script, the following errors may occur.
Use the Test button to make sure OS Command configuration is valid. If there are any
errors, they will appear in the console.
Invalid path or no read permissions on file:
Could not find /bin/myscript (machineb10.oracle.com_Management_Service) (SYSMAN,
myrule )
No execute permission on executable:
Error calling /bin/myscript: java.io.IOException: /bin/myscript: cannot execute
(machineb10.oracle.com_Management_Service) (SYSMAN, myrule )
Timeout because OS Command ran too long:
Timeout occurred running /bin/myscript (machineb10.oracle.com_Management_Service)
(SYSMAN, myrule )
Any errors such as out of memory or too many processes running on OMS machine
will be logged as appropriate.
Always use the Test button to make sure OS Command configuration is valid.
Using Notifications 4-83
Notification Reference
4.10.4.6 SNMP Trap Errors
Use the Test button to make sure SNMP Trap configuration is valid.
The OMS will not report an error if the SNMP trap cannot reach the third party SNMP
console as this is sent via UDP. If the SNMP trap encounters problems when trying to
reach the third party SNMP console, possible SNMP trap problems include: invalid
host name, port, community for a machine running an SNMP Console or a network
issue such as a firewall problem.
Other possible SNMP trap problems include: invalid host name, port, or community
for a machine running an SNMP Console.
4.10.4.7 PL/SQL Errors
When attempting to execute an PL/SQL procedure, the following errors may occur.
Use the Test button to make sure the procedure is valid. If there are any errors, they
will appear in the console.
Procedure name is invalid or is not fully qualified. Example: SCOTT.PKG.PROC
Error calling PL/SQL procedure plsql_proc: ORA-06576: not a valid function or
procedure name (SYSMAN, myrule)
Procedure is not the correct signature. Example: PROCEDURE event_proc(s IN
GC$NOTIF_EVENT_MSG)
Error calling PL/SQL procedure plsql_proc: ORA-06553: PLS-306: wrong number or
types of arguments in call to 'PLSQL_PROC' (SYSMAN, myrule)
Procedure has bug and is raising an exception.
Error calling PL/SQL procedure plsql_proc: ORA-06531: Reference to uninitialized
collection (SYSMAN, myrule)
Care should be taken to avoid leaking cursors in your PL/SQL. Any exception due to
this condition will result in delivery failure with the message being displayed in the
Details section of the alert in the Cloud Control console.
Always use the Test button to make sure the PL/SQL configuration is valid.
4-84 OracleВ® Enterprise Manager Administration
5
Using Blackouts
5
Blackouts allow Enterprise Manager administrators to suspend all data collection
activity on one or more monitored targets. The primary reason for blacking out targets
is to allow Enterprise Manager administrators to perform scheduled maintenance on
those targets.
A blackout can be defined for individual target(s), a group of multiple targets that
reside on different hosts, or for all targets on a host. The blackout can be scheduled to
run immediately or in the future, and to run indefinitely or stop after a specific
duration. Blackouts can be created on an as-needed basis, or scheduled to run at
regular intervals. If, during the maintenance period, the administrator discovers that
he needs more (or less) time to complete his maintenance tasks, he can easily extend
(or stop) the blackout that is currently in effect.
Blackout functionality is available from both the Enterprise Manager console as well as
via the Enterprise Manager command-line interface (EMCLI). EMCLI is often useful
for administrators who would like to incorporate the blacking out of a target within
their maintenance scripts.
5.1 Working with Blackouts
Blackouts allow you to collect accurate monitoring data. For example, you can stop
data collections during periods where a managed target is undergoing routine
maintenance, such as a database backup or hardware upgrade. If you continue
monitoring during these periods, the collected data will show trends and other
monitoring information that are not the result of normal day-to-day operations. To get
a more accurate, long-term picture of a target's performance, you can use blackouts to
exclude these special-case situations from data analysis.
Blackout Access
Enterprise Manager administrators who have at least Blackout Target privileges on all
Selected Targets in a blackout will be able to create, edit, stop, or delete the blackout.
In case an administrator has at least Blackout Target privileges on all Selected Targets
(targets directly added to the blackout), but does not have Blackout Target privileges
on some or all of the Dependent Targets, then that administrator will be able to edit,
stop, or delete the blackout. For more information on Blackout access, see About
Blackouts Best Effort.
5.1.1 Creating a Blackout
To create a blackout:
1.
From the Enterprise menu, select Monitoring, then select Blackouts.
Using Blackouts
5-1
Working with Blackouts
2.
From the table, click Create. Enterprise Manager displays a wizard page to guide
you through the steps required to create a blackout. Click Help from any wizard
page for more information on specific steps.
3.
To display the latest blackout information, click the refresh icon.
5.1.2 Editing a Blackout
To edit a blackout:
1.
From the Enterprise menu, select Monitoring, then select Blackouts.
2.
If necessary, use the Search and display options to show the blackouts you want to
change in the blackouts table.
3.
Select the desired blackout and click Edit.
Enterprise Manager also allows you to edit blackouts after
they have already started.
Note:
5.1.3 Viewing Blackouts
To view information and current status of a blackout:
1.
From the Enterprise menu, select Monitoring, then select Blackouts.
2.
If necessary, you can use the Search and display options to show the blackouts you
want to view in the blackouts table.
3.
Select the desired blackout and click View. Alternatively, if you are in View By 'Targets in Blackout', then you can click on blackout status in the table to access the
5-2 OracleВ® Enterprise Manager Administration
Working with Blackouts
View Blackout page. If you are in View By - 'Blackout Name', then you can click on
a blackout name in the table to access the View Blackout page.
You can bookmark the View Blackout page as a quick way to monitor the status of a
particular blackout. Click Refresh to display the most recent blackout information.
5.1.3.1 Viewing Blackouts on Targets Monitored by a Specific Management Agent
To view the blackouts configured for the targets monitored by the Management Agent:
1.
From the Cloud Control home page, click Targets and then All Targets. In the All
Targets page, locate the Management Agent in the list of targets. Click on the
Management Agent's name. This brings you to the Management Agent's home
page.
2.
The list of targets monitored by the Management Agent are listed in the Monitored
Targets section.
3.
For each of target in the list:
a.
Click the target name. This brings you to the target's home page.
b.
From the <Target> menu, select Monitoring and then click Blackouts. This
allows you to check any currently running blackouts or blackouts that are
scheduled in the future for this target.
5.1.3.2 Viewing Blackouts from Target Home Pages
For most target types, you can view a blackout information from the target home page
for any target currently under blackout. A blackout message provides pertinent
blackout status information for that target.
5.1.3.3 Viewing Blackouts from Groups and Systems Target Administration Pages
For Groups and Systems, you can view blackout information about the number of
active/scheduled blackouts on a group/system and its member targets.
Using Blackouts
5-3
Controlling Blackouts Using the Command Line Utility
5.1.4 Purging Blackouts that have Ended
When managing a large number of targets, the number of completed blackouts, or
those blackouts that have been ended by an administrator can become quite large.
Removing these ended blackouts facilitates better search an display for current
blackouts.
To purge ended blackouts from Enterprise Manager:
1.
From the Enterprise menu, select Monitoring, then select Blackouts.
2.
Use the search criteria to filter for the desired targets.
3.
From the Show drop-down menu, select History.
4.
In the table, select the ended blackouts you want to remove and click Delete. The
purge confirmation page appears.
5.
Click Yes to complete the purge process.
5.2 Controlling Blackouts Using the Command Line Utility
You can control blackouts from the Oracle Enterprise Manager 12c Cloud Control
Console or from the Enterprise Manager command line utility (emctl). However, if
you are controlling target blackouts from the command line, you should not attempt to
control the same blackouts from the Cloud Control Console. Similarly, if you are
controlling target blackouts from the Cloud Control Console, do not attempt to control
those blackouts from the command line.
From the command line, you can perform the following blackout functions:
в– Starting Immediate Blackouts
в– Stopping Immediate Blackouts
в– Checking the Status of Immediate Blackouts
When you start a blackout from the command line, any
Enterprise Manager jobs scheduled to run against the blacked out
targets will still run. If you use the Cloud Control Console to
control blackouts, you can optionally prevent jobs from running
against blacked out targets.
Note:
To use the Enterprise Manager command-line utility to control blackouts:
1.
Change directory to the AGENT_HOME/bin directory (UNIX) or the
AGENT_INSTANCE_HOME\bin directory (Windows).
2.
Enter the appropriate command as described in Table 5–1, " Summary of Blackout
Commands".
When you start a blackout, you must identify the target or
targets affected by the blackout. To obtain the correct target name
and target type for a target, see "Listing the Targets on a Managed
Host" on page 20-13.
Note:
5-4 OracleВ® Enterprise Manager Administration
Controlling Blackouts Using the Command Line Utility
Table 5–1
Summary of Blackout Commands
Blackout Action
Command
Set an immediate blackout
on a particular target or list
of targets
emctl start blackout <Blackoutname>
[<Target_name>[:<Target_Type>]]....
[-d <Duration>]
Be sure to use a unique name for the blackout so you can refer to
it later when you want to stop or check the status of the
blackout.
The -d option is used to specify the duration of the blackout.
Duration is specified in [days] hh:mm where:
в– days indicates number of days, which is optional
в– hh indicates number of hours
в– mm indicates number of minutes
If you do not specify a target or list of targets, Enterprise
Manager will blackout the local host target. All monitored
targets on the host are not blacked out unless a list is specified or
you use the -nodelevel argument.
If two targets of different target types share the same name, you
must identify the target with its target type.
Stop an immediate blackout emctl stop blackout <Blackoutname>
Set an immediate blackout
for all targets on a host
emctl start blackout <Blackoutname>
[-nodeLevel] [-d <Duration>]
The -nodeLevel option is used to specify a blackout for all the
targets on the host; in other words, all the targets that the
Management Agent is monitoring, including the Management
Agent host itself. The -nodeLevel option must follow the
blackout name. If you specify any targets after the -nodeLevel
option, the list is ignored.
Check the status of a
blackout
emctl status blackout [<Target_name>[:<Target_
Type>]]....
Use the following examples to learn more about controlling blackouts from the
Enterprise Manager command line:
в– To start a blackout called "bk1" for databases "db1" and "db2," and for Oracle
Listener "ldb2," enter the following command:
$PROMPT> emctl start blackout bk1 db1 db2 ldb2:oracle_listener -d 5 02:30
The blackout starts immediately and will last for 5 days 2 hours and 30 minutes.
в– To check the status of all the blackouts on a managed host:
$PROMPT> emctl status blackout
в– To stop blackout "bk2" immediately:
$PROMPT> emctl stop blackout bk2
в– To start an immediate blackout called "bk3" for all targets on the host:
$PROMPT> emctl start blackout bk3 -nodeLevel
в– To start an immediate blackout called "bk3" for database "db1" for 30 minutes:
$PROMPT> emctl start blackout bk3 db1 -d 30
Using Blackouts
5-5
About Blackouts Best Effort
в– To start an immediate blackout called "bk3" for database "db2" for five hours:
$PROMPT> emctl start blackout bk db2 -d 5:00
5.3 About Blackouts Best Effort
The Blackouts Best Effort feature allows you to create blackouts on aggregate targets,
such as groups or systems, for which you do not have Blackout Target (or Higher)
privileges on all members of the aggregate target.
Here, an Enterprise Manager administrator has Blackout Target privilege on an
aggregate target but do not have OPERATOR privilege on its member/associated
targets. You should ideally create a Full Blackout on this aggregate target. When
defining the blackout, you are allowed to select any member target, even those
member targets for which you have no Blackout Target privileges.
When the blackout actually starts, Enterprise Manager checks privileges on each
member target and only blackout those on which you have Blackout Target( or Higher)
privileges. This automated privilege check and target blackout selection is Enterprise
Manager's "best effort" at blacking out the aggregate target.
5.3.1 When to Use Blackout Best Effort
The Blackout Best Effort functionality is targeted towards the creation of blackouts on
targets of any aggregate type, such as Group, Hosts, Application Servers, Web
Applications, Redundancy Groups, or Systems.
All targets the blackout creator has Blackout Target (or higher) privilege on will be
displayed in the first step of Create/Edit Blackout Wizard. Once the blackout creator
selects an aggregate type of target to be included in the Blackout Definition, this
Blackout is "Full Blackout" by default.
The creator has the option of choosing the Blackout to run on “All Current” or
“Selected” Targets, by selecting the appropriate values from the List box. Only when
the "Full Blackout" option is chosen, will Blackout Best Effort affect targets for which
the creator does not have Blackout Target (or higher) privileges.
Example Use Case
Consider 3 targets T1,T2 and T3 (all databases). A Group G1 contains all these 3
targets.
User U1 has OPERATOR privilege on T1,T2 and G1. User U1 has VIEW privilege on
T3.
User U1 creates a scheduled full blackout on target G1. Scheduled implies that the
blackout will start at a later point in time.
At the time of blackout creation, the tip text Needs Blackout Target privilege, see Tip below
the table would be shown beside target T3.
When this blackout starts, if by that time User U1 has been granted OPERATOR
privileges on target T3, then target T3 would also be under blackout. Otherwise only
targets T1, T2 and G1 will be under blackout.
5-6 OracleВ® Enterprise Manager Administration
6
Managing Groups
6
This chapter introduces the concept of group management and contains the following
sections:
в– Introduction to Groups
в– Managing Groups
в– Using Out-of-Box Reports
6.1 Introduction to Groups
Groups are an efficient way to logically organize, manage, and monitor the targets in
your global environments. Each group has its own group home page. The group home
page shows the most important information for the group and enables you to drill
down for more information. The home page shows the overall status of the group and
other information such as current availability, incidents, and patch recommendations
for members of the group.
Group Management Tasks
You can use Enterprise Manager to perform the following group management
functions:
в– в– в– Edit the configuration of a selected group, remove groups, and, in the case of an
Administration Group, associate or disassociate a Template Collection.
View the status and health of the group from the System Dashboard
Drill down from a specific group to collectively monitor and manage its member
targets.
в– View a roll-up of member statuses and open incidents for members of the group.
в– Apply blackouts to all targets in a group.
в– Run jobs against a group
в– Run a report
в– Apply monitoring templates
в– Associate compliance standards
In addition to creating groups, you can also create specific types of groups, such as
redundancy groups, privilege propagating groups, and dynamic groups. The
following sections explain the different types of groups.
Managing Groups
6-1
Introduction to Groups
6.1.1 Overview of Groups
Groups enable you to collectively monitor and administer many targets as a single
logical unit. For example, you can define a group to contain all the databases serving
an enterprise application, and define another group to contain all the hosts in a host
farm. You can then use these groups to perform administrative operations. To create a
group, you can manually select and add the members of the group. If you add an
aggregate target, such as a Cluster Database, all of its member targets are
automatically added to the group.
A group can include targets of the same type, such as all your production databases, or
it could include all the targets on a host which would be comprised of different target
types. You can nest static groups inside each other. In the target selector when you are
selecting group members, choose Group as the target type, or choose a parent group as
part of the process of creating a group.
If a system target is added to a group, it automatically pulls in its member targets. This
could be the case of a regular group where a system such as WebLogic Server is added
and also pulls in its members, or in a dynamic group where you specify a Target type
to be an Oracle WebLogic Server and it also pulls in members of the WebLogic Server
even though it does not match the dynamic group criteria. In this scenario, the group
operations (for example, runnning jobs, blackouts, and so on) apply to all members of
the group.
After you configure a group, you can perform various administrative operations, such
as:
в– View a summary status of the targets within the group.
в– View a roll-up of member statuses and open incidents for members of the group.
в– View a summary of critical patch advisories.
в– View configuration changes during the past 7 days.
в– Create jobs and view the status of job executions.
в– Create blackouts and view the status of current blackouts.
6.1.2 Overview of Privilege Propagating Groups
Privilege propagating groups enable administrators to propagate privileges to
members of a group. You can grant a privilege on a group once to an administrator or
a role and have that same privilege automatically propagate to any new member of the
group. For example, granting operator privilege on a privilege propagating group to an
Administrator grants him the operator privilege on its member targets and also to any
members that will be added in the future. Privilege propagating groups can contain
individual targets or other privilege propagating groups. Any aggregate that you add
to a privilege propagating group must also be privilege propagating as well. For
example, any group that you add to a privilege propagating group must also be
privilege propagating.
Privileges on the group can be granted to an Enterprise Manager administrator or a
role. Use a role if the privileges you want to grant are to be granted to a group of
Enterprise Manager administrators.
For example, suppose you create a privilege propagating group and grant a privilege
to a role which is then granted to administrators. If new targets are later added to the
privilege propagating group, then the administrators receive the privileges on the
target automatically. Additionally, when a new administrator is hired, you only need
6-2 OracleВ® Enterprise Manager Administration
Introduction to Groups
to grant the role to the administrator for the administrator to receive all the privileges
on the targets automatically.
6.1.3 Overview of Dynamic Groups
The membership management for groups is typically manual or static in nature.
Manually managing memberships works well for small deployments but not
necessarily in large, dynamic environments where new targets come into the system
frequently. Groups whose members are added frequently would be easier to maintain
if they were to be defined by membership criteria instead of adding targets directly
into the group. When the membership criteria is defined once, Enterprise Manager
will automatically add targets.
A dynamic group is a group whose membership is determined by membership
criteria. The owner of a dynamic group specifies the membership criteria during
dynamic group creation (or modification) and membership in the group is determined
solely by the criteria specified. Membership in a dynamic group cannot be modified
directly because targets cannot be directly added to a dynamic group. Enterprise
Manager automatically adds targets that match membership criteria when a dynamic
group is created. It also updates group membership as new targets are added or target
properties are changed and the target matches the group’s membership criteria.
It is important to note that static groups can contain dynamic groups as members but
not the other way around. You cannot include a static group as a member of a dynamic
group.
Use the Define Membership Criteria function of Dynamic Groups to define the criteria
for group membership. Once you have defined criteria, the targets selected by the
criteria will be displayed in a read-only table in the Members region of the Groups
page. Since dynamic groups are defined by criteria, you can intentionally or
unintentionally define criteria that could result in very large groups.
The following requirements apply to dynamic groups:
в– в– в– в– в– в– Dynamic groups cannot contain static groups, other dynamic groups, or
administration groups.
Administration groups cannot contain dynamic groups, however, a static group
can contain dynamic groups as a member.
OR-based criteria is not supported. All criteria selected on the criteria page are
AND-based.
Supported properties are limited to global properties plus other attributes
specifically supported for administration groups such as Version, Platform, Target
Name and Type, and so on. Specifically user-defined properties and other instance
properties, plus config data elements are not supported as criteria.
The View Any Target and Add Any Target privileges are required to create a
dynamic group.
The Full Any Target, Add Any Target, and Create Privilege Propagating Group
privileges are required to create a privilege-propagating dynamic group.
6.1.4 Overview of Administration Groups
Administration Groups greatly simplify the process of setting up targets for
management in Enterprise Manager by automating the application of management
settings such as monitoring settings or compliance standards. Typically, these settings
are manually applied to individual target, or perhaps semi-automatically using custom
Managing Groups
6-3
Managing Groups
scripts. However, by defining Administration Groups, Enterprise Manager uses
specific target properties to direct the target to the appropriate Administration Group
and then automatically apply the requisite monitoring and management settings. This
level of automation simplifies the target setup process and also enables a datacenter to
easily scale as new targets are added to Enterprise Manager for management.
Administration groups are a special type of group used to fully automate application
of monitoring and other management settings upon joining the group. When a target
is added to the group, Enterprise Manager applies these settings using a Template
Collection consisting of Monitoring Templates, compliance standards, and cloud
policies. This completely eliminates the need for administrator intervention.
To watch Part 1 of a video about using administrative groups and template collections,
click here.
To watch Part 2 of the video about using administrative groups and template
collections, click here.
6.1.5 Choosing Which Type of Group To Use
There are two major types of groups you can choose to manage targets: Static Groups/
Dynamic groups, which can be one or more groups that you define, and
Administration Groups for automating monitoring setup using templates.
You should carefully consider the purpose of your group and the function it serves
before determining which type of group to use.
The following table diagrams when you should use Administration Groups or
Dynamic Groups.
Table 6–1
When To Use Administration Groups vs. Dynamic Groups
Additional
Membership
Requirements
Type of Group
Main Purpose
Membership Based
on Criteria
Privilege
Propagating
Administration
Group
Auto-apply monitoring
templates
Yes, based on target
properties
Target can belong to at Yes (always)
most one
administration group
Dynamic Group
Perform any group
operation.
Yes, based on target
properties
Target can belong to
one or more groups
User-specified
option
The main purpose of an Administration Group is to automate the application of
management settings, such as monitoring settings or compliance standards. When a
target is added to the group, Enterprise Manager automatically applies these settings
using templates to eliminate the need for administrator action.
Dynamic groups, on the other hand, can be used to manage many targets as a single
unit where you can define the group membership by defining the properties that
constitute the group. For example, you could use dynamic groups to manage
privileges or groups that you create containing the targets that are managed for
different support teams.
6.2 Managing Groups
By combining targets in a group, Enterprise Manager provides management features
that enable you to efficiently manage these targets as one group. Using the Group
functionality, you can:
в– View a summary status of the targets within the group.
6-4 OracleВ® Enterprise Manager Administration
Managing Groups
в– Monitor incidents for the group collectively, rather than individually.
в– Monitor the overall performance of the group.
в– Perform administrative tasks, such as scheduling jobs for the entire group, or
blacking out the group for maintenance periods.
You can also customize the console to provide direct access to group management
pages.
When you choose Groups from the Targets menu in the Enterprise Manager, the
Groups page appears. You can view the currently available groups and perform the
following tasks:
в– View a list of all the defined groups.
в– Search for existing groups and save search criteria for future searches.
в– View a member status summary and rollup of incidents for members in a group.
в– в– в– в– Create Groups, Dynamic Groups or the Administration Group hierarchy, edit the
configuration of a selected group, remove groups, and, in the case of an
Administration Group, associate or disassociate a Template Collection.
Add groups or privilege propagating groups, remove groups, and change the
configuration of currently defined groups.
Drill down from a specific group to collectively monitor and manage its member
targets.
Customize the homepage of a specific group
Redundancy systems and special high availability groups are not accessed from this
Groups page. You can access them from the All Targets page or you can access
Redundancy Systems and other systems from the Systems page.
6.2.1 Creating and Editing Groups
Enterprise Manager Groups enable administrators to logically organize distributed
targets for efficient and effective management and monitoring.
To create a group, follow these steps:
1.
From the Enterprise Manager Console, choose Targets then choose Groups.
Alternately, you can choose Add Target from the Setup menu and choose the
menu option to add the specific type of group.
2.
Click Create and choose the type of group you want to create. The Enterprise
Manager Console displays a set of Create Group pages that function similarly to a
wizard.
3.
On the General tab of the Create Group page, enter the Name of the Group you
want to create. If you want to make this a privilege propagating group, then
enable the Privilege Propagation option by clicking Enabled. If you enable
Privilege Propagation for the group, the target privileges granted on the group to
an administrator or a role are propagated to the member targets. As with regular
groups with privilege propagation, the Create Privilege Propagating Group
privilege is required for creation of privilege propagating dynamic groups. In
addition, the Full any Target privilege is required to enable privilege propagation
because only targets on which the owner has Full Target privileges can be
members, and any target can potentially match the criteria and a system wide Full
privilege is required. To create a regular dynamic group, the View any Target
Managing Groups
6-5
Managing Groups
system wide privilege is required as the group owner must be able to view any
target that can potentially match the membership criteria.
4.
Configure each page, then click OK. You should configure all the pages before
clicking OK. For more information about these steps, see the online help.
After you create the group, you always have immediate access to it from the Groups
page.
You can edit a group to change the targets that comprise the group, or change the
metrics that you want to use to summarize a given target type. To edit a group, follow
these steps:
1.
From the Enterprise Manager Console, choose Targets then choose Groups.
2.
Click the group Name for the group you want to edit.
3.
Click Edit from the top of the groups table.
4.
Change the configuration for a page or pages, then click OK.
6.2.2 Creating Dynamic Groups
The owner of a dynamic group specifies the membership criteria during dynamic
group creation (or modification) and membership in the group is determined solely by
the criteria specified. Membership in a dynamic group cannot be modified directly.
Enterprise Manager automatically adds targets that match membership criteria when a
dynamic group is created. It also updates group membership as new targets are added
or target properties are changed and the targets match the group’s membership
criteria.
To create a dynamic group, follow these steps:
1.
From the Groups page, click Create and then select Dynamic Group from the
drop-down list. Alternately, you can choose Add Target from the Setup menu and
then select Group.
2.
On the General tab of the Create Dynamic Group page, enter the Name of the
Dynamic Group you want to create. If you want to make this a privilege
propagating dynamic group, then enable the Privilege Propagation option by
clicking Enabled. If you enable Privilege Propagation for the group, the target
privileges granted on the group to an administrator or a role are propagated to the
member targets. As with regular groups with privilege propagation, the Create
Privilege Propagating Group privilege is required for creation of privilege
propagating dynamic groups. In addition, the Full any Target privilege is required
to enable privilege propagation because only targets on which the owner has Full
Target privileges can be members, and any target can potentially match the criteria
and a system wide Full privilege is required. To create a regular dynamic group,
the View any Target system wide privilege is required as the group owner must be
able to view any target that can potentially match the membership criteria.
The privilege propagating group feature contains two privileges:
в– Create Privilege Propagating Group
This privileged activity allows the administrators to create the privilege
propagating groups. Administrators with this privilege can create propagating
groups and delegate the group administration activity to other users.
в– Group Administration
6-6 OracleВ® Enterprise Manager Administration
Managing Groups
Grant this privilege to an administrator or role that enables him to become
group administrator for the group. This means he can perform operations on
the group, share privileges on the group with other administrators, etc.
The Group Administration Privilege is available for both Privilege
Propagating Groups and conventional groups. If you are granted this
privilege, you can grant full privilege access to the group to other Enterprise
Manager users without having to be the SuperAdministrator to grant the
privilege.
3.
In the Define Membership Criteria section, define the criteria for the dynamic
group membership by clicking Define Membership Criteria.
The Define Membership Criteria page appears where you can Add or Remove
properties of targets to be included in the group. Group members must match one
value in each of the populated target properties. Use the Member Preview section
to review a list of targets that match the criteria. Click OK to return to the General
page.
At least one of the criteria on the Define Membership Criteria page must be
specified. You cannot create a Dynamic group without at least one of the target
types, on hosts or target properties specified. Use the following criteria for
dynamic groups:
в– Target type(s)
в– Department
в– On Host
в– Target Version
в– Lifecycle Status
в– Operating System
в– Line of Business
в– Platform
в– Location
в– CSI
в– Cost Center
в– Contact
в– Comment
You can add or remove properties using the Add or Remove Target Properties
button on the Define Membership Criteria page.
4.
Enter the Time Zone. The time zone you select is used for scheduling operations
such as jobs and blackouts on this group. The groups statistics charts will also use
this time zone.
5.
Click the Charts tab. Specify the charts that will be shown in the Dynamic Group
Charts page. By default, the commonly used charts for the target types contained
in the Dynamic Group are added.
6.
Click the Columns tab to add columns and abbreviations that will be seen in the
Members page and also in the Dashboard.
7.
Click the Dashboard tab to specify the parameters for the System Dashboard. The
System Dashboard displays the current status and incidents and compliance
Managing Groups
6-7
Managing Groups
violations associated with the members of the Dynamic Group in graphical
format.
8.
Click the Access tab. Use the Access page to administer access privileges for the
group. On the Access page you can grant target access to Enterprise Manager roles
and grant target access to Enterprise Manager administrators.
9.
Click OK to create the Dynamic Group.
6.2.3 Adding Members to Privilege Propagating Groups
The target privileges granted on a propagating group are propagated to member
targets. The administrator grants target objects scoped to another administrator, and
the grantee maintains the same privileges on member targets. The propagating groups
maintain the following features:
в– в– The administrator with a Create Privilege Propagating Group privilege will be
able to create a propagating group
To add a target as a member of a propagating group, the administrator must have
Full target privileges on the target
You can add any non-aggregate target as the member of a privilege propagating
group. For aggregate targets in Cloud Control version 12c, cluster and RAC databases
and other propagating groups can be added as members. Cloud Control version 12c
supports more aggregate target types, such as redundancy systems, systems and
services.
If you are not the group creator, you must have at least the Full target privilege on the
group to add a target to the group.
6.2.4 Converting Conventional Groups to Privilege Propagating Groups
In Enterprise Manager release 12c you can convert conventional groups to privilege
propagating groups (and vice-versa) through the use of the specified EM CLI verb.
Two new parameters have been added in the modify_group EM CLI verb:
в– privilege_propagation
This parameter is used to modify the privilege propagation behavior of the group.
The possible value of this parameter is either true or false.
в– drop_existing_grants
This parameter indicates whether existing privilege grants on that group are to be
revoked at the time of converting a group from privilege propagation to normal
(or vice versa). The possible values of this parameter are yes or no. The default
value of this parameter is yes.
These same enhancements have been implemented on the following EM CLI verbs:
modify_system, modify_redundancy_group, and modify_aggregrate_service.
The EM CLI verb is listed below:
emcli modify_group
-name="name"
[-type=<group>]
[-add_targets="name1:type1;name2:type2;..."]...
[-delete_targets="name1:type1;name2:type2;..."]...
[-privilege_propagation = true/false]
[-drop_existing_grants = Yes/No]
6-8 OracleВ® Enterprise Manager Administration
Managing Groups
For more information about this verb and other EM CLI verbs, see the EM CLI
Reference Manual.
6.2.5 Viewing and Managing Groups
Enterprise Manager enables you to quickly view key information about members of a
group, eliminating the need to navigate to individual member targets to check on
availability and performance. You can view the entire group on a single screen and
drill down to obtain further details. The Group Home page provides the following
sections:
в– в– в– A General section that displays the general information about the group, such as
the Owner, Group Type, and whether the group is privilege propagating. You can
drill down to the Edit Group page to enable or disable privilege propagating by
clicking on the Privilege Propagating field.
A Status section that shows how many member targets are in Up, Down, and
Unknown states. For nested groups, this segment shows how many targets are in
up, down, and unknown states across all its sub-groups. The status roll up count is
based on the unique member targets across all sub-groups. Consequently, even if a
target appears more than once in sub-groups, it is counted only once in status roll
ups.
An Overview of Incidents and Problems section that displays the summary of
incidents on members of the group that have been updated in the recent period of
time. It also shows a count of open problems as well as problems updated in
recent period of time.
The rolled up information is shown for all the member targets regardless of their
status. The status roll up count is based on the unique member targets across all
sub-groups. Consequently, even if a target appears more than once in sub-groups,
its alerts are counted only once in alert roll ups.
Click the number in the Problems column to go to the Incident Manager page to
search, view, and manage exceptions and issues in your environment. By using
Incident Manager, you can track outstanding incidents and problems.
в– в– в– в– A Compliance Summary section that shows the compliance of members of the
group against the compliance standards defined for the group. This section also
shows a rollup of violations by severity (critical, warning, minor warning) as well
as the average compliance score(%).
A Job Activity section that displays a summary of jobs for the targets in the group
whose start date is within the last 7 days. You can click Show to see the latest run
or all runs. Click View to select and reorder the columns that appear in the table or
to adjust scrolling and expanding the table.
A Blackouts section that displays information about current or pending blackouts.
You can also create a blackout from this section.
A Patch Recommendations section that displays the Oracle patch
recommendations that are applicable to your enterprise. You can view patch
recommendations by classification or target type.
You can navigate to My Oracle Support to view all recommendations by clicking
the All Recommendations link.
в– An Inventory and Usage section where you can view inventory summaries for
deployments such as hosts, database installations, and fusion middleware
installations on an enterprise basis or for specific targets. You can also view
Managing Groups
6-9
Managing Groups
inventory summary information in the context of different dimensions. From here
you can click See Details to display the Inventory and Usage page.
в– A Configuration Changes section that displays the number of configuration
changes to the group in the previous 7 days. You can click the number to display a
page that displays detailed information about the changes. Enterprise Manager
automatically collects configuration information for group targets and changes to
configurations are recorded and may be viewed from that page.
Viewing a Group
To view a group, follow these steps:
1.
From the Enterprise Manager Console, choose Targets then choose Groups. A
summary table lists all defined groups.
2.
Click the desired group to go to the Home page of that group.
You can use View By filters (located in the upper right corner of the home page) to
change the view of the homepage to members of targets of a specific type. When you
do this, the Group homepage refreshes to only show information for targets of that
type. Additional regions of interest might display. For example, DBAs might switch to
the Database filter to view information specifically on Database targets in the group.
You can also personalize the home page by clicking the Actions icon in the upper right
corner of each region on the home page to move that region up or down on the page.
You can also expand or contract a region by clicking the arrow icon in the upper left
corner of each region.
You can also navigate to other management operations on the group using the Group
menu. For example, you can view all the members in a group by choosing Member
from the Group menu. Likewise you can view the Membership History of the group
by choosing Membership History from the Group menu.
6.2.6 Overview of Group Charts
Group Charts enable you to monitor the collective performance of a group. Out-of-box
performance charts are provided based on the type of members in the group. For
example, when databases are part of the group, a Wait Time (%) chart is provided that
shows the top databases with the highest wait time percentage values. You can view
this performance information over the last 24 hours, last 7 days, or last 31 days. You
can also add your own custom charts to the page.
6.2.7 Overview of Group Members
Enterprise Manager allows you to summarize information about the member targets
in a group. It provides information on their current availability status, roll-up of open
incidents and compliance violations, and key performance metrics based on the type
of targets in the group.
You can visually assess availability and relative performance across all member
targets. You can rank members by a certain criterion (for example, database targets in
order of decreasing wait time percentage). You can display default key performance
metrics based on the targets you select, but you can customize these to include
additional metrics that are important for managing your group.
You can view the members of a group by choosing Members from the Group menu.
Enterprise Manager displays the Members page where you can view the table of
members filtered by All Members, Direct Members, or Indirect Members. Direct
members are targets directly added to the group. Indirect members are targets that are
6-10 OracleВ® Enterprise Manager Administration
Managing Groups
members of a direct member target, and are automatically included into the group
because their parent target was added to the group. The page provides the option to
Export or Edit the group.
You can also access information about membership history by choosing Membership
History from the Group menu. The Membership History page displays changes in the
group membership over time.
6.2.8 Viewing Group Status History
You can view Status History for a group to see the historical availability of a member
during a specified time period or view the current status of all group members. You
can access the Status History page by choosing Monitoring from the Group menu and
then selecting Status History.
Bar graphs provide a historical presentation of the availability of group members
during a time period you select from the View Data drop-down list. The color-coded
graphs can show statuses of Up, Down, Under Blackout, Agent Down, Metric
Collection Error, and Status Pending. You can select time periods of 24 hours, 7 days,
or 31 days.
To view the current status of a member, you can click a Status icon on the View Group
Status History page to go to the Availability page, which shows the member's current
and past availability status within the last 24 hours, 7 days, or 31 days. Click a member
Name to go to the member's Home page. You can use this page as a starting point
when evaluating the performance of the selected member.
6.2.9 About the System Dashboard
The System Dashboard enables you to proactively monitor the status, incidents and
compliance violations in the group as they occur. The color-coded interface is designed
to highlight problem areas — targets that are down are highlighted in red, metrics in
critical severity are shown as red dots, metrics in warning severity are shown as
yellow dots, and metrics operating within normal boundary conditions are shown as
green dots.
Using these colors, you can easily determine the problem areas for any target and drill
down for details as needed. An incident table is also included to provide a summary
for all open incidents in the group. The incidents in the table are presented in reverse
chronological order to show the most recent incidents first, but you can also click any
column in the table to change the sort order. The colors in top bar of the Member
Targets table change based on the incident's critical level. The priority progresses from
warning to critical to fatal. If the group has at least one fatal incident (irrespective of
critical or warning incidents), the top bar becomes dark red. If the group has at least
one critical incident (irrespective of warning incidents), the top bar becomes faint red.
If the group has only warning incidents, the top bar turns yellow. If the group has no
incidents, the top bar remains colorless.
The Dashboard auto-refreshes based on the Refresh Frequency you set on the
Customize Dashboard page.
The Dashboard allows you to drill down for more detailed information. You can click
the following items in the Dashboard for more information:
в– A target name to access the target home page
в– A group or system name to access the System Dashboard
в– Status icon corresponding to specific metric columns to access the metric detail
page
Managing Groups 6-11
Using Out-of-Box Reports
в– в– в– Status icon for a metric with key values to access the metric page with a list of all
key values
Dashboard header to access the group home page
Incidents and Problems table to view summary information about all incidents or
specific categories of incidents.
Click Customize to access the Customize Dashboard page. This page allows you to
change the refresh frequency and display options for the Member Targets table at the
top of the dashboard. You can either show all individual targets or show by target
type. There is also the option to expand or contract the Incidents and Problems table at
the bottom. To change the columns shown in the Member Targets table, go to the
Columns tab of the Edit Group page which you can access by choosing Target Setup
from the Group menu.
In the Group by Target Type mode, the Dashboard displays information of the targets
based on the specific target types present in the group or system. The statuses and
incidents displayed are rolled up for the targets in that specific target type.
If you minimize the dashboard window, pertinent alert information associated with
the group or system is still displayed in the Microsoft Windows toolbar.
You can use Information Publisher reports to make the System Dashboard available to
non-Enterprise Manager users. First, create a report and include the System
Monitoring Dashboard reporting element. In the report definition, choose the option,
Allow viewing without logging in to Enterprise Manager. Once this is done, you can
view it from the Enterprise Manager Information Publisher Reports website.
6.3 Using Out-of-Box Reports
Enterprise Manager provides several out-of-box reports for groups as part of the
reporting framework, called Information Publisher. These reports display important
administrative information, such as hardware and operating system summaries across
all hosts within a group, and monitoring information, such as outstanding alerts and
incidents for a group.
You can access these reports from the Information Publisher Reports menu item on
the Groups menu.
See Also:
Chapter 26, "Using Information Publisher"
6-12 OracleВ® Enterprise Manager Administration
7
Using Administration Groups
7
Administration groups greatly simplify the process of setting up targets for
management in Enterprise Manager by automating the application of management
settings such as monitoring settings or compliance standards. Typically, these settings
are manually applied to individual target, or perhaps semi-automatically using custom
scripts. However, by defining administration groups, Enterprise Manager uses specific
target properties to direct the target to the appropriate administration group and then
automatically apply the requisite monitoring and management settings. Any change to
the monitoring setting will be automatically applied to the appropriate targets in the
administration group. This level of automation simplifies the target setup process and
also enables a datacenter to easily scale as new targets are added to Enterprise
Manager for management.
This chapter covers the following topics:
в– What is an Administration Group?
в– Planning
в– Implementing Administration Groups and Template Collections
в– Removing Administration Groups
For a video tutorials on using administration
groups and template collections, see:
Instructional Videos:
Use Administration Groups and Template Collections - Part 1
https://apex.oracle.com/pls/apex/f?p=44785:24:64247952489
65:::24:P24_CONTENT_ID%2CP24_PREV_PAGE:5732%2C24
Use Administration Groups and Template Collections - Part 2
https://apex.oracle.com/pls/apex/f?p=44785:24:15101831740
469:::24:P24_CONTENT_ID%2CP24_PREV_PAGE:5733%2C24
7.1 What is an Administration Group?
Administration groups are a special type of group used to fully automate application
of monitoring and other management settings targets upon joining the group. When a
target is added to the group, Enterprise Manager applies these settings using a
template collection consisting of monitoring templates, compliance standards, and
cloud policies. This completely eliminates the need for administrator intervention. The
following illustration demonstrates the typical administration group workflow:
Using Administration Groups
7-1
What is an Administration Group?
The first step involves setting a target's Lifecycle Status property when a target is first
added to Enterprise Manager for monitoring. At that time, you determine where in the
prioritization hierarchy that target belongs; the highest level being "mission critical"
and the lowest being "development."
Target Lifecycle Status prioritization consists of the following levels:
в– Mission Critical (highest priority)
в– Production
в– Stage
в– Test
в– Development (lowest priority)
As shown in step two of the illustration, once Lifecycle Status is set, Enterprise Manger
uses it to determine which administration group the target belongs.
In order to prevent different monitoring settings to be applied to the same target,
administration groups were designed to be mutually exclusive with other
administration groups in terms of group membership. Administration groups can also
be used for hierarchically classifying targets in an organization, meaning a target can
belong to at most one administration group. This also means you can only have one
administration group hierarchy in your Enterprise Manager deployment.
For example, in the previous illustration, you have an administration group hierarchy
consisting of two subgroups: Production targets and Test targets, with each subgroup
having its own template collections. In this example, the Production group inherits
7-2 OracleВ® Enterprise Manager Administration
Planning
monitoring settings from monitoring template A while targets in the Test subgroup
inherit monitoring settings from monitoring template B.
7.1.1 Developing an Administration Group
In order to create an administration group, you must have both Full Any Target and
Create Privilege Propagating Group target privileges.
Developing an administration group is performed in two phases:
в– в– Planning
–
Plan your administration group hierarchy by creating a group hierarchy in a
way reflects how you monitor your targets.
–
Plan the management settings associated with the administration groups in
the hierarchy.
*
Management settings: Monitoring settings, Compliance standard settings,
Cloud policy settings
*
For Monitoring settings, you can have additional metric settings or
override metric settings lower in your hierarchy
*
For Compliance standards or Cloud policies, additional rules/policies
lower in the hierarchy are additive
Implementation
–
Enter the group hierarchy definition and management settings in Enterprise
Manager.
*
Create the administration group hierarchy.
*
Create the monitoring templates, compliance standards, cloud policies
and add these to template collections.
*
Associate template collections with administration groups.
*
Add targets to the administration group by assigning the appropriate
values to the target properties such that Enterprise Manager automatically
adds them to the appropriate administration group.
7.2 Planning
As with any management decision, the key to effective implementation is planning
and preparation. The same holds true for administration groups.
Step 1: Plan Your Group Hierarchy
You can only have one administration group hierarchy in your Enterprise Manager
deployment, thus ensuring that administration group member targets can only
directly belong to one administration group. This prevents monitoring conflicts from
occurring as a result of having a target join multiple administration groups with
different associated monitoring settings.
To define the hierarchy, you want to think about the highest (root) level as consisting
of all targets that have been added to Enterprise Manager. Next, think about how you
want to divide your targets along the lines of how they are monitored, where targets
that are monitored in one way are in one group, and targets that are monitored in
another way are part of another group. For example, Production targets might be
monitored one way and Test targets might be monitored in another way. You can
further divide individual groups if there are further differences in monitoring. For
Using Administration Groups
7-3
Planning
example, your Production targets might be further divided based on the line of
business they support because they might have additional metrics that need to be
monitored for that line of business. Eventually, you will end up with a hierarchy of
groups under a root node.
The attributes used to define each level of grouping and thus the administration group
membership criteria are based on global target properties, which are attributes of every
target that specify operational information within the organization. For example,
location, line of business to which it belongs, and lifecycle status. Target properties that
can be used in the definition of administration groups are:
в– Lifecycle Status
Note: Lifecycle Status target property is of particular importance
because it denotes a target’s operational status. Lifecycle Status can be
any of the following: Mission Critical, Production, Staging, Test, or
Development.
в– Location
в– Line of Business
в– Department
в– Cost Center
в– Contact
в– Platform
в– Operating System
в– Target Version
в– Customer Support Identifier
в– Target Type (Allowed but not a global target property.)
You cannot manually add targets to an administration group. Instead, you set the
target properties of the target (prospective group member) to match the membership
criteria defined for the administration group. Once the target properties are set,
Enterprise Manager automatically adds the target to the appropriate administration
group.
7-4 OracleВ® Enterprise Manager Administration
Planning
Enterprise Manager Administrators and Target Properties
When creating an Enterprise Manager administrator, you can associate properties such
as Contact, Location, and Description. However, there are additional resource
allocation properties that can be associated with their profile. These properties are:
в– Department
в– Cost Center
в– Line of Business
It is important to note that these properties are persistent--when associated with an
administrator, the properties (which mirror, in part, the target properties listed above)
are automatically passed to any targets that are discovered or created by the
administrator.
Example
In the following administration group hierarchy, two administration groups are
created under the node Root Administration Group, Production and Test, because
monitoring settings for production targets will differ from the monitoring settings for
test targets.
In this example, the group membership criteria are based on the Lifecycle Status target
property. Targets whose Lifecycle Status is 'Production' join the Production group and
targets whose Lifecycle Status is 'Test' join the Test group. For this reason, Lifecycle Status
is the target property that determines the first level in the administration group
hierarchy. The values of Lifecycle Status property determine the membership criteria
of the administration groups in the first level: Production group has membership
criteria of "Lifecycle Status = Production" and Test group has membership criteria of
"Lifecycle Status = Test' membership criteria.
Additional levels in the administration group hierarchy can be added based on other
target properties. Typically, additional levels are added if there are additional
monitoring (or management) settings that need to be applied and these could be
different for different subsets of targets in the administration group. For example, in
the Production group, there could be additional monitoring settings for targets in
Finance line of business that are different from targets in Sales line of business. In this
case, an additional level based on Line of Business target property level would be
added.
The end result of this hierarchy planning exercise is summarized in the following
table.
Using Administration Groups
7-5
Planning
Root Level (First Row)
Level 1 target property
(second row)
Level 2 target property
(third row)
Lifecycle Status
Line of Business
Production or Mission
Critical
Finance
Staging or Test or
Development
Finance
Sales
Root Administration Group
Sales
Each cell of the table represents a group. The values in each cell represent the values of
the target property that define membership criteria for the group.
It is possible to have the group membership criteria be based on more than one target
property value. In that case, any target whose target property matches any of the
values will be added to the group. For example, in the case of the Production group, if
the Lifecycle Status of a target is either Production or Mission Critical, then it will be
added to the Production group.
It is also important to remember that group membership criteria is cumulative. For
example, for the Finance group under Production or Mission Critical group, a target must
have its Lifecycle Status set to Production or Mission Critical AND its Line of Business set
to Finance before it can join the group. If the target has its Lifecycle Status set to
Production but does not have its Line of Business set to Finance or Sales, then it does not
join any administration group.
For this planning example, the resulting administration group hierarchy would appear
as shown in the following graphic.
It is important to note that a target can become part of hierarchy if and only if its
property values match criteria at both the levels. A target possessing matching values
for lifecycle status cannot become member of the administration group at the first level.
Also, all targets in the administration group hierarchy will belong to the lowest level
groups.
Step 2: Assign Target Properties
After establishing the desired administration group hierarchy, you must make sure
properties are set correctly for each target to ensure they join the correct administration
group. Using target properties, Enterprise Manager automatically places targets into
the appropriate administration group without user intervention. For targets that have
already been added to Enterprise Manager, you can also set the target properties via
the console or using the EM CLI verb set_target_property_value, See the Enterprise
7-6 OracleВ® Enterprise Manager Administration
Planning
Manager Command Line Interface guide for more information. Note that when
running set_target_property_value, any prior values of the target property are
overwritten. If you set target properties before hierarchy creation, it will join the group
after it is created. The targets whose properties are set using EM CLI will automatically
join their appropriate administration groups. Target properties can, however, be set
after the administration group hierarchy is created.
For small numbers of targets, you can change target properties directly from the
Enterprise Manager console.
1.
From an Enterprise Manager target’s option menu, select Target Setup, then select
Properties.
2.
On the Target Properties page, click Edit to change the property values.
Using Administration Groups
7-7
Planning
To help you specify the appropriate target property values used as administration
group criteria, pay attention to the instructional verbiage at the top of the page.
3.
Once you have set the target properties, click OK.
For large numbers of targets, it is best to use the Enterprise Manager Command Line
Interface (EM CLI) set_target_property_value verb to perform a mass update.
For more information about this EM CLI verb, see the Enterprise Manager Command
Line Interface guide.
Administration groups are privilege-propagating: Any privilege that you grant on the
administration group to a user (or role) automatically applies to all members of the
administration group. For example, if you grant Operator privilege on the Production
administration group to a user or role, then the user or role automatically has Operator
privileges on all targets in the administration group. Because administration groups
are always privilege propagating, any aggregate target that is added to an
administration group must also be privilege propagating.
An aggregate target is a target containing other member
targets. For example, a Cluster Database (RAC) is an aggregate target
has RAC instances.
Note:
A good example of aggregate target is the Privilege Propagating Group. See
"Managing Groups" on page 6-1 for more information.
At any time, you can use the All Targets page to view properties across all targets. To
view target properties:
1.
From the Targets menu, select All Targets to display the All Targets page.
2.
From the View menu, select Columns, then select Show All.
3.
Alternatively, if you are interested in specific target properties, choose Columns
and then select Show More Columns to display column selector, as shown in
following graphic.
7-8 OracleВ® Enterprise Manager Administration
Implementing Administration Groups and Template Collections
Step 3: Prepare for Creating Template Collections
Template collections contain the monitoring settings and other management settings
that are meant to be applied to targets as they join the administration group.
Monitoring settings for targets are defined in monitoring templates. Monitoring
templates are defined on a per target type basis, so you will need to create monitoring
templates for each of the different target types in your administration group. You will
most likely create multiple monitoring templates to define the appropriate monitoring
settings for an administration group. For example, you might create a database
Monitoring template containing the metric settings for your production databases and
a separate monitoring template containing the settings for your non-production
databases. Other management settings that can be added to a template collection
include Compliance Standards and Cloud Policies. Ensure all of these entities that you
want to add to your template collection are correctly defined in Enterprise Manager
before adding them to template collections.
If you have an administration group hierarchy defined with more than two levels,
such as the hierarchy shown in the following figure, it is important to understand how
management settings are applied to the targets in the administration group.
Each group in the administration group hierarchy can be associated with a template
collection (containing monitoring templates, compliance standards, and cloud
policies). If you associate a template collection containing monitoring settings with the
Production group, then the monitoring settings will apply to the Finance and Sales
subgroup under Production. If the Finance group under Production has additional
monitoring settings, then you can create a monitoring template with only those
additional monitoring settings. (Later, this monitoring template should be added to
another template collection and associated with the Finance group). The monitoring
settings from the Finance Template Collection will be logically combined with the
monitoring settings from the Production Template Collection. In case there are duplicate
metric settings in both template collections, then the metric settings from the Finance
Template Collection takes precedence and will be applied to the targets in the Finance
group. This precedence rule only applies to the case of metric settings. In the case of
compliance standard rules and cloud policies, even if there are duplicate compliance
standard rules and cloud policies in both template collections, they will be all applied
to the targets in the Finance group.
Once you have completed all the planning and preparation steps, you are ready to
begin creating an administration group.
7.3 Implementing Administration Groups and Template Collections
With the preparatory work complete, you are ready to begin the four step process of
creating an administration group hierarchy and template collections. The
Using Administration Groups
7-9
Implementing Administration Groups and Template Collections
administration group user interface is organized to guide you through the creation
process, with each tab containing the requisite operations to perform each step.
This process involves:
1.
Creating the administration group hierarchy.
2.
Create monitoring templates.
3.
Creating template collections.
4.
Associating template collections to administration group.
5.
Synchronizing the targets with the selected items.
The following graphic shows a completed administration group hierarchy with
associated template collections. It illustrates how Enterprise Manager uses this to
automate the application of target monitoring settings.
7.3.1 Creating the Administration Group Hierarchy
The following four primary tasks summarize the administration group creation
process. These tasks are conveniently arranged in sequence via tabbed pages.
In order to create the administration group hierarchy, you
must have both Full Any Target and Create Privilege Propagating
Group target privileges.
Important:
Task 1: Access the Administration Group and Template Collections page.
Task 2: Define the hierarchy.
From the Hierarchy tab, you define the administration group hierarchy that matches
the way you manage your targets. See Section 7.3.1.2, "Defining the Hierarchy".
7-10 OracleВ® Enterprise Manager Administration
Implementing Administration Groups and Template Collections
Task 3: Define the Template Collections.
From the Template Collections tab, you define the monitoring and management
settings you want applied to targets. See Section 7.3.1.3, "Defining Template
Collections".
Task 4: Associate the Template Collections with the Administration Groups
From the Associations tab, you tie the monitoring and management settings to the
appropriate administration group. See Section 7.3.1.4, "Associating Template
Collections with Administration Groups".
7.3.1.1 Accessing the Administration Group Home Page
All administration group operations are performed from the Administration Groups
home page.
From the Setup menu, select Add Target and then select Administration Groups. The
Administration Groups home page displays.
Read the relevant information on the Getting Started page. The information contained
in this page summarizes the steps outlined in this chapter. For your convenience, links
are provided that take you to appropriate administration group functions, as well as
the Enterprise Manager All Targets page where you can view target properties.
7.3.1.2 Defining the Hierarchy
On this page you define the administration group hierarchy that reflects the
organizational hierarchy you planned earlier and which target properties are
associated with a particular hierarchy level.
On the left side of the page are two tables: Hierarchy Levels and Hierarchy Nodes.
Using Administration Groups 7-11
Implementing Administration Groups and Template Collections
The Hierarchy Levels table allows you to add the target properties that define
administration group hierarchy. The Hierarchy Nodes table allows you to define the
values associated with the target properties in the Hierarchy Levels table. When you
select a target property, the related property values are made available in the
Hierarchy Nodes table, where you can add/remove/merge/split the values. In the
Hierarchy Nodes table, each row corresponds to a single administration group. The
Short Value column displays abbreviated value names that are used to auto-generate
group names.
The Hierarchy Levels table allows you to add the target properties that define each
level in the administration group hierarchy. The Hierarchy Nodes table allows you to
define the values associated with the target properties in the Hierarchy Levels table.
Each row in the Hierarchy Nodes table will correspond to a node or group in the
administration group hierarchy for that level. When you select a target property in the
Hierarchy Levels table, the related property values are made available in the
Hierarchy Nodes table, where you can add/remove/merge/split the values. Merge
two or more values if either value should be used as membership criteria for the
corresponding administration group. The Short Value column displays abbreviated
value names that are used to auto-generate group names.
Adding a Hierarchy Level
1. On the Administration Group page, click the Hierarchy tab.
2.
From the Hierarchy Levels table, click Add and choose one of the available target
properties. You should add one property/level at a time instead of all properties at
once.
3.
With the target property selected in the Hierarchy Levels table, review the list of
values shown in the Hierarchy Nodes table. The values of the target property in
the Hierarchy Nodes table.
Enterprise Manager finds all existing values of the target property across all
targets and displays them in the Hierarchy Nodes table. For some target
properties, such as Lifecycle Status, predefined property values already exist and
7-12 OracleВ® Enterprise Manager Administration
Implementing Administration Groups and Template Collections
are automatically displayed in the Hierarchy Nodes table. You can select and
remove target property values that will not be used as membership criteria in any
administration group. However, property values that are not yet available but will
be used as administration group membership criteria, will need to be added.
The next step shows you how to add property values.
4.
From the Hierarchy Nodes table, click Add. The associated property value add
dialog containing existing values from various targets displays. Add the requisite
value(s). Multiple values can be specified using a comma separated list. For
example, to add multiple locations such as San Francisco and Zurich, add the
Location target property to the Hierarchy Level table. Select Location and then
click Add in the Hierarchy Nodes table. The Values for Hierarchy Nodes dialog
displays. Enter "San Francisco,Zurich" as shown in the following graphic.
Extending Administration Group Hierarchy Maximum Limits
There is a default maximum for the number of values that can be supported for a
target property as administration group criteria. If you see a warning message
indicating that you have reached this maximum value, you can extend it using the
OMS property admin_groups_width_limit. Specify the maximum number of values
that should be supported for a target property. For example, to support up to 30
values for a target property that will be used in administration group criteria, set
the admin_groups_width_limit as follows (using the OMS emctl utility):
emctl set property -name admin_groups_width_limit -value 30 -module emoms
You can also add up to four levels after the root node of an administration group
hierarchy. If there is a need to add additional level, you will first need to change
the OMS admin_groups_height_limit property to the maximum height limit. For
example, if you want to create to administration group hierarchy consisting of five
levels after the root node, set the admin_groups_height_limit property as follows
(using the OMS emctl utility):
emctl set property -name admin_groups_height_limit -value 5 -module emoms
This is a global property and only needs to be set once using the emctl utility of
any OMS. This is also a dynamic property and does not require a stop/restart of
the OMS in order to take effect.
Click OK. The two locations "San Francisco" and "Zurich" appear as nodes in the
Preview pane as shown in the following graphic.
Using Administration Groups 7-13
Implementing Administration Groups and Template Collections
Under certain circumstances, it may be useful to treat multiple property values as
one: Targets may have different target property values, but should belong to the
same administration group because they have same monitoring profile/settings.
For example, if a combination of values is needed, such as Production or Mission
Critical for the Lifecycle Status property, they need to be merged (combined into a
single node).
To merge property values:
1.
Select a target property from the list of chosen properties in the Hierarchy
Levels table. The associated property values are displayed.
2.
Select two or more property values by holding down the Shift key and clicking
on the desired values.
3.
Click Merge.
5.
Continue adding hierarchy levels until the group hierarchy is complete. The
Preview pane dynamically displays any changes you make to your administration
group hierarchy.
6.
Set the time zone for the group.
1.
Click on the group name. The Administration Group Details dialog displays
allowing you to select the appropriate time zone.
7-14 OracleВ® Enterprise Manager Administration
Implementing Administration Groups and Template Collections
The administration group time zone is used for displaying group charts and
also for scheduling operations on the group. Because this is also the default
time zone for all subgroups that may be created under this group, you should
specify the time zone at the highest level group in the administration group
hierarchy before the subgroups are created. Note that the parent group time
zone will be used when creating any child subgroups, but user can always
select a child subgroup and change its time zone.
The auto-generated name can also be changed.
7.
Click Create to define the hierarchy.
Review and define the complete hierarchy before
clicking Create.
IMPORTANT:
Even after your administration group hierarchy has been created, you can always
make future updates if organizational needs change. For example,
adding/removing group membership criteria property values, which equates to
creating/deleting additional administration groups for a given level. Using the
previous example, if in addition to San Francisco and Zurich you add more
locations, say New York and Bangalore, you can click Add in the Hierarchy Node
table to add additional locations, as shown in the following graphic. For more
information about changing the administration group hierarchy, see "Changing the
Administration Group Hierarchy" on page 7-28.
Click Update to save your changes.
7.3.1.3 Defining Template Collections
A template collection is an assemblage of monitoring/management settings to be
applied to targets in the administration group. Multiple monitoring templates can be
added to a template collection that in turn is associated with an administration group.
However, you can only have one monitoring template of a particular target type in the
template collection. The monitoring template should contain the complete set of metric
settings for the target in the administration group. You should create one monitoring
template for each type of target in the administration group. For example, you can
Using Administration Groups 7-15
Implementing Administration Groups and Template Collections
have a template collection containing a template for database and a template for
listener, but you cannot have a template collection containing two templates for
databases. When members targets are added to an administration group, the template
monitoring and management settings are automatically applied. A template will
completely replace all metric settings in the target. This means applying the template
copies over metric settings (thresholds, corrective actions, collection schedule) to the
target, removes the thresholds of the metrics that are present in the target, but not
included in the template. Removing of thresholds disables alert functionality for these
metrics. Metric data will continue to be collected.
Template collections may consist of three types of monitoring/management setting
categories:
в– Monitoring Templates (monitoring settings)
в– Compliance Standards (compliance policy rules)
в– Cloud Policies (cloud policies such as determining when to start virtual machines
or scale out clusters).
When creating a template collection, you can use the default monitoring templates,
compliance standards, or cloud templates supplied with Enterprise Manager or you
can create your own. For more information, see Chapter 8, "Using Monitoring
Templates."
To create a template collection:
1.
Click the Template Collections tab. The Template Collection page displays.
2.
Click Create.
3.
In the Name field, specify the template collection name.
4.
Click the template collection member type you want to add (Monitoring Template,
Compliance Standard, Cloud Polices). The requisite definition page appears.
5.
Click Add. A list of available template entities appears.
7-16 OracleВ® Enterprise Manager Administration
Implementing Administration Groups and Template Collections
6.
Select the desired template entities you want added to the template collection.
7.
Click OK.
8.
Continue adding template entities (Monitoring Template, Compliance Standard,
Cloud Policies) as required.
9.
Click Save. The newly defined collection appears in the Template Collections
Library.
10. To create another template collection, click Create and create and repeat steps two
through eight. Repeat this process until you have created all required template
collections.
Note: When editing existing template collections, you can back out
of any changes made during the editing session by clicking Cancel.
This restores the template collection to its state when it was last saved.
7.3.1.3.1 Required Privileges To create a template collection, you must have the Create
Template Collection resource privilege. To include a monitoring template into a template
collection, you need at least View privilege on the specific monitoring template or View
Any Monitoring Template privilege, which allows you to view any monitoring template
and add it to the template collection. The following table summarizes privilege
requirements for all Enterprise Manager operations related to template collection
creation.
Enterprise Manager Operation
Minimum Privilege Requirement
Create administration group hierarchy.
Full Any Target
Create Privilege Propagating Group
Create monitoring templates.
Create Monitoring Template
Using Administration Groups 7-17
Implementing Administration Groups and Template Collections
Enterprise Manager Operation
Minimum Privilege Requirement
Create template collection.
Create template collection (resource
privilege).
VIEW on the monitoring template to
be added to the template collection
or
View any monitoring template
(resource privilege).
Create compliance standards.
Create Compliance Entity
No privileges are required to view
compliance standards.
Create cloud policies.
Create Any Policy
View Cloud Policy
Associate template collection with administration
group.
VIEW on the specific template
collection.
Manage Target Metrics on the group.
Perform on-demand synchronization.
OPERATOR on the group or Manage
Target Metrics.
Define global synchronization schedule.
Enterprise Manager Super
Administrator privileges.
Set the value of target properties for a target (allows
the target to "join" an administration group).
Configure Target on the specific target
Delete an administration group hierarchy.
Full Any Target
7.3.1.3.2 Corrective Action Credentials A corrective action is an automated task that is
executed in response to a metric alert. When a corrective action is part of a monitoring
template/template collection, the credentials required to execute the corrective action
will vary depending on how the template is applied.
The two situations below illustrate the different credential requirements.
в– The corrective action is part of a monitoring template that is manually applied to a target.
When the corrective action runs, it can use one of the following:
–
The preferred credentials of the user who is applying the template
or
–
The user-specified named credentials.
The user selects the desired credential option during the template apply operation.
в– The corrective action is part of a monitoring template within a template collection that is
associated with an administration group.
When the corrective action runs, the preferred credentials of the user who is
associating the template collection with the administration group is used.
7.3.1.4 Associating Template Collections with Administration Groups
Once you have defined one or more template collections, you need to associate them
to administration groups in the hierarchy. You can associate a template collection with
one or more administration groups. As a rule, you should associate the template
collection with the applicable administration group residing at the highest level in the
7-18 OracleВ® Enterprise Manager Administration
Implementing Administration Groups and Template Collections
hierarchy as the template collection will also be applied to targets joining any
subgroup.
The Associations page displays the current administration group hierarchy diagram.
Each administration group in the hierarchy can only be associated with one template
collection.
Associating a Template Collection with an administration group
Note: For users that do not have View privilege on all administration
groups, you can also perform the association/disassociation operation
from the Groups page (from the Targets menu, select Groups).
1.
Click the Associations tab. The Associations page displays.
2.
Select the desired administration group in the hierarchy.
3.
Click Associate Template Collection. The Choose a Template Collection dialog
displays.
4.
Choose the desired template collection and click Select. The list of targets affected
by this operation is displayed. Confirm or discard the operation.
All sub-nodes in the hierarchy will inherit the selected
template collection.
Note:
5.
Repeat steps 1-3 until template collections have been associated with the desired
groups.
The target privileges of the administrator who performs the
association will be used when Enterprise Manager applies the
template to the group. The administrator needs at least Manage Target
Metrics privileges on the group.
Note:
Using Administration Groups 7-19
Implementing Administration Groups and Template Collections
Settings from monitoring templates applied at lower levels in
the hierarchy override settings inherited from higher levels. This does
not apply to compliance standards or cloud policies.
Note:
Searching for Administration Groups
While the administration group UI is easy to navigate, there may be cases where the
administration group hierarchy is inordinately large, thus making it difficult to find
individual groups. At the upper right corner of the Associations page is a search
function that greatly simplifies finding groups in a large hierarchy.
Figure 7–1 Administration Group Search Dialog
To search for a specific administration group:
1.
If not already displayed, expand the Search interface.
2.
Enter either a full or partial group name and click Search.
As shown in the graphic, the search results display a list of administration groups that
match the search criteria. You can then choose an administration group from the list by
double-clicking on the entry. The administration group hierarchy will then display a
vertical slice (subset) of the administration group hierarchy from the root node to the
group you selected.
7-20 OracleВ® Enterprise Manager Administration
Implementing Administration Groups and Template Collections
Figure 7–2 Administration Group Search: Graphical Display
To restore the full administration group hierarchy, click Clear.
Group Names and Searches
In order to perform effective searches for specific administration groups, it is helpful to
know how Enterprise Manager constructs an administration group name: Enterprise
Manger uses the administration group criteria to generate names. For example, you
have an administration group with the following criteria:
в– Lifecycle Status: Development or Mission Critical
в– Department: DEV
в– Line of Business: Finance or HR
в– Location: Bangalore
Enterprise Manager assembles a group name based on truncated abbreviations. In this
example, the generated administration group name is DC-DEV-FH-Bang-Grp
As you are building the hierarchy, you can change the abbreviation associated with
each value (this is the Short Value column next to the property value in the Hierarchy
Nodes table. Hence, you can specify a short value and Enterprise Manager will use
that value when constructing new names for any subgroups created.
During the design phase of an administration group, you have the option of specifying
a custom name. However, if there is large number of groups, it is easier to allow
Enterprise Manager to generate unique names.
Setting the Global Synchronization Schedule
In order to apply the template collection/administration group association, you must
set up a global synchronization schedule. This schedule is used to perform
synchronization operations, such as applying templates to targets in administration
groups. If no synchronization schedule is set up, when a target joins an administration
group, Enterprise Manager will auto-apply the associated template. However, if there
are changes to the template later on, then Enterprise Manager will only apply these
based on synchronization schedule, otherwise these operations are pending.When
Using Administration Groups 7-21
Implementing Administration Groups and Template Collections
there are any pending synchronization operations, they will be scheduled on the next
available date based on the synchronization schedule.
Important: You must set the synchronization schedule as there is no
default setting. You can specify a non-peak time such as weekends.
To set up the synchronization schedule:
1.
Click Synchronization Schedule. The Synchronization Schedule dialog displays.
2.
Click Edit and then choose a date and time you want any pending sync operations
(For example, template apply operations) to occur. By default, the current date and
time is shown.
You can specify a start date for synchronization operations and
interval in days. Whenever there are any pending sync operations,
then they will be scheduled on the next available date based on this
schedule.
Note:
3.
Click Save.
When Template Collection Synchronization Occurs
The following table summarizes when Template Collection Synchronization
operations (such as apply operations) occur on targets in administration groups.
Action
When Synchronization Occurs
Target is added to an administration group (by Immediate upon joining the administration
setting its target properties)
group.
Template collection is associated with the
administration group.
7-22 OracleВ® Enterprise Manager Administration
Targets in an administration group will be
synchronized based on next scheduled date in
global Synchronization Schedule.
Implementing Administration Groups and Template Collections
Action
When Synchronization Occurs
Changes are made to any of the templates in
the template collection.
Targets in an administration group will be
synchronized based on the next scheduled
date in global Synchronization Schedule.
Target is removed from an administration
group (by changing its target properties).
No change in target's monitoring settings.
Compliance Standards and Cloud Policies will
be disassociated with the target.
Immediate synchronization operation occurs.
Template collection is disassociated with
administration group.
No change in target's monitoring settings for
all the targets in the administration group.
Compliance Standards and Cloud Policies will
be disassociated with the target.
Targets under the administration group will
be synchronized based on next schedule date
in Global Synchronization Schedule.
User performs an on-demand synchronization Immediate synchronization operation occurs.
by clicking on the Start Synchronization
button in the Synchronization Status region
in the administration group's homepage.
Viewing Synchronization Status
You can check the current synchronization status for a specific administration group
directly from the group’s homepage.
1.
Select an administration group in the hierarchy.
2.
Click Goto Group Homepage.
3.
From the Synchronization Status region, you can view the status of the
monitoring template, compliance standard, and/or cloud policies synchronization
(In Sync, Pending, or Failed).
You can initiate an immediate synchronization by clicking Start Synchronization.
Using Administration Groups 7-23
Implementing Administration Groups and Template Collections
Group Member Type and Synchronization
There are two types of administration group member targets: Direct and Indirect
в– в– Direct Members: Group members whose target properties match the
administration group criteria. Monitoring settings, compliance standards, cloud
policies from the associated template collection are applied to direct members.
Indirect Members: Indirect members are targets whose target properties DO NOT
match administration group criteria. However, they have been added to the
administration group because their parent target are direct members of the
administration group. These targets are categorized as aggregate targets because
they have other member targets. When such targets are added to a group
(administration group or other types of groups), all members of the aggregate
target are also added to the group. An example of an aggregate target is Oracle
WebLogic Server. If that is added to a group, then all Application Deployment
targets on it are also pulled into the group. Indirect group members will NOT be
part of any template apply/sync operations.
Only direct members are represented in the targets count in the Synchronization Status
region.
1.
From the hierarchy diagram, click on a group name to access the group’s home
page. You can also access this information from All Targets groups page.
2.
From the Group menu, select Members. The Members page displays.
System Targets and Administration Groups
If a system target gets added to an administration group because it matches group
criteria, then the system target and its constituent members are also added. However,
for template apply purposes, it will only operate on the direct members that also
match the administration group criteria. Template apply operations will not occur on
member targets whose target properties do not match administration group criteria.
All other group operations, such as jobs and blackouts, will apply on all members,
both direct and indirect.
Disassociating a Template Collection from a Group
To disassociate a template collection from an administration group.
1.
From the Setup menu, select Add Target and then Administration Groups. The
Administration Group home page displays.
7-24 OracleВ® Enterprise Manager Administration
Implementing Administration Groups and Template Collections
2.
Click on the Associations tab to view the administration group hierarchy diagram.
3.
From the hierarchy diagram, select the administration group with the template
collection you wish to remove. If necessary, use the Search option to locate the
administration group.
4.
Click Disassociate Template Collection. The number of targets affected by this
operation is displayed. Click Continue or Cancel.
The template collection is immediately removed. See "When Template Collection
Synchronization Occurs" on page 7-22 for more information.
Viewing Aggregate (Group Management) Settings
For any administration group, you can easily view what template collection
components (monitoring templates, compliance standards, and/or cloud policies) are
associated with individual group members.
For monitoring templates, the settings for a target could be a
union of two or more monitoring templates from different template
collections.
Note:
1.
From the Setup menu, select Add Target and then Administration Groups. The
Administration Group home page displays.
2.
Click on the Associations tab to view the administration group hierarchy diagram.
3.
From the hierarchy diagram, select the desired administration group.
4.
Click Show Group Management Settings.
The Administration Group Details page displays.
This page displays all aggregate settings for monitoring templates, compliance
standards and cloud policies that will be applied to members of the selected
administration group (listed by target type). The page also displays the
synchronization status of group members.
To can change the display to show a different branch of the administration group
hierarchy, click Select Branch at the upper-right area of the page. This function lets
you display hierarchy branches by choosing different target property values
Using Administration Groups 7-25
Implementing Administration Groups and Template Collections
Viewing the Administration Group Homepage
Like regular groups, each administration group has an associated group homepage
providing a comprehensive overview of group member status and/or activity such as
synchronization status, details of the Associated Template Collection for the group
selected in hierarchy viewer, job activity, or critical patch advisories. To view
administration group home pages:
1.
From the hierarchy diagram, select an administration group.
2.
Click Goto Group Homepage. The homepage for that particular administration
group displays.
Alternatively, from the Enterprise Manger Targets menu, choose Groups. From the
table, you can expand the group hierarchy.
7-26 OracleВ® Enterprise Manager Administration
Implementing Administration Groups and Template Collections
Identifying Targets Not Part of Any Administration Group
From the Associations page, you can determine which targets do not belong to any
administration group by generating an Unassigned Targets Report.
1.
From the Actions menu, select Unassigned Targets Report. The report lists all the
targets that are not part of any administration group. The values for the target
properties defining the administration groups hierarchy are shown.
2.
From the View menu, choose the customization options to display only the
desired information.
Using Administration Groups 7-27
Changing the Administration Group Hierarchy
Note: The Non-Privilege Propagating Aggregate column indicates
whether a target is a non-privilege propagating aggregate. This type of
target cannot be added to an administration group, which are by
design privilege propagating. For this reason, any aggregate target
added to administration group must also be privilege propagating. To
make an aggregate target privilege propagating, use the EM CLI verb
modify_system with -privilege_propagation=true option. For more
information see the Enterprise Manager Command Line Reference.
On this page, you can review the list to see if there any targets that need to be
added to the administration group. Click on the target names shown in this page
to access the target’s Edit Target Properties page where you can change the target
property values. After making the requisite changes and clicking OK, you are
returned to the Unassigned Targets page.
For information on changing target properties, see "Planning" on page 7-3.
3.
Click your browser back button to return to the Administration Groups and
Template Collections homepage.
7.4 Changing the Administration Group Hierarchy
Organizations are rarely static--new lines of business may be added or perhaps groups
are reorganized due to organizational expansion. To accommodate these changes, you
may need to make changes to the existing administration group hierarchy.
Beginning with Enterprise Manager 12c Release 12.1.0.3, you can change the
administration group hierarchy without having to rebuild the entire hierarchy. You can
easily perform administration group alterations such as adding more groups to each
hierarchy level, merging two or more groups, or adding/deleting entire hierarchy
levels. All of these operations can be performed from the Hierarchy page.
7-28 OracleВ® Enterprise Manager Administration
Changing the Administration Group Hierarchy
Important: After making any change to the administration group
hierarchy, click Update to save your changes.
7.4.1 Adding a New Hierarchy Level
Adding a new hierarchy level equates to adding a new target property to the
administration group criteria. For this reason, you must set the value of this target
property for all your targets in order for them to continue to be part of the
administration group hierarchy. Any new target property added/hierarchy level
added will always be added as the bottom-most level of the hierarchy. You cannot
insert a new level between levels.
To insert a hierarchy level, you must remove a hierarchy level, then add the levels you
want. Think carefully before removing a hierarchy level as removing a level will result
in the deletion of groups corresponding to that hierarchy level.
See "Adding a Hierarchy Level" on page 7-12 for step-by-step instructions on adding a
new level.
7.4.2 Removing a Hierarchy Level
Removing a hierarchy level equates to deleting a target property, which in turn causes
groups at that level to be deleted. For this reason, think carefully about the groups that
will be removed when you remove the hierarchy level, especially if those groups are
used in other functional areas of Enterprise Manager.
To remove a hierarchy level:
1.
On the Administration Group page, click the Hierarchy tab.
2.
From the Hierarchy Levels table, select a hierarchy level and click Remove
3.
Click Update to save your changes.
If any of those groups have an associated template collection, then the monitoring
settings of the subgroups of the deleted group will be impacted since the subgroups
obtained monitoring settings from the associated template collection. You may need to
review the remaining template collections and re-associate the template collection with
the appropriate administration group.
7.4.3 Merging Administration Groups
If you want to merge two or more administration groups, you merge their
corresponding target property criteria in the administration group hierarchy
definition. The group merge operation consists of retaining one of the groups to be
merged and then moving over the targets from the other groups into the group that is
retained. Once the targets have been moved, the other groups will be deleted.
You choose which group is retained by choosing its corresponding target property
value. The group(s) containing the selected target property value as part of its criteria
is retained. If the retained target property criteria corresponds to multiple groups, i.e.
group containing subgroups, the movement of targets will actually occur at the lowest
level administration groups since the targets only reside in the lowest level
administration groups. The upper-level administration groups' criteria will be
updated to include the criteria of the other groups that have been merged into it.
To merge groups:
Using Administration Groups 7-29
Changing the Administration Group Hierarchy
1.
Select a target property from the list of chosen properties in the Hierarchy Levels
table. You choose the target property corresponding to the groups you want to
merge.
For example, let us assume you want to merge <Devt-Group> with <Test or Stage
Group>. In the hierarchy, this corresponds to target property Lifecycle Status. The
associated property values are displayed.
2.
Select two or more property values corresponding to the groups you would like to
merge by holding down the Shift or CTRL key and clicking on the desired values.
3.
Click Merge.
The Merge Values dialog displays.
7-30 OracleВ® Enterprise Manager Administration
Changing the Administration Group Hierarchy
Again, by merging membership criteria (target properties), you are merging
administration groups and their respective subgroups. You choose the
administration group to be retained. The other groups will be merged into that
group.
4.
Choose the group you wish to retain and specify whether you want to use the
existing name of the retained group or specify a new name.
Important: When deciding which group to retain, consider choosing
the group that is used in most group operations such as incident rule
sets, system dashboard, or roles. These groups will be retained and the
members of the other merged groups will join the retained groups.
After the merge, group operations on the retained groups will also
now apply to the members from the other merged groups. Doing so
minimizes the impact of the merge.
5.
Click OK to merge the groups.
6.
Click Update to save the new hierarchy.
Example
Your administration group consists of the following:
Hierarchy Levels
в– Lifecycle Status
в– Line of Business
Hierarchy Nodes
в– в– Lifecycle Status
–
Development
–
Mission Critical or Production
–
Staging or Test
Line of Business
–
Online Store
–
Sales
–
Finance
The following graphic shows the administration group hierarchy.
Using Administration Groups 7-31
Changing the Administration Group Hierarchy
You decide that you want to merge the Mission Critical or Production group with the
Staging or Test group because they have the same monitoring settings.
Choose Lifecycle Status from the Hierarchy Levels table.
From the Hierarchy Nodes table, choose both Mission Critical or Production and Staging
or Test.
Select Merge from the Hierarchy Nodes menu.
The Merge Values dialog displays. In this case, you want to keep original name
(Mission Critical or Production) of the retained group.
After clicking OK to complete the merge, the resulting administration group hierarchy
is displayed. All targets from Test-Sales group moved to the Prod-Sales group. The
Test-Sales group was deleted. All targets from the Test-Finance group moved to the
Prod-Finance group. The Test-Finance group got deleted.
Click Update to save the changes.
7.4.4 Removing Administration Groups
You can completely remove an administration group hierarchy or just individual
administration groups from the hierarchy. Deleting an administration group will not
delete targets or template collections, but it will remove associations. Any stored
membership criteria is removed. When you delete an administration group, any stored
membership criteria is removed.
To remove the entire administration group hierarchy:
1.
From the Setup menu, select Add Target, then select Administration Groups.
2.
Click on the Hierarchy tab.
3.
Click Delete.
To remove individual administration groups from the hierarchy:
7-32 OracleВ® Enterprise Manager Administration
Changing the Administration Group Hierarchy
1.
From the Setup menu, choose Add Target, then select Administration Groups.
2.
Click on the Hierarchy tab.
3.
From the Hierarchy Levels table, choose the target property that corresponds to
the hierarchy level containing the administration group to be removed.
4.
From the Hierarchy Nodes table, select the administration group (Property Value
for Membership Criteria) to be removed.
5.
Choose Remove from the drop-down menu.
6.
Click Update.
Using Administration Groups 7-33
Changing the Administration Group Hierarchy
7-34 OracleВ® Enterprise Manager Administration
8
Using Monitoring Templates
8
Monitoring templates simplify the task of setting up monitoring for large numbers of
targets by allowing you to specify the monitoring and Metric and Collection Settings
once and applying them to many groups of targets as often as needed.
This chapter covers the following topics:
в– About Monitoring Templates
в– Definition of a Monitoring Template
в– Default Templates (Auto Apply Templates)
в– Viewing a List of Monitoring Templates
в– Creating a Monitoring Template
в– Editing a Monitoring Template
в– Applying Monitoring Templates to Targets
в– Comparing Monitoring Templates with Targets
в– Comparing Metric Settings Using Information Publisher
8.1 About Monitoring Templates
Monitoring templates let you standardize monitoring settings across your enterprise
by allowing you to specify the monitoring settings once and apply them to your
monitored targets. You can save, edit, and apply these templates across one or more
targets or groups. A monitoring template is specified for a particular target type and
can only be applied to targets of the same type. For example, you can define one
monitoring template for test databases and another monitoring template for
production databases.
A monitoring template defines all Enterprise Manager parameters you would
normally set to monitor a target, such as:
в– в– Target type to which the template applies.
Metrics (including metric extensions), thresholds, metric collection schedules, and
corrective actions.
Once a monitoring template is defined, it can be applied to your targets. This can be
done either manually through the Enterprise Manager console, via the command line
interface (EM CLI), or automatically using template collections. See "Defining
Template Collections" on page 7-15 for more information. For any target, you can
preserve custom monitoring settings by specifying metric settings that can never be
overwritten by a template.
Using Monitoring Templates 8-1
Definition of a Monitoring Template
Oracle Certified Templates and Oracle Provided Templates
In addition to templates that you create, there are also Oracle-supplied templates.
в– в– Oracle-provided templates: Contains the out-of-box monitoring settings for the
target type. So, if you updated a target's monitoring settings, and later wanted to
revert to the original out-of-box settings, then you can use and apply the
Oracle-provided templates.
Oracle Certified templates: Contains a specific set of metrics for a specific purpose
and the purpose is indicated in the description associated with the template.
Example: The template called Oracle Certified - Enable AQ Metrics for SI Database
contains AQ-related metrics for single instance databases. You would use this
template (or copy the metric settings into your own template) if you want to use
the AQ metrics.
8.2 Definition of a Monitoring Template
A monitoring template defines all Enterprise Manager parameters you would
normally set to monitor a target. A template specifies:
в– Name: A unique identifier for the template. The template name must be globally
unique across all templates defined within Enterprise Manager.
в– Description: Optional text describing the purpose of the template.
в– Target Type: Target type to which the template applies.
в– Owner: Enterprise Manager administrator who created the template.
в– в– Metrics: Metrics for the target type. A monitoring template allows you to specify a
subset of all metrics for a target type. With these metrics, you can specify
thresholds, collection schedules and corrective actions.
Other Collected Items: Additional collected information (non-metric) about your
environment.
8.3 Default Templates (Auto Apply Templates)
Under certain circumstances, Oracle's out-of-box monitoring settings may not be
appropriate for targets in your monitored environment. Incompatible Metric and
Collection Settings for specific target types can result in unwanted/unintended alert
notifications. Enterprise Manager allows you to set default monitoring templates that
are automatically applied to newly added targets, thus allowing you to apply
monitoring settings that are appropriate for your monitored environment.
Note: Super Administrator privileges are required to define default
monitoring templates.
8.4 Viewing a List of Monitoring Templates
To view a list of all Monitoring Templates, from the Enterprise menu, select
Monitoring and then Monitoring Templates.
The Monitoring Templates page displays all the out-of-box templates and the
templates for which you have has at least VIEW privilege on. Enterprise Manager
Super Administrators can view all templates.
8-2 OracleВ® Enterprise Manager Administration
Creating a Monitoring Template
Figure 8–1 Monitoring Templates
You can begin the monitoring template creation process from this page.
8.5 Creating a Monitoring Template
Monitoring template allow you to define and save monitoring settings for specific
target types. To define a new template:
1.
From the Enterprise menu, select Monitoring and then Monitoring Templates.
2.
Click Create. Enterprise Manager gives you the option of selecting either a specific
target or a target type. Template monitoring settings are populated according to
the selected target or target type. Click Continue.
If the selected target type is either Web Application or Service,
you will only be able to select those targets for which you have
Operator privilege.
Note:
3.
Enter requisite template information on the General, Metric Thresholds, and Other
Collected Items tabs.
On the Metric Thresholds tab, you can delete or add monitoring template metrics.
To delete existing metrics, select one or more metrics and click Remove Metrics
from Template.
To add metrics, click Add Metrics to Template. The Add Metrics to Template page
displays as shown in the following graphic.
Using Monitoring Templates 8-3
Editing a Monitoring Template
On this page, you can select a source from which you can copy metrics to the
template. Sources include specific targets, other monitoring templates, or metric
extensions. Click Continue once your have finished modifying the template
metrics.
4.
Once you have finished entering requisite information, click OK.
8.6 Editing a Monitoring Template
The Monitoring Templates page lists all viewable templates. To edit a template, you
must have FULL access privileges.
To edit a Monitoring Template:
1.
From the Enterprise menu, select Monitoring, and then Monitoring Templates.
2.
Choose the desired template from the table.
3.
Click Edit.
4.
Once you have finished making changes, click OK.
Sharing Access with Other Users
By default, template owners (creators) have FULL access privileges on the template
and Enterprise Manager Super Administrators have FULL access privileges on all
templates. Only the template owner can change access to the template. You, as owner,
can grant VIEW (view the template) or FULL (edit or delete the template) on the
template to a user or role.
8.7 Applying Monitoring Templates to Targets
As mentioned earlier, a monitoring template can be applied to one or more targets of
the same target type, or to composite targets such as groups. For composite targets, the
template is applied to all member targets that are of the appropriate type. If you
applied the template manually or via EM CLI, once a template is applied, future
changes made to the template will not be automatically propagated to the targets: You
must reapply the template to all affected targets
Administration Groups and Template Collections: Applying Monitoring
Templates Automatically
Monitoring templates can be automatically applied whenever a new targets are added
to your Enterprise Manager environment. Automation is carried out through
Administration Groups and Template Collections Administration Groups are a special
type of group used to automate application of monitoring settings to targets upon
joining the group. When a target is added to the administration group Enterprise
Manager applies monitoring settings from the associated template collection
consisting of monitoring templates, compliance standards, and cloud policies. If
changes are later made to the monitoring template, Enterprise Manager automatically
applies the changes to the relevant targets based on the synchronization schedule. For
more information, see "Using Administration Groups" on page 7-1.
8.7.1 Applying a Monitoring Template
To apply a template, you must have at least Manage Target Metrics target privileges on
the destination target(s).
1.
From the Enterprise menu, select Monitoring and then Monitoring Templates.
8-4 OracleВ® Enterprise Manager Administration
Applying Monitoring Templates to Targets
2.
Select the desired template from the table.
3.
Click Apply.
4.
Select the desired apply options and the target(s) to which you want the templates
applied. See Section 8.7.2, "Monitoring Template Application Options" for
additional information.
5.
Click OK.
8.7.2 Monitoring Template Application Options
You can choose aggregate targets such groups, systems or clusters as destination
targets. The templates will apply to the appropriate members of the
group/system/cluster as they currently exist. If new members are later added to the
group, you will need to re-apply the template to those new members. Template
application is performed in the background as asynchronous jobs, so after the apply
operation is performed, you can click on the link under the Pending Apply Operations
column in the main templates table to see any apply operations that still are pending.
When applying a Monitoring Template, metric settings such as thresholds, comparison
operators, and corrective actions are copied to the destination target. In addition,
metric collection schedules including collection frequency and upload interval are also
copied to the target. You determine how Enterprise Manager applies the metric
settings from the template to the target by choosing an apply option.
8.7.2.1 Apply Options
Template apply options control how template metric and policy settings are applied to
a target. Two template apply options are available:
в– в– Template will completely replace all metric settings in the target: When the
template is applied, all metrics and policies defined in the template will be applied
to the target. Pre-existing target monitoring settings not defined in the template
will be disabled: Metric thresholds will be set to NULL or blank. Policies will be
disabled. This effectively eliminates alerts from these metrics and policies by
clearing current severities and violations.
Template will only override metrics that are common to both template and
target: When the template is applied, only metrics and policies common to both
the template and target are updated. Existing target metric and policies that do not
exist in the template will remain unaffected. When this option is selected,
additional template apply options are made available for metrics with key value
settings.
8.7.2.2 Metrics with Key Value Settings
A metric with key value settings is one that can monitor multiple objects at different
thresholds. For example, the Filesystem Space Available(%) metric can monitor
different mount points using different warning and critical thresholds for each mount
point.
When the template contains a metric that has key value settings, you can choose one of
three options when applying this template to a target. As an example, consider the
case where the template has the following metric:
Filesystem Space Available(%)
Using Monitoring Templates 8-5
Applying Monitoring Templates to Targets
Mount Point
Operator
Warning Threshold
Critical Threshold
/
ВЈ
40
20
/private
ВЈ
30
20
/u1
ВЈ
30
20
And a host target has the same metric at different settings:
Mount Point
Operator
Warning Threshold
Critical Threshold
/
ВЈ
30
10
/private
ВЈ
25
15
/private2
ВЈ
20
20
All Others
ВЈ
25
15
These are the results for each option:
1) All key value settings in the template will be applied to the target, any additional
key values settings on the target will not be removed
When the template is applied to the target using this copy option, all the template
settings for the mount points, /, /private, and /U1 will be applied. Existing target
settings for mount points not covered by the template remain unaffected. Thus, the
resulting settings on the target for this metric will be:
Mount Point
Operator
Warning Threshold
Critical Threshold
/
ВЈ
40
20
/private
ВЈ
30
20
/u1
ВЈ
30
20
2) All key value settings in the template will be applied to target, any additional key
value settings on the target will be removed.
When the template is applied to the target using this copy option, all template settings
will be applied to the target. Any object-specific threshold settings that exist only on
the target will be removed, any object-specific thresholds that are only in the template
will be added to the target. Thus, the final settings on the target will be:
Mount Point
Operator
Warning Threshold
Critical Threshold
/
ВЈ
40
20
/private
ВЈ
30
20
/u1
ВЈ
30
20
All Others
ВЈ
25
15
3) Only settings for key values common to both template and target will be applied
to the target
When the template is applied to the target using this copy option, only the settings for
the common mount points, / and /private will be applied. Thus, the resulting settings
on the target for this metric will be:
8-6 OracleВ® Enterprise Manager Administration
Comparing Monitoring Templates with Targets
Mount Point
Operator
Warning Threshold
Critical Threshold
/
ВЈ
40
20
/private
ВЈ
30
20
/private2
ВЈ
20
10
All Others
ВЈ
25
15
8.8 Comparing Monitoring Templates with Targets
The intended effect of applying Monitoring Templates to destination targets is not
always clear. Deciding how and when to apply a template is simplified by using the
Compare Monitoring Template feature of Enterprise Manager. This allows you to see
at a glance how metric and collection settings defined in the template differ from those
defined on the destination target. You can easily determine whether your targets are
still compliant with the monitoring settings you have applied in the past. This
template comparison capability is especially useful when used with aggregate targets
such as groups and systems. For example, you can quickly compare the metric and
collection settings of group members with those of a template, and then apply the
template as appropriate.
Performing a Monitoring Template-Target comparison:
From the Enterprise menu, select Monitoring and then Monitoring Templates.
1.
2.
Choose the desired template from the table.
3.
Click Compare Settings. The Compare Monitoring Template page displays.
4.
Click Add to add one or more destination targets. The Search and Select dialog
displays.
5.
Select a one or more destination targets and then click Select. The selected targets
are added to the list of destination targets.
6.
Select the newly added destination targets and then click Continue. A
confirmation message displays indicating the Compare Template Settings job was
successfully submitted.
7.
Click OK to view the job results. Note: Depending on the complexity of the job
run, it may take time for the job to complete.
8.8.1 When is a metric between a template and a target considered "different"?
The metric is said to be different when any or all the following conditions are true
(provided the target does not have "Template Override" set for that metric):
в– The Warning Threshold settings are different
в– The Critical Threshold settings are different.
в– The Collection Schedules are different.
в– The Upload Intervals are different.
в– в– The number of occurrences (for which the metric has to remain at a value above
the threshold before an alert is raised) are different.)
For user-defined metrics, in addition to the above, the OS Command/SQL
statement used to evaluate the user-defined metric is different. Note that this
applies only if the user-defined metric name and the return type are the same.
Using Monitoring Templates 8-7
Comparing Metric Settings Using Information Publisher
в– The metric extension marked for delete will be shown as "different" on the
destination target and the template only if:
в– в– в– A metric extension with the same name exists on both the destination target
and template.
The return type (String, Numeric) of the metric extension is the same on both
the destination target and template.
The metric type is the same on both the destination target and the template.
8.9 Comparing Metric Settings Using Information Publisher
In addition to viewing metric differences between Monitoring Templates and
destination targets using the Compare Monitoring Template user-interface, you can
also use Information Publisher to generate reports containing the target-template
differences. Using Information Publisher's reporting capabilities gives you more
flexibility for displaying and distributing metric comparison data. For more
information, see "Using Information Publisher" on page 26-1.
Create a Report Definition
1. From the Enterprise menu, select Reports and then Information Publisher
Reports.
2.
Click Create. The Create Report Definition user interface is displayed.
3.
On the General page, specify the report name, how targets should be included,
target privileges, report time period, and display options.
4.
On the Elements page, click Add to access the Add Element page.
5.
Select the Monitoring Template Comparison element and click Continue to return
to the Element page.
6.
Once you have added the report element, click the Set Parameter icon to specify
requisite operational parameters. On this page, you specify a report header, select
a monitoring template, destination targets, and template application settings for
multiple threshold metrics. Click Continue to return to the Elements page.
7.
Click Layout to specify how information should be arranged in the report.
8.
Click Preview to validate that you are satisfied with the data and presentation of
the report.
9.
On the Schedule page, define when reports should be generated, and whether
copies should be saved and/or sent via e-mail, and how stored copies should be
purged.
10. On the Access page, click Add to specify which Enterprise Manager
administrators and/or roles will be permitted to view this generated report.
Additionally, if you have GRANT_ANY_REPORT_VIEWER system privilege, you
can make this report definition accessible to non-credential users via the
Enterprise Manager Reports Website
11. Click OK when you are finished.
12. Validate the report definition. If the parameters provided conflict, validation errors
or warnings will appear and let you know what needs attention.
13. Once the report definition has been saved successfully, it appears in the Report
Definition list under the Category and Subcategory you specified on the General
page.
8-8 OracleВ® Enterprise Manager Administration
Exporting and Importing Monitoring Templates
Viewing the Report
1. Find the template comparison report definition in the Report Definition list. You
can use the Search function to find or filter the list of report definitions.
2.
Click on the report definition title. If the report has a specified target, the report
will be generated immediately. If the report does not have a specified target, you
will be prompted to select a target.
Scheduling Reports for Automatic Generation
1. Create or edit a report definition.
2.
On the Schedule page, choose the Schedule Report option.
3.
Specify a schedule type. The schedule parameters on this page change according
to the selected schedule type.
When reports are scheduled for automatic generation, you have the option of saving
copies to the Management Repository and/or sending an e-mail version of the report
to designated recipients.
If a report has been scheduled to save copies, a copy of the report is saved each time a
scheduled report completes. When a user views a report with saved copies by clicking
on the report title, the most recently saved copy of the report is rendered. To see the
complete list of saved copies click on the Saved Copies link at the top of the report.
Enterprise Manager administrators can generate a copy of the report on-demand by
clicking on the Refresh icon on the report.
8.10 Exporting and Importing Monitoring Templates
For portability, monitoring templates can be exported to an XML file and then
imported into another Enterprise Manager installation as an active template.
Important: You can export templates from Enterprise Manager 10g
release 2 or higher and import them into Enterprise Manager 12c. .
Exporting a Monitoring Template
To export a template to an XML file:
1.
From the Enterprise menu, select Monitoring and then Monitoring Templates.
2.
Select the desired monitoring template from the table.
3.
Click Export.
Note: If you are running an Enterprise Manager 11g or earlier release, use the EM
CLI export_template verb to perform the export operation. Note that if the
monitoring template contains policy rules from earlier Enterprise Manager releases
(pre-12c) , these will not be imported into Enterprise Manager 12c as policy rules
no longer exist in this release.
Importing a Monitoring Template
To import a template from an XML file:
1.
From the Enterprise menu, select Monitoring and then Monitoring Templates.
2.
Click Import. The Import Template page displays.
3.
Specify the monitoring template XML file you want to import.
Using Monitoring Templates 8-9
Upgrading Enterprise Manager: Comparing Monitoring Templates
4.
Click Import.
8.11 Upgrading Enterprise Manager: Comparing Monitoring Templates
When upgrading from one Enterprise Manager release to the next, you will
accumulate monitoring templates that have been created for different releases.
Beginning with Enterprise Manager release 12.1.0.4, you can generate a post-upgrade
Monitoring Template Difference Report that allows you to view what templates had
been created for various Enterprise Manager releases. To generate the Monitoring
Template Difference Report, from the Setup menu, select Manage Cloud Control, and
then Post Upgrade Tasks.
8.12 Changing the Monitoring Template Apply History Retention Period
You can view monitoring template apply history using the predefined report in
Information Publisher. From the Enterprise menu, select Reports and then
Information Publisher. On the Information Publisher page, you can enter "template"
in the Title text entry field and click Go. The predefined report Monitoring Template
Apply History (last 7 days) appears in the report list.
By default, Enterprise Manager retains the monitoring template apply history for a
period of 180 days. If required, you can change the retention period to a value suitable
for your monitoring needs. Although the retention period cannot be indefinite, it can
be set to an extremely long period of time. Enterprise Manager provides the following
PL/SQL API to change the retention period.
mgmt_template_ui.modify_purge_policy(p_retention_days=><num_days>)
This procedure takes a NUMBER as input (num_days).
8-10 OracleВ® Enterprise Manager Administration
9
Using Metric Extensions
9
Metric extensions provide you with the ability to extend Oracle's monitoring
capabilities to monitor conditions specific to your IT environment. This provides you
with a comprehensive view of your environment. Furthermore, metric extensions
allow you to simplify your IT organization’s operational processes by leveraging
Enterprise Manager as the single central monitoring tool for your entire datacenter
instead of relying on other monitoring tools to provide this supplementary
monitoring.
This chapter covers the following:
в– What are Metric Extensions?
в– Metric Extension Lifecycle
в– Working with Metric Extensions
в– Adapters
в– Converting User-defined Metrics to Metric Extensions
в– Metric Extension Command Line Verbs
Instructional Videos:
For video tutorials on using metric extensions,
see:
Metric Extensions Part 1: Create Metric Extensions
https://apex.oracle.com/pls/apex/f?p=44785:24:11551596047
5402:::24:P24_CONTENT_ID%2CP24_PREV_PAGE:5741%2C24
Metric Extensions Part 2: Deploy Metric Extensions
https://apex.oracle.com/pls/apex/f?p=44785:24:15296555051
584:::24:P24_CONTENT_ID%2CP24_PREV_PAGE:5742%2C24
9.1 What are Metric Extensions?
Metric extensions allow you to create metrics on any target type. Unlike user-defined
metrics (used to extend monitoring in previous Enterprise Manager releases), metric
extensions allow you to create full-fledged metrics for a multitude of target types, such
as:
в– Hosts
в– Databases
в– Fusion Applications
Using Metric Extensions
9-1
What are Metric Extensions?
в– IBM Websphere
в– Oracle Exadata databases and storage servers
в– Siebel components
в– Oracle Business Intelligence components
You manage metric extensions from the Metric Extensions page. This page lists all
metric extensions in addition to allowing you to create, edit, import/export, and
deploy metric extensions.
The cornerstone of the metric extension is the Oracle Integration Adapter. Adapters
provide a means to gather data about targets using specific protocols. Adapter
availability depends on the target type your metric extension monitors.
How Do Metric Extensions Differ from User-defined Metrics?
In previous releases of Enterprise Manager, user-defined metrics were used to extend
monitoring capability in a limited fashion: user-defined metrics could be used to
collect point values through execution of OS scripts and a somewhat more complex set
of values (one per object) through SQL. Unlike metric extensions, user-defined metrics
have several limitations:
в– в– в– Limited Integration: If the OS or SQL user-defined metric executed custom
scripts, or required atonal dependent files, the user needed to manually transfer
these files to the target’s file system.
Limited Application of Query Protocols: OS user-defined metrics cannot model
child objects of servers by returning multiple rows from a metric (this capability
only exists for SQL user-defined metrics).
Limited Data Collection: Full-fledged Enterprise Manager metrics can collect
multiple pieces of data with a single query and reflect the associated data in alert
context. However, in the case of user-defined metrics, multiple pieces of data must
be collected by creating multiple user-defined metrics. Because the data is being
collected separately, it is not possible to refer to the associated data when alerts are
generated.
9-2 OracleВ® Enterprise Manager Administration
Metric Extension Lifecycle
в– в– Limited Query Protocols: User-defined metrics can only use the "OS" and "SQL"
protocols, unlike metric extensions which can use additional protocols such as
SNMP and JMX.
Limited Target Application: User-defined metrics only allow OS user-defined
metrics against host targets and SQL user-defined metrics against database targets.
No other target types are permitted. If, for example, you want to deploy a
user-defined metric against WebLogic instances in your environment, you will not
be able to do so since it is neither a host or database target type.
Most importantly, the primary difference between metric extensions and user-defined
metrics is that, unlike user-defined metrics, metric extensions are full-fledged metrics
similar to Enterprise Manager out-of-box metrics. They are handled and exposed in all
Enterprise Manager monitoring features as any Enterprise Manager-provided metric
and will automatically apply to any new features introduced.
9.2 Metric Extension Lifecycle
Developing a metric extension involves the same three phases you would expect from
any programmatic customization:
в– Developing Your Metric Extension
в– Testing Your Metric Extension
в– Deploying and Publishing Your Metric Extension
Developing Your Metric Extension
The first step is to define your monitoring requirements. This includes deciding the
target type, what data needs to be collected, what mechanism (adapter) can be used to
collect that data, and if elevated credentials are required. After making these decisions,
you are ready to begin developing your metric extension. Enterprise Manager
provides an intuitive user interface to guide you through the creation process.
Using Metric Extensions
9-3
Metric Extension Lifecycle
The metric extension wizard allows you to develop and refine your metric extension in
a completely editable format. And more importantly, allows you to interactively test
your metric extension against selected targets without having first to deploy the
extension to a dedicated test environment. The Test page allows you to run real-time
metric evaluations to ensure there are no syntactical errors in your script or metric
extension definition.
When you have completed working on your metric extension, you can click Finish to
exit the wizard. The newly created metric extension appears in the Metric Extension
Library where you can edit can be opened for further editing or saved as a deployable
draft that can be tested against multiple targets.
9-4 OracleВ® Enterprise Manager Administration
Metric Extension Lifecycle
You can edit a metric extension only if its status is editable.
Once it is saved as a deployable draft, you must create a new version
to implement further edits.
Note:
Testing Your Metric Extension
Once your metric extension returns the expected data during real-time target testing,
you are ready to test its robustness and actual behavior in Enterprise Manager by
deploying it against targets and start collecting data. At this point, the metric extension
is still private (only the developer can deploy to targets), but is identical to Oracle
out-of-box metrics behavior wise. This step involves selecting your editable metric
extension in the library and generating a deployable draft.
You can now deploy the metric extension to actual targets by going through the
“Deploy To Targets…” action. After target deployment, you can review the metric data
returned and test alert notifications. As mentioned previously, you will not be able to
edit the metric extension once a deployable draft is created: You must create a new
version of the metric extension.
Deploying Your Metric Extension
After rigorous testing through multiple metric extension versions and target
deployments, your metric extension is ready for deployment to your production
environment. Until this point, your metric extension is only viewable by you, the
metric extension creator. To make it accessible to all Enterprise Manager
administrators, it must be published.
Using Metric Extensions
9-5
Working with Metric Extensions
Now that your metric extension has been made public, your metric extension can be
deployed to intended production targets. If you are monitoring a small number of
targets, you can select the Deploy To Targets menu option and add targets one at a
time. For large numbers of targets, you deploy metric extensions to targets using
monitoring templates. An extension is added to a monitoring template in the same
way a full-fledged metric is added. The monitoring template is then deployed to the
targets.
You cannot add metric extensions to monitoring templates
before publishing the extension. If you attempt to do so, the
monitoring template page will warn you about it, and will not
proceed until you remove the metric extension.
Note:
Updating Metric Extensions
Beginning with Enterprise Manager Release 12.1.0.4, metric extensions can be updated
using the Enterprise Manager Self-update feature. See Chapter 14, "Updating Cloud
Control" for more information.
9.3 Working with Metric Extensions
Most all metric extension operations can be carried out from the Metric Extension
home page. If you need to perform operations on published extensions outside of the
UI, Enterprise Manger also provides EM CLI verbs to handle such operations as
importing/exporting metric extensions to archive files and migrating legacy
user-defined metrics to metric extensions. This section covers metric extension
operations carried out from the UI.
9.3.1 Administrator Privilege Requirements
In order to create, edit, view, deploy or undeploy metric extensions, you must have the
requisite administrator privileges. Enterprise Manager administrators must have the
following privileges:
9-6 OracleВ® Enterprise Manager Administration
Working with Metric Extensions
в– Create Metric Extension: System level access that:
Lets administrators view and deploy metric extensions
Allows administrators to edit and delete extensions.
в– в– в– Edit Metric Extension: Lets users with "Create Metric Extension" privilege edit
and create next versions of a particular metric extensions. The metric extension
creator has this privilege by default. This privilege must be granted on a
per-metric extension basis.
Full Metric Extension: Allows users with 'Create Metric Extension' privilege to
edit and create new versions of a particular metric extension.
Manage Metrics: Lets users deploy and un-deploy extensions on targets
Note: The Manage Metrics privilege must be granted on a per-target basis.
9.3.2 Granting Create Metric Extension Privilege
To grant create metric extension privileges to another administrator:
1.
From the Setup menu, select Security, then select Administrators.
2.
Choose the Administrator you would like to grant the privilege to.
3.
Click Edit.
4.
Go to the Resource Privileges tab, and click Manage Privilege Grants for the
Metric Extension resource type.
5.
Under Resource Type Privileges, click the Create Metric Extension check box.
6.
Click Continue, review changes, and click Finish in the Review tab.
9.3.3 Managing Administrator Privileges
Before an Enterprise Manager administrator can edit or delete a metric extension
created by another administrator, that administrator must have been granted requisite
access privileges. Edit privilege allows editing and creating next versions of the
extension. Full privilege allows the above operations and deletion of the extension.
To grant edit/full access to an existing metric extension to another administrator:
1.
From the Setup menu, select Security, then select Administrators.
2.
Choose the Administrator you would like to grant access to.
3.
Click Edit.
4.
Go to Resource Privileges and click Manage Privilege Grants (pencil icon) for the
Metric Extensions resource type.
5.
Under Resource Privileges, you can search for and add existing metric extensions.
Add the metric extensions you would like to grant privileges to. This allows the
user to edit and create next versions of the metric extension.
On this page, you can also grant an administrator the Create Metric Extension
privilege, which will allow them to manage metric extension access. See
"Managing Administrator Access to Metric Extensions" for more information.
6.
If you would additionally like to allow delete operations, then click the pencil icon
in the Manage Resource Privilege Grants column, and select Full Metric
Extension privilege in the page that shows up.
7.
Click Continue, review changes, and click Finish in the review tab.
Using Metric Extensions
9-7
Working with Metric Extensions
9.3.4 Managing Administrator Access to Metric Extensions
Administrators commonly share the responsibility of monitoring and managing
targets within the IT environment. Consequently, creating and maintaining metric
extensions becomes a collaborative effort involving multiple administrators. Metric
extension owners can control access directly from the metric extension UI.
9.3.4.1 Granting Full/Edit Privileges on a Metric Extension
As metric extension owner or Super Administrator, perform the following actions to
assign full/edit privileges on a metric extension to another administrator:
1.
From the Enterprise menu, select Monitoring, then select Metric Extensions.
2.
Choose a metric extension requiring update.
3.
From the Actions menu, select Manage Access.
4.
Click Add. The administrator selection dialog box appears. You can filter the list
by administrator, role, or both.
5.
Choose one or more administrators/roles from the list.
6.
Click Select. The chosen administrators/roles appear in the access list.
In the Privilege column, Edit is set by default. Choose Full from the drop-down
menu to assign Full privileges on the metric extension.
Edit Privilege: Allows an administrator to make changes to the metric extension but
not delete it.
Full Privilege: Allows an administrator to edit and also delete the metric extension.
The privilege granted to a user or role applies to all versions of the metric
extension.
7.
Click OK.
9.3.4.2 Revoking Access Privileges on a Metric Extension
As metric extension owner or Super Administrator, perform the following actions to
revoke metric extension privileges assigned to another administrator:
1.
From the Enterprise menu, select Monitoring, then select Metric Extensions.
2.
Choose a metric extension requiring update.
3.
From the Actions menu, select Manage Access.
4.
Choose one or more administrators/roles from the list.
5.
Click Remove. The chosen administrators/roles is deleted from the access list.
6.
Click OK.
Enterprise Manager allows metric extension ownership to be transferred from the
current owner of the metric extension to another administrator as long as that
administrator has been granted the Create Metric Extension privilege.
The Enterprise Manager Super Administrator has full
managerial access to all metric extensions (view, edit, and ownership
transfer).
Note:
As mentioned above, manage access is only enabled for the owner of the extension or an
Enterprise Manager Super User. Once the ownership is transferred, the previous
9-8 OracleВ® Enterprise Manager Administration
Working with Metric Extensions
owner does not have any management privileges on the metric extension unless
explicitly granted before ownership transfer. The Change Owner option is only
available to users and not roles.
Manage access allows the metric extension owner or Super Administrator to grant other
Enterprise Manager users or roles the ability to edit, modify, or delete metric
extensions.
9.3.4.3 Transferring Metric Extension Ownership
Enterprise Manager allows metric extension ownership to be transferred from the
current owner of the metric extension to another administrator as long as that
administrator has been granted the Create Metric Extension privilege.
The Enterprise Manager Super Administrator has full
managerial access to all metric extensions (view, edit, and ownership
transfer).
Note:
As mentioned above, manage access is only enabled for the owner of the extension or an
Enterprise Manager Super User. Once the ownership is transferred, the previous
owner does not have any management privileges on the metric extension unless
explicitly granted before ownership transfer. The Change Owner option is only
available to users and not roles.
Manage access allows the metric extension owner or Super Administrator to grant other
Enterprise Manager users or roles the ability to edit, modify, or delete metric
extensions.
9.3.5 Creating a New Metric Extension
To create a new metric extension:
1.
From the Enterprise menu, select Monitoring, then select Metric Extensions.
2.
From the Create menu, select Metric Extension. Enterprise Manager will
determine whether you have the Create Extension privilege and guide you
through the creation process.
3.
Decide on a metric extension name. Be aware that the name (and Display Name)
must be unique across a target type.
4.
Enter the general parameters.
The selected Adapter type defines the properties you must specify in the next step
of the metric extension wizard. The following adapter types are available:
в– OS Command Adapter - Single Column
Executes the specified OS command and returns the command output as a
single value. The metric result is a 1 row, 1 column table.
в– OS Command Adapter- Multiple Values
Executes the specified OS command and returns each command output line as
a separate value. The metric result is a multi-row, 1 column table.
в– OS Command Adapter - Multiple Columns
Executes the specified OS command and parses each command output line
(delimited by a user-specified string) into multiple values. The metric result is
a mult-row, multi-column table.
Using Metric Extensions
9-9
Working with Metric Extensions
в– SQL Adapter
Executes custom SQL queries or function calls against single instance
databases and instances on Real Application Clusters (RAC).
в– SNMP (Simple Network Management Protocol) Adapter
Allow Enterprise Manager Management Agents to query SNMP agents for
Management Information Base (MIB) variable information to be used as
metric data.
в– JMX (Java Management Extensions) Adapter
Retrieves JMX attributes from JMX-enabled servers and returns these
attributes as a metric table.
Refer to the Adapters section for specific information on the selected adapter
needed in the Adapter page (step 2) of the wizard.
Be aware that if you change the metric extension Adapter, all
your previous adapter properties (in Step 2) will be cleared.
Note:
Collection Schedule
You defined the frequency with which metric data is collected and how it is used
(Alerting Only or Alerting and Historical Trending) by specifying collection
schedule properties.
Depending on the target type selected, an Advanced option region may appear.
This region may (depending on the selected target type) contain one or two
options that determine whether metric data continues to be collected under certain
target availability/alert conditions. The options are:
в– в– Option 1: Continue metric data collection even if the target is down. This
option is visible for all target types except for Host target types as it is not
possible to collect metric data when the host is down.
Option 2: Continue metric data collection when an alert severity is raised for a
specific target metric. This metric is defined in such a way (AltSkipCondition
element is defined on this metric) that when a severity is generated on this
metric, the metric collections for other target metrics are stopped. The
explanatory text above the checkbox for this option varies depending on the
selected target type.
The Management Agent has logic to skip evaluation of metrics for targets that
are known to be down to reduce generation of metric errors due to connection
failures. If the AltSkipCondition element is defined for that target metric, other
metrics are skipped whenever there is an error in evaluating the Response
metric or there is a non-clear severity on the Response:Status metric. There are
two situations where a metric collection will be skipped or not happen:
–
When a target is down (option 1). This is same as the Severity on
Response/Status metric.
–
When a target is UP, but there is a severity on any other metric. Such
conditions are called Alt Skip (Alternate Skip) conditions.
Option 2 is only visible if an AltSkipCondition defined for one of the target’s
metrics. For example, this option will not be visible if the selected target type
is Oracle Weblogic Domain, but will be visible if the selected target type is
Database Instance.
9-10 OracleВ® Enterprise Manager Administration
Working with Metric Extensions
The following graphic shows the Advanced collection schedule options.
5.
From the Columns page, add metric columns defining the data returned from the
adapter. Note that the column order should match the order with which the
adapter returns the data.
в– Column Type
A column is either a Key column, or Data column. A Key column uniquely
identifies a row in the table. For example, employee ID is a unique identifier of
a table of employees. A Data column is any non-unique data in a row. For
example, the first and last names of an employee. You can also create rate and
delta metric columns based on an existing data column. See Rate and Delta
Metric Columns below.
в– Value Type
A value type is Number or String. This determines the alert comparison
operators that are available, and how Enterprise Manager renders collection
data for this metric column.
в– Alert Thresholds
The Comparison Operation, Warning, and Critical fields define an alert
threshold.
в– Alert Thresholds By Key
The Comparison Operation, Warning Thresholds By Key, and Critical
Thresholds By Key fields allow you to specify distinct alert thresholds for
different rows in a table. This option becomes available if there are any Key
columns defined. For example, if your metric is monitoring CPU Usage, you
can specify a different alert threshold for each distinct CPU. The syntax is to
specify the key column values in a comma separated list, the "=" symbol,
followed by the alert threshold. Multiple thresholds for different rows can be
separated by the semi-colon symbol ";". For example, if the key columns of the
CPU Usage metric are cpu_id and core_id, and you want to add a warning
threshold of 50% for procecessor1, core1, and a threshold of 60% for
processor2, core2, you would specify:
procecessor1,core1=50;processor2,core2=60
в– Manually Clearable Alert
You must expand the Advanced region in order to view the
Manually Clearable Alert option.
Note:
Using Metric Extensions 9-11
Working with Metric Extensions
If this option is set to true, then the alert will not automatically clear when the
alert threshold is no longer satisfied. For example, if your metric is counting
the number of errors in the system log files, and you set an alert threshold of
50, if an alert is raised once the threshold is met, the alert will not
automatically clear once the error count falls back below 50. The alert will
need to be manually cleared in the Alerts UI in the target home page or
Incident Manager.
в– Number of Occurrences Before Alert
The number of consecutive metric collections where the alert threshold is met,
before an alert is raised.
в– Alert Message / Clear Message
The message that is sent when the alert is raised / cleared. Variables that are
available for use are: %columnName%, %keyValue%, %value%, %warning_
threshold%, %critical_threshold%
You can also retrieve the value of another column by surrounding the desired
column name with "%". For example, if you are creating an alert for the cpu_
usage column, you can get the value of the core_temperature column by using
%core_temperature%. Note that the same alert / clear message is used for
warning or critical alerts.
Think carefully and make sure all Key columns are added,
because you cannot create additional Key columns in newer versions
of the metric extension. Once you click Save As Deployable Draft, the
Key columns are final (edits to column display name, alert thresholds
are still allowed). You can still add new Data columns in newer
versions. Also be aware that some properties of existing Data columns
cannot be changed later, including Column Type, Value Type,
Comparison Operator (you can add a new operator, but not change an
existing operator), and Manually Clearable Alert.
Note:
в– Metric Category
The metric category this column belongs to.
Rate and Delta Metric Columns
You can create additional metric columns based on an existing data column that
measures the rate at which data changes or the difference in value (delta) since the
last metric collection. The rate/delta metric definition will be allowed when a
metric's collection frequency is periodic. For example, collected every 10 minutes.
Converseley, a metric that is computed every Monday and Tuesday only cannot
have a rate/delta metric as data sampling is too infrequent.
After at least one data column has been created, three additional options appear in
the Add menu as shown in the following graphic.
9-12 OracleВ® Enterprise Manager Administration
Working with Metric Extensions
в– Add Delta metric columns based on another metric column
Example: You want to know the difference in the table space used since the
last collection.
Delta Calculation:
current metric value - previous metric value
в– Add Rate Per Minute metric column based on another metric column
Example: You want to know the average table space usage per minute based
on the table space column metric which is collected every 1 hr.
Rate Per Minute Calculation:
(current metric value - previous metric value)/ collection schedule
where the collection schedule is in minutes.
в– Add Rate Per Five Minutes metric column based on another metric column
Example: You want to know the average table space usage every five minutes
based on the table space column which is collected say every 1 hour]
Rate Per Five Minute Calculation:
[(current metric value - previous metric value)/ collection schedule ] * 5
where the collection schedule is in minutes.
To create a rate/delta metric column, click on an existing data column in the table
and then select one of the rate/delta column options from the Add menu.
6.
From the Credentials page, you can override the default monitoring credentials by
using custom monitoring credential sets. By default, the metric extension wizard
chooses the existing credentials used by Oracle out-of-box metrics for the
particular target type. For example, metric extensions will use the dbsnmp user for
database targets. You have the option to override the default credentials, by
creating a custom monitoring credential set through the "emcli create_credential_
set" command. Refer to the Enterprise Manager Command Line Interface Guide for
additional details. Some adapters may use additional credentials, refer to the
Adapters section for specific information.
7.
From the Test page, add available test targets.
8.
Click Run Test to validate the metric extension. The extension is deployed to the
test targets specified by the user and a real-time collection is executed. Afterwards,
the metric extension is automatically undeployed. The results and any errors are
added to the Test Results region.
Using Metric Extensions 9-13
Working with Metric Extensions
9.
Repeat the edit /test cycle until the metric extension returns data as expected.
10. Click Finish.
9.3.6 Creating a New Metric Extension (Create Like)
To create a new metric extension based on an existing metric extension:
1.
From the Enterprise menu, select Monitoring, then select Metric Extensions.
2.
From the Metric Extensions page, determine which extensions are accessible. The
page displays the list of metric extensions along with target type, owner,
production version and deployment information.
3.
Select an existing metric extension.
4.
From the Actions menu, select Create Like. Enterprise Manager will determine
whether you have the Create Extension privilege and guide you through the
creation process.
5.
Make desired modifications.
6.
From the Test page, add available test targets.
7.
Click Run Test to validate the metric extension. The extension is deployed to the
test targets specified by the user and a real-time collection is executed. Afterwards,
the metric extension is automatically undeployed. The results and any errors are
added to the Test Results region.
8.
Repeat the edit /test cycle until the metric extension returns data as expected.
9.
Click Finish.
9.3.7 Editing a Metric Extension
Before editing an existing metric extension, you must have Edit privileges on the
extension you are editing or be the extension creator. Note: Once a metric extension is
saved as a deployable draft, it cannot be edited, you can only create a new version.
To edit an existing metric extension:
1.
From the Enterprise menu, select Monitoring, then select Metric Extensions.
2.
From the Metric Extensions page, determine which extensions are accessible. The
page displays the list of metric extensions along with target type, owner,
production version and deployment information.
3.
Select the metric extension to be edited.
4.
From the Actions menu, select Edit.
5.
Update the metric extension as needed.
6.
From the Test page, add available test targets.
7.
Click Run Test to validate the metric extension. The extension is deployed to the
test targets specified by the user and a real-time collection is executed. Afterwards,
the metric extension is automatically undeployed. The results and any errors are
added to the Test Results region.
8.
Repeat the edit /test cycle until the metric extension returns data as expected.
9.
Click Finish.
9-14 OracleВ® Enterprise Manager Administration
Working with Metric Extensions
9.3.8 Creating the Next Version of an Existing Metric Extension
Before creating the next version of an existing metric extension, you must have Edit
privileges on the extension you are versioning or be the extension creator.
To create next version of an existing metric extension:
1.
From the Enterprise menu, select Monitoring, then select Metric Extensions.
2.
From the Metric Extensions page, determine which extensions are accessible. The
page displays the list of metric extensions along with target type, owner,
production version and deployment information.
3.
Select the metric extension to be versioned.
4.
From the Actions menu, select Create Next Version.
5.
Update the metric extension as needed. The target type, and extension name
cannot be edited, but all other general properties can be modified. There are also
restrictions on metric columns modifications. See Note in Creating a New Metric
Extension section for more details.
6.
From the Test page, add available test targets.
7.
Click Run Test to validate the metric extension. The extension is deployed to the
test targets specified by the user and a real-time collection is executed. Afterwards,
the metric extension is automatically undeployed. The results and any errors are
added to the Test Results region.
8.
Repeat the edit /test cycle until the metric extension returns data as expected.
9.
Click Finish.
9.3.9 Importing a Metric Extension
Metric extensions can be converted to portable, self-contained packages that allow you
to move the metric extension to other Enterprise Manager installations, or for
storage/backup. These packages are called Metric Extension Archives (MEA) files.
MEA files are zip files containing all components that make up the metric extension:
metric metadata, collections, and associated scripts/jar files. Each MEA file can
contain only one metric extension. To add the metric extension back to your Enterprise
Manager installation, you must import the metric extension from the MEA.
To import a metric extension from an MEA file:
1.
From the Enterprise menu, select Monitoring, then select Metric Extensions.
2.
Click Import.
3.
Browse to file location, and select the MEA file. Enterprise Manager checks if the
target type and metric extension name combination is already used in the system.
If not, the system will create a new metric extension. If the extension name is
already in use, the system will attempt to create a new version of the existing
extension using the MEA contents. This will require the MEA to contain a superset
of all the existing metric extension's metric columns. You also have the option to
rename the metric extension.
4.
Clicking on OK creates the new metric extension or the new version of an existing
metric extension.
5.
From the Actions menu, select Edit to verify the entries.
6.
From the Test page, add available test targets.
Using Metric Extensions 9-15
Working with Metric Extensions
7.
Click Run Test to validate the metric extension. The extension is deployed to the
test targets specified by the user and a real-time collection is executed. Afterwards,
the metric extension is automatically undeployed. The results and any errors are
added to the Test Results region.
8.
Repeat the edit /test cycle until the metric extension returns data as expected.
9.
Click Finish.
9.3.10 Exporting a Metric Extension
Existing metric extensions can be package as self-contained zip files (exported) for
portability and/or backup and storage.
To export an existing metric extension:
1.
From the Enterprise menu, select Monitoring, then select Metric Extensions.
2.
From the Metric Extensions page, determine which extensions are accessible. The
page displays the list of metric extensions along with target type, owner,
production version and deployment information.
3.
Select the metric extension to be exported.
4.
From the Actions menu, select Export. Enterprise Manager prompts you to enter
the name and location of the MEA file that is to be created.
5.
Enter the name and location of the package. Enterprise Manager displays the
confirmation page after the export is complete.
Note: You can only export Production, Deployable Draft and Published metric
extension versions.
6.
Confirm the export file is downloaded.
9.3.11 Deleting a Metric Extension
Initiating the deletion of a metric extension is simple. However, the actual deletion
triggers a cascade of activity by Enterprise Manager to completely purge the metric
extension from the system. This includes closing open metric alerts, and purging
collected metric data (if the latest metric extension version is deleted).
Before a metric extension version can be deleted, it must be undeployed from all
targets, and removed from all monitoring templates (including templates in pending
apply status).
To delete a metric extension:
1.
From the Enterprise menu, select Monitoring, then select Metric Extensions.
2.
From the Metric Extensions page, determine which extensions are accessible. The
page displays the list of metric extensions along with target type, owner,
production version and deployment information.
3.
Select the metric extension that is to be deleted.
4.
From the Actions menu, select Delete. Enterprise Manager prompts you to
confirm the deletion.
5.
Confirm the deletion.
9.3.12 Deploying Metric Extensions to a Group of Targets
A metric extension must be deployed to a target in order for it to begin collecting data.
9-16 OracleВ® Enterprise Manager Administration
Working with Metric Extensions
To deploy a metric extension to one or more targets:
1.
From the Enterprise menu, select Monitoring, then select Metric Extensions.
2.
From the Metric Extensions page, determine which extensions are accessible. The
page displays the list of metric extensions along with target type, owner,
production version and deployment information.
3.
Select the metric extension that is to be deployed.
4.
From the Actions menu, select Manage Target Deployments. The Manage Target
Deployments page appears showing you on which target(s) the selected metric
extension is already deployed.
5.
Return to the Metric Extensions page.
6.
Select the metric extension.
7.
From the Actions menu, select Deploy to Targets. Enterprise Manager determines
whether you have "Manage Target Metrics" privilege, and only those targets where
you do show up in the target selector.
8.
Add the targets where the metric extension is to be deployed and click Submit.
Enterprise Manager submits a job deploying the metric extension to each of the
targets. A single job is submitted per deployment request.
9.
You are automatically redirected to the Pending Operations page, which shows a
list of currently scheduled, executing, or failed metric extension deploy operations.
Once the deploy operation completes, the entry is removed from the pending
operations table.
9.3.13 Creating an Incident Rule to Send Email from Metric Extensions
One of the most common tasks administrators want Enterprise Manager to perform is
to send an email notification when a metric alert condition occurs. Specifically,
Enterprise Manager monitors for alert conditions defined as incidents. For a given
incident you create an incident rule set to tell Enterprise Manager what actions to take
when an incident occurs. In this case, when an incident consisting of an alert condition
defined by a metric extension occurs, you need to create an incident rule to send email
to administrators. For instructions on sending email for metric alerts, see "Sending
Email for Metric Alerts" on page 3-78.
For information incident management see Chapter 3, "Using Incident Management."
9.3.14 Updating Older Versions of Metric Extensions Already deployed to a Group of
Targets
When a newer metric extension version is published, you may want to update any
older deployed instances of the metric extension.
To update old versions of the metric extension already deployed to targets:
1.
From the Enterprise menu, select Monitoring, then select Metric Extensions.
2.
From the Metric Extensions page, determine which extensions are accessible. The
page displays the list of metric extensions along with target type, owner,
production version and deployment information.
3.
Select the metric extension to be upgraded.
Using Metric Extensions 9-17
Working with Metric Extensions
4.
From the Actions menu, select Manage Target Deployments. The Manage Target
Deployments page appears showing a list of targets where the metric extension is
already deployed.
5.
Select the list of targets where the extension is to be upgraded and click Upgrade.
Enterprise Manager submits a job for the deployment of the newest Published
metric extension to the selected targets. A single job is submitted per deployment
request.
6.
You are automatically redirected to the Pending Operations page, which shows a
list of currently scheduled, executing, or failed metric extension deploy operations.
Once the deploy operation completes, the entry is removed from the pending
operations table.
9.3.15 Creating Repository-side Metric Extensions
Beginning with Enterprise Manager Release 12.1.0.4, you can create repository-side
metric extensions. This type of metric extension allows you to use SQL scripts to
extract information directly from the Enterprise Manager repository and raise alerts
for the target against which the repository-side extension is run. For example, you can
use repository-side metric extensions to raise an alert if the total number of alerts for a
host target is greater than 5. Or perhaps, raise an alert if the CPU utilization on that
host is greater than 95% AND the number of process running on that host is greater
than 500. Repository-side metrics allows you to monitor your Enterprise Manager
infrastructure with greater flexibility.
To create a repository-side metric:
1.
From the Enterprise menu, select Monitoring, then select Metric Extensions.
2.
From the Create menu, select Repository-side Metric Extension. Enterprise
Manager will determine whether you have the Create Extension privilege and
guide you through the creation process.
3.
Decide on a target type and metric extension name. Be aware that the name (and
Display Name) must be unique across a target type.
4.
Enter the general parameters.
Collection Schedule
You defined the frequency with which metric data is collected and how it is used
(Alerting Only or Alerting and Historical Trending) by specifying collection
schedule properties.
5.
Create the SQL query to be run against the Enterprise Manager Repository.
Explicit instructions for developing the query as well as examples are provide on
the SQL Query page.
9-18 OracleВ® Enterprise Manager Administration
Working with Metric Extensions
Click Validate SQL to test the query.
If you already have a SQL script, you can click Upload to load the SQL from an
external file.
6.
From the Columns page, you can view/edit columns returned by the SQL query.
You may edit the columns, however, you cannot add or delete columns from this
page.
в– Column Type
A column is either a Key column, or Data column. A Key column uniquely
identifies a row in the table. For example, employee ID is a unique identifier of
a table of employees. A Data column is any non-unique data in a row. For
example, the first and last names of an employee. You can also create rate and
delta metric columns based on an existing data column. See Rate and Delta
Metric Columns below.
в– Value Type
A value type is Number or String. This determines the alert comparison
operators that are available, and how Enterprise Manager renders collection
data for this metric column.
в– Alert Thresholds
The Comparison Operation, Warning, and Critical fields define an alert
threshold.
в– Alert Thresholds By Key
The Comparison Operation, Warning Thresholds By Key, and Critical
Thresholds By Key fields allow you to specify distinct alert thresholds for
different rows in a table. This option becomes available if there are any Key
columns defined. For example, if your metric is monitoring CPU Usage, you
can specify a different alert threshold for each distinct CPU. The syntax is to
specify the key column values in a comma separated list, the "=" symbol,
followed by the alert threshold. Multiple thresholds for different rows can be
separated by the semi-colon symbol ";". For example, if the key columns of the
CPU Usage metric are cpu_id and core_id, and you want to add a warning
threshold of 50% for procecessor1, core1, and a threshold of 60% for
Using Metric Extensions 9-19
Working with Metric Extensions
processor2, core2, you would specify:
procecessor1,core1=50;processor2,core2=60
в– Manually Clearable Alert
You must expand the Advanced region in order to view the
Manually Clearable Alert option.
Note:
If this option is set to true, then the alert will not automatically clear when the
alert threshold is no longer satisfied. For example, if your metric is counting
the number of errors in the system log files, and you set an alert threshold of
50, if an alert is raised once the threshold is met, the alert will not
automatically clear once the error count falls back below 50. The alert will
need to be manually cleared in the Alerts UI in the target home page or
Incident Manager.
в– Number of Occurrences Before Alert
The number of consecutive metric collections where the alert threshold is met,
before an alert is raised.
в– Alert Message / Clear Message
The message that is sent when the alert is raised / cleared. Variables that are
available for use are: %columnName%, %keyValue%, %value%, %warning_
threshold%, %critical_threshold%
You can also retrieve the value of another column by surrounding the desired
column name with "%". For example, if you are creating an alert for the cpu_
usage column, you can get the value of the core_temperature column by using
%core_temperature%. Note that the same alert / clear message is used for
warning or critical alerts.
Think carefully and make sure all Key columns are added,
because you cannot create additional Key columns in newer versions
of the metric extension. Once you click Save As Deployable Draft, the
Key columns are final (edits to column display name, alert thresholds
are still allowed). You can still add new Data columns in newer
versions. Also be aware that some properties of existing Data columns
cannot be changed later, including Column Type, Value Type,
Comparison Operator (you can add a new operator, but not change an
existing operator), and Manually Clearable Alert.
Note:
в– Metric Category
The metric category this column belongs to.
в– Add Delta metric columns based on another metric column
Example: You want to know the difference in the table space used since the
last collection.
Delta Calculation:
current metric value - previous metric value
в– Add Rate Per Minute metric column based on another metric column
9-20 OracleВ® Enterprise Manager Administration
Adapters
Example: You want to know the average table space usage per minute based
on the table space column metric which is collected every 1 hr.
Rate Per Minute Calculation:
(current metric value - previous metric value)/ collection schedule
where the collection schedule is in minutes.
в– Add Rate Per Five Minutes metric column based on another metric column
Example: You want to know the average table space usage every five minutes
based on the table space column which is collected say every 1 hour]
Rate Per Five Minute Calculation:
[(current metric value - previous metric value)/ collection schedule ] * 5
where the collection schedule is in minutes.
To create a rate/delta metric column, click on an existing data column in the table
and then select one of the rate/delta column options from the Add menu.
7.
From the Test page, add available test targets.
8.
Click Run Test to validate the metric extension. The extension is deployed to the
test targets specified by the user and a real-time collection is executed. Afterwards,
the metric extension is automatically undeployed. The results and any errors are
added to the Test Results region.
9.
Repeat the edit /test cycle until the metric extension returns data as expected.
10. Click Finish.
9.4 Adapters
Oracle Integration Adapters provide comprehensive, easy-to-use monitoring
connectivity with a variety of target types. The adapter enables communication with
an enterprise application and translates the application data to standards-compliant
XML and back.
The metric extension target type determines which adapters are made available from
the UI. For example, when creating a metric extension for an Automatic Storage
Management target type, only three adapters (OS Command-Single Column, OS
Command-Multiple Columns, and SQL) are available from the UI.
Using Metric Extensions 9-21
Adapters
A target type’s out-of-box metric definition defines the adapters for which it has native
support, and only those adapters will be shown in the UI. No other adapters are
supported for that target type.
A complete list of all adapters is shown below.
в– OS Command Adapter - Single Column
в– OS Command Adapter- Multiple Values
в– OS Command Adapter - Multiple Columns
в– SQL Adapter
в– SNMP (Simple Network Management Protocol) Adapter
в– JMX Adapter
9.4.1 OS Command Adapter - Single Column
Executes the specified OS command and returns the command output as a single
value. The metric result is a 1 row, 1 column table.
Basic Properties
The complete command line will be constructed as: Command + Script + Arguments.
в– в– в– Command - The command to execute. For example, %perlBin%/perl. The
complete command line will be constructed as: Command + Script + Arguments.
Script - A script to pass to the command. For example,
%scriptsDir%/myscript.pl. You can upload custom files to the agent, which
will be accessible under the %scriptsDir% directory.
Arguments - Additional arguments to be appended to the Command.
Advance Properties
в– Input Properties - Additional properties can be passed to the command through
its standard input stream. This is usually used for secure content, such as
username or passwords, that you don't want to be visible to other users. For
example, you can add the following Input Property:
Name=targetName, Value=%NAME%
which the command can read through it's standard input stream as
"STDINtargetName=<target name>".
в– Environment Variables - Additional properties can be accessible to the command
from environment variables. For example, you can add Environment Variable:
Name=targetType, Value="%TYPE%", and the command can access the target
type from environment variable "ENVtargetType".
Credentials
в– Host Credentials - The credential used to launch the OS Command.
в– Input Credentials - Additional credentials passed to the OS Command's standard
input stream.
Example 1
Read the contents of a log file, and dump out all lines containing references to the
target.
9-22 OracleВ® Enterprise Manager Administration
Adapters
в– Approach 1 - Use the grep command, and specify the target name using
%NAME% parameter.
Command = /bin/grep %NAME% mytrace.log
в– Approach 2 - Run a perl script
Command = %perlBin%/perl
Script = %scriptsDir%/filterLog.pl
Input Properties:
targetName = %NAME%
targetType = %TYPE%
filterLog.pl:
require "emd_common.pl";
my %stdinVars = get_stdinvars();
my $targetName = $stdinVars{"targetName"};
my $targetType = $stdinVars{"targetType"};
open (MYTRACE, mytrace.log);
foreach $line (<MYTRACE >)
{
# Do line-by-line processing
}
close (MYTRACE);
Example 2
Connect to a database instance from a PERL script and query the HR.JOBS sample
schema table.
в– Approach 1 - Pass credentials from target type properties into using Input
Properties:
Command = %perlBin%/perl
Script = %scriptsDir%/connectDB.pl
Input Properties:
EM_DB_USERNAME = %Username%
EM_DB_PASSWORD = %Password%
EM_DB_MACHINE = %MachineName%
EM_DB_PORT = %Port%
EM_DB_SID = %SID%
connectDB.pl
use DBI;
require "emd_common.pl";
my
my
my
my
%stdinVars = get_stdinvars();
$dbUsername = $stdinVars{"EM_DB_USERNAME"};
$dbPassword = $stdinVars{"EM_DB_PASSWORD"};
$dbMachine = $stdinVars{"EM_DB_MACHINE"};
Using Metric Extensions 9-23
Adapters
my $dbPort = $stdinVars{"EM_DB_PORT"};
my $dbSID = $stdinVars{"EM_DB_SID"};
my $dbAddress =
"(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=$dbMachine)(Port=$dbPort))(CONNECT_
DATA=(SID=$dbSID)))";
# Establish Target DB Connection
my $db = DBI->connect('dbi:Oracle:', "[email protected]".$dbAddress, "$dbPassword",
{PrintError => 0, RaiseError => 0, AutoCommit => 0})
or die (filterOraError("em_error=Could not connect to
$dbUsername/$dbAddress: $DBI::errstr\n", $DBI::err));
my $query = "SELECT JOB_TITLE, MIN_SALARY FROM HR.JOBS";
my $st = $db->prepare($query);
$st->execute();
while ( my ($job_title, $min_sal) = $st->fetchrow_array() )
{
print "$job_title|$min_sal\n";
}
$db->disconnect
or warn "disconnect $DBI::errstr\n";
exit 0;
в– Approach 2 - Pass monitoring credential set using Input Credentials
Command = %perlBin%/perl
Script = %scriptsDir%/connectDB.pl
Input Credentials:
dbCreds = MyCustomDBCreds
connectDB.pl
use DBI;
require "emd_common.pl";
my
my
my
my
my
%stdinVars = get_stdinvars();
$credType = getCredType("dbCred", \%stdinVars);
%credProps = getCredProps("dbCreds", \%stdinVars);
$dbUsername = $credProps{"DBUserName"};
$dbPassword = $credProps{"DBPassword"};
Example 3
Overriding default monitoring credentials by creating and using a custom monitoring
credential set for host target.
Creating host credentials for the host target type:
> emcli create_credential_set -set_name=myCustomCreds -target_type=host -auth_
target_type=host -supported_cred_types=HostCreds -monitoring -description='My
Custom Credentials'
When you go to the Credentials page of the Metric Extension wizard, and choose to
"Specify Credential Set" for Host Credentials, you will see "My Custom Credentials"
show up as an option in the drop down list.
9-24 OracleВ® Enterprise Manager Administration
Adapters
Note that this step only creates the Monitoring Credential Set for the host target type,
and you need to set the credentials on each target you plan on deploying this metric
extension to. You can set credentials from Enterprise Manager by going to Setup, then
Security, then Monitoring Credentials. Alternatively, this can be done from the
command line.
> emcli set_monitoring_credential -target_name=target1 -target_type=host -set_
name=myCustomCreds -cred_type=HostCreds -auth_target_type=host
-attributes='HostUserName:myusername;HostPassword:mypassword'
9.4.2 OS Command Adapter- Multiple Values
Executes the specified OS command and returns each command output line as a
separate value. The metric result is a multi-row, 1 column table.
For example, if the command output is:
em_result=out_x
em_result=out_y
then three columns are populated with values 1,2,3 respectively.
Basic Properties
в– Command - The command to execute. For example, %perlBin%/perl.
в– Script - A script to pass to the command. For example,
%scriptsDir%/myscript.pl. You can upload custom files to the agent, which
will be accessible under the %scriptsDir% directory.
в– Arguments - Additional arguments to be appended to the Command.
в– Starts With - The starting string of metric result lines.
Example: If the command output is:
em_result=4354
update
test
setting Starts With = em_result specifies that only lines starting with em_result will
be parsed.
Advanced Properties
Input Properties - Additional properties to be passed to the command through its
standard input stream. For example, you can add Input Property:
Name=targetName, Value=%NAME%, which the command can read through its
standard input stream as "STDINtargetName=<target name>". See usage examples
in OS Command Adapter - Single Columns.
в– в– Environment Variables - Additional properties can be accessible to the command
from environment variables. For example, you can add Environment Variable:
Name=targetType, Value="%TYPE%", and the command can access the target type
from environment variable "ENVtargetType". See usage examples in OS
Command Adapter - Single Columns.
Credentials
в– Host Credentials - The credential used to launch the OS Command. See usage
examples in OS Command Adapter - Single Columns.
Using Metric Extensions 9-25
Adapters
в– Input Credentials - Additional credentials passed to the OS Command's standard
input stream. See usage examples in OS Command Adapter - Single Columns.
9.4.3 OS Command Adapter - Multiple Columns
Executes the specified OS command and parses each command output line (delimited
by a user-specified string) into multiple values. The metric result is a mult-row,
multi-column table.
Example: If the command output is
em_result=1|2|3
em_result=4|5|6
and the Delimiter is set as "|", then there are two rows of three columns each:
Table 9–1
Multi-Column Output
1
2
3
4
5
6
Basic Properties
The complete command line will be constructed as: Command + Script + Arguments
в– в– Command - The command to execute. For example, %perlBin%/perl.
Script - A script to pass to the command. For example, %scriptsDir%/myscript.pl.
You can upload custom files to the agent, which will be accessible under the
%scriptsDir% directory.
в– Arguments - Additional arguments.
в– Delimiter - The string used to delimit the command output.
в– Starts With - The starting string of metric result lines.
Example: If the command output is
em_result=4354
out_x
out_y
setting Starts With = em_result specifies that only lines starting with em_result will
be parsed.
в– в– Input Properties - Additional properties can be passed to the command through
its standard input stream. For example, you can add Input Property:
Name=targetName, Value=%NAME%, which the command can read through it's
standard input stream as STDINtargetName=<target name>. To specify multiple
Input Properties, enter each property on its own line.
Environment Variables - Additional properties can be accessible to the command
from environment variables. For example, you can add Environment Variable:
Name=targetType, Value="%TYPE%, and the command can access the target type
from environment variable "ENVtargetType".
Advanced Properties
в– Input Properties - Additional properties can be passed to the command through
its standard input stream. For example, you can add Input Property:
9-26 OracleВ® Enterprise Manager Administration
Adapters
Name=targetName, Value=%NAME%, which the command can read through its
standard input stream as STDINtargetName=<target name>. See usage examples
in OS Command Adapter - Single Columns.
в– Environment Variables - Additional properties can be accessible to the command
from environment variables. For example, you can add Environment Variable:
Name=targetType, Value="%TYPE%, and the command can access the target type
from environment variable "ENVtargetType". See usage examples in OS
Command Adapter - Single Columns.
Credentials
Host Credentials - The credential used to launch the OS Command. See usage
examples in OS Command Adapter - Single Columns
в– в– Input Credentials - Additional credentials passed to the OS Command's standard
input stream. See usage examples in OS Command Adapter - Single Columns.
9.4.4 SQL Adapter
Executes custom SQL queries or function calls supported against single instance
databases and instances on Real Application Clusters (RAC).
Properties
в– SQL Query - The SQL query to execute. Normal SQL statements should not be
semi-colon terminated. For example, SQL Query = "select a.ename, (select count(*)
from emp p where p.mgr=a.empno) directs from emp a". PL/SQL statements are
also supported, and if used, the "Out Parameter Position" and "Out Parameter
Type" properties should be populated.
в– в– в– в– SQL Query File - A SQL query file. Note that only one of "SQL Query" or "SQL
Query File" should be used. For example, %scriptsDir%/myquery.sql. You can
upload custom files to the agent, which will be accessible under the %scriptsDir%
directory.
Transpose Result - Transpose the SQL query result.
Bind Variables - Declare bind variables used in normal SQL statements here. For
example, if the SQL Query = "select a.ename from emp a where a.mgr = :1", then
you can declare the bind variable as Name=1, Value=Bob.
Out Parameter Position - The bind variable used for PL/SQL output. Only
integers can be specified.
Example: If the SQL Query is
DECLARE
l_output1 NUMBER;
l_output2 NUMBER;
BEGIN
.....
OPEN :1 FOR
SELECT l_output1, l_output2 FROM dual;
END;
you can set Out Parameter Position = 1, and Out Parameter Type = SQL_CURSOR
в– Out Parameter Type - The SQL type of the PL/SQL output parameter. See
comment for Out Parameter Position
Using Metric Extensions 9-27
Adapters
Credentials
в– Database Credentials - The credential used to connect to the database.
Example
Overriding default monitoring credentials by creating and using a custom monitoring
credential set for database target.
Creating host credentials for the database target type:
> emcli create_credential_set -set_name=myCustomDBCreds -target_type=oracle_
database -auth_target_type=oracle_database -supported_cred_types=DBCreds
-monitoring -description='My Custom DB Credentials'
When you go to the Credentials page of the Metric Extension wizard, and choose to
"Specify Credential Set" for Database Credentials, you will see "My Custom DB
Credentials" show up as an option in the drop down list.
Note that this step only creates the Monitoring Credential Set for the host target type,
and you need to set the credentials on each target you plan on deploying this metric
extension to. You can set credentials from Enterprise Manager by going to Setup, then
selecting Security, then selecting Monitoring Credentials. Alternatively, this can be
performed using the Enterprise Manager Command Line Interface.
> emcli set_monitoring_credential -target_name=db1 -target_type=oracle_database
-set_name=myCustomDBCreds -cred_type=DBCreds -auth_target_type=oracle_database
-attributes='DBUserName:myusername;DBPassword:mypassword'
9.4.5 SNMP (Simple Network Management Protocol) Adapter
Allow Enterprise Manager Management Agents to query SNMP agents for
Management Information Base (MIB) variable information to be used as metric data.
Basic Properties
в– Object Identifiers (OIDs): Object Identifiers uniquely identify managed objects in
a MIB hierarchy. One or more OIDs can be specified. The SNMP adapter will
collect data for the specified OIDs. For example, 1.3.6.1.4.1.111.4.1.7.1.1
Advanced Properties
Delimiter - The delimiter value used when specifying multiple OID values for an
OID's attribute. The default value is space or \n or \t
в– в– в– Tabular Data - Indicates whether the expected result for a metric will have
multiple rows or not. Possible values are TRUE or FALSE. The default value is
FALSE
Contains V2 Types - Indicates whether any of the OIDs specified is of SNMPV2
data type. Possible values are TRUE or FALSE. The default value is FALSE. For
example, if an OID value specified is of counter64 type, then this attribute will be
set to TRUE.
9.4.6 JMX Adapter
Retrieves JMX attributes from JMX-enabled servers and returns these attributes as a
metric table.
9-28 OracleВ® Enterprise Manager Administration
Converting User-defined Metrics to Metric Extensions
Properties
в– Metric -- The MBean ObjectName or ObjectName pattern whose attributes are to
be queried. Since this is specified as metric metadata, it needs to be instanceagnostic. Instance-specific key properties (such as servername) on the MBean
ObjectName may need to be replaced with wildcards.
в– ColumnOrder -- A semi-colon separated list of JMX attributes in the order they
need to be presented in the metric.
Advanced Properties
в– IdentityCol -- The MBean key property that needs to be surfaced as a column
when it is not available as a JMX attribute. For example:
com.myCompany:Name=myName,Dept=deptName, prop1=prop1Val, prop2=prop2Val
In this example, setting identityCol as Name;Dept will result in two additional key
columns representing Name and Dept besides the columns representing the JMX
attributes specified in the columnOrder property.
в– AutoRowPrefix -- Prefix used for an automatically generated row. Rows are
automatically generated in situations where the MBean ObjectName pattern
specified in metric property matches multiple MBeans and none of the JMX
attributes specified in the columnOrder are unique for each. The autoRowId value
specified here will be used as a prefix for the additional key column created. For
example, if the metric is defined as:
com.myCompany:Type=CustomerOrder,* columnOrder
is
CustomerName;OrderNumber;DateShipped
and assuming CustomerName;OrderNumber;Amount may not be unique if an order
is shipped in two parts, setting autoRowId as "ShipItem-" will populate an
additional key column for the metric for each row with ShipItem-0, ShipItem-1,
ShipItem-2...ShipItem-n.
в– Metric Service -- True/False. Indicate whether MetricService is enabled on a target
Weblogic domain. This property would be false (unchecked) in most cases for
Metric Extensions except when metrics that are exposed via the Oracle DMS
MBean needs to be collected. If MetricService is set to true, then the basic property
metric becomes the MetricService table name and the basic property columnOrder
becomes a semicolon-separated list of column names in the MetricService table.
Refer to the Monitoring Using Web Services and JMX chapter
in the OracleВ® Enterprise Manager Extensibility Programmer's Reference
for an in-depth example of creating a JMX based Metric Extension.
Note:
9.5 Converting User-defined Metrics to Metric Extensions
For targets monitored by Enterprise Manager 12c Agents, both older user-defined
metrics and metric extensions will be supported. After release 12c, only metric
extensions will be supported. If you have existing user-defined metrics, it is
recommended that you migrate them to metric extensions as soon as possible to
prevent potential monitoring disruptions in your managed environment.
Using Metric Extensions 9-29
Converting User-defined Metrics to Metric Extensions
Migration of user-defined metric definitions to metric extensions is not automatic and
must be initiated by an administrator. The migration process involves migrating
user-defined metric metadata to metric extension metadata.
Migration of collected user-defined metric historic data is not
supported.
Note:
After the user-defined metric is migrated to the metric extension and the metric
extension has been deployed successfully on the target, the user-defined metric should
be either disabled or deleted. Disabling the collection of the user-defined metric will
retain the metadata definition of the user-defined metric) but will clear all the open
alerts, remove the metric errors and prevent further collections of the user-defined
metric. Deleting the user-defined metric will delete the metadata, historic data, clear
open alerts and remove metric errors.
9.5.1 Overview
The User Defined Metric (UDM) to Metric Extension (ME) migration replaces an
existing UDM with a new or existing ME. The idea behind the migration process is to
consolidate UDMs with the same definition that have been created on different targets
into a single ME. In addition, MEs support multiple metric columns, allowing the user
to combine multiple related UDMs into a single ME.
This migration process is comprised of the following steps:
1.
Identify the UDMs that need to be migrated.
2.
Use the provided EM CLI commands to create or select a compatible metric
extension.
3.
Test and publish the metric extension.
4.
Deploy the metric extension to all targets and templates where the original UDMs
are located. Also update the existing notification rules to refer to the ME.
5.
Delete the original UDMs. Note that the historical data and alerts from the old
UDM is still accessible from the UI, but the new ME will not inherit them.
Note that the credentials being used by the UDM are NOT migrated to the newly
created ME. The user interface allows a user to specify the credential set required by
the metric extension. If the ME does not use the default monitoring credentials, the
user will need to create a new credential set to accommodate the necessary credentials
through the relevant EM CLI commands. This set will then be available in the
credentials page of the metric extension wizard.
The migration process is categorized by migration sessions. Each session is responsible
for migrating one or more UDMs. The process of migrating an individual session is
referred to as a task. Therefore, a session is comprised of one or more tasks. In general
terms, the migration involves creating a session and providing the necessary input to
complete each tasks within that session. The status of the session and tasks is viewable
throughout the workflow.
9.5.2 Commands
A number of EM CLI commands are responsible for completing the various steps of
this process. For a more detailed explanation of the command definition, please use
the 'EM CLI help <command>' option.
9-30 OracleВ® Enterprise Manager Administration
Converting User-defined Metrics to Metric Extensions
в– list_unconverted_udms - Lists the UDMs that have yet to be migrated and not in
a session
в– create_udmmig_session - Creates a session to migrate one or more UDMs
в– udmmig_summary - Lists the migration sessions in progress
в– udmmig_session_details - Provides the details of a specific session
в– в– в– udmmig_submit_metricpics - Provides a mapping between the UDM and the ME
in order to create a new ME or use an existing one
udmmig_retry_deploys - Deploys the ME to the targets where the UDM is
present. Note that the ME has to be in a deployable draft or published state for this
command to succeed
udmmig_request_udmdelete - Deletes the UDM and completing the migration
process
Usage Examples
The following exercise outlines a simple use case to showcase the migration
Consider a system with one host (host1) that has one host UDM (hostudm1) on it. The
goal is to create a new ME (me1) that represents the UDM. The sequence of commands
would be as follows
$ emcli list_unconverted_udms
-------------+----------------------+-----------+-------------------Type
| Name
| Metric
| UDM
-------------+----------------------+-----------+-------------------host
| host1
|UDM
| hostudm1
The command indicates that there is only one UDM that has not been migrated or in
the process of migration at this stage. Now proceed with the creation of a session.
$ emcli create_udmmig_session -name=migration1 -desc="Convert UDMs for host
target" -udm_choice=hostudm1 -target=host:host1
Migration session created - session id is 1
The command creates a migration session with name migration1 and the description
"convert UDMs for host target". The udm_choice flag indicates the UDM chosen and
the target flag describes the target type and the target on which the UDM resides.
Migration sessions are identified by session IDs. The current session has an ID of 1.
$ emcli udmmig_summary
------+--------------+------------------+------+------+--------+------+-------ID
| Name
| Description
|#Tgts |Todo |#Tmpls |Todo |IncRules
------+--------------+------------------+------+------+--------+------+-------1
|migration1
|Convert UDMS
|
| 1/1 | 0
| -/0 | -/0
------+--------------+------------------+------+------+--------+------+--------
The command summarizes all the migrations sessions currently in progress. The name
and description fields identify the session. The remaining columns outline the number
of targets, templates and incident rules that contain references to the UDM that is
being converted to a metric extension. The 'Todo' columns indicate the number of
targets, templates and incident rules whose references to the UDM are yet to be
updated. Since a migration session can be completed over a protracted period of time,
Using Metric Extensions 9-31
Converting User-defined Metrics to Metric Extensions
the command provides an overview of the portion of the session that was been
completed.
$ emcli list_unconverted_udms
There are no unconverted udms
Since the UDM is part of a migration session, it no longer shows up in the list of
unconverted UDMs.
$ emcli udmmig_session_details -session_id=1
Name: migration1
Desc: Convert UDMs for host target
Created: <date> <time>
UDM Pick: [hostudm1]
UDMs being converted:
----------+----------+---------+------+------------+---------+---------+----Type
|Name
|UDM
|#MC
|Metric
|Column
|DepS
|DelS
----------+----------+---------+------+------------+---------+---------+----host
|host1
|hostudm1 | 0
|
|
|WAIT
|WAIT
----------+----------+---------+------+------------+---------+---------+-----
The command provides the status of a single migration session. It lists the name of the
UDM and the target type and name of the target on which the UDM resides. In
addition, it also outlines the metric extensions currently in the EM instance that match
the UDM. The user can elect to use one of the existing choices or create an entirely new
metric extension.
The system attempts to find compatible metric extensions by matching the properties
of the UDM. For example, in the case of a host UDM, the system tries to find a metric
extension that has the same command, script and argument fields. In the case of a
database UDM, the system attempts to match the SQL query.
Finally, the DepS column indicates whether the metric extension that was matched to
the UDM has been deployed to the target on which the UDM is defined. The DelS
column tells the user whether the UDM has been deleted after the metric extension has
been deployed. As the user proceeds with the migration, the above table is updated
from left to right. When the delete status column is set to complete, the migration
session has ended.
$ emcli udmmig_submit_metricpicks -session_id=1 -input_file=metric_picks:filename
Successfully submitted metric picks for migration session
The command instructs the Enterprise Manager instance to use an existing metric
extension or create a new one to replace the UDM. The various options are presented
through a file, which is filename in the above command. The contents of the file are
shown below
"host,host1,hostudm1,N,ME$me1,Usage"
Each line in the file represents a mapping from n UDM to an ME. The line provides the
target type, the name of the target, the name of the UDM, a flag to indicate whether
the metric extension is new (N) or existing (E), the name of the metric extension (note
that ME$ must be prefixed) and the column name.
The types of UDMs supported are:
в– Host (host)
в– Database (oracle_database)
9-32 OracleВ® Enterprise Manager Administration
Converting User-defined Metrics to Metric Extensions
в– RAC (rac_database)
A user can only specify the names of the data columns via the collection item portion
of the file. A metric extension created through migration will always have two
columns to represent the structure of the UDM. The first column is an index column
for single column UDMs while the second column uses the column name mentioned
in the file. In the case of two column UDMs, the first column of the ME is termed as
the 'KEY' column and the collection name is used for the second column.
At this stage, the metric extension has been created and is visible in the metric
extensions library.
$ emcli udmmig_session_details -session_id=1
Name: migration1
Desc: Convert UDMs for host target
Created: <date> <time>
UDM Pick: [hostudm1]
Udms being converted:
----------+--------+---------+------+----------+------------+---------+----Type
|Name
|UDM
|#MC
|Metric
|Column
|DepS
|DelS
----------+--------+---------+------+----------+------------+---------+----host
|host1
|hostudm1 |
1 | ME$me1
| Usage
|WAIT
|WAIT
----------+--------+---------+------+----------+------------+---------+----#MC : There are 1 matches for udms in this session.
Use emcli udmmig_list_matches to list available matches
The session details command indicates that there is one matching metric extension for
this UDM (the value of the MC column is 1) and that metric extension is named as
ME$me1. At this stage, we are ready to test the metric extension through the library
page. Once the testing is complete and the user is satisfied with the metric extension
that has been created, it is ready to be deployed. In order to deploy, the metric
extension has to be minimally saved as a deployable draft.
$ emcli udmmig_retry_deploys -session_id=1 -input_file=metric_tasks:filename2
Metric Deployments successfully submitted
Note that the system will trigger a job to automatically deploy the metric extension to
all targets where the UDM was present once the metric extension is published. If the
user is interested in manually controlling the operation, the above command will
perform the necessary steps. The command is similar to the submit_metricpicks option
in that a file with the UDM to target mapping is provided. It is referred to by filename2
above. The contents of the file are as follows
"host,host1,hostudm1"
Each line in the file is a mapping from the UDM to the targets type and target on
which it resides. Once the command is executed, jobs to deploy the metric extensions
to various targets have been launched and can be tracked through the user interface.
$ emcli udmmig_request_udmdelete -session_id=1 -input_file=metric_tasks:demo_tasks
Udm deletes successfully submitted
The final command deletes the UDMs that were migrated to metric extensions. Note
that this command might partially finish based on how many of the deployments were
completed when the command was run.
$ emcli udmmig_session_details -session_id=1
Using Metric Extensions 9-33
Metric Extension Command Line Verbs
Name: migration1
Desc: Convert UDMs for host target
Created: <date > <time>
Completed: <date > <time>
UDM Pick: [hostudm1]
Udms being converted:
--------+----------+---------+------+------------+------------+---------+----Type
|Name
|UDM
|#MC
|Metric
|Column
|DepS
|DelS
--------+----------+---------+------+------ ----+------------+---------+----host
|host1
|hostudm1 | 1
| ME$me1
| Usage
|COMP
|COMP
--------+----------+---------+------+------------+------------+---------+----#MC : There are 1 matches for udms in this session.
Use emcli udmmig_list_matches to list available matches
The session details command shows that the migration process is indeed complete.
9.6 Metric Extension Command Line Verbs
Metric extensions can be manipulated outside the UI via the Enterprise Manager
Command Line Interface (EM CLI). Two categories of verbs are available:
в– в– Metric Extension Verbs
–
export_metric_extension: Export a metric extension to an archive file
–
get_unused_metric_extensions: Get a list of unused metric extensions.
–
import_metric_extension: Import a metric extension archive file.
–
publish_metric_extension: Publish a metric extension for use by all
administrators.
–
save_metric_extension_draft: Save a deployable draft of a metric extension.
User-defined Metric Migration Verbs
–
abort_udmmig_session: Abort (partially) user-defined metric migration session.
–
analyze_unconverted_udms: Analyze the unconverted user-defined metrics.
–
create_udmmig_session: Create a user-defined metric migration session.
–
list_unconverted_udms: List the user-defined metrics that are not yet in a
migration session.
–
udmmig_list_matches: List the matching metrics per user-defined metric in a
specific user-defined metric migration session.
–
udmmig_request_udmdelete: Request deletion of user-defined metrics from
targets.
–
udmmig_retry_deploys: Retry deployment of metric extensions to targets.
–
udmmig_session_details: Retrieve the details of a specific user-defined metric
migration session.
–
udmmig_submit_metricpicks: Select the metrics to replace user-defined metrics
in a session.
–
udmmig_summary: Summarize the status of all user-defined metric migration
sessions.
9-34 OracleВ® Enterprise Manager Administration
Metric Extension Command Line Verbs
–
udmmig_update_incrules: Update user-defined metric incident rules to include
replacement metric references.
Metric Extension Verbs
emcli export_metric_extension
-file_name=<name of the metric extension archive>
-target_type=<target type of the metric extension>
-name=<name of the metric extension
-version=<version of the metric extension>
Description:
Export a metric extension archive file.
Options:
-file_name=<file name>
The name of the metric extension archive file to export into.
-target_type=<target type>
Target type of the metric extension.
-name=<name>
Name of the metric extension.
-version=<version>
Version of the metric extension to be exported.
emcli get_unused_metric_extensions
Description:
Get a list of metric extensions that are deployed to agents but not attached
to any targets.
emcli import_metric_extension
-file_name=<name of the metric extension archive>
-rename_as=<name of the metric extension to import as>
Description:
Import a metric extension archive file.
Options:
-file_name=<file name>
The name of the metric extension archive file to be imported.
-rename_as=<metric extension name>
Import the metric extension using the specified name, replacing the name
given in the archive.
emcli publish_metric_extension
-target_type=<target type of the metric extension>
-name=<name of the metric extension
-version=<version of the metric extension>
Description:
Publish a metric extension for use by all administrators.
The metric extension must currently be a deployable draft.
Options:
-target_type=<target type>
Target type of the metric extension.
-name=<name>
Name of the metric extension.
Using Metric Extensions 9-35
Metric Extension Command Line Verbs
-version=<version>
Version of the metric extension to be published.
emcli save_metric_extension_draft
-target_type=<target type of the metric extension>
-name=<name of the metric extension
-version=<version of the metric extension>
Description:
Save a deployable draft of a metric extension. The metric
extension must currently be in editable state. Once saved as
draft, the metric extension will no longer be editable.
Options:
-target_type=<target type>
Target type of the metric extension.
-name=<name>
Name of the metric extension.
-version=<version>
Version of the metric extension to be saved to draft.
User-Defined Metric Verbs
emcli abort_udmmig_session
-session_id=<sessionId>
[-input_file=specific_tasks:<complete path to file>]
Description:
Abort the migration of user-defined metrics to MEs in a session
Options:
-session_id=<id of the session>
Specify the id that was returned at time of session created,
or from the output of udmmig_summary
[-input_file=specific_tasks:<complete file path>]
This optional parameter points at a file name that contains a
target, user-defined metric,
one per line in the following format:
<targetType>,<targetName>,<collection name>
Use targetType=Template to indicate a template
Use * for collection name to abort all user-defined metrics for a target
emcli analyze_unconverted_udms [-session_id=<sessionId>]
Description:
Analyze user-defined metrics and list unique user-defined metrics, any
possible matches, and
templates that can apply these matching metric extensions
Options:
-session_id=<id of a session to be reanalyzed>
Not specifying a session id causes the creation of a analysis
session that contains all unconverted user-defined metrics. You can specify
this session id in future invocations to get fresh analysis.
emcli create_udmmig_session
-name=<name of the session>
9-36 OracleВ® Enterprise Manager Administration
Metric Extension Command Line Verbs
-desc=<description of the session>
[-udm_choice=<specific udm to convert>]*
{-target=<type:name of the target to migrate> }*
| {-input_file=targetList:<complete path to file>};
the template to update> }*
| {-input_file=templateList:<complete path to file>}
[-allUdms]
{-template=<name of
Description:
Creates a session to migrate user-defined metrics to metric extensions for
targets.
Options:
-name=<session name>
The name of the migration session to be created.
-desc=<session session description>
A description of the migration session to be created.
-udm_choice=<udm name>
If the session should migrate specific user-defined metrics, specify them
Otherwise, all user-defined metrics will be migrated
-target=<type:name of target to migrate>
The type:name of the target to be updated.
Multiple values may be specified.
-input_file=targetList:<complete file path>
This takes a file name that contains a list of targets,
one per line in the following format:
<targetType>:<targetName>
-template=<name of template to migrate>
The name of the template to update.Multiple values may be specified
-input_file=templateList:<complete file path>
This takes a file name that contains a list of templates,
one name per line
-allUdms
This forces the session to contain all user-defined metrics from targets and
templates (default behavior just picks those not in a session)
emcli list_unconverted_udms [-templates_only]
Description:
Get the list of all user-defined metrics that are not yet in a migration
session
Options:
-templates_only
Only lists unconverted user-defined metrics in templates.
emcli udmmig_list_matches
-session_id=<sessionId>
Description:
Lists the matching metrics per user-defined metric in a migration session
Options:
-session_id=<id of the session>
Specify the id that was returned at time of session created,
or from the output of udmmig_summary
emcli udmmig_request_udmdelete
-session_id=<sessionId>
Using Metric Extensions 9-37
Metric Extension Command Line Verbs
-input_file=metric_tasks:<complete path to file>
Description:
Delete the user-defined metrics that have been replaced by Metric Extenions
Options:
-session_id=<id of the session>
Specify the id that was returned at time of session created,
or from the output of udmmig_summary
-input_file=metric_tasks:<complete file path>
This takes a file name that contains a target, user-defined metric,
one per line in the following format:
<targetType>,<targetName>,<collection name>
emcli udmmig_retry_deploys
-session_id=<sessionId>
-input_file=metric_tasks:<complete path to file>
Description:
Retry the deployment of metric extensions to a target
Options:
-session_id=<id of the session>
Specify the id that was returned at time of session created,
or from the output of udmmig_summary
-input_file=metric_tasks:<complete file path>
This takes a file name that contains a target, user-defined metric,
one per line in the following format:
<targetType>,<targetName>,<collection name>
emcli udmmig_submit_metricpicks
-session_id=<sessionId>
-input_file=metric_picks:<complete path to file>
Description:
Supply the metric picks to use to replace user-defined metrics per target in a
session
Options:
-session_id=<id of the session>
Specify the id that was returned at time of session created,
or from the output of udmmig_summary
-input_file=metric_picks:<complete file path>
This takes a file name that contains a target, user-defined metric, metric
pick,
one per line in the following format:
<targetType>,<targetName>,<collection name>,[N/E],<metric>,<column>
using N if a new metric should be created or E if an existing
metric is referenced.
emcli udmmig_summary
[-showAll]
Description:
Gets the summary details of all migration sessions in progress
Options:
9-38 OracleВ® Enterprise Manager Administration
Metric Extension Command Line Verbs
-showAll
This prints out all sessions including those that are complete.
By default, only in-progress sessions are listed.
emcli udmmig_update_incrules
-session_id=<sessionId>
-input_file=udm_inc_rules:<complete path to file>
Description:
Update Incident Rules that reference user-defined metrics with a reference to
replacing metric extension.
Options:
-session_id=<id of the session>
Specify the id that was returned at time of session created,
or from the output of udmmig_summary
-input_file=udm_inc_rules:<complete file path>
This takes a file name that contains rule, user-defined metric, metric,
one per line in the following format:
<ruleset id>,<rule id>,<udm name>,<metric name>
Using Metric Extensions 9-39
Metric Extension Command Line Verbs
9-40 OracleВ® Enterprise Manager Administration
10
Advanced Threshold Management
10
There are monitoring situations in which different workloads for a target occur at
regular (expected) intervals. Under these conditions, a static alert threshold would
prove to be inaccurate. For example, the accurate alert thresholds for a database
performing Online Transaction Process (OLTP) during the day and batch processing at
night would be different. Similarly, database workloads can change based purely on
different time periods, such as weekday versus weekend. In both these situations,
fixed, static values for thresholds might result in false alert reporting.
Advanced Thresholds allow you to define and manage alert thresholds that are either
adaptive (self-adjusting) or time-based (static).
в– в– Adaptive Thresholds are thresholds based on statistical calculations from the target's
observed behavior (metrics).
Time-based Thresholds are user-defined threshold values to be used at different
times of the day/week to account for changing target workloads.
This chapter covers the following topics:
в– Accessing the Advanced Threshold Management Page
в– Adaptive Thresholds
в– Time-based Static Thresholds
в– Determining What is a Valid Metric Threshold
10.1 Accessing the Advanced Threshold Management Page
You manage advanced thresholds from the Enterprise Manager console. The
Advanced Threshold Management page allows you to create time-based static
thresholds and adaptive thresholds. To access this page:
1.
From a target home page (host, for example), navigate to the Metric Collection
and Settings page.
2.
From the Related Links region, click Advanced Threshold Management.
The Advanced Threshold Management page displays as shown in the following
graphic.
Advanced Threshold Management
10-1
Adaptive Thresholds
Figure 10–1 Advanced Threshold Management Page
10.2 Adaptive Thresholds
Adaptive thresholds are statistically computed thresholds that adapt to target
workload conditions. Adaptive thresholds apply to all targets (both Agent and
repository-monitored).
Important Concepts
Creating an adaptive threshold is based on the following key concepts:
в– Baseline periods
For the purpose of performance evaluation, a baseline period is a period of time
used to characterize the typical behavior of the system. You compare system
behavior over the baseline period to that observed at some other time.
There are two types of baseline periods:
в– Moving window baseline periods: Moving window baselines are defined as some
number of days prior to the current date. This "window" of days forms a rolling
interval that moves with the current time. The number of days that can be used to
define moving window baseline in Enterprise Manager are:
–
7 days
–
14 days
–
21 days
–
30 days
Example: Suppose you have specified trailing 7 days as a time period while
creating moving window baseline. In this situation, the most recent 7-day
period becomes the baseline period for all metric observations and
comparisons today. Tomorrow, this reference period drops the oldest day and
picks up today.
Moving window baselines allow you to compare current metric values with
recently observed history, thus allowing the baseline to incorporate changes to
10-2 OracleВ® Enterprise Manager Administration
Adaptive Thresholds
the system over time. Moving window baselines are suitable for systems with
predictable workload cycles.
Enterprise Manager computes moving window statistics every
day rather than sampling.
Note:
10.2.1 Registering Adaptive Threshold Metrics
Adaptive threshold metrics are not immediately available by default; they must be
defined and added to the system (registered) in order for them to become available for
use by Enterprise Manager. Not all metrics can have adaptive thresholds: Adaptive
Threshold metrics must fall into one of the following categories:
в– Load
в– LoadType
в– Utilization
в– Response
There are two methods for registering adaptive metric:
в– Standard Registration Method
в– Quick Configuration Method
10.2.1.1 Standard Registration Method
You can manually register adaptive threshold metrics from the Advanced Threshold
Management page.
1.
From a target menu (Host is used in this example), select Monitoring and then
Metric and Collection Settings.
2.
In the Related Links area, click Advanced Threshold Management.The Advanced
Threshold Management page displays.
3.
From the Select Active Adaptive Setting menu, select Moving Window,
additional controls are displayed allowing you to define the moving window’s
Threshold Change Frequency and the Accumulated Trailing Data that will be used to
compute the adaptive thresholds.
Advanced Threshold Management
10-3
Adaptive Thresholds
в– Threshold Change Frequency (The target timezone is used.)
–
None: One set of thresholds will be calculated using past data. This set of
thresholds will be valid for the entire week.
None should be used when there is no usage pattern between daytime versus nighttime or within hours of a day.
–
By Day and Night: Two sets of thresholds (day and night) will be
calculated using past data. Day thresholds will be calculated using
previous day’s daytime data, Night thresholds will be calculated using
previous day’s nighttime data. Thresholds will be changed every day and
night.
By Day and Night should be used when there are distinct performance and
usage variations between day hours and night hours.
–
By Weekdays and Weekend: Two sets of thresholds (weekdays and
weekend) will be calculated using past data. Weekdays thresholds will be
calculated using the previous weekdays data. Weekend thresholds will be
calculated using the previous weekend data. Thresholds will be changed
at start of the weekdays and start of the weekend.
–
By Day and Night, over Weekdays and Weekend: Four sets of thresholds
will be calculated using past data. Weekdays Day thresholds will be
calculated using the previous weekday’s daytime data, Weekdays Night
thresholds will be calculated using the previous weekday’s nighttime
data. Weekends Day thresholds will be calculated using previous
weekend’s daytime data, Weekends Night thresholds will be calculated
using previous weekend’s nighttime data. Thresholds will be changed
each day and night.
Weekday day hours (7a.m. to 7p.m)
Weekend day hours (7am to 7pm)
Weekday night hours (7pm to 7am)
Weekend night hours (7pm to 7am)
–
By Day of Week: Seven sets of thresholds will be calculated, one for each
day of the week. Thresholds will be calculated using the previous week’s
same-day data. Thresholds will be changed every day.
By Day of Week should be used when there is significant daily variation in
usage for each day of week.
–
в– By Day and Night, per Day of Week: Fourteen sets of thresholds will be
calculated, one for each day of the week and one for each night of the
week. Day thresholds will be calculated using previous weeks same-day
daytime data, Night thresholds will be calculated using the previous
week’s same-day nighttime data. Thresholds will be changed every day
and night each day of the week.
Accumulated Trailing Data
Total time period for which metric data will be collected. Options are 7, 14, 21,
and 28 days. In general, you should select the larger value as the additional
data helps in computing more accurate thresholds.
4.
The Register Metrics button becomes active in the Register Adaptive Metrics
region. Click Register Metrics. The Metric Selector dialog displays.
10-4 OracleВ® Enterprise Manager Administration
Adaptive Thresholds
5.
Select the desired metric(s) and then click OK. A confirmation dialog displays
stating that the selected metric(s) will be added to this target's Adaptive Setting.
Click Yes to confirm the action. The selected metrics appear in the Register
Adaptive Metrics region.
6.
Once registered as adaptive metrics, you can then select individual metrics to
configure thresholds. When metrics are first registered, by default, Enterprise
Manager enables Significance Level and sets the warning and critical thresholds at
95 and 99 percentile respectively.
Advanced Threshold Management
10-5
Adaptive Thresholds
10.2.1.2 Quick Configuration Method
If you have a large number of metrics to configure for many different target types,
using the standard registration method can be cumbersome. The Quick Configuration
method is typically used by integrators to specify predefined settings for specific
usage patterns, such as a database running in batch processing mode versus a
database running in an online transaction processing (OLTP) mode.
The Quick Configuration method of registering adaptive threshold metrics involves
updating the target metadata using Enterprise Manager’s Metric Registration Service
(MRS). The MRS allows you to upload one or more updated target XML metadata files
to the Oracle Management Service and have it registered with the Enterprise Manager
framework. For more information about the MRS and the Quick Configuration
method, see the OracleВ® Enterprise Manager Cloud Control Extensibility
Programmer's Reference.
10.2.2 Configuring Adaptive Thresholds
Once you have registered the adaptive metrics, you now have the option of
configuring the thresholds if the predefined thresholds do not meet your monitoring
requirements.
To configure adaptive thresholds:
1.
From the Register Adaptive Metrics region, select the metric(s) you wish to
configure and click Configure Thresholds. The Configure Thresholds dialog
displays.
2.
Choose whether you want your threshold to be based on:
Significance Level: Thresholds based on significance level use statistical relevance
to determine which current values are statistical outliers. The primary reason to
use Significance Level for alerting is that you are trying to detect statistical outliers
in metric values as opposed to simply setting a threshold value. Hence, thresholds
are percentile based. For example, if the significance level is set to .95 for a
warning threshold, the metric threshold is set where 5% of the collected metric
values fall outside this value and any current values that exceed this value trigger
an alert. A higher significance level of .98 or .99 will cause fewer alerts to be
triggered.
Percentage of Maximum: These types of thresholds compute the threshold values
based on specified percentages of the maximum observed over the period of time
you selected. Percentage-of-maximum-based alerts are generated if the current
10-6 OracleВ® Enterprise Manager Administration
Adaptive Thresholds
value is at or above the percentage of maximum you specify. For example, if a
maximum value of 1000 is encountered during a time group, and if 105 is specified
as the Warning level, then values above 1050 (105% of 1000 = 1050) will raise an
alert.
For both types of alerts you can set the Occurrences parameter, which is the
number of times the metric crosses a threshold value before an alert is generated.
Clear Threshold: Thresholds for the selected metrics will be cleared. No Alert will
be generated. Use this option when you do not want any thresholds set for the
metrics but you do not want to remove historical data. Important: Deregistering
metrics will remove the historical data.
Occurrences: Consecutive number of occurrences before raising an alert.
Depending on the option selected, the Warning, Critical, and Occurrence setting
options will change.
The Threshold action for insufficient data menu allows you set the appropriate
action for Enterprise Manager to take if there is not enough data to calculate a
valid metric threshold. There are two actions available: Preserve the prior threshold
and Suppress Alerts.
3.
Click OK to set the changes.
10.2.3 Determining whether Adaptive Thresholds are Correct
Even though Enterprise Manager will use the adaptive threshold settings to determine
an accurate target workload-metric threshold match, it is still be necessary to match
the metric sampling schedule with the actual target workload. For example, your
moving window baseline period (see Moving Window Baseline Periods on page 10-2)
should match the target workloads. In some situations, you may not know the actual
target workloads, in which case setting adaptive thresholds may be problematic.
To help you determine the validity of your adaptive thresholds, Enterprise Manager
allows you to analyze threshold using various adaptive settings to determine whether
the settings are correct.
To analyze existing adaptive thresholds:
1.
From the Register Adaptive Metrics region, click Analyze Thresholds.
The Analyze Threshold page displays containing historical metric data charts (one
for each metric).
Advanced Threshold Management
10-7
Adaptive Thresholds
2.
Modify the adaptive threshold parameters to closely match metric threshold
settings with the target workload. You can experiment with the following adaptive
metric parameters:
Threshold Change Frequency
Threshold Based On
Metric Warning and Critical Thresholds
3.
Once you are satisfied with the modifications for the Threshold Change Frequency
or any of the individual metrics, click Save to set the new parameters.
10-8 OracleВ® Enterprise Manager Administration
Adaptive Thresholds
10.2.4 Testing Adaptive Metric Thresholds
Because adaptive metric thresholds utilize statistical sampling of data over time, the
accuracy of the thresholds will rely on the quantity and quality of the data collected.
Hence, a sufficient amount of metric data needs to have been collected in order for the
thresholds to be valid. To verify whether enough data has been collected for metrics
registered with adaptive thresholds, use the Test All function.
1.
From the Registered Adaptive Metrics regions, click Test All.
Enterprise Manager evaluates the adaptive threshold metrics and then displays
the results in the Test Metrics window.
If there is sufficient data collected to compute the adaptive threshold, a green
check appears in the results column. A red ’x’ appears in the results column if
there is insufficient data collected. To resolve this situation, you can use a longer
accumulating trailing data window. Additionally, you can have metric data
collected more frequently.
2.
Click OK once you are finished viewing the results.
10.2.5 Deregistering Adaptive Threshold Metrics
If you no longer want specific metrics to be adaptive, you can deregister them at any
time. To deregister an adaptive threshold metric:
1.
From the Register Adaptive Metrics regions, select the metric(s) you wish to
deregister.
Advanced Threshold Management
10-9
Time-based Static Thresholds
2.
Click Deregister. A confirmation displays asking if you want the metric removed
from the target’s adaptive setting.
3.
Click Yes.
10.3 Time-based Static Thresholds
Time-based static thresholds allow you to define specific threshold values to be used at
different times to account for changing workloads over time. Using time-based static
thresholds can be used whenever the workload schedule for a specific target is well
known or if you know what thresholds you want to specify.
10.3.1 Registering Time-based Static Thresholds
To register metrics with time-based static thresholds:
1.
From the target menu (Host is used in this example), select Monitoring and then
Metric and Collection Settings.
2.
In the Related Links area, click Advanced Threshold Management.The Advanced
Threshold Management page displays.
3.
Click on the Time Based Static Settings tab.
4.
Select the Threshold Change Frequency.
5.
Click Register Metrics.
The Metric Selector dialog displays.
10-10 OracleВ® Enterprise Manager Administration
Time-based Static Thresholds
6.
Select the desired metric(s) and click OK.
The selected metrics appear in the Registered Metrics table.
7.
Enter the desired metric thresholds and click Save once you are done.
If you want to set the thresholds for multiple metrics simultaneously, check the
Select box for the metrics you want to update and click Configure Thresholds. The
Configure Thresholds dialog displays.
Enter the revised Warning and Critical threshold values and click OK. A
confirmation dialog displays stating that existing metric threshold values will be
overwritten. Click Yes.
8.
Optionally, you can change the Threshold Change Frequency. To do so, from the
Time Based Static Thresholds Settings page, click Modify. The Modify Threshold
Advanced Threshold Management 10-11
Determining What is a Valid Metric Threshold
Change Frequency dialog displays allowing you to select a new change frequency.
Select a new frequency and click OK.
A confirmation dialog displays stating that changing Threshold Change
Frequency will affect all the registered metrics and whether you want to continue.
Click Yes to proceed.
9.
Click Save to ensure all changes have been saved to the Enterprise Manager
repository.
10.3.2 Deregistering Time-based Static Thresholds
If you no longer require time-based static threshold metrics, you can deregister them
from the target.
To deregister time-based static metric thresholds:
1.
From the Time Based Static Thresholds tab, select the metric(s) you want to
deregister.
2.
Click Remove. The metric entry is removed from list of
3.
Click Save to save the changes to the Enterprise Manager repository.
10.4 Determining What is a Valid Metric Threshold
As previously discussed, static thresholds do not account for expected performance
variation due to increased/decreased workloads encountered by the target, such as the
workload encountered by a warehouse database target against which OLTP
transactions are performed. Workloads can also change based on different time
periods, such as weekday versus weekend, or day versus night. These types of
workload variations present conditions where fixed static metric threshold values may
cause monitoring issues, such as the generation of false and/or excessive metric alerts.
Ultimately, your monitoring needs dictate how to best go about obtaining accurate
metric thresholds.
10-12 OracleВ® Enterprise Manager Administration
11
Utilizing the Job System and Corrective
Actions
11
The Enterprise Manager Cloud Control Job System can automate routine
administrative tasks and synchronize components in your environment so you can
manage them more efficiently.
This chapter facilitates your usage of the Job System by presenting instructional
information in the following sections:
в– Job System Purpose and Overview
в– Preliminary Considerations
в– Creating Jobs
в– Viewing and Analyzing Job Status
в– Generating Job Event Criteria
в– Creating Event Rules For Job Status Change
в– Using Diagnostic Tools
в– Creating Corrective Actions
11.1 Job System Purpose and Overview
The Enterprise Manager Job System serves these purposes:
в– Automates many administrative tasks; for example: backup, cloning, and patching
в– Enables you to create your own jobs using your own custom OS and SQL scripts
в– Enables you to create your own multi-task jobs comprised of multiple tasks
в– Centralizes environment job scheduling into one robust tool
A job is a unit of work that you define to automate commonly-run tasks. Scheduling
flexibility is one of the advantages of jobs. You can schedule a job to start immediately
or start at a later date and time. You can also run the job once or at a specific interval,
such as three times every month.
The Job Activity page (Figure 11–1) is the hub of the Job System. From this page, you
can:
в– в– Search for existing job runs and job executions filtered by name, owner, status,
scheduled start, job type, target type, and target name
Create a job
Utilizing the Job System and Corrective Actions 11-1
Job System Purpose and Overview
в– View or edit the job definition
в– Create like, copy to library, suspend, resume, stop, and delete a job
в– View results, edit, create like, suspend, resume, retry, stop, and delete a job run or
execution
Figure 11–1 Job Activity Page
Besides accessing the Job Activity page from the Enterprise menu, you can also access
this page from any target-specific menu for all target types by selecting Job Activity
from the target type’s menu. When you access this page from these alternate locations,
rather than showing the entire list of jobs, the Job Activity page shows a subset of the
jobs associated with the particular target.
11.1.1 What Are Job Executions and Job Runs?
The following sections explain the characteristics of each of these.
11.1.1.1 Job Executions
Job executions are usually associated with one target, such as a patch job on a
particular database. However, job executions are not always a one-to-one mapping to a
target. Some executions have multiple targets, such as comparing hosts. When a job is
run against multiple targets, it either runs with all targets in a single execution or with
each target in a separate execution.
Job executions are usually associated with one target, such as a patch job on a
particular database. These are called single-target jobs because each execution has only
one target. However, job executions are not always a one-to-one mapping to a target.
Some executions have multiple targets, such as comparing hosts. These jobs are called
single-execution jobs, since there is only one execution for all the targets. When a job is
run against multiple targets, it runs in one or many executions depending on whether
it is a single-execution or single-target job. A few jobs have no target. These jobs are
called targetless jobs and run in one execution.
When you submit a job to many targets, it would be tedious to examine the status of
each execution of the job against each target. For example, suppose you run a backup
11-2 OracleВ® Enterprise Manager Administration
Preliminary Considerations
job against several databases. A typical question would be: Were all the backup jobs
successful, and if not, which jobs failed? If this backup job runs every week, you
would want to know which backups were successful and those that failed each week.
11.1.1.2 Job Runs
With the Job System, you can easily get these answers by viewing the job run. A job
run is the summary of all job executions of a job that ran on a particular scheduled
date. For example, if you have a job scheduled for March 5th, you will have a March 5
job run. The job table that shows the job run provides a roll-up of the status of the
executions, such as Succeeded, Failed, or Error.
11.1.2 Operations on Job Executions and Job Runs
Besides supporting the standard job operations of create, edit, create like, and delete,
the Job System enables you to:
■Suspend jobs —
You can suspend individual executions or entire jobs. For example, you may need
to suspend a job if a needed resource was unavailable, or the job needs to be
postponed.
If a job is scheduled to repeat but is suspended past the scheduled repeat time, or a
maximum of one day, the execution of this job would be marked "Skipped." A job
is also skipped when the scheduled time plus the grace period has passed.
■Resume jobs —
After you suspend a job, any scheduled executions do not occur until you decide
to resume the job.
■Retry all failed executions in a job run —
When analyzing individual executions or entire jobs, it is useful to retry a failed
execution after you determine the cause of the problem. This alleviates the need to
create a new job for that failed execution. When you use the Retry operation in the
Job System, Enterprise Manager provides links from the failed execution to the
retried execution and vice versa, should it become useful to retroactively examine
the causes of the failed executions. Only the most recent retry is shown in the Job
Run page.
With regard to job runs, the Job System enables you to:
в– Delete old job runs
в– Stop job runs
в– Retry all failed executions in a job run. Successful executions are never retried.
See Also: For more information on job executions and runs, refer to
Enterprise Manager Cloud Control online help.
11.2 Preliminary Considerations
Before proceeding to the procedural information presented in Section 11.3, "Creating
Jobs" on page 11-5, it is suggested that you read the topics presented in the sections
below:
в– Administrator Roles
в– Creating Scripts
Utilizing the Job System and Corrective Actions 11-3
Preliminary Considerations
в– Sharing Job Responsibilities
в– Submitting Jobs for Groups
11.2.1 Administrator Roles
Enterprise Manager provides the following administrator types:
■■■Administrator — Most jobs and other activities should be initiated using this
"normal" user type
Super Administrator — There may be limited use cases for a super administrator
to run jobs, create blackouts, or own targets, but generally, this should be avoided.
Repository owner (SYSMAN) — The special repository owner user SYSMAN
should almost never own or do any of the tasks listed for the other two types
above. This user should only be reserved for top-level actions, such as setting up
the site and so forth.
11.2.2 Creating Scripts
Besides predefined job tasks, you can define your own job tasks by writing code to be
included in OS and SQL scripts. The advantages of using these scripts include:
в– в– в– в– When defining these jobs, you can use target properties.
When defining these jobs, you can use the job library, which enables you to share
the job and make updates as issues arise. However, you need to resubmit modified
library jobs for them to take effect.
You can submit the jobs against multiple targets.
You can submit the jobs against a group. The job automatically keeps up with
changes to group membership.
в– For host command jobs, you can submit to a cluster.
в– For SQL jobs, you can submit to a Real Application Cluster.
11.2.3 Sharing Job Responsibilities
To allow you to share job responsibilities, the Job System provides job privileges. These
job privileges allow you to share the job with other administrators. Using privileges,
you can:
в– в– Grant access to the administrators who need to see the results of the job.
Grant Full access to the administrators who may need to edit the job definition or
control the job execution (suspend, resume, stop).
You can grant these privileges on an as-needed basis.
11.2.4 Submitting Jobs for Groups
Rather than listing a large number of targets individually, you can use a group as the
target of a job. All member targets in the group that match the selected target type of
the job are selected as actual targets of the job when it runs. If the membership of the
group changes, the actual target list of the job changes with it. If the job repeats, each
iteration (or "run") of the job executes on the matching targets in the group at the time
of the run.
11-4 OracleВ® Enterprise Manager Administration
Creating Jobs
Overriding the Target Type Selection
To override the target type selection for a group, set targetType=<override_target_
type> in the input file for the create_job verb. For example, the default target type for
OSCommand jobs is "host". To submit a job against a group of databases, specify:
target_list=my_db_group:composite
targetType=oracle_database
Note that any targets in the group that do not match the target type selected are
ignored.
See Also:
Chapter 6, "Managing Groups"
11.3 Creating Jobs
Your first task in creating a job from the Job Activity page is to choose a job type,
which the next section, Selecting a Job Type, explains. The most typical job types are
OS command jobs, script jobs, and multi-task jobs, which are explained in these
subsequent sections:
в– Creating an OS Command Job
в– Creating a SQL Script Job
в– Creating a Multi-task Job
11.3.1 Selecting a Job Type
Using the Job System, you can create a job by selecting one of the job types from the
Create Job drop-down in the Job Activity page. The most commonly used types are as
follows:
■OS Command — Runs an operating system command or script.
■SQL Script — Runs a user-defined SQL or PL/SQL script.
■Multi-Task — Use to specify primary characteristics for multi-task jobs or
corrective actions. Multi-task jobs enable you to create composite jobs by defining
tasks, with each task functioning as an independent job. You edit and define tasks
similarly to a regular job.
11.3.2 Creating an OS Command Job
Use this type of job to run an operating system command or script. Tasks and their
dependent steps for creating an OS command are discussed below.
Task 1 Initiate Job Creation
1. From the Enterprise menu, select Jobs, then Job Activity.
2.
Select OS Command from the Create Job drop-down, then click Go. The General
property page of the Create OS Command Job page appears.
Task 2 Specify General Job Information
Perform these steps on the General property page:
1.
Provide a required Name for the job, then select a Target Type from the
drop-down.
After you have selected a target of a particular type for the job, only targets of that
same type can be added to the job. If you change target types, the targets you have
Utilizing the Job System and Corrective Actions 11-5
Creating Jobs
populated in the Targets table disappear, as well as parameters and credentials for
the job.
If you specify a composite as the target for this job, the job executes only against
targets in the composite that are of the selected target type. For example, if you
specify a target type of host and a group as the target, the job only executes against
the hosts in the group, even if there are other non-host targets in the group. You
can also include clusters in the target list if they are of the same base target type.
For example, a host cluster would be selected if the target type is "host" and a RAC
database would be selected if the target type is database.
2.
Click Add, then select one or more targets from the Search and Select: Targets
pop-up window. The targets now appear in the Targets table.
3.
Click the Parameters property page link.
Task 3 Specify Parameters
Perform these steps on the Parameters property page:
1.
Select either Single Operation or Script from the Command Type drop-down.
The command or script you specify executes against each target specified in the
target list for the job. The Management Agent executes it for each of these targets.
Depending on your objectives, you can choose one of the following options:
в– в– Single Operation to run a specific command
Script to run an OS script and optionally provide an interpreter, which
processes the script; for example, %perlbin%/perl or /bin/sh .
Sometimes, a single command line is insufficient to specify the commands to run,
and you may not want to install and update a script on all hosts. In this case, you
can use the Script option to specify the script text as part of the job.
2.
Based on your objectives, follow the instructions in Section 11.3.2.1, "Specifying a
Single Operation" or Section 11.3.2.2, "Specifying a Script".
3.
Click the Credentials property page link.
The OS Command relies on the target host's shell to execute
the command/interpreter specified. On *nix systems, it is
/bin/sh -c and on Windows systems, it is cmd /c. The command line
specified is interpreted by the corresponding shell.
Note:
Task 4 Specify Credentials - (optional)
You do not need to provide input on this page if you want to use the system default of
using preferred credentials.
On the Credentials property page, you can specify the credentials that you want the
Oracle Management Service to use when it runs the OS Command job against target
hosts. The job can use either the job submitter's preferred host-based credentials for the
selected targets, or you can specify other credentials to override the preferred
credentials.
You do not need to provide input on this page if you have already set preferred
credentials.
11-6 OracleВ® Enterprise Manager Administration
Creating Jobs
Tip: preferred credentials are useful when a job is submitted on
multiple targets and each target needs to use different credentials for
authentication.
в– To use preferred credentials:
1.
Select the Preferred Credential radio button, which is the default selection.
If the target for the OS Command job is a host or host group, the preferred
host credentials are used. You specify these for the host target on the Preferred
Credentials page, and they are different from the host credentials for the host
on which the database resides.
2.
Select either Normal Host Credentials or Privileged Host Credentials from
the Host Credentials drop-down.
You specify these separately on the Preferred Credentials page, which you can
access by selecting Security from the Setup menu, then Preferred Credentials.
The Preferred Credentials page appears, where you can click the Manage
Preferred Credentials button to set credentials.
в– To use named credentials:
1.
Select the Named Credential radio button to override database or host
preferred credentials.
The drop-down list is a pre-populated credential set with values saved with
names. These are not linked to targets, and you can use them to provide
credential and authentication information to tasks.
в– To use other credentials:
1.
Select the New Credential radio button to override previously defined
preferred credentials.
Note that override credentials apply to all targets. This applies even for named
credentials.
2.
Optionally select Sudo or PowerBroker as the run privilege.
Sudo enables you to authorize certain users (or groups of users) to run some
(or all) commands as root while logging all commands and arguments.
PowerBroker provides access control, manageability, and auditing of all types
of privileged accounts.
If you provide Sudo or PowerBroker details, they must be applicable to all
targets. It is assumed that Sudo or PowerBroker settings are already applied
on all the hosts on which this job is to run.
See your Super Administrator about setting up these features if they are not
currently enabled.
Tip:
For information on using Sudo, see the Sudo Manual at:
http://www.sudo.ws/sudo/man/1.7.4p6/sudo.man.html
For information on using PowerBroker, see the PowerBroker Desktops
User Guide at:
http://www.ubm-global.com/docs/powerbroker/PBWD_User_Guide_
V5%200.pdf
Utilizing the Job System and Corrective Actions 11-7
Creating Jobs
Task 5 Schedule the Job - (optional)
You do not need to provide input on this page if you want to proceed with the system
default of running the job immediately after you submit it.
1.
Select the type of schedule:
в– One Time (Immediately)
If you do not set a schedule before submitting a job, Enterprise Manager
executes the job immediately with an indefinite grace period. You may want to
run the job immediately, but specify a definite grace period in case the job is
unable to start for various reasons, such as a blackout, for instance.
A grace period is a period of time that defines the maximum permissible delay
when attempting to start a scheduled job. The job system sets the job status to
Skipped, if it cannot start the execution between the scheduled time and the
time equal to the scheduled time plus the grace period, or within the grace
period from the scheduled time.
в– One Time (Later)
–
Setting up a custom schedule:
You can set up a custom schedule to execute the job at a designated time
in the future. When you set the Time Zone for your schedule, the job runs
simultaneously on all targets when this time zone reaches the start time
you specify. If you select each target's time zone, the job runs at the scheduled time using the time zone of the managed targets. The time zone you
select is used consistently when displaying date and time information
about the job, such as on the Job Activity page, Job Run page, and Job Execution page.
For example, if you have targets in the Western United States (US Pacific
Time) and Eastern United States (US Eastern Time), and you specify a
schedule where Time Zone = US Pacific Time and Start Time = 5:00 p.m.,
the job runs simultaneously at 5:00 p.m. against the targets in the Western
United States and at 8:00 p.m. against the targets in the Eastern United
States. If you specify 5:00 p.m. in the Agent time zone, the executions do
not run concurrently. The EST target would run 3 hours earlier.
–
Specifying the Grace Period:
The grace period controls the latest start time for the job in case the job is
delayed. A job might not start for many reasons, but the most common
reasons are that the Agent was down or there was a blackout. By default,
jobs are scheduled with indefinite grace periods.
A job can start any time before the grace period expires. For example, a job
scheduled for 1 p.m. with a grace period of 1 hour can start any time
before 2 p.m., but if it has not started by 2 p.m., it is designated as skipped.
в– Repeating
–
Defining the repeat interval:
Specify the Frequency Type (time unit) and Repeat Every (repeat interval)
parameters to define your job's repeat interval. The Repeat Until options
are as follows:
Note that both the end date and time determine the last execution. For
example, for a job that runs daily at 6 p.m., where...
11-8 OracleВ® Enterprise Manager Administration
Creating Jobs
Start Time is June 1, 2010 at 6 p.m.
End Time is June 30, 2010 at 4 a.m.
... the last execution runs on June 29, not June 30, since the June 30 end
time occurs before the daily time of the job.
–
eart:
See the description above for Grace Period under the One Time schedule
type.
2.
Click the Access property page link.
Task 6 Specify Who Can Access the Job - (optional)
You do not need to provide input on this page if you want to proceed with the system
default of not sharing the job. The table shows the access that administrators and roles
have to the job. Only the job owner (or Super Administrator) can make changes on the
Job Access page.
1.
Change access levels for administrators and roles, or remove administrators and
roles. Your ability to make changes depends on your function.
If you are a job owner, you can:
в– в– Change the access of an administrator or role by choosing the Full or View
access privilege in the Access Level column in the table.
Remove all access to the job for an administrator or role by clicking the icon in
the Remove column for the administrator or role. All administrators with
Super Administrator privileges have the View access privilege to a job. If you
choose to provide access privileges to a role, you can only provide the View
access privilege to the role, not the Full access privilege. For private roles, it is
possible to grant Full access privileges.
If you are a Super Administrator, you can:
в– Grant View access to other Enterprise Manager administrators or roles.
в– Revoke all administrator access privileges.
Neither the owner nor a super user can revoke View access
from a super user. All super users have View access.
Note:
For more information on access levels, see Section 11.3.2.3, "Access Level Rules".
2.
Click Add to add administrators and roles. The Create Job Add Administrators
and Roles page appears.
a.
Specify a Name and Type in the Search section and click Go. If you just click
Go without specifying a Name or Type, all administrators and roles in the
Management Repository appear in the table.
The value you specify in the Name field is not case-sensitive. You can specify
either * or % as a wildcard character at any location in a string (the wildcard
character is implicitly added to the end of any string). For example, if you
specify %na in the Name field, names such as ANA, ANA2, and CHRISTINA
may be returned as search results in the Results section.
b.
Select one or more administrators or roles in the Results section, then click
Select to grant them access to the job. Enterprise Manager returns to the
Utilizing the Job System and Corrective Actions 11-9
Creating Jobs
Create Job Access page or the Edit Job Access page, where you can modify the
access of administrators and roles.
3.
Define a notification rule.
You can use the Notification system (rule creation) to easily associate specific jobs
with a notification rule. The Cloud Control Notification system enables you to
define a notification rule that sends e-mail to the job owner when a job enters one
of these chosen states:
в– Scheduled
в– Running
в– Suspended
в– Succeeded
в– Problems
в– Action Required
Before you can specify notifications, you need to set up your
email account and notification preferences. See Chapter 4, "Using
Notifications" for this information.
Note:
Task 7 Conclude Job Creation
At this point, you can either submit the job for execution or save it to the job library.
■Submitting the job —
Click Submit to send the active job to the job system for execution, and then view
the job's execution status on the main Job Activity page. If you are creating a
library job, Submit saves the job to the library and returns you to the main Job
Library page where you can edit or create other library jobs.
If you submit a job that has problems, such as missing parameters or credentials,
an error appears and you will need to correct these issues before submitting an
active job. For library jobs, incomplete specifications are allowed, so no error
occurs.
If you click Submit without changing the access, only Super
Administrators can view your job.
Note:
■Saving the job to the library —
Click Save to Library to the job to the Job Library as a repository for frequently
used jobs. Other administrators can then share and reuse your library job if you
provide them with access privileges. Analogous to active jobs, you can grant View
or Full access to specific administrators. Additionally, you can use the job library
to store:
–
Basic definitions of jobs, then add targets and other custom settings before
submitting the job.
–
Jobs for your own reuse or to share with others. You can share jobs using
views or giving Full access to the jobs.
–
Critical jobs for resubmitting later, or revised versions of these jobs as issues
arise.
11-10 OracleВ® Enterprise Manager Administration
Creating Jobs
11.3.2.1 Specifying a Single Operation
Note: The following information applies to step 2 in Task 3, "Specify
Parameters" on page 11-6.
Enter the full command in the Command field. For example:
/bin/df -k /private
Note the following points about specifying a single operation:
в– You can use shell commands as part of your command. The default shell for the
platform is used, which is /bin/sh for Linux and cmd/c for Windows.
ls -la /tmp > /tmp/foobar.out
в– If you need to execute two consecutive shell commands, you must invoke the shell
in the Command field and the commands themselves in the OS Script field. You
would specify this as follows in the Command field:
sleep 3; ls
в– The job status depends on the exit code returned by the command. If the
command execution returns 0, the job returns a status of Succeeded. If it returns
any other value, it returns a job status of Failed.
11.3.2.2 Specifying a Script
Note: The following information applies to step 2 in Task 3, "Specify
Parameters" on page 11-6.
The value you specify in the OS Script field is used as stdin for the command
interpreter, which defaults to /bin/sh on Linux and cmd/c on Windows. You can
override this with another interpreter; for example: %perlbin%/perl. The shell scripts
size is limited to 2 GB.
To control the maximum output size, set the mgmt_job_output_size_limit parameter in
MGMT_PARAMETERS to the required limit. Values less than 10 KB and greater than 2
GB are ignored. The default output size is 10 MB.
The job status depends on the exit code returned by the last command in the script. If
the last command execution returns 0, the job returns a status of Succeeded. If it
returns any other value, it returns a job status of Failed. You should implement proper
exception handling in the script and return non-zero exit codes when appropriate. This
will avoid situations in which the script failed, but the job reports the status as
Succeeded.
You can run a script in several ways:
■OS Scripts — Specify the path name to the script in the OS Script field. For
example:
OS Script field: /path/to/mycommand
Interpreter field:
Utilizing the Job System and Corrective Actions
11-11
Creating Jobs
■List of OS Commands — You do not need to enter anything in the Interpreter
field for the following example of standard shell commands for Linux or Unix
systems. The OS’s default shell of /bin/sh or cmd/c will be used.
/usr/local/bin/myProg arg1 arg2
mkdir /home/$USER/mydir
cp /dir/to/cp/from/file.txt /home/$USER/mydir
/usr/local/bin/myProg2 /home/$USER/mydir/file.txt
When submitting shell-based jobs, be aware of the syntax you use and the targets
you choose. This script does not succeed on NT hosts, for example.
■Scripts Requiring an Interpreter — Although the OS shell is invoked by default,
you can bypass the shell by specifying an alternate interpreter. For example, you
can run a Perl script by specifying the Perl script in the OS Script field and the
location of the Perl executable in the Interpreter field:
OS Script field: <Enter-Perl-script-commands-here>
Interpreter field: %perlbin%/perl
The following example shows how to run a list of commands that rely on a certain
shell syntax:
setenv VAR1 value1
setenv VAR2 value2
/user/local/bin/myProg $VAR1 $VAR2
You would need to specify csh as the interpreter. Depending on your system
configuration, you may need to specify the following string in the Interpreter field:
/bin/csh
You have the option of running a script for a list of Windows shell commands, as
shown in the following example. The default shell of cmd/c is used for Windows
systems.
C:\programs\MyApp arg1 arg2
md C:\MyDir
copy C:\dir1x\copy\from\file.txt \home\$USER\mydir
11.3.2.3 Access Level Rules
The following rules apply to Task 6, "Specify Who Can Access
the Job - (optional)" on page 11-9.
Note:
в– в– в– в– в– Super Administrators always have View access on any job.
The Enterprise Manager administrator who owns the job can make any access
changes to the job, except revoking View from Super Administrators.
Super Administrators with a View or Full access level on a job can grant View (but
not Full) to any new user. Super Administrators can also revoke Full and View
from normal users, and Full from Super Administrators.
Normal Enterprise Manager administrators with Full access levels cannot make
any access changes on the job.
If the job owner performs a Create Like operation on a job, all access privileges for
the new job are identical to the original job. If the job owner grants other
administrators View or Full job access to other administrators, and any of these
11-12 OracleВ® Enterprise Manager Administration
Creating Jobs
administrators perform a Create Like operation on the job, ALL administrators
will, by default, have View access on the newly created job.
11.3.3 Creating a SQL Script Job
The basic process for creating a SQL script job is the same as described in
Section 11.3.2, "Creating an OS Command Job." The following sections provide
supplemental information specific to script jobs:
в– Specifying Targets
в– Specifying Options for the Parameters Page
в– Specifying Host and Database Credentials
в– Returning Error Codes from SQL Script Jobs
11.3.3.1 Specifying Targets
You can run a SQL Script job against database and cluster database target types. You
select the targets to run the job against by doing the following:
1.
Click Add in the Targets section.
2.
Select the database target(s) from the pop-up.
Your selection(s) now appears in the Target table.
For a cluster host or RAC database, a job runs only once for
the target, regardless of the number of database instances.
Consequently, a job cannot run on all nodes of a RAC cluster.
Note:
11.3.3.2 Specifying Options for the Parameters Page
In a SQL Script job, you can specify any of the following in the SQL Script field of the
Parameters property page:
в– Any directives supported by SQL*Plus
в– Contents of the SQL script itself
в– Fully-qualified SQL script file; for example:
@/private/oracle/scripts/myscript.sql
Make sure that the script file is installed in the appropriate location on all targets.
в– PL/SQL script using syntax supported by SQL*Plus; for example, one of the
following:
EXEC plsql_block;
or
DECLARE
local_date DATE;
BEGIN
SELECT SYSDATE INTO local_date FROM dual;
END;
/
Utilizing the Job System and Corrective Actions
11-13
Creating Jobs
You can use target properties in the SQL Script field, a list of which appears in the
Target Properties table. Target properties are case-sensitive. You can enter optional
parameters to SQL*Plus in the Parameters field.
11.3.3.3 Specifying Host and Database Credentials
In the Credentials property page, you specify the host credentials and database
credentials. The Management Agent uses the host credentials to launch the SQL*Plus
executable, and uses database credentials to connect to the target database and run the
SQL script. The job can use either the preferred credentials for hosts and databases, or
you can specify other credentials that override the preferred credentials.
■Use Preferred Credentials —
Select this choice if you want to use the preferred credentials for the targets for
your SQL Script job. The credentials used for both host and database are those you
specify in the drop-down. If you choose Normal Database Credentials, your
normal database preferred credentials are used. If you choose SYSDBA Database
Credentials, the SYSDBA preferred credentials are used. For both cases, the host
credentials associated with the database target are used. Each time the job
executes, it picks up the current values of your preferred credentials.
■Named Credentials —
Select this choice if you want to override the preferred credentials for all targets,
then enter the named credentials you want the job to use on all targets.
Many IT organizations require that passwords be changed on regular intervals.
You can change the password of any preferred credentials using this option. Jobs
and corrective actions that use preferred credentials automatically pick up these
new changes, because during execution, Enterprise Manager uses the current
value of the credentials (both user name and password). Named credentials are
also centrally managed. A change to a named credential is propagated to all jobs
or corrective actions that use it.
For corrective actions, if you specify preferred credentials, Enterprise Manager
uses the preferred credentials of the last Enterprise Manager user who edited the
corrective action. For this reason, if a user attempts to edit the corrective action
that a first user initially specified, Enterprise Manager requires this second user to
specify the credentials to be used for that corrective action.
11.3.3.4 Returning Error Codes from SQL Script Jobs
The SQL Script job internally uses SQL*Plus to run a user's SQL or PL/SQL script. If
SQL*Plus returns 0, the job returns a status of Succeeded. If it returns any other value,
it returns a job status of Failed. By default, if a SQL script runs and encounters an
error, it may still result in a job status of Succeeded, because SQL*Plus still returned a
value of 0. To make such jobs return a Failed status, you can use SQL*Plus EXIT to
return a non-zero value.
The following examples show how you can return values from your PL/SQL or SQL
scripts. These, in turn, will be used as the return value of SQL*Plus, thereby providing
a way to return the appropriate job status (Succeeded or Failed). Refer to the SQL*Plus
User's Guide and Reference for more information about returning EXIT codes.
Example 1
WHENEVER SQLERROR EXIT SQL.SQLCODE
select column_does_not_exist from dual;
11-14 OracleВ® Enterprise Manager Administration
Creating Jobs
Example 2
-- SQL*Plus will NOT return an error for the next SELECT statement
SELECT COLUMN_DOES_NOT_EXIST FROM DUAL;
WHENEVER SQLERROR EXIT SQL.SQLCODE;
BEGIN
-- SQL*Plus will return an error at this point
SELECT COLUMN_DOES_NOT_EXIST FROM DUAL;
END;
/
WHENEVER SQLERROR CONTINUE;
Example 3
variable exit_code number;
BEGIN
DECLARE
local_empno number(5);
BEGIN
-- do some work which will raise exception: no_data_found
SELECT 123 INTO local_empno FROM sys.dual WHERE 1=2;
EXCEPTION
WHEN no_data_found THEN
:exit_code := 10;
WHEN others THEN
:exit_code := 2;
END;
END;
/
exit :exit_code;
11.3.4 Creating a Multi-task Job
The basic process for creating a multi-task job is the same as described in
Section 11.3.2, "Creating an OS Command Job." The following sections provide
supplemental information specific to multi-task jobs:
в– Job Capabilities
в– Specifying Targets for a Multi-task Job
в– Adding Tasks to the Job
11.3.4.1 Job Capabilities
Multi-task jobs enable you to create complex jobs consisting of one or more distinct
tasks. Because multi-task jobs can run against targets of the same or different type,
they can perform ad hoc operations on one or more targets of the same or different
type.
The Job System’s multi-task functionality makes it easy to create extremely complex
operations. You can create multi-task jobs in which all tasks run on a single target. You
can also create a multi-task job consisting of several tasks, each of which has a different
job type, and with each task operating on separate (and different) target types. For
example:
в– в– Task 1 (OS Command job type) performs an operation on Host 1.
If Task 1 is successful, run Task2 (SQL Script job type) against Database 1 and
Database 2.
Utilizing the Job System and Corrective Actions
11-15
Viewing and Analyzing Job Status
11.3.4.2 Specifying Targets for a Multi-task Job
You can run a multi-task job against any targets for which jobs are defined that can be
used as tasks. Not all job types can be used as tasks.
The Target drop-down in the General page enables you to choose between running the
job against the same targets for all tasks, or different targets for different tasks. Because
each task of a multi-task job can be considered a complete job, when choosing the
Same targets for all tasks option, you add all targets against which the job is to run
from the General page. If you choose the Different targets for different tasks option,
you specify the targets (and required credentials) the tasks will run against as you
define each task.
After making your choice from the Target drop-down, you then select the targets to
run the job against by clicking Add in the Targets section.
11.3.4.3 Adding Tasks to the Job
You can use the Tasks page to:
в– Add, delete, or edit tasks of various job types
в– Set task condition and dependency logic
в– Add task error handling
You must define at least two tasks in order to set Condition and Depends On options.
Task conditions define states in which the task will be executed. Condition options
include:
■■■Always — Task is executed each time the job is run.
On Success — Task execution Depends On the successful execution of another
task.
On Failure — Task execution Depends On the execution failure of another task.
The Error Handler Task is often a "clean-up" step that can undo the partial state of the
job. The Error Handler Task executes if any task of the multi-task job has an error.
Errors are a more severe form of failure, usually meaning that the job system could not
run the task. Failures normally indicate that the task ran, but failed. The Error Handler
Task does not affect the job execution status. Use the Select Task Type page to specify
the job type of the task to be used for error handling.
11.4 Viewing and Analyzing Job Status
Viewing the Aggregate Status of All Jobs
After you submit jobs, the status of all job executions across all targets is automatically
rolled up and available for review on the Enterprise Summary page. Figure 11–2
shows the Jobs section at the bottom of the Enterprise Summary page.
11-16 OracleВ® Enterprise Manager Administration
Viewing and Analyzing Job Status
Figure 11–2 Summary of Target Jobs on the Enterprise Summary Page
This information is particularly important when you are examining jobs that execute
against hundreds or thousands of systems. You can determine the job executions that
have failed. By clicking the number associated with a particular execution, you can
drill down to study the details of the failed jobs.
Viewing the General Status of a Particular Job
To find out general status information for a particular job or jobs you have submitted,
search for them in the Job Activity page, shown in Figure 11–1.
Viewing the Status of Job Executions
You can view detailed information about a single execution or multiple executions. A
single execution can have a single step or multiple steps.
To view the status of executions:
1.
From the Job Activity page, click the Name link or Status link for the job of
interest.
The Job Run page appears, as shown in Figure 11–3.
Utilizing the Job System and Corrective Actions
11-17
Viewing and Analyzing Job Status
Figure 11–3 Job Run Page
2.
Click on the Status link of a particular execution or step for further information.
Switching to Enhanced View
Beginning with Cloud Control version 12.1.0.4, you can optionally invoke a view of job
runs that combines the views of several drill-downs on one page. To enable the
enhanced view, execute the following command:
emctl set property -name oracle.sysman.core.jobs.ui.useAdfExecutionUi -value true
To revert to the standard view, execute the following command:
emctl set property -name oracle.sysman.core.jobs.ui.useAdfExecutionUi -value false
Note:
These commands do not require you to restart the OMS.
Viewing the Enhanced Status of a Job Run with a Single Execution
A single execution can have a single step or multiple steps. To view the status of a
single execution:
1.
From the Job Activity page, click the Name link or Status link for a job containing
a single execution.
2.
Click on the task or step in the List of Tasks table.
The details for the particular execution appears on the right side of the page, as
shown in Figure 11–4.
11-18 OracleВ® Enterprise Manager Administration
Viewing and Analyzing Job Status
Figure 11–4 Enhanced Execution Summary in Jobs Page
Viewing the Enhanced Status of a Job Run with Multiple Executions
To view the status of multiple executions:
1.
From the Job Activity page, click the Name link or Status link for a job containing
multiple executions.
The Job Run page appears.
2.
Click on an execution of interest in the left table.
The details for the particular execution appears on the right side of the page, as
shown in Figure 11–5.
Figure 11–5 Execution Summary in Job Run Page
Utilizing the Job System and Corrective Actions
11-19
Generating Job Event Criteria
11.5 Generating Job Event Criteria
The job system publishes status change events when a job changes its execution status,
and these events have different severities based on the execution status.
Use the Job Event Generation Criteria page (Figure 11–6) to set up targets for job event
notifications. This page enables you to decide about the jobs or targets or statuses for
which you want to raise events or notifications. This ensures that users raise only
useful events. Any settings you make on this page do not change the job behavior
whatsoever. You can set up notifications on job events through incident rule sets.
To access this page, from the Setup menu, select Incidents, then Job Events.
Figure 11–6 Job Event Generation Criteria Page
11.5.1 Enabling Events For Job Status, Status Severity, and Targetless Jobs
To enable events for job status and targetless jobs, do the following:
1.
Ensure that you have Super Administrator privileges to select the job status for
which you want to generate events.
2.
Ensure that you are an administrator with View Target privileges to add targets for
which you want to generate events for the job status set by the Super
Administrator.
3.
Log into Cloud Control as a Super Administrator.
4.
From the Setup menu, select Incidents and then select Job Events. The Job Event
Generation Criteria Page is displayed.
5.
In the Job Event Generation Criteria page, do the following:
a.
In the "Enable Events for Job Status"region, select the statuses for which you
want to publish events.
11-20 OracleВ® Enterprise Manager Administration
Creating Event Rules For Job Status Change
6.
b.
In the "Enable Events for Status Severity" region, select whether you want to
enable events for a critical status, informational status, for both.
c.
In the "Enable Events for Jobs Without Target(s)" section, select Yes if you
want to create events for jobs that are not associated with any target.
d.
In the "Events for Targets" section, click Add to add targets for which you
want the job events to be enabled.
Click Apply.
11.5.2 Adding Targets To Generate Events For Job Status
After a Super Administrator selects events for which job status will be published,
administrators can add targets to generate events. To add targets to generate events for
job status, do the following:
1.
Ensure that you are an administrator with View Target privileges to add targets for
which you want to generate events for the job status set by a Super Administrator.
2.
Log into Cloud Control as an administrator.
3.
From the Setup menu, select Incidents and then select Job Events. The Job Event
Generation Criteria Page is displayed.
4.
In the Job Event Generation Criteria page, do the following:
a.
In the Events For Job Status And Targetless Jobs section, you can view the
status for which events can be published. You can also see if events have been
enabled for targetless job filters.
b.
In the Events For Targets section, click Add to add targets for which you want
the job events to be enabled. You can also remove targets for which you do not
want the job events to be enabled by clicking Remove.
Your selected settings in the Events for Targets section are
global. Adding or removing targets for events also affect other
Enterprise Manager users.
Note:
5.
Click Apply.
11.6 Creating Event Rules For Job Status Change
Enterprise Manager enables you to create and apply rules to events, incidents, and
problems. A rule is applied when a newly created or updated event, incident, or
problem matches the conditions defined in the rule. The following sections explain
how to create event rules for job status change events:
в– Creating Job Status Change Event Rules For Jobs
в– Creating Job Status Change Event Rules For Targets
11.6.1 Creating Job Status Change Event Rules For Jobs
To create job status change event rules for jobs, do the following:
1.
Ensure that the relevant job status is enabled and required targets have been
added to job event generation criteria.
Utilizing the Job System and Corrective Actions
11-21
Creating Event Rules For Job Status Change
2.
Ensure that you have administrator privileges to create event rules for job status
change events.
3.
Log into Cloud Control as an administrator.
4.
From the Setup menu, select Incidents and then Incident Rules. The Incident
Rules Page appears.
5.
In the Incident Rules page, click Create Rule Set to create rule sets for incidents.
6.
Specify the Name, Description, and select Enabled to enable the rule set. Select
Type as Enterprise if you want to set the rule for all Enterprise Manager users or
Private if you want to set the rule for a specific user only. Select Applies to Job.
In the Job tab, click Add to add jobs for which you want to create event rules.
7.
In the Add Jobs dialog box, if you select the job By Pattern, provide Job name like
and select the Job Type. Specify Job owner like. For the Specific jobs choice,
select the job. Click OK.
8.
In the Rules tab, click Create.
In the Select Type of Rule to Create dialog box that appears, you can select from
the following choices according to the rule set you want to create:
в– в– в– Incoming events and updates to events to receive notification or create
incidents for job rules. If you are operating on events (for example, if you want
to create incidents for incoming events, such as job failed, or notify someone),
choose this option.
Newly created incidents or updates to incidents receive notifications or
create rules for incidents even though the events for which incidents are
generated do not have associated rules. If you are operating on incidents
already created or newly created (for example, you want to direct all incidents
related to a group, say foo, to a particular user or escalate all incidents open
for more than 3 days), choose this option.
Newly created problems or updates to problems to receive notifications or
create rules for problems even though the incidents for which problems are
generated do not have associated rules. This option does not apply for jobs.
11-22 OracleВ® Enterprise Manager Administration
Creating Event Rules For Job Status Change
9.
Select Incoming events and updates to events, and in the Create New Rule: Select
Events page, do the following:
a.
Select By Type to Job Status Change. Select All events of type Job Status
Change if you want to take an action for all job state change events for the
selected jobs. Select Specific events of type Job Status Change if you only
want to act on specific job states. If you have selected Specific events of type
Job Status Change, select Job Status for events for which you want to create
the rule.
b.
Set the other criteria for which you want to set the rule as displayed in the
graphic below.
10. Select Newly created incidents or updates to incidents if you want to create rules
for an incident, though the event associated with the incident does not have
notification rules. In the Create New Rule: Select Incidents page, select any of the
following:
в– All new incidents and updated incidents to apply the rule to all new and
updated incidents
в– All new incidents to apply the rule to all new incidents
в– Specific incidents and then select the criteria for the incidents
Utilizing the Job System and Corrective Actions
11-23
Creating Event Rules For Job Status Change
11. In the Create New Rule: Add Actions page, click Add to add actions to the rule.
12. In the Add Conditional Actions page, specify actions to be performed when the
event matches the rule.
In the Conditions for actions section, select:
в– в– Always execute the actions to execute actions regardless of event.
Only execute the actions if specified conditions match to execute actions to
match specific criteria.
When adding actions to events, specify the following:
в– в– в– в– Select Create Incident to create an incident for the event to manage and track
its resolution.
In the Notifications section, specify recipients for notifications in the E-mail
To, E-mail Cc, and Page fields who will receive e-mail when the event for
which a condition is set occurs. If Advanced and Repeat Notifications options
have been set, specify them.
In the Clear events section, select Clear permanently if you want to clear an
event after the issue that generated the event is resolved.
If you have configured event connections, in the Forward to Event Connectors
section, you can send the events to third-party event management systems.
When adding actions to incidents, specify the following:
в– в– в– In the Notifications section, specify recipients for notifications in the E-mail
To, E-mail Cc, and Page fields who will receive e-mail when the event for
which a condition is set occurs. If Advanced and Repeat Notifications options
have been set, specify them.
In the Update Incident section, specify the details to triage incidents when
they occur. Specify Assign to, Set priority to, Set status to, and Escalate to
details.
In the Create Ticket section, if a ticket device has been configured, specify
details to create the ticket.
Click Continue.
11-24 OracleВ® Enterprise Manager Administration
Creating Event Rules For Job Status Change
13. In the Specify Name and Description page, specify a Name and Description for
the event rule. Click Next.
14. In the Review page, verify the details you have selected for the event rule and
click Continue to add this rule in the rule set.
15. On the Create Rule Set page, click Save to save the rule set.
11.6.2 Creating Job Status Change Event Rules For Targets
To create job status change event rules for targets, do the following:
1.
Ensure that the relevant job status is enabled and required targets have been
added to job event generation criteria.
2.
Ensure that you have administrator privileges to create event rules for job status
change events.
3.
Log into Cloud Control as an administrator.
4.
From the Setup menu, select Incidents, then Incident Rules. The Incident Rules
Page is displayed.
5.
In the Incident Rules page, click Create Rule Set to create rule sets for incidents.
6.
Specify the Name, Description, and select Enabled to enable the rule set. Select
Type as Enterprise if you want to set the rule for all Enterprise Manager users, or
Private if you want to set the rule for a only specific user. Select Applies to
Targets.
In the Targets tab, select one of the following:
в– в– All targets to apply to all targets. In the Excluded Targets section, click Add to
search and select the target that you want to exclude from the rule set. Click
Select.
All targets of types to select the types of targets to which you want to apply
the rule set.
Utilizing the Job System and Corrective Actions
11-25
Creating Event Rules For Job Status Change
в– Specific targets to individually specify the targets. Select to Add Groups or
Targets to add groups or targets and click Add to search and select the targets
to which you want to apply the rule set. Click Select. In the Excluded Targets
section, click Add to search and select the target that you want to exclude from
the rule set. Click Select.
7.
In the Rules tab, click Create.
8.
In the Select Type of Rule to Create dialog box, select from the following choices
according to the rule set you want to create:
в– в– в– 9.
Incoming events and updates to events to receive notifications or create
incidents for job rules. If you are operating on events (for example, if you want
to create incidents for incoming events, such as job failed, or notify someone),
choose this option.
Newly created incidents or updates to incidents receive notifications or
create rules for incidents even though the events for which incidents are
generated do not have associated rules. If you are operating on incidents
already created or newly created (for example, you want to direct all incidents
related to a group, say foo, to a particular user or escalate all incidents open
for more than 3 days), choose this option.
Newly created problems or updates to problems to receive notifications or
create rules for problems even though the incidents for which problems are
generated do not have associated rules. This option does not apply for jobs.
Select Incoming events and updates to events, and in the Create New Rule: Select
Events page, do the following:
в– в– Select By Type to Job Status Change. Select All events of type Job Status
Change if you want to take an action for all job state change events for the
selected jobs. Select Specific events of type Job Status Change if you only
want to act on specific job states. If you have selected Specific events of type
Job Status Change, select Job Status for events for which you want to create
the rule.
Set the other criteria for which you want to set the rule as displayed in the
above graphic.
10. Select Newly created incidents or updates to incidents if you want to create rules
for an incident, though the event associated with the incident does not have
11-26 OracleВ® Enterprise Manager Administration
Creating Event Rules For Job Status Change
notification rules. In the Create New Rule: Select Incidents page, select any of the
following:
в– All new incidents and updated incidents to apply the rule to all new and
updated incidents.
в– All new incidents to apply the rule to all new incidents.
в– Specific incidents and then select the criteria for the incidents.
11. In the Create New Rule: Add Actions page, click Add to add actions to the rule.
12. In the Add Conditional Actions page, specify actions to be performed when the
event matches the rule.
In the Conditions for actions section, select:
в– в– Always execute the actions to execute actions regardless of event.
Only execute the actions if specified conditions match to execute actions to
match specific criteria.
When adding actions to events, specify the following:
в– в– в– в– Select Create Incident to create an incident for the event to manage and track
its resolution.
In the Notifications section, specify recipients for notifications in the E-mail
To, E-mail Cc, and Page fields who will receive e-mail when the event for
which a condition is set occurs. If Advanced and Repeat Notifications options
have been set, specify them.
In the Clear events section, select Clear permanently if you want to clear an
event after the issue that generated the event is resolved.
If you have configured event connections, in the Forward to Event Connectors
section, you can send the events to third-party event management systems.
When adding actions to incidents, specify the following:
в– In the Notifications section, specify recipients for notifications in the E-mail
To, E-mail Cc, and Page fields who will receive e-mail when the event for
which a condition is set occurs. If Advanced and Repeat Notifications options
have been set, specify them.
Utilizing the Job System and Corrective Actions
11-27
Using Diagnostic Tools
в– в– In the Update Incident section, specify the details to triage incidents when
they occur. Specify Assign to, Set priority to, Set status to, and Escalate to
details.
In the Create Ticket section, if a ticket device has been configured, specify the
details to create the ticket.
Click Continue.
13. In the Specify Name and Description page, specify a Name and Description for
the event rule. Click Next.
14. In the Review page, verify the details you have selected for the event rule and
click Continue to add this rule in the rule set.
15. On the Create Rule Set page, click Save to save the rule set.
11.7 Using Diagnostic Tools
The following sections provided procedures for these diagnostic topics:
в– Enabling Job Logging
в– Viewing Job Logging
в– Debugging a Failed Job
в– Checking for Incidents Related to a Failed Job
в– Packaging an Incident Generated by a Job Step
в– Viewing Remote Log Files
в– Diagnosing Problems with Cloud Control Management Tools
11.7.1 Enabling Job Logging
You can enable and disable object logging for purposes.
To enable job logging for a scheduled job:
1.
From the Enterprise menu of the Cloud Control console, select Job, then Activity.
2.
In the Top Activity table, click the link for the job you want to log.
3.
In the Job Run table, click the Scheduled link for the job.
4.
In the Execution page that appears, click Debug.
A confirmation message appears that states "Successfully enabled logging at
DEBUG level."
11.7.2 Viewing Job Logging
If there is user-visible logging for a particular job, you can view the job execution log
by doing the following:
1.
In the Top Activity page, click the link of the job for which you want to view the
log.
2.
In the Job Run table, click Log Report.
You can also access job logging from the Execution page by clicking the link that
appears in the Status column of the Job Run page, then clicking Log Report on the
Execution page as shown below.
11-28 OracleВ® Enterprise Manager Administration
Using Diagnostic Tools
To view the log for a job step, do the following:
1.
In the Top Activity page, click the link of the job for which you want to view the
log.
2.
In the Job Run table, click the link that appears in the Status column for the step
you want to examine.
The Output Log appears for the step.
11.7.3 Debugging a Failed Job
If an execution fails and you had not previously set debug as the logging level, you
can choose to set the debug level when you retry the execution. For new job
executions, you can set logging at the debug level in advance by clicking the Debug
button. The Object Logging field indicates whether logging is enabled at the debug
level.
Perform the following procedure if you encounter a job that fails.
1.
View the job steps that failed.
2.
Check the output for the failed step(s). Aggregated job output is displayed for all
steps, and also for specific steps.
3.
If the output does not contain the reason for the failure, view the logging output.
You may also want to check for any incidents that have occurred while the job was
running.
4.
Determine the cause of the failure and fix the problem.
5.
Enable debug mode, then resubmit the job.
Note that the checkbox for Debug mode only appears on the confirmation page if
the earlier execution was not in Debug mode. If the earlier execution was already
in Debug mode, the retried execution is automatically in Debug mode.
Utilizing the Job System and Corrective Actions
11-29
Using Diagnostic Tools
11.7.4 Checking for Incidents Related to a Failed Job
It is possible for a job to fail because of an internal code error, a severe scaling issue, or
other Enterprise Manager issue for which you may be able to investigate an incident or
event trail. For example, if the OMS bounced because all Job Workers were stuck, this
would cause many jobs to fail. If the loader were failing, that could also cause some
jobs to fail.
1.
Check for incidents or alerts in the time-frame of the job.
в– To check for incidents, select Summary from the Enterprise menu of the Cloud
Control console, then view the Incidents section of the Enterprise Summary
page.
For more detailed information, select Monitoring from the Enterprise menu,
then select Support Workbench.
в– 2.
To check for alerts,
Submit a service request with the related incident(s) or event data.
All step output, error output, logging, remote log files, and incident dump files for
a given job are captured for an incident.
в– в– To submit a service request, select My Oracle Support from the Enterprise
menu of the console, then select Service Requests.
To create a technical SR, click Create SR on the Service Requests Home page.
To create a contact us SR, click Create "Contact Us” SR at the top of the
Contact Us Service Requests region, or click Contact Us at the top of any My
Oracle Support page. If you are creating a technical service request, depending
on the Support IDs registered in your profile, you can create hardware or
software SRs, or both.
The Create Service Request wizard guides you through the process of
specifying product information and attaching configuration information to the
SR when it is filed with Oracle Support. To ensure that Oracle Support has the
most accurate target information, select the Configuration tab in the What is
the Problem? section of Step 1: Problem, then select a target.
3.
Apply a patch that support provides.
в– 4.
Select Provisioning and Patching from the Enterprise menu of the console,
then select Patches & Updates.
в– Provide login credentials, then click Go.
в– Access the online help for assistance with this page.
Try to submit the job again after applying the patch.
To package an incident or manually trigger an incident:
1.
Access Support Workbench.
2.
Gather all job-related dumps and log files, as well as other data from the same
time, and package it for Support.
3.
Review the incident-related data in Support Workbench, searching for relevant
errors.
4.
If you determine the root cause without support intervention, fix the job and
resubmit it.
11-30 OracleВ® Enterprise Manager Administration
Using Diagnostic Tools
11.7.5 Packaging an Incident Generated by a Job Step
Incidents (and the problems that contain them) are not packaged by default. You will
need to package the problems of interest or concern in Support Workbench. You can
choose whether to package all problems or only a portion thereof.
If a job with remote log files is involved in an incident, the
remote files are automatically included in the incident as part of
packaging. For more information on remote log files, see
Section 11.7.6, "Viewing Remote Log Files."
Note:
To package an incident generated by a job step:
1.
On the Log Report page, click the Incident ID link as shown below.
2.
Click the Problem Key link on the Support Workbench Incident Details page as
shown below.
Utilizing the Job System and Corrective Actions
11-31
Using Diagnostic Tools
3.
In the Support Workbench Problem Details page, either click the Package the
Problem link or the Quick Package button as shown below, then follow the
instructions in the Quick Packaging wizard and online help.
11.7.6 Viewing Remote Log Files
Some jobs or Provisioning Adviser Framework (PAF) procedures run external
commands, such as DBCA or the installer. These commands generate their own log
files local to the system and Oracle home where they ran.
To view remote log files:
1.
In the Top Activity page, click the link of the job for which you want to view the
log.
2.
In the Job Run table, click Log Report.
3.
In the Log Report page, click the Remote Log Files link as shown below.
11-32 OracleВ® Enterprise Manager Administration
Using Diagnostic Tools
4.
Specify host credentials, then click OK.
The Remote File Viewer appears and displays the file contents.
11.7.7 Diagnosing Problems with Cloud Control Management Tools
The Cloud Control management portion of the console provides several tools that can
assist you in assessing the current state of the job system and determining a proper
course of action for optimum performance. All of these tools are accessible from the
Setup menu of the Cloud Control console.
The following sections provide information on each of the available tools.
11.7.7.1 Health Overview
1.
From the Setup menu of the Cloud Control console, select Manage Cloud Control.
2.
Select the Health Overview sub-menu.
The Job System Status region of the Health Overview page displays the following
information:
в– Step Scheduler Status
The job step scheduler processes the job steps that are ready to run. If the status
indicates that job step scheduler is running in warning or error mode, the job
system is not functioning normally. In this case, the job system may run in
fail-over mode, where the job dispatcher process may also run the task performed
by the job step scheduler periodically. However, the job system may be running
below its potential capacity, so resolving this situation would be beneficial.
Several possible messages can appear:
–
DBMS_SCHEDULER job for step-scheduler not found
This message is very rare and usually indicates a potentially serious issue. The
job was likely removed inadvertently or due to some special processing
Utilizing the Job System and Corrective Actions
11-33
Using Diagnostic Tools
(patch installation, for example, that requires recycling all DBMS_
SCHEDULER jobs). No automatic resolution is possible here, and this would
need to be addressed on a case by case basis.
–
Failure in checking status
This is a rare occurrence. The error message is usually shown. The error may
disappear on its own as this error indicates that the status could not be
calculated.
–
DBMS_SCHEDULER is disabled
All of the DBMS_SCHEDULER jobs are disabled in the environment. This
should not occur unless a type of installation is in progress. Resolve this by
starting DBMS_SCHEDULER processes.
–
All job queue processes are in use
The DBMS_SCHEDULER processes have been expended. Increase the
parameter job_queue_processes in the repository RDBMS.
–
All slave processes are in use
The cause is similar to the above case. In this situation, you need to increase
MAX_JOB_SLAVE_PROCESSES of the DBMS_SCHEDULER.
–
All sessions are in use
No RDBMS sessions were available for the DBMS_SCHEDULER. Increase the
PROCESSES for the RDBMS.
–
Reason for delay could not be established
This usually appears because none of the above criteria were met, and is the
most common warning. The dispatcher may just be overloaded because there
is more work than available workers. Check the backlog in this case. The
situation should resolve automatically, but if it persists, the number of workers
available for the job system may be insufficient for the load the site
experiences.
в– Job Backlog
The job backlog indicates the number of job steps that have passed their scheduled
time but have not executed yet. If this number is high and has not decreased for a
long period, the job system is not functioning normally. This situation usually
arises if job engine resources are unable to meet the inflow of jobs from system or
user activity.
A high backlog can also happen because of the abnormal processing of specific
jobs because they are stuck for extended periods. For more information on stuck
job worker threads, do the following:
1.
From the OMS and Repository menu of the Health Overview page, select
Monitoring, then Diagnostic Metrics.
If the jobs system has a backlog for long periods of time, or if you would like to
process the backlog faster, set the following parameters with the emctl set property
command. These settings assume that sufficient database resources are available to
support more load. These parameters are likely to be needed in a Large
configuration with 2 OMS nodes.
11-34 OracleВ® Enterprise Manager Administration
Using Diagnostic Tools
Table 11–1
Large Job System Backlog Settings
Parameter
Value
oracle.sysman.core.jobs.shortPoolSize
50
oracle.sysman.core.jobs.longPoolSize
24
oracle.sysman.core.jobs.longSystemPoolSize
20
oracle.sysman.core.jobs.systemPoolSize
50
oracle.sysman.core.conn.maxConnForJobWorkers
144 *
* This setting may require an increase of the processes setting in the database of 144 * number
of OMS servers.
11.7.7.2 Repository Home Page
1.
From the Setup menu of the Cloud Control console, select Manage Cloud Control.
2.
Select the Repository sub-menu.
The Repository Scheduler Jobs Status table in the Management Services and
Repository page displays the job system purge status and next run schedule.
11.7.7.3 Management Services and Repository: All Metrics
There are two navigation paths for accessing the All Metrics page:
в– From the Setup menu
в– From the Targets menu
Setup Menu Navigation
From the Setup menu of the Cloud Control console, select Manage Cloud Control.
1.
2.
Select the Health Overview sub-menu.
3.
From the Management Services and Repository page that appears, select
Monitoring from the OMS and Repository menu, then select All Metrics.
4.
Scroll down to DBMS Job Status in the left pane, then select a metric as shown
below.
Utilizing the Job System and Corrective Actions
11-35
Using Diagnostic Tools
5.
Scroll down further and expand Repository Job Scheduler Performance as shown
below.
Definitions for the available metrics are as follows:
■■■Average number of steps marked as ready by the scheduler — Average
number of steps processed by the job step scheduler to mark the steps "ready"
for execution. This number usually depends on the job system load over a
time period.
Estimated time for clearing current Job steps backlog (Mins) — Estimated
time to clear the backlog assuming the current inflow rate of the job system.
Job step backlog — Number of job steps that have passed their scheduled
time but have not executed yet. If this number is high and has not decreased
for a long period, the job system is not functioning normally. This situation
usually arises if job engine resources are unable to meet the inflow of jobs from
system or user activity.
11-36 OracleВ® Enterprise Manager Administration
Using Diagnostic Tools
в– в– в– 6.
Latency in marking steps as ready by the scheduler — The job step scheduler
moves scheduled steps to ready queue. This metric indicates the average
latency in marking the steps to ready queue. High latency means abnormal
functioning of the job step scheduler process.
Overall job steps per second — Average number of steps the job system
executes per second.
Scheduler cycles — Frequency of dbms scheduler process. Executes a
minimum of 5 cycles per min, and may increase depending on the job system
load. A low number usually indicates a problem in the job step scheduler
process.
Scroll down further and expand the Usage Summary entries for Jobs, then select
the metric for which you are interested as shown below.
Target Menu Navigation
From the Targets menu of the Cloud Control console, select All Targets.
1.
2.
In the left pane of the All Targets page, scroll down and expand Internal, then
select OMS and Repository.
3.
Click on the OMS and Repository table entry in the page that follows.
4.
From the OMS and Repository menu in the Health Overview page that appears,
select Monitoring, then All Metrics.
11.7.7.4 OMS and Repository: Diagnostic Metrics
1.
From the Setup menu of the Cloud Control console, select Manage Cloud Control.
2.
Select the Health Overview sub-menu.
Utilizing the Job System and Corrective Actions
11-37
Using Diagnostic Tools
3.
From the Management Services and Repository page that appears, select
Monitoring from the OMS and Repository menu, then select Diagnostic Metrics.
pbs_* metrics are relevant for diagnosing issues in the job system. This information is
useful if you are searching for more information on stuck job system threads, or job
threads usage statistics to determine outliers preventing other jobs from running.
11.7.7.5 OMS and Repository: Charts
1.
From the Setup menu of the Cloud Control console, select Manage Cloud Control.
2.
Select the Health Overview sub-menu.
3.
From the Management Services and Repository page that appears, select
Monitoring from the OMS and Repository menu, then select Charts.
Assuming that the job had steps, this page shows historical charts for the overall
upload backlog, job step backlog, and overall job steps per second.
11.7.7.6 Management Servers and Job Activity Details Pages
1.
From the Targets menu of the Cloud Control console, select All Targets.
2.
On the left pane under Groups, Systems, and Services, click Management Servers.
3.
Click Management Servers in the Target Name table.
The Job System region displays a snapshot of job system status and details of
processed executions. The Recent Job Executions Summary table displays the total
user job executions that are expected to run within a specific time period, the
completed count, running count, and the count of executions that are neither
completed nor running. This helps you to determine if various user jobs are
running as expected in the system.
11-38 OracleВ® Enterprise Manager Administration
Using Diagnostic Tools
4.
Click the More Details link below the summary table.
5.
Select the desired time frame in the drop-down for when executions are expected
to start.
Jobs and their status, if any, appear in the table.
6.
Select the Job Dispatchers tab.
If more than one management server is configured, the page displays the job
dispatcher and thread pool utilization information for each management servers.
■■■Dispatcher Utilization (%) — Measures how frequently the job dispatcher
picks up the job steps. High utilization indicates a heavy job system load.
Throughput (steps dispatched/min) — Indicates the average number of steps
other than internal steps processed by dispatcher every minute.
Thread Pool Utilization — Displays the total number of threads configured
for each pool, the average steps selected by the thread pool per minute, and
the average number of available threads.
11.7.7.7 Job System Reports
The job system provides both a diagnostic report and usage report.
Diagnostic Report
1.
From the Setup Enterprise of the Cloud Control console, select Reports.
2.
Select the Information Publisher Reports sub-menu.
3.
Search for job in the Title field, then click Go.
4.
Click the Job System Diagnostic Report link in the table.
This report provides an overview of the job system's health and displays diagnostic
information about executing jobs or jobs that are possibly delayed beyond their
scheduled time. This information is usually relevant for an Oracle Support engineer
diagnosing problems in the job system.
Usage Report
Follow the steps above to access this report, except click the Job Usage Report link in
step 4.
Utilizing the Job System and Corrective Actions
11-39
Creating Corrective Actions
This report provides an overview of the job system usage information over the past 7
days.
11.8 Creating Corrective Actions
Corrective Actions enable you to specify automated responses to metric alerts.
Corrective Actions ensure that routine responses to metric alerts are automatically
executed, thereby saving you time and ensuring problems are dealt with before they
noticeably impact end users.
Corrective actions share many features in common with the Job System. By default, a
corrective action runs on the target on which the metric alert is triggered.
Alternatively, you can specify a corrective action to contain multiple tasks, with each
task running on a different target. You can also receive notifications for the success or
failure of corrective actions.
You define corrective actions for individual metrics for monitored targets. The
following sections provide instructions on setting up corrective actions and viewing
the details of a corrective action execution:
в– Providing Credentials
в– Creating Corrective Actions for Metrics
в– Creating a Library Corrective Action
в– Specifying Access to Corrective Actions
в– Setting Up Notifications for Corrective Actions
в– Providing Agent-side Response Actions
в– Viewing the Details of a Corrective Action Execution
11.8.1 Providing Credentials
Since corrective actions are associated with a target's metric thresholds, you can define
corrective actions if you have been granted OPERATOR or greater privilege on the
target. You can define separate corrective actions for both Warning and Critical
thresholds. Corrective actions must run using the credentials of a specific user. For this
reason, whenever a corrective action is created or modified, you must specify the
credentials that the modified action runs with.
11.8.2 Creating Corrective Actions for Metrics
For any target, the Metric and Collection Settings page shows whether corrective
actions have been set for various metrics. For each metric, the Corrective Actions
column shows whether Critical and/or Warning severities of corrective actions have
been set.
1.
From any target’s home page menu, select Monitoring, then Metric and
Collection Settings. The Metric and Collection Settings page appears.
Tip: For instance, on the home page for a host named
dadvmn0630.us.oracle.com, you would select the Host menu, then
Monitoring, then Metric and Collection Settings.
2.
Click the pencil icon for a specific metric to access the Edit Advanced Settings
page for the metric.
11-40 OracleВ® Enterprise Manager Administration
Creating Corrective Actions
3.
In the Corrective Actions section, click Add for the metric severity (Warning
and/or Critical) for which you want to associate a corrective action.
4.
Select the task type on the Add Corrective Actions page, then click Continue.
в– в– в– If you want to use a corrective action from the library, select From Library as
the task type. Using a library corrective action copies the description,
parameters, and credentials from the library corrective action. You must still
define a name for the new corrective action. You can provide corrective action
parameters if necessary.
If you want to create a corrective action to store in the library, see
Section 11.8.3, "Creating a Library Corrective Action."
If you want to provide an Agent-side response action, select Agent Response
Action as the task type. See Section 11.8.6, "Providing Agent-side Response
Actions" for more information.
5.
On the Corrective Action page, provide input for General, Parameters, and
Credentials as you would similarly do when creating a job.
6.
Click Continue to save the corrective action and return to the Edit Advanced
Settings page, where your corrective action now appears.
7.
Optional: To prevent multiple instances of a corrective action from operating
simultaneously, enable the Allow only one corrective action for this metric to run
at any given time checkbox.
This option specifies that both Critical and Warning corrective actions will not run
if a severity is reported to the Oracle Management Services when an execution of
either corrective action is currently running. This can occur if a corrective action
runs longer than the collection interval of the metric it corrects; the value of the
metric may be oscillating back and forth across one of the thresholds (leading to
multiple executions of the same corrective action), or may be rising or falling
quickly past both thresholds (in which case an execution of the Warning corrective
action may overlap an execution of the Critical corrective action).
If you do not select this option, multiple corrective action executions are launched
under the aforementioned circumstances. It is the administrator’s responsibility to
ensure that the simultaneous corrective action executions do not conflict.
8.
Click Continue when you have finished adding corrective actions to return to the
Metric and Collection Settings page.
The page shows the corrective action value you have provided for the metric in
the Corrective Actions column. Possible values are:
■■■■None — No corrective actions have been set for this metric.
Warning — A corrective action has been set for Warning, but not Critical,
alerts for this metric.
Critical — A corrective action has been set for Critical, but not Warning, alerts
for this metric.
Warning and Critical — Corrective actions have been set for both Warning
and Critical alerts for this metric. If an Agent-side response action is associated
with the metric, the value is also Warning and Critical, since Agent-side
response actions are always triggered on either Critical or Warning alert
severities.
Utilizing the Job System and Corrective Actions
11-41
Creating Corrective Actions
9.
Continue the process from step 2 forward, then click OK on the Metric and
Collection Settings page to save your corrective actions and return to the target
page you started from in step 1.
11.8.3 Creating a Library Corrective Action
For corrective actions that you use repeatedly, you can define a library corrective
action. After a corrective action is in the library, you can reuse the corrective action
definition whenever you define a corrective action for a target metric or policy rule.
1.
From the Enterprise menu, select Monitoring, then Corrective Actions. The
Corrective Action Library page appears.
2.
Select a job type from the Create Library Corrective Action drop-down, then click
Go.
3.
Define the corrective action as you would for creating a job in Section 11.3,
"Creating Jobs" for General, Parameters, and Credentials. For Access, go to the
following optional step.
4.
Optional: Select Access to define or modify the access you want other users to have
for this corrective action.
For more information, see Section 11.8.4, "Specifying Access to Corrective Actions."
5.
Click Save to Library when you have finished. The Corrective Action Library page
reappears, and your corrective action appears in the list.
You can now create another corrective action based on this one (Create Like
button), edit, or delete this corrective action.
You can access this library entry whenever you define a corrective action for a metric
severity by selecting From Library as the task type in the Add Corrective Actions page.
See step 4 in Section 11.8.2, "Creating Corrective Actions for Metrics," for more
information.
11.8.4 Specifying Access to Corrective Actions
As mentioned in the procedure above, you can determine the access to corrective
actions by other users. You do not need to provide input for this page if you do not
want to share the corrective action.
11.8.4.1 Defining or Modifying Access
The table on the Access page shows the access that administrators and roles have to
the corrective action. Only the corrective action owner (or Super Administrator) can
make changes on this page.
As the corrective action owner, you can do the following:
в– в– в– Add other administrators and roles to the table by clicking Add, then selecting the
appropriate type in the subsequent page that appears.
Change the access of an administrator or role by choosing the Full or View access
right in the Access Level column in the table.
Remove all access to the corrective action for an administrator or role by clicking
the icon in the Remove columns for this administrator or role. All administrators
with Super Administrator privileges have the View access right to a corrective
action.
11-42 OracleВ® Enterprise Manager Administration
Creating Corrective Actions
If you choose to provide access rights to a role, you can only provide the View access
right to the role, not the Full access right.
If you are a Super Administrator, you can:
в– Grant View access to other Enterprise Manager administrators or roles.
в– Revoke all administrator access privileges.
If a new user is being created, the user should have the
CREATE_JOB privilege to create Corrective Actions.
Note:
11.8.4.2 Access Level Rules
Access level rules are as follows:
в– в– в– в– в– Super Administrators always have View access for any corrective action.
The Enterprise Manager administrator who owns the corrective action can make
any access changes to the corrective action (except revoking View from Super
Administrators).
Super Administrators with a View or Full access level for a corrective action can
grant View (but not Full) access to any new user. Super Administrators can also
revoke Full and View access from normal users, and Full access from Super
Administrators.
Normal Enterprise Manager administrators with Full access levels cannot make
any access changes on the corrective action.
If the corrective action owner performs a Create Like operation on a corrective
action, all access privileges for the new corrective action become identical to the
original corrective action. If the corrective action owner grants other
administrators View or Full access to other administrators, and any of these
administrators perform a Create Like operation on this corrective action, all
administrators will, by default, have View access on the newly created corrective
action.
11.8.5 Setting Up Notifications for Corrective Actions
Corrective actions are associated with metrics whose alerts trigger them. Any
Enterprise Manager administrator with View or higher privileges on a target can
receive notifications following the success or failure of a corrective action.
A single incident rule can contain any combination of alert and corrective action states.
All metrics and targets selected by the incident rule are notified for the same alert and
corrective action states. Therefore, if you want to be notified of corrective action
success or failure for one metric, but only on failure for another, you need to use two
incident rules. An incident rule can include corrective action states for metrics with
which no corrective actions have been associated. In this case, no notifications are sent.
Notifications cannot be sent for Agent-side response actions,
regardless of the state of any incident rules applied to the target.
Note:
To create incident rules for notifications:
1.
From the Setup menu, select Incidents, then Incident Rules.
2.
Click Create Rule Set. The Create Rule Set wizard appears.
Utilizing the Job System and Corrective Actions
11-43
Creating Corrective Actions
3.
Provide the requisite information at the top of the Create Rule Set page, then select
one of the target choices in the Targets sub-tab, supplying additional information
as needed for the "All targets of types" and "Specific targets" choices.
4.
Select the Rules sub-tab, then click Create.
5.
In the pop-up that appears, select the default Incoming events and update to
events choice, the click Continue.
6.
On the Select Events page, enable the Type checkbox, then select Metric Alert.
7.
Click the Specific events of type Metric alert radio button, then click Add in the
table that appears.
8.
In the pop-up that appears, select the Target Type, filter and select the metric,
select a severity, then enable the desired corrective action status. Click OK.
9.
From the Add Actions page, click Add.
10. Specify recipients in the Basic Notifications section of the Add Conditional Actions
page.
11. Proceed through the final two pages of the wizard, then click Continue. Your new
rule appears in the Create Rule Set page.
12. Click Save to save this rule.
After you have created one or more rule sets, you need to set up notification methods
as follows:
1.
From the Setup menu, select Notifications, then Notification Methods.
2.
From the Notification Methods page, select Help, then Enterprise Manager Help
for assistance on providing input for this page.
11.8.6 Providing Agent-side Response Actions
Agent-side response actions perform simple commands in response to an alert. When
the metric triggers a warning or critical alert, the Management Agent automatically
runs the specified command or script without requiring coordination with the Oracle
Management Service (OMS). The Agent runs this command or script as the OS user
who owns the Agent executable. Specific target properties can be used in the Agent
response action script.
Use the Agent-side Response Action page to specify a single
command-line action to be executed when a Warning or Critical
severity is reached for a metric. For tasks that require alert context,
contain more complex logic, or require that notifications be sent on
success or failure, corrective actions should be used instead of an
Agent-side response action.
Note:
To access this page, follow steps 1 through 4 in Section 11.8.2, "Creating Corrective
Actions for Metrics."
11.8.6.1 Specifying Commands and Scripts
You can specify a single command or execute a script. You cannot specify special shell
command characters (such as > and <) as part of the response action command. If you
must include these types of special characters in your response action commands, you
should use them in a script, then specify the script as the response action command.
11-44 OracleВ® Enterprise Manager Administration
Creating Corrective Actions
If using a script, make sure the script is installed on the host machine that has the
Agent. If using shell scripts, make sure the shell is specified either in the Response
Action command line:
Script/Command: /bin/csh myScript
... or within the body of the script itself:
Script/Command: myScript
... where myScript contains the following:
!#/bin/csh<
<rest of script>
11.8.6.2 Using Target Properties in Commands
You can use target properties in a command. Click Show Available Target Properties
to display target properties you can use in the Script/Command field. The list of
available target properties changes according to the type of target the response action
is to run against.
Use Target Properties as command-line arguments to the script or command, then
have the script reference these command-line arguments. For example, to use the
%OracleHome% and %SID% target properties, your command might appear as
follows:
/bin/csh MyScript %OracleHome% %SID%
.... and your script, MyScript, can reference these properties as command-line
arguments. For example:
IF $1 = 'u1/bin/OracleHome' THEN...
Target properties are case-sensitive. For example, if you want to access the
Management Agent's Perl interpreter, you can specify %perlBin%/perl <my_perl_
script> in the Script/Command field.
11.8.6.3 Using Advanced Capabilities
You can get other target properties from the target's XML file in the
OracleHome/sysman/admin/metadata directory, where OracleHome is the Oracle
home of the Management Agent that is monitoring the target. In the XML file, look for
the PROP_LIST attribute of the DynamicProperties element to get a list of properties
that are not listed in the targets.xml entry for the target.
The following example is an excerpt from the hosts.xml file:
<InstanceProperties>
<DynamicProperties NAME="Config" FORMAT="ROW"
PROP_LIST="OS;Version;OS_patchlevel;Platform;Boottime;IP_address">
<ExecutionDescriptor>
<GetTable NAME="_OSConfig"/>
<GetView NAME="Config" FROM_TABLE="_OSConfig">
<ComputeColumn NAME="osName" EXPR="Linux" IS_VALUE="TRUE"/>
<Column NAME="osVersion"/>
<Column NAME="osPatchLevel"/>
<Column NAME="Platform"/>
<Column NAME="Boottime"/>
<Column NAME="IPAddress"/>
</GetView>
</ExecutionDescriptor>
</DynamicProperties>
Utilizing the Job System and Corrective Actions
11-45
Creating Corrective Actions
<InstanceProperty NAME="Username" OPTIONAL="TRUE" CREDENTIAL="TRUE">
<ValidIf>
<CategoryProp NAME="OS" CHOICES="Linux"/>
</ValidIf>
<Display>
<Label NLSID="host_username_iprop">Username</Label>
</Display>
</InstanceProperty>
<InstanceProperty NAME="Password" OPTIONAL="TRUE" CREDENTIAL="TRUE">
<ValidIf>
<CategoryProp NAME="OS" CHOICES="Linux"/>
</ValidIf>
<Display>
<Label NLSID="host_password_iprop">Password</Label>
</Display>
</InstanceProperty>
</InstanceProperties>
11.8.7 Viewing the Details of a Corrective Action Execution
There are two methods of displaying the outcome of a corrective action execution.
в– Incident Manager method
1.
From the Enterprise Manager Cloud Control console Enterprise menu, select
Monitoring, then Incident Manager.
2.
Click the Search icon, select Events from the Type drop-down, then click Get
Results.
3.
Double-click the message of interest in the search results table.
The Corrective Action History table now appears at the bottom of the page, as
shown below.
11-46 OracleВ® Enterprise Manager Administration
Creating Corrective Actions
4.
Select the desired message in the history table, then click the glasses icon as
shown below.
The Corrective Action Execution page now appears, which displays the
output of the corrective action, status, start time, end time, and so forth.
в– All Metrics method
1.
From the target’s home page, select Monitoring, then All Metrics.
2.
From the tree panel on the left, click the desired metric name.
A row for the metric alert now appears in the Metric Alert History table.
3.
Click the glasses icon in the Details column as shown below.
The Incident Manager Event Details page now appears.
4.
In the Corrective Action History table at the bottom of the page, select the
message in the history table, then click the glasses icon.
The Corrective Action Execution page now appears, which displays the
output of the corrective action, status, start time, end time, and so forth.
Utilizing the Job System and Corrective Actions
11-47
Creating Corrective Actions
11-48 OracleВ® Enterprise Manager Administration
Part II
Part II
Administering Cloud Control
This section contains the following chapters:
в– Maintaining Enterprise Manager
в– Maintaining and Troubleshooting the Management Repository
в– Updating Cloud Control
в– Configuring a Software Library
в– Managing Plug-Ins
в– Patching Oracle Management Service and the Repository
в– Patching Oracle Management Agents
в– Personalizing Cloud Control
в– Starting and Stopping Enterprise Manager Components
в– Enterprise Manager Command Line Utility Commands
в– Locating and Configuring Enterprise Manager Log Files
в– Configuring and Using Services
в– Introducing Enterprise Manager Support for SNMP
12
Maintaining Enterprise Manager
12
Enterprise Manager provides extensive monitoring and management capabilities for
various Oracle and non-Oracle products. Used to manage your heterogeneous IT
infrastructure, Enterprise Manager plays an integral role in monitoring and
maintaining the health of your IT resources. It is therefore essential to ensure
Enterprise Manager itself is operating at peak efficiency.
To help you maintain your Enterprise Manager installation, a variety of enhanced
self-monitoring and diagnostic functionality is available from the Enterprise Manager
console. These functions are designed to help you understand and monitor various
components of Enterprise Manger, monitor/measure the quality of services Enterprise
Manager provides, diagnose failures quickly, and manage Agents more easily.
This chapter covers the following topics:
в– Overview: Managing the Manager
в– Health Overview
в– Repository
в– Controlling and Configuring Management Agents
в– Management Servers
12.1 Overview: Managing the Manager
Although Enterprise Manager functions as a single entity to manage your IT
infrastructure, in reality it is composed of multiple components working in concert to
provide a complete management framework from a functional standpoint. All major
components of Enterprise Manager have been grouped into a single system. A special
set of services has been created (based on the system) to model Enterprise Manager
functions.
Management Features
в– Topology view that allows you to see all major components of Enterprise Manager
and their current status.
в– в– Dashboard displaying the overall health of Enterprise Manager.
Full control of the Agent directly from the Enterprise Manager console. Functions
include:
–
View/edit Agent configuration properties.
–
View Agent(s) configuration history and compare the results against other
Agents.
Maintaining Enterprise Manager 12-1
Health Overview
–
Perform Agent control operations (start/stop/secure).
–
Upgrade Management Agents
12.2 Health Overview
The Health Overview provides a comprehensive overview of OMS and Repository
operation and performance, and therefore allows you to view the overall health of
your Enteprise Manager environment.
Accessing the Health Overview
From the Setup menu, select Manage Cloud Control and then Health Overview.
All major areas of Enterprise Manager are represented.
в– в– в– в– в– в– Overview: Provides key information for active Management Services such as the
Management Agents, the WebLogic Administration Server, total number of
monitored targets, number of administrators, and server load balancer (SLB)
upload and console URLs, provided SLB is configured. If configured, the SLB
upload and console URLs are also displayed.
Repository Details: Provides physical information about the Management
Repository and the host on which the database is located. You can drill down into
the database home page for more information and carry out administrative
operations.
Job System Status: Displays key operational parameters of the Enterprise
Manager Job service. For detailed information, you can click on the status icon to
drill down into the Enterprise Manager Job Service home page.
Console Activity: Displays the overall load on the Enterprise Manager console
through the average number of requests per minute and the average time required
to process those requests.
Alerts: Provides details on the metric errors recorded and when an alert was
triggered. In-context links to Incident Manager are also provided.
Performance Charts: Upload Backlog and Upload Rate, Backoff Requests,
Notification backlog. You can drill down into any chart to view detailed metric
information.
12-2 OracleВ® Enterprise Manager Administration
Health Overview
Figure 12–1 Health Overview Page
From this page, you can carry out all monitoring and management operations using
the OMS and Repository menu.
The Diagnostic Metrics page is intended for use by Oracle
Support when diagnosing issues with the OMS. The page can be
accessed by selecting Monitoring and then Diagnostic Metrics from
the OMS and Repository menu.
Note:
Maintaining Enterprise Manager 12-3
Health Overview
Figure 12–2 OMS and Repository Menu
12.2.1 Viewing Enterprise Manager Topology and Charts
The Enterprise Manager Topology page provides a graphical representation of the
Enterprise Manager infrastructure components and their association. Each node in the
hierarchy displays key information about the member type, the host on which it
resides, and the number of incidents, if any. The incident icons on each of the nodes
expand to display a global view of current status for each node in the hierarchy.
In order for the Enterprise Manager repository database to
appear in the Topology page, you must first manually discover the
database. Manual discovery is also required in order to have the
database’s metric data (Database Time (centiseconds per second))
displayed in the charts.
Note:
Accessing the Enterprise Manager Topology
1. From the Setup menu, select Manage Cloud Control and then Health Overview.
2.
Click on the OMS and Repository menu to display available operations that can
be performed from this page.
3.
Select Members and then Topology.
12-4 OracleВ® Enterprise Manager Administration
Health Overview
Figure 12–3 Enterprise Manager Topology
Enterprise Manager Charts
The Enterprise Manager Charts page displays eight charts representing key areas that
together indicate the overall health of Enterprise Manager. These are Overall Files
Pending Load -Agent, Job Step Backlog, Job Step Throughput (per second), Request
Processing Time (ms), Database Time (centiseconds per second), CPU Utilization (%),
Pages Paged-in (per second), Pages Paged-out (per second). Data can be viewed for
the Last 24 hours, last 7 days or last 31 days.
Accessing the Enterprise Manager Charts
1. From the Setup menu, select Manage Cloud Control and then Health Overview.
2.
Click on the OMS and Repository menu to display available operations that can
be performed from this page.
3.
Select Monitoring and then Charts.
Maintaining Enterprise Manager 12-5
Health Overview
Figure 12–4 Enterprise Manager Charts
12.2.2 Determining Enterprise Manager Page Performance
Page Performance Monitoring and diagnosis feature provides you with the ability to
identify and diagnose performance issues with Enterprise Manager pages without
having to contact Oracle support.
The Enterprise Manager page performance tracing
feature requires Agent version 12.1.0.4 or later.
Important:
The charts and tables in this page will display data only if the Agent
that is monitoring Management Services and Repository target is
version 12.1.0.4 or later. If you have upgraded Enterprise Manager
12.1.0.4, you should also upgrade the Agents on Management Service
(OMS) machines to 12.1.0.4 in order to access the latest monitoring
capabilities for the Management Services and Repository, as well as
related targets.
See the OracleВ® Enterprise Manager Cloud Control Upgrade Guide
for more information on upgrading Agents.
To access Page Performance Monitoring and Diagnosis functionality:
1.
From the Setup menu, select Manage Cloud Control, and then Health Overview
or Repository.
2.
From the OMS and Repository menu, select Monitoring and then Page
Performance. The Page Performance page displays.
12-6 OracleВ® Enterprise Manager Administration
Health Overview
Overview
The overview tab provides details of the overall page performance in Enterprise
Manager.
The charts display the Page Accesses and Sessions, Current Page Accesses and
Sessions Distribution across OMSs and the Overall Statistics of page performance in
the last 24 hours. There are details of the page performance in each of the OMSs as
well as the details of the available repositories.
The Overall Statistics table provides the breakdown of times spent in the Repository,
the OMS and network and the number of page accesses, the maximum time taken by
page in the last 24 hours.
Page Level Performance
The page level performance tab shows the list of pages accessed in the last 24 hours.
Maintaining Enterprise Manager 12-7
Health Overview
The page also displays the breakdown of time spent in the Repository and the OMS
and network in a line graph format for each page.
Performance Correlation
The performance correlation tab displays graphs for page performance that allow you
to correlate performance trends.
This tab provides details of page accesses and sessions, page processing time,
SQL/PLSQL executions, and average active sessions.
Symptom Diagnosis
Symptom diagnosis can be performed for both overall page processing time and
individual page times. Symptom diagnosis is triggered when the set metric thresholds
for overall page processing time are exceeded. Diagnosis is accessed by means of an
icon in the Overview tab in the Overall Statistics section when the overall page
performance threshold is exceeded, as shown in the following graphic.
12-8 OracleВ® Enterprise Manager Administration
Health Overview
Figure 12–5 Symptom Diagnosis Icon
For individual pages, the symptom diagnosis icon is displayed in the table in the
Current Severity column if the page performance metric threshold is exceeded.
When the icon is displayed in the Overall Statistics section, it indicates that the overall
performance of the Enterprise Manager pages has exceeded the threshold in the last 10
minutes. Clicking on the icon, you are taken to another tab where the details of the
diagnosis are presented. The diagnosis indicates the root cause for the overall page
performance exceeding the metric threshold, the findings that were deduced on
diagnosis and the checks that were performed to analyze the overall page performance
issue.
The checks are performed at the database level, middle-tier level and the
browser/network level to isolate which part of the system might be the cause of the
issue. Each check is analyzed and the checks that are identified as the top causes are
reported as findings. The topmost finding is then reported as the root cause for the
performance issue.
Target Availability Symptom Diagnosis:
Symptom diagnosis can be performed on the availability of the Agent as well. The icon
is displayed in the Agent List and Agent Home pages in the event that the Agent
target is unreachable or in pending status.
Maintaining Enterprise Manager 12-9
Health Overview
Figure 12–6 Agent List Page
Figure 12–7 Agent Home Page
On clicking the icon, the user is navigated to another tab where the details of the
diagnosis are presented. The diagnosis indicates the root cause for the Agent's
unreachable/pending state, the findings that were deduced from the diagnosis and the
checks that were performed to analyze the Agent availability issue.
The checks performed to diagnose the issue consist of the following:
в– if the communication between the Management Service and the Agent is
successful
в– if the Agent has communicated with the Management Service
в– if reasons can be deduced from the Repository
в– if further reasons can be deduced by performing checks from the Agent side
(whether communication between the OMS and the Agent exists).
12-10 OracleВ® Enterprise Manager Administration
Health Overview
Each check is analyzed and the checks that are identified as the top causes are reported
as findings. The topmost finding is then reported as the root cause for the performance
issue.
Figure 12–8 Agent Symptom Diagnosis
If communication between the OMS and Agent cannot be
established, then the diagnosis will report findings based on the data
available in Management Repository, which may not be the real cause
for the issue.
Note:
The symptom diagnosis feature is also available in the All Targets page for targets in
unreachable or pending status by clicking on the status icon.
Figure 12–9 All Targets Page
Maintaining Enterprise Manager
12-11
Repository
12.3 Repository
The Repository page provides you with an overview of the status and performance of
the Repository DBMS Jobs that handle part of Enterprise Manager's maintenance and
monitoring functionality. These DBMS jobs run within the Management Repository
and require no user input. Charts showing the key Repository Details and Backlog in
Repository Collection are provided The Scheduler Status region provides the status of
the scheduler and the number of Job Queue Processes.
Accessing Repository Information
From the Setup menu, select Manage Cloud Control and then Repository.
Three tabs are displayed providing a comprehensive view of repository attributes,
performance, as well as access to requisite operational parameters.
в– Repository Tab
в– Metrics Tab
в– Schema Tab
Figure 12–10
Repository Page
12.3.1 Repository Tab
As shown in Figure 12–10, the Repository tab provides a comprehensive snapshot of
repository-specific monitoring.
Repository Details
The Repository Details region provides high-level database information for the
Enterprise Manager repository. From this region, you can click on the number
Management Service Repository Sessions details to view the exact number of
repository connections per individual Enterprise Manager subcomponent such as the
event system, console, job system, or connector framework.
12-12 OracleВ® Enterprise Manager Administration
Repository
Figure 12–11
Repository Sessions Per Subcomponent
Incidents and Problems
The Incidents and Problems region displays all incidents an problems associated with
the repository database. For more detailed incident or problem information, you can
click on the Summary link to access the issue in Incident Manager.
Figure 12–12
Accessing Incident/Problem Information
Initialization Parameter Compliance for Instance
This region displays the currents initialization parameter settings, recommended
standards, and whether the current parameter values comply with those standard
values.
Figure 12–13
Initialization Parameter Compliance
If you are running the repository in a RAC environment, this region also lets you select
individual database instances in order to view initialization parameter compliance for
that specific instance.
Maintaining Enterprise Manager
12-13
Repository
Repository Scheduler Job Status
The Repository Scheduler Jobs Status region provides details of the DBMS Jobs
regarding their status, duration, and the next scheduled run time.
Figure 12–14
Repository Scheduler Job Status Region
If the Status of a job is down, you can run the job again by clicking Restart Job.
For high-cost jobs requiring greater resources that, when run, can reduce repository
performance, an edit icon (pencil) appears in the Edit column. Clicking on the icon
displays a dialog allowing you to reschedule the next run time.
Figure 12–15
Job Reschedule Dialog
Repository Collection Performance
The Repository Collection Performance region provides information on the
performance of repository collections. They are collected by background DBMS jobs in
the repository database called collection workers.
Figure 12–16
Repository Collection Performance Region
12-14 OracleВ® Enterprise Manager Administration
Repository
Repository metrics are sub-divided into long and short running metrics. These are
called task classes (short task class and long task class). Some collection workers
process the short task class and some process long task class. Repository collection
performance metrics measure the performance data for repository metric collections
for each task class. This metric is a repository metric and hence collected by the
collection workers.
You can select between Short Running and Long Running collection workers. When
viewing Short Running workers, you can click Configure to change short worker
settings.
Figure 12–17
Short Worker Configuration Dialog
Clicking Save submits a job to change the worker configuration. For this reason, the
change will not be instantaneous and may require a minute or so in order to take
effect.
Clicking on an item in the legend allows you to drill down into Problem Analysis,
Metric Details, or the Target Home.
Figure 12–18
Collection Performance Information
Metric Data Rollup Performance
This region displays the rollup performance by graphically displaying the quantity of
data being rolled up (Number of Records Rolled Up) and speed (Throughput per
Minute) over time.
Maintaining Enterprise Manager
12-15
Repository
Figure 12–19
Metric Data Rollup Performance Region
The graphs for Number of Records Rolled Up and Throughput per Minute may increase
over time as more targets are added, but on a daily basis should remain about the
fairly level. Large spikes could indicate that agents are not communicating properly to
the OMS
Clicking Configure allows you to change the number of rollup worker threads that
will be started.
12.3.2 Metrics Tab
The Metrics tab provides a graphical rollup of key repository performance
measurements. Information includes:
в– Top 25 Metric Data Loading Target Types In Last 30 Days
в– Top 10 Data Loading Metrics In Last 30 Days
в– Metric Alerts Per Day In Last 30 Days
в– Top 10 Metric Collection Errors By Target Type In Last 30 Days
Figure 12–20
Metrics Tab
The graphs allow you to drill down to access information in greater detail.
12-16 OracleВ® Enterprise Manager Administration
Repository
Top 25 Metric Data Loading Target Types In Last 30 Days
Figure 12–21
Top 25 Metric Data Loading Target Types In Last 30 Days
If you wish to view only metrics for a specific target type, click on a specific metric
target type area within the graph. A new graph displays showing only metrics for that
specific target type. You have the option grouping the results by metric or target.
Figure 12–22
Top 25 Metric Data Loading for a Single Target Type
Click Clear to return to the original graph containing all target types.
Maintaining Enterprise Manager
12-17
Repository
Top 10 Data Loading Metrics In Last 30 Days
Figure 12–23
Metric Data Load Volume
Top 10 Metric Collection Errors By Target Type In Last 30 Days
Figure 12–24
Metric Alerts Per Day In Last 30 Days
The Metric Alerts Per Day In Last 30 Days graphically displays the number of open,
closed, and backlogged metric alerts over time. If you wish to focus on a narrower
time span, click Zoom.
12-18 OracleВ® Enterprise Manager Administration
Repository
Figure 12–25
Metric Alerts Per Day
12.3.3 Schema Tab
The Schema tab provides physical attribute and performance data pertaining to the
repository database schema. Information includes:
в– Tablespace Growth Rate
You can select the specific tablespace: MGMT_TABLESPACE, MGMT_ECM_
DEPOT_TS, or MGMT_AD4J_TS.
Top 20 Large Tables/Indexes are also displayed.
в– Top 20 Tables with Unused Space in Repository
в– Purge Policies
в– Partition Retention
Figure 12–26
Schema Tab
Maintaining Enterprise Manager
12-19
Controlling and Configuring Management Agents
12.4 Controlling and Configuring Management Agents
Beginning with Enterprise Manager Cloud Control 12c, controlling Management
Agents can be performed directly from the Enterprise Manager console. This provides
a central point where all Management Agents within your monitored environment can
be compared, configured and controlled.
12.4.1 Manage Cloud Control Agents Page
The Agents page lists all Management Agents within your monitored environment.
This page also includes misconfigured, blocked and both upgradable and nonupgradable Agents.
Accessing the Agent Page
From the Setup menu, select Manage Cloud Control and then Agents.
Figure 12–27
Agent List Page
Misconfigured and Blocked Agents
A misconfigured Agent is an Agent that is not able to perform a heartbeat or upload
data to the Oracle Management Service (OMS) due to invalid configuration or invalid
data. Agent misconfiguration alerts are triggered by the following metrics:
в– Consecutive metadata upload failure count
в– Consecutive ping failure count
в– Consecutive severity upload failure count
в– OMS Agent time skew
If the Agent heartbeat or upload requests are failing consistently, and the problem
cannot be resolved in a timely manner, you can manually block the Agent to prevent
excessive load on the OMS. When you block an Agent, the OMS rejects all heartbeat or
upload requests from the blocked Agent. However, even though blocked Agents
continue to collect monitoring data, it will not be able to upload any alerts or metric
data to the OMS. Once the Agent configuration problem is resolved, you must
manually unblock the Agent to resume normal operation.
12-20 OracleВ® Enterprise Manager Administration
Controlling and Configuring Management Agents
Before unblocking the Agent, ensure that all issues related to
Agent misconfiguration have been resolved.
Note:
From this page, you can also initiate the Agent upgrade process. For more information
about upgrading Agents see "Upgrading Multiple Management Agents" on
page 12-25.
12.4.2 Agent Home Page
The Agent home page provides details for a single Agent. This page also lets you drill
down for more detailed information. You can access an Agent home page by clicking
on a specific Agent from in the Agent list page or by selecting it from the All Targets
page.
Figure 12–28
в– в– в– в– в– Agent Home Page
The Summary region provides primary details of the Agent such as its status and
availability. The Interaction with Management Service region provides details on
the communication between the OMS and the Agent and metric extensions and
management plug-ins deployed in the Agent.
The Status region provides further details on the Agent status such as the number
of restarts, the action that the Agent is performing currently.
The Performance, Usage and Resource Consumption charts provide further
details on the Agent in graphical format.
The Incidents region lists the incidents recorded for the Agent.
The Monitoring region provides details on the targets that are being monitored by
the Agent. You can filter targets in this region by All, Broken, and Not Uploading.
Maintaining Enterprise Manager
12-21
Controlling and Configuring Management Agents
Separate tabs within the Monitoring section display Metric Issues and Top
Collections.
12.4.3 Controlling a Single Agent
Control operations for a single Agent can be performed on the Agent home page for
that Agent.
1.
Navigate to the desired Agent home page.
2.
From the Agent drop-down menu, choose Control and then one of the control
operations (Start Up/Shut Down, or Restart)
You must have at least operator privileges in order to perform
Agent control operations.
Note:
Figure 12–29
Control Operations from the Agent Home Page
Upon choosing any of the above control menu options, a pop-up dialog requesting the
credentials of the user displays. These operations require the credentials of the OS user
who owns the Agent, or credentials of a user who has SUDO or PowerBroker privilege
of the Agent owner. At this point, you can either choose from a previously stored
username/password, preferred or named credential. You also have the option of
choosing a new set of credentials which could also be saved as the preferred credential
or as a named credential for future use.
Once you are authenticated, the chosen control operation begins and continues even if
the pop-up dialog is closed. Any message of failure/success of the task is displayed in
the pop-up dialog.
When choosing the Secure/Resecure/Unsecure options, you must provide the
requisite Registration Password.
12-22 OracleВ® Enterprise Manager Administration
Controlling and Configuring Management Agents
Agent Control When Using a Server Load Balancer
When choosing the Agent Secure/Resecure options in a multi-OMS environment with
a server load balancer (SLB), the Agent will be secured/resecured against the SLB
automatically without administrator intervention.
12.4.4 Configuring Single Management Agents
Configuration operations for a single Agent can be performed from the Agent home
page. To access the Agent properties page:
1.
Navigate to the desired Agent home page.
2.
From the Agent drop-down menu, select Properties.
You must have at least Configure privileges in order to
perform Agent configuration operations.
Note:
Figure 12–30
Agent Properties Page
The properties on this page can be filtered to show All Properties, Basic Properties, or
Advanced Properties. The Basic Properties are a simple name, value combination of a
property and its value. Advanced Properties are also a combination of name and
value but can also be grouped into categories. You must have at least configure
privileges in order to modify the existing properties and set custom properties.
12.4.5 Controlling Multiple Management Agents
In order to perform control operations on multiple Management Agents, Enterprise
Manager makes use of the Job system to automate repetitive tasks. Therefore, you
must have Job privileges for controlling multiple Management Agents through a
single action. To access
1.
From the Setup menu, select Manage Cloud Control and then Agents. The Agent
page displays.
2.
Select multiple Management Agents from the list.
3.
Click one of the control operation buttons (Start Up/Shut
Down/Restart/Secure/Resecure/Unsecure).
When you click on any of the control operations, you are taken to the Job creation
wizard where you schedule a new job to perform the action on the selected Agents.
Maintaining Enterprise Manager
12-23
Controlling and Configuring Management Agents
Figure 12–31
Multiple Agent Control Operation: Job Creation
In the Jobs page, you can view the chosen Management Agents in Target section in the
General tab. You can add more Management Agents by clicking the Add button. You
then provide the parameters for the operation in the Parameters tab, if needed. The
credentials must be specified in the Credentials tab where you can either choose from
a previously stored username/password, preferred, or named credential. You also
have the option of choosing a new set of credentials which could also be saved as the
preferred credential or as a named credential for future use.
You are given the option to start the job immediately or schedule the job for a later
time. At this point, you can also create a repeating job by specifying the job start time,
the frequency, and the end time.
The Access tab displays the Administrator details and the access levels they have to
the job. You can then add a new administrator or modify the access level to View or
Full, if you have the requisite privileges.
Figure 12–32
Job Creation: Access Tab
Administrators with insufficient privileges can also schedule
jobs for these control operations, but in this situation, the jobs will not
complete successfully.
Note:
12.4.6 Configuring Multiple Agents
As with multi-Agent control operations, you can also perform Agent configuration on
multiple Agents in the same way. This greatly simplifies standardizing Agent
configurations across your enterprise. To access Agent properties:
12-24 OracleВ® Enterprise Manager Administration
Controlling and Configuring Management Agents
1.
From the Setup menu, select Manage Cloud Control and then Agents. The Agent
page displays.
2.
Select multiple Management Agents from the list.
3.
Click Properties. As with any multi-Agent operation, configuration is
implemented using the Job system.
Figure 12–33
Agent Properties Page
In the Jobs page, you can view the chosen Management Agents in the Target section of
the General tab. You can add more Management Agents by clicking the Add button if
necessary. In the Parameters tab, you provide the modified value for a particular set of
properties that you want to change. You can also set a custom property for the chosen
agents. No credentials are required for modifying Agent properties.
The Access tab displays the administrator details and the access levels they have to the
job. You can then add a new administrator or modify the access level to View or Full if
you have the requisite privileges.
Figure 12–34
Multi-Agent Configuration: Job Access
12.4.7 Upgrading Multiple Management Agents
When you upgrade to the current Enterprise Manager Cloud Control 12c release, you
upgrade your Oracle Management Services (OMS) to the current release, but not your
target Oracle Management Agents (Management Agents). To mass-upgrade your
Management Agents, access the Upgrade Agents page. To access this page:
Maintaining Enterprise Manager
12-25
Management Servers
1.
From the Setup menu, select Manage Cloud Control, then select Agents.
2.
Click Upgradable, then select the Management Agents you want to upgrade.
3.
Click Upgrade.
Alternatively, to access the Upgrade Agents page, from the Setup menu, select
Manage Cloud Control, then select Upgrade Agent. For more information on
upgrading Management Agents, refer Upgrading Oracle Management Agents.
12.5 Management Servers
A Management Server is a composite target consisting of multiple Enterprise Manager
Management Services.
The Management Servers page displays the list of Management Services, their status,
incidents, the loader throughput, CPU usage, and the JVM memory usage metrics. In
addition, the Management Services displayed can be filtered by Normal Mode,
Console Only, PBS only and Standby Management Services.
Accessing the Management Servers Page
From the Setup menu, select Manage Cloud Control and then Management Services.
Figure 12–35
Management Servers Page
This page consists of the following sections:
в– в– Summary: Displays the high-level information about WebLogic administration
server and Load balancer.
Job System: Displays information about the status of jobs over past time periods
(such as the last 30 minutes, 1 hour, or 2 hours).
12-26 OracleВ® Enterprise Manager Administration
Management Servers
в– в– Servers: Displays information about individual Management Services of the
Management Server.
Loader: Displays information that provides insight into the Loader subsystem
performance as a whole.
There are primarily 3 graphs as follows.
–
Throughtput (Rows processed per second): Indicates the rate (rows processed
per second) at which the Loader is processing files.
–
Files Processed vs Backoff: Indicates the number of files processed versus
backed off (rejected) by the Loader. Note: You should contact Oracle Support if
consistent backoffs are being generated.
–
% Utilized Capacity: Sows the current Loader CPU utilization. If the Loader
consistently runs at more than 85% capacity, contact Oracle Support to confirm
whether your system capacity needs to be increased.
To view detailed IP reports of Loader statistics, click the Loader Statistics link
located below the graphs.
в– Incidents: This displays the incidents and problems that have occurred against
individual targets hosting Management Services.
Maintaining Enterprise Manager
12-27
Management Servers
12-28 OracleВ® Enterprise Manager Administration
13
Maintaining and Troubleshooting the
Management Repository
13
This chapter describes maintenance and troubleshooting techniques for maintaining a
well-performing Management Repository.
Specifically, this chapter contains the following sections:
в– Management Repository Deployment Guidelines
в– Management Repository Data Retention Policies
в– Dropping and Recreating the Management Repository
в– Troubleshooting Management Repository Creation Errors
в– Cross Platform Enterprise Manager Repository Migration
13.1 Management Repository Deployment Guidelines
To be sure that your management data is secure, reliable, and always available,
consider the following settings and configuration guidelines when you are deploying
the Management Repository:
в– в– в– Install a RAID-capable Logical Volume Manager (LVM) or hardware RAID on the
system where the Management Repository resides. At a minimum the operating
system must support disk mirroring and stripping. Configure all the Management
Repository data files with some redundant configuration.
Use Real Application Clusters to provide the highest levels of availability for the
Management Repository.
If you use Enterprise Manager to alert administrators of errors or availability
issues in a production environment, be sure that the Cloud Control components
are configured with the same level of availability. At a minimum, consider using
Oracle Data Guard to mirror the Management Repository database. Configure
Data Guard for zero data loss. Choose between Maximum Availability or
Maximum Protection based on your environment and needs.
See Also: Oracle Database High Availability Architecture and Best
Practices
Oracle Data Guard Concepts and Administration
в– Oracle strongly recommends that archive logging be turned on and that a
comprehensive backup strategy be in place prior to an Enterprise Manager
implementation going live in a production environment. The backup strategy
Maintaining and Troubleshooting the Management Repository
13-1
Management Repository Data Retention Policies
should include archive backups and both incremental and full backups as
required.
Oracle Enterprise Manager Cloud Control Installation and
Basic Configuration for information about the database initialization
parameters required for the Management Repository
See Also:
в– в– в– в– Oracle recommends that you not use SQL Plan Management (SQL plan baselines
and capture) with the Enterprise Manager Cloud Control repository. If you do
need to use it for a specific problem, shut it off immediately after using. Issues
with the Enterprise Manager Cloud Control repository may occur when using SQL
Plan Management, such as very poor SQL performance using unverified plans,
and deadlocks between SQL Plan Management capture and the Enterprise
Manager security VPD.
After enabling auditing for the repository database and for audit entries related to
ORA- errors, error messages should be ignored if they are not reported in the
Enterprise Manager application logs; for example, emoms.trc, the
MGMT_SYSTEM_ERROR_LOG table, or in the alert.log of the repository
database. In these cases the errors are harmless.
To see a list of the regular maintenance activities that need to be performed for the
repository, see the Sizing Your Enterprise Manager Deployment in the OracleВ®
Enterprise Manager Cloud Control Advanced Installation and Configuration Guide.
To monitor the repository database activities using the Enterprise Manager user
interface, see Chapter 12, "Maintaining Enterprise Manager".
13.2 Management Repository Data Retention Policies
When the various components of Enterprise Manager are configured and running
efficiently, the Oracle Management Service gathers large amounts of raw data from the
Management Agents running on your managed hosts and loads that data into the
Management Repository. This data is the raw information that is later aggregated,
organized, and presented to you in the Cloud Control console.
After the Oracle Management Service loads information into the Management
Repository, Enterprise Manager aggregates and purges the data over time.
The following sections describe:
в– в– The default aggregation and purging policies used to maintain data in the
Management Repository.
How you can modify the length of time the data is retained before it is aggregated
and then purged from the Management Repository.
13.2.1 Management Repository Default Aggregation and Purging Policies
Enterprise Manager aggregates collected metric data by hour and by day to enhance
query performance and help minimize the size of the Management Repository. Before
the data is aggregated, each data point is stored in a raw metric data table. Once a day,
the previous day’s raw metric data is rolled up, or aggregated, into a one-hour and a
one-day table. These hourly and daily records will have hourly and daily metric data
averages, minimums, maximums and standard deviations respectively.
After Enterprise Manager aggregates the data, the data is then considered eligible for
purging. A certain period of time must pass for data to actually be purged. This period
of time is called the retention time.
13-2 OracleВ® Enterprise Manager Administration
Management Repository Data Retention Policies
The raw data, with the highest insert volume, has the shortest default retention time,
which is set to 7 days. As a result, 7 days after it is aggregated into a one-hour record, a
raw data point is eligible for purging.
Note:
This data retention policy varies for JVMD and ADP data.
Hourly aggregate metric data records are purged after 31 days. The highest level of
aggregation, one day, is kept for 12 months (roughly 365 days).
The default data retention policies are summarized in Table 13–1.
Table 13–1
Default Repository Purging Policies
Aggregate Level
Retention Time
Raw metric data
7 days
Hourly aggregated metric data
31 days
Daily aggregated metric data
12 months (~365 days)
If you have configured and enabled Application Performance Management, Enterprise
Manager also gathers, saves, aggregates, and purges response time data. The response
time data is purged using policies similar to those used for metric data. The
Application Performance Management purging policies are shown in Table 13–2.
Table 13–2 Default Repository Purging Policies for Application Performance
Management Data
Aggregate Level
Retention Time
Raw response time data
24 hours
One-hour aggregated
response time data
7 days
One-hour distribution
response time data
24 hours
One-day aggregated
response time data
31 days
One-day distribution
aggregated response time
data
31 days
If you do not want to keep severity data for the default period (6 months), and want to
reduce the retention period for the EVENTS purge policy, you can use the following
command:
em_purge.modify_purge_policy_group('EVENTS',NULL,*l_new_purge_
hours*);
Events data is partitioned and maintains six months of historical data by default. You
can change the default retention period using the procedure described above. The
severity data is tied to the events data purge policy and will be adjusted accordingly.
The fixed set of tables affected by this data purge are listed below:
EM_EVENT_SEQUENCES
EM_EVENT_RAW
EM_EVENT_MSGS
EM_EVENT_CONTEXT
Maintaining and Troubleshooting the Management Repository
13-3
Management Repository Data Retention Policies
EM_EVENT_ANNOTATIONS
EM_EVENTS_INCIDENT
EM_ISSUES_INTERNAL
EM_ISSUES_MSG
EM_ISSUES_ANNOTATIONS
EM_INCIDENT_ISSUE
EM_PROBLEM_ISSUE
EM_INCIDENTS_PROBLEM
The following list is a dynamic set of tables that store data for different event types
supported by Enterprise Manager. This list can vary over time as new event types or
unsupported event types are added or removed:
EM_EV_CS_RULE_VIOLATION
EM_EV_CS_SCORE
EM_EV_JOB_STATUS_CHANGE
EM_EV_METRIC_ALERT
EM_EV_METRIC_ERROR
EM_EV_MEXT_UPDATE
EM_EV_MNTR_DISRUPTION
EM_EV_SELFUPDATE
EM_EV_SLA_ALERT
EM_EV_TARGET_AVAILABILITY
EM_EV_USER_REPORTED
EM_EV_ADP_ALERT
EM_EV_APM_KPI_ALERT
EM_EV_JVMDIAG_ALERT
EM_EV_HA_EVENT
13.2.2 Management Repository Default Aggregation and Purging Policies for Other
Management Data
Besides the metric data and Application Performance Monitoring data, other types of
Enterprise Manager data accumulates over time in the Management Repository.
For example, the last availability record for a target will also remain in the
Management Repository indefinitely, so the last known state of a target is preserved.
13.2.3 Modifying the Default Aggregation and Purging Policies
The Enterprise Manager default aggregation and purging policies were designed to
provide the most available data for analysis while still providing the best performance
and least disk-space requirements for the Management Repository. As a result, you
should not modify these policies to improve performance or increase your available
disk space.
However, if you plan to extract or review the raw or aggregated data using data
analysis tools other than Enterprise Manager, you may want to increase the amount of
raw or aggregated data available in the Management Repository. You can accomplish
this by increasing the retention times for the raw or aggregated data.
A PL/SQL API has been provided to modify the default retention time for the core
metric data tables in the Enterprise Manager repository. Table 13–3 shows the default
number of partitions retained for each of the three tables and the size of the partitions
for each table. The API will allow you to change the number of partitions retained
only.
13-4 OracleВ® Enterprise Manager Administration
Management Repository Data Retention Policies
Table 13–3
Repository
Core EM Metric Data Tables and Default Data Retention in the Management
Table Name
Partitions Retained
Partition Size
EM_METRIC _VALUES
7
DAY
EM_METRIC_VALUES_HOURLY
32
DAY
EM_METRIC_VALUES_DAILY
24
MONTH
To modify the retention period for any of the above tables, execute the following
command:
SQL> execute gc_interval_partition_mgr.set_retention('SYSMAN',
<table name>, <number of partitions to retain>);
Replace the <table name> by name of table as listed above. The API will allow you to
change the number of partitions retained only.
For example, to modify the default retention time for the table EM_METRIC_VALUES
from 7 partitions to 14 partitions, follow these steps:
1.
Use SQL*Plus to connect to the repository database as the SYSMAN user.
2.
Check the current value of the retention periods:
SQL> select table_name, partitions_retained
from em_int_partitioned_tables
where table_name in ('EM_METRIC_VALUES','EM_METRIC_VALUES_HOURLY','EM_METRIC_
VALUES_DAILY');
TABLE_NAME
------------------------EM_METRIC_VALUES
EM_METRIC_VALUES_HOURLY
EM_METRIC_VALUES_DAILY
3.
PARTITIONS_RETAINED
------------------7
32
24
To modify the default retention time for the table EM_METRIC_VALUES from 7
partitions to 14, execute the following command:
SQL> execute gc_interval_partition_mgr.set_
retention('SYSMAN', 'EM_METRIC_VALUES', 14);
4.
Verify that the retention period has been modified:
SQL> select table_name, partitions_retained
from em_int_partitioned_tables
where table_name in ('EM_METRIC_VALUES','EM_METRIC_VALUES_HOURLY','EM_METRIC_
VALUES_DAILY');
TABLE_NAME
PARTITIONS_RETAINED
------------------------- ------------------EM_METRIC_VALUES
14
EM_METRIC_VALUES_HOURLY
32
EM_METRIC_VALUES_DAILY
24
13.2.4 How to Modify the Retention Period of Job History
Enterprise Manager Cloud Control has a default purge policy which removes all
finished job details which are older than 30 days. This section provides details for
modifying this default purge policy.
Maintaining and Troubleshooting the Management Repository
13-5
Management Repository Data Retention Policies
The actual purging of completed job history is implemented via a DBMS_
SCHEDULER job that runs once a day in the repository database. When the job runs, it
looks for finished jobs that are 'n' number of days older than the current time (value of
sysdate in the repository database) and deletes these jobs. The value of 'n' is, by
default, set to 30 days.
The default purge policy cannot be modified via the Enterprise Manager console, but it
can be changed using SQL*Plus.
To modify this purge policy, follow these steps:
1.
Log in to the repository database as the SYSMAN user, via SQL*Plus.
2.
Check the current values for the purge policies using the following command:
SQL> select * from mgmt_job_purge_policies;
POLICY_NAME
TIME_FRAME
-------------------------------- ---------SYSPURGE_POLICY
30
REFRESHFROMMETALINKPURGEPOLICY
7
FIXINVENTORYPURGEPOLICY
7
OPATCHPATCHUPDATE_PAPURGEPOLICY
7
The purge policy responsible for the job deletion is called SYSPURGE_POLICY. As
seen above, the default value is set to 30 days.
3.
To change the time period, you must drop and recreate the policy with a different
time frame:
SQL> execute MGMT_JOBS.drop_purge_policy('SYSPURGE_POLICY');
PL/SQL procedure successfully completed.
SQL> execute MGMT_JOBS.register_purge_policy('SYSPURGE_
POLICY', 60, null);
PL/SQL procedure successfully completed.
SQL> COMMIT;
Commit complete.
SQL> select * from mgmt_job_purge_policies;
POLICY_NAME
TIME_FRAME
-------------------------------- ---------SYSPURGE_POLICY
60
....
The above commands increase the retention period to 60 days. The time frame can also
be reduced below 30 days, depending on the requirement.
You can check when the purge job will be executed next. The actual time that the
purge runs is set to 5 AM repository time and can be verified using these steps:
1.
Login to the Repository database using the SYSMAN account.
2.
Execute the following command:
SQL> select job_name,
to_char(last_start_date, 'DD-MON-YY HH24:MI:SS') last_run,
to_char(next_run_date,
'DD-MON-YY HH24:MI:SS'