Getting Optimal Performance from Oracle E

Getting Optimal Performance from Oracle E
Getting Optimal Performance
from Oracle E-Business Suite
Samer Barakat
Director, Applications Performance Group
Applications Technology Group, Oracle E-Business Suite Development
ORACLE
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Program Agenda
1
Performance Triage & Resolution
2
Optimizing E-Business Suite Applications Tier
3
Optimizing E-Business Suite Database Tier
4
Optimizing E-Business Suite on RAC
5
Optimizing E-Business Data Management
6
Upgrade Performance Best Practices
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
3
Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features or
functionality described for Oracle’s products remains at the sole discretion of Oracle.
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
4
Performance Triage & Resolution
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
5
Performance Triage & Resolution
How to Approach a Performance Issue
• DEFINE the problem clearly
• GATHER the right data to analyze the issue
• Identify the ROOT CAUSE of the problem, possibly gather additional data
• Search for a KNOWN SOLUTION or workaround that addresses the root cause of the problem
• If it is a product issue, PASS ON the right information to support/dev through the regular
channels
• Try to identify a TEMPORARY WORKAROUND to alleviate the issue while you get a permanent
fix
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
6
Performance Triage & Resolution
Define the Problem
• Get a clear understanding & quantitative definition of the issue
• Where is the time going ? Get the right diagnostics
– What?
– Where?
– When?
– Why?
– How?
• Identify the Techstack
components in-play
• Refer to the architecture
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
7
Performance Triage & Resolution
Gather Diagnostic Data
~ 80% of issues are DB processing related (flow chart slide)
Start with MOS ID 1121043.1 for How-to & Best Practices
• SQL Tuning
•
Trace files
•
SQLT output (MOS ID: 215187.1)
•
Trace Analyzer (MOS ID: 224270.1)
•
AWR Report (MOS ID: 748642.1)
•
11g SQL Monitor Report
•
AWR SQL Report (awrsqrpt.sql)
• PL/SQL Tuning
•
Product logs
•
PL/SQL Profiler (MOS ID: 808005.1)
• Reports Tracing (MOS ID: 111311.1)
• Database Tuning
• AWR Report (MOS ID: 748642.1)
• ADDM report (MOS ID: 250655.1)
• Active Session History (ASH)
• Forms Tuning
• Forms Tracing (MOS ID: 373548.1)
• FRD Log (MOS ID: 445166.1)
• Generic note (MOS ID: 438652.1)
• Middle tier Tuning
• JVM Logs
• JVM Sizing/Tuning (MOS ID: 362851.1,278868.1)
• OS - OSWatcher (MOS ID: 301137.1)
• Network Test Utilities Best Practices (MOS ID: 556738.1)
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
8
Performance Triage & Resolution
Gather Diagnostic Data
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
9
Performance Triage & Resolution
Gather Enhanced Performance Diagnostics for Oracle E-Business Suite (MOS ID: 1362660.1)
•
Enables gathering all the relevant diagnostic data in one round trip.
•
Available from 12.1.3 as a standalone patch# 12544073 (Mandatory Pre-Requisite AD Patch#
12991557)
•
Enhancement to FND_TRACE: New procedure “set_preferences” enables you to
– Enable tracing for sessions
– Enable tracing for one or more SQL statements using SQL_ID (*)
– Generate CBO trace when tracing SQL statements (*)
– Trace PL/SQL using PL/SQL Hierarchical Profiler (*)
– Set Statistics Level
(*) - Only in Oracle Database 11g+
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
10
Performance Triage & Resolution
Gather Enhanced Performance Diagnostics for Oracle E-Business Suite (MOS ID: 1362660.1)
•
One or more of the following outputs can generated automatically after the end of traced
session. The concurrent program 'Gather Diagnostic Data for Traced Sessions' is automatically
submitted via 'System Administrator responsibility to gather this output
–
AWR, ADDM and ASH Reports
–
Session Statistics
–
SQLT output (*)
–
Trace Analyzer output, which includes SQLT output for top SQLs
–
Objects statistics when a SQL statement is traced, but SQLT output is not chosen or not installed
–
Real-Time SQL Monitor Report when a SQL statement is traced (**)
–
SQL Detail Report when a SQL Statement is traced (***)
(* )Requires SQLT tool - Note 215187.1,
(** )Oracle DB 11g+
(*** )Oracle DB 11.2.0.1+
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
11
Performance Triage & Resolution
Gather Diagnostic Data
• Establish baselines for different workloads by collecting system level performance
information
– AWR (Automatic Workload Repository) Report
– ADDM (Automatic Database Diagnostic Monitor) Report
– ASH (Active Session History) Report
– Real-Time SQL Monitoring Report (11g+) (Parallel queries and queries taking > 5
seconds)
• Monitor Operating system statistics
– OSW Document MOS ID301137.1
– OSWg Document MOS ID 461053.1
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
12
Performance Triage & Resolution
Gather Diagnostic Data
• Monitor and trace critical flows & concurrent programs
– ASH (Active Session History) Report
– SQL Trace Level 8 (with waits) or Level 12 (with waits and binds) followed by TKPROF
• Check both raw trace and TKPROF
• Interpreting Raw SQL_TRACE files MOS ID 39817.1
– Trace Analyzer (TRCA) reads an Event 10046 SQL Trace file and provides a comprehensive
report for performance analysis and tuning MOS ID 224270.1
• Monitor and trace specific SQLs
– SQLTXPLAIN (aka SQLT) MOS ID 215187.1
• Maintained by Oracle Support - often requested
• Collects information on why the execution plan was chosen
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
13
Performance Triage & Resolution
Gather Diagnostic Data
Trace/TKPROF … Get the complete session story
– Set Trace Parameters
– max_dump_file_size=unlimited
• Ensure there is sufficient disk space in the tracing directory
– user_dump_dest or if diagnostic_dest is set in Oracle11g
• (diagnostic_dest)/diag/rdbms/(dbname)/(instname)/trace
• background_dump_dest for parallel slave processes
– STATISTICS_LEVEL=ALL
Generate Trace
– Forms: Help -> Diagnostics -> Trace
– OA HTML: Diagnostics -> Tracing
– Set the profile: “Initialization SQL Statement - Custom“. Use FND_CTL.FND_SESS_CTL
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
14
Performance Triage & Resolution
Gather Diagnostic Data
Trace/TKPROF … Get the complete session story
– Raw trace must be
• From before the user action began To the point the DB session ends**
• A complete, non-truncated trace file
– *** DUMP FILE SIZE IS LIMITED TO 12345 BYTES***
– Run TKPROF from the correct Oracle home
• Use the sort options = fchela,exeela,prsela
– TKPROF output MUST have
• Runtime Execution Plans and Row Counts & Row Source Stats
• Run Time Execution Plan = “Row Source Operation” != “Explain Plan”
– Best practices, detailed how-to steps & recommended tracing patches.
• MOS ID 1121043.1 (Also review Appendix B – Getting In-Memory partial runtime stats)
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
15
Performance Triage & Resolution
Gather Diagnostic Data
Automatic Workload Repository (AWR) … Get the complete system story
• Regularly Review Automatic Workload Repository (AWR) data for different workloads
• Use DBMS_WORKLOAD_REPOSITORY or Enterprise Manager to create snapshots
• Generate reports using $ORACLE_HOME/rdbms/admin/awrrpt.sql
• Review the Advisory sections in AWR to fine tune SGA & PGA
• Monitor and correlate top SQL in AWR to business flows
• Review Load Profile and Top 5 Timed Events section to determine utilization or bottlenecks and
review relevant detailed sections for these events
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
16
Performance Triage & Resolution
Gather Diagnostic Data
Automatic Workload Repository (AWR) … Get the complete system story
Contention … Latch Related Waits
I/O Related Waits … Bad SQLs
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
17
Performance Triage & Resolution
Analyze Root Cause
• Regularly Review Automatic Workload Repository (AWR) data for different workloads
– For Latch related waits
• Often due to: non-sharable SQL, sub-optimal SQL which performs full table or full index
scans, dynamic object creation/removal, etc.
• Review the latch Statistics section to determine the hot latches
• Trace waiter and holder sessions to determine actual cause
– For I/O related waits
• Review SQL sections by Logical/Physical reads/writes and the Segment Statistics sections by
I/O
– For Enqueue related or buffer busy waits
• Review the following sections: enqueue, segment statistics by buffer busy waits, row lock &
ITL waits
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
18
Performance Triage & Resolution
Analyze Root Cause
If bottleneck is I/O related...
(db file sequential read, db file scattered read, log file sync, free buffer waits, etc...)
– Tune Top SQL is always the first step
– Make sure that system has updated statistics
– Maximize Memory availability, allocate generously to buffer cache, review AWR advisories &
monitor swapping & paging
– Use ASM or alternatively use the SAME methodology for db files
– IO Sub-system:
• RAID 10 still preferred most often for high end requirements Ideal avg. response times of < 10ms
– Check for excessive/redundant indexing
– Configure async IO, use quick I/O technologies
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
19
Performance Triage & Resolution
Analyze Root Cause
If the bottleneck is concurrency related (enq%, latch%, buffer busy%, etc… )
– Use global hash-partitioned indexes for hot leaf blocks - Identify via AWR "Top logical IOs by
Segment"
– Some of these waits are commonly caused by bad SQL execution plans
• For example: latch: cache buffer chains, %buffer busy waits, read by other session
– Make sure to use ASSM and OATM table space model
– Increase INITRANS to alleviate ITL contention
– Increase sequence caches
– Work with support/consulting to evaluate if table/index partitioning will help (is not supported
to change out of the box partitioned table)
– If the program allows, more frequent commits (batch size) can help by reducing the CR
(consistent read) work required
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
20
Optimizing E-Business Suite
Applications Tier
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
21
Optimizing E-Business Suite Applications Tier
Stay Current
• Keep the system current on AD/ATG/OAM code e.g. apply latest AD/ATG RUPs
• Apply the latest EBS Family Packs + Recommended Patch List
• For 12.2, review Oracle E-Business Suite Release 12.2: Consolidated List of Patches and Technology Bug Fixes
(MOS ID 1594274.1)
• Critical EBS patches are also listed in the E-Business Suite 12.2 readmes.
• That database versions 11.2.0.3 and 12.1.0.1 will be end of error correction July 22nd 2015 and August 27th
respectively.
• For 12.2, apply the Latest AD and TXK Release Update Packs to Oracle E-Business Suite Release 12.2 (MOS ID
1617461.1)
• Apply “Recommended Performance Patches for the Oracle E-Business Suite”, MOS ID 244040.1
– Recommended performance patches for all the modules and tech stack components are consolidated in this
note
• Upgrade techstack components to the latest certified levels
– Check http://blogs.oracle.com/stevenChan/resource/certifications.html
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
22
Optimizing E-Business Suite Applications Tier
Use Optimal Logging Settings
• As Logging and enabling debug messages have considerable impact on performance, always
make sure that the associated profiles are set properly while keeping logging levels to
minimum.
• FND: Debug Log Enabled - Set it to ‘Yes’ (12.0 RUP3+, 12.1.x, 12.2)
• FND: Debug Log Level - Set to Unexpected (Level=6)
• Query FND_LOG_MESSAGES to check the logging traffic
• Note that With above profile configuration
– If the query shows a large number of rows being created, there could be exceptions and
errors that need to be investigated
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
23
Optimizing E-Business Suite Applications Tier
Optimize Workflow Processes
• Purge runtime data (Concurrent Program: FNDWFPR)
– Can run by ITEM_TYPE
– Monitor/Troubleshoot old workflows not properly closed
– Purge frequently
• For background engines via Concurrent Manager, set the ‘Process Stuck’ parameter to ‘No’
– Identification of stuck workflows is resource intensive
– Start a separate background engine to handle stuck/timed out processes with a low
frequency –i.e. once a day
• Use deferred activities to improve online response times for flows such as Scheduling, PO
Document Approval, etc.
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
24
Optimizing E-Business Suite Applications Tier
Optimize Workflow Processes
• Run $FND_TOP/sql/wffngen.sql to translate the activity function calls into static calls
– Edit & add heavily used ITEM_TYPES
– itemtypelist_t (‘WFSTD’,’FNDFFWF’)
– Generates wffncal2b.pls in utl_file_dir to recreate package.
• Disable retention on Workflow Queues
– DBMS_AQADM.ALTER_QUEUE(queue_name=>:b1,retention_time=>0);
• Verify that runtime tables are partitioned for higher scalability – MOS ID: 260884.1
• For high volume batch processing in RAC use ITEM_TYPE to Node affinity
• Define node affinity at the program level (MOS ID 1129203.1)
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
25
Optimizing E-Business Suite Applications Tier
Optimize Forms Processes
• Reduce load on the database server: train applications users to optimally utilize the professional
Forms interfaces by:
– Avoiding Blind queries with unselective filtering criteria
– Providing selective criteria in Find windows and LOVs
• Minimize network traffic and form open times: train applications users to
– Avoid opening and closing forms across transactions
– Combine forms from multiple-products on a single menu
• Reducing system load and network overhead, is especially important for deployments with high
user concurrency levels and high latency connections
• Monitor concurrency levels and system load on the applications tier
• Generate complete SQL trace and Forms Runtime Diagnostics (FRD) to debug performance
issues
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
26
Optimizing E-Business Suite Applications Tier
Optimize JVM Processes
• Total available memory is the most important factor affecting GC performance.
• Monitor the frequency of collections, especially major collections (i.e. Full GC)
• Enable verbose GC to tune heap sizes based on the GC traffic
• Start with: -Xms1024M and –Xmx1024M
• Setting -Xms and -Xmx to the same value avoids the need for heap memory re-allocation
during runtime
• If full GCs are too frequent, consider increasing Xms and Xmx
• Consider using jstat to see and review each of the heap generations for possible tuning
• Increase the number of JVMs to scale up for more users with faster GC times.
– Each JVM has a smaller Xmx,Xms memory footprint so full GCs will be faster!
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
27
Optimizing E-Business Suite Applications Tier
Size The Middle Tier For Concurrency
• The main contributors to the middle tier memory are the JVMs heaps, forms (frmweb) process
memory and concurrent manager components, especially java concurrent programs.
• To calculate physical memory for oacore JVM heap, the following formula can be used:
M= (N/ 150 ) * 1 GB
Where M = total memory used by oacore VMs
N = total number of concurrent Self-Service users
• Use one JVM per 2 CPUs
• ~100-150 concurrent users per JVM
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
28
Optimizing E-Business Suite Applications Tier
Size The Middle Tier For Concurrency
• To calculate physical memory for Forms frmweb processes, the following formula can be
uses used:
M= N * 40MB
Where
M = total memory used by Forms frmweb processes
N = total number of concurrent Forms users
• Care should be taken to size for the concurrent manager components, especially java
concurrent programs.
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
29
Optimizing E-Business Suite Applications Tier
Size The Middle Tier For Concurrency
•
Managed instances JVM sizing should consider both memory and CPU domains.
•
On 64bit environment, we do not recommend allocating huge heap sizes, but rather
have more managed instances in the cluster to scale up to the target concurrency levels.
•
For 12.2, for Admin Server sizing, the default size of 512M is not enough for most
installations, we recommend setting the XMS to at least 1 GB and the XMX to 2GB
•
For 12.2, consider additional sizing requirements for online patching. An initial
guidance on 12.2 environment standard sizing can be found in the “Oracle E-Business
Suite Installation Guide: Using Rapid Install (Part No. E22950)”
http://docs.oracle.com/cd/E26401_01/doc.122/e22950/T422699i4773.htm#T610671
Managing Configuration of Oracle HTTP Server and Oacore, Oafm, Forms and Formsc4ws Applications in Oracle E-Business Suite Release 12.2 (MOS ID 1905593.1)
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
30
Optimizing E-Business Suite Applications Tier
Optimize the Concurrent Manager
• Manage CM Jobs Lifecycle: 50% of performance tuning is in the business!
– Review long-running auto resubmitted jobs
– Review short/long-running jobs – selective parameters?
– Don’t purge jobs that the users then have to run again
– Trim the fat: Review and eliminate concurrent jobs that are not required and/or are not
being used by users.
• Manage CM Managers: More managers/workers != more throughput
– Avoid enabling an excessive number of standard or specialized managers.
– A common guideline is between 1-2 target process per CPU, but this needs to be balanced
with resources required by online activity
– Utilize Parallel Concurrent Processing (PCP) to leverage the Cluster (RAC)
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
31
Optimizing E-Business Suite Applications Tier
Optimize the Concurrent Manager
• Manage CM State Data:
– Purge the FND tables on a regular basis using the “Purge Concurrent Request and Manager Data”
program
– High throughput: Keep *.out and *.log on fast disks. Use Separate disks if throughput seems slow
especially on NFS Filers
– Truncate the reports.log file in log directory. Watch for 2GB limit on any output files
(MOS ID 842850.1)
• Manage CM Schedules:
–
–
–
–
Use specialization rules and work shifts to bind specific jobs to specific time windows .
Avoid scheduling resource intensive batch requests during peak activity.
Reschedule some programs to run when the concurrent managers have excess capacity.
Define Workload Management Strategy based on job average duration and system usage profile.
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
32
Optimizing E-Business Suite Applications Tier
Optimize the Concurrent Manager: Workload Management Strategy
Avoid Short Sleep Times
– *Enough for a couple of minutes of work
– **Set cache size to at least twice the number of workers
• Specialization rules and work shifts
– Bind specific jobs to specific time windows
– Profile Option: Concurrent: Active Request Limit can be used restrict the number of concurrent requests that
may be run simultaneously by each user
– Specialize if there are too many jobs in a specific category
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
33
Optimizing E-Business Suite Applications Tier
Optimize the Concurrent Manager
• Conflict Resolution Manager Sleep Time:
– To maximize throughput for jobs which spawn parallel workers (i.e. Auto Invoice, Payroll), consider
reducing the sleep time of the Conflict Resolution Manager (CRM).
– Default is 60s, consider 5 or 10 seconds
• Concurrent Manager Sleep Time:
– Define dedicated queues for short and frequent requests and increase the sleep times for managers
which do not require near real-time job execution
– Reducing sleep time to a very low value many cause excessive CPU utilization.
• Transaction Manager Sleep Time:
– Set the profile “Concurrent:Wait for Available TM” to 1 (second) to minimize TM latency. The profile
sets the total time to wait for a TM before switchover to next available TM
– Set Sleep time on Transaction Managers to 30 minutes. this avoids constant polls to check for
shutdown requests.
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
34
Optimizing E-Business Suite
Database Tier
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
35
Optimizing E-Business Suite Database Tier
Optimize Database
• Tune Init.ora Parameters:
– Ensure mandatory init.ora parameters are set correctly.
– MOS ID 396009.1 & 216205.1 (R12 & 11i respectively)
(MOS now has a HOT TOPICS feature to be alerted when favorite notes have been updated)
• Apply Required Patches & Fixes/Workarounds for Known Issues
– Recommended Performance Patches - MOS ID 244040.1
– Required Patching – Interoperability Notes
• Size SGA and PGA memory for maximum projected concurrency levels, buffer cache and shared pool
sizing specially critical
• Use Large/Huge pages. For example on Linux Huge Pages – See MOS ID 744769.1
– Only SGA allocation benefits from large/huge pages, PGA allocation does not
– Set the parameter USE_LARGE_PAGES=’only’ for each instance so that the instance will only start if
sufficient Huge Pages are available See MOS ID 1392497.1
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
36
Optimizing E-Business Suite Database Tier
Optimize Disk IO
• Ensure kernel asynchronous I/O is enabled and supported for the underlying file system
(if file systems are being used for the DB files)
• Configure & verify direct I/O is used (if file systems are being used for the DB files)
– Improves performance/scalability by simulating raw devices, avoiding file system cache traffic
• Ensure your I/O subsystem can handle your peak I/O load & IOPS
– Consider RAID, S.A.M.E. configuration with sufficient spindles
– Measure with ORION tool or in 11g
DBMS_RESOURCE_MANAGER.CALIBRATE_IO – MOS ID 727062.1
• Reduce the I/O workload
– Tune expensive SQL
– Trim the workload of unnecessary tasks
– Do 3X less I/O on average with 11g Advanced Compression Option
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
37
Optimizing E-Business Suite Database Tier
Optimize Backend Code Execution Path
• Utilize PL/SQL Native Compilation.
• Pin Top Apps PL/SQL packages by executions to help avoid shared pool fragmentation and reloads.
• Check cursor leaks and SQL statements using literals with large execution counts. Cursor leaks/literals
can result in ORA-4031 errors.
• Tune sequence cache sizes for hot sequences (e.g. 1,000 or higher) in order to minimize index key
contention for the Apps surrogate keys.
• As each online patching cycle is completed, the database will accumulate an additional old database
edition. If the number of these grows too large, system performance will start to be affected. When the
number of old database editions reaches 25 or more, you should consider dropping all old database
editions by running the adop actualize_all phase and then performing a full cleanup.
• Make sure to use FND_STATS to gather statistics.
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
38
Optimizing E-Business Suite Database Tier
Gather Applications Statistics
• Statistics gathering strategy should not be based on time.
• It should always be based on rate of change in the data and is an iterative process.
• Disable the 10g/11g automatic job to gather stats
• Do not gather statistics excessively on entire schemas or the entire database such as
nightly or weekly.
• Do not gather statistics on Global Temporary Tables.
• Avoid gathering statistics during peak hours.
• Gather statistics after large data load and when mix/max values have changed.
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
39
Optimizing E-Business Suite Database Tier
Gather Applications Statistics
• New Published MOS ID 1586374.1 for "Best Practices for Gathering Statistics with Oracle EBusiness Suite"
• Use only FND_STATS or the Gather Schema / Table Statistics Concurrent Programs
– Do NOT USE the analyze or dbms_stats command directly. It is not supported, and can result
in sub-optimal plans.
– When Gather Schema Statistics concurrent program is used fnd_stats does the bookkeeping
for the run.
• FND_STATS supports column histograms.
– Histograms can be seeded by calling to fnd_stats.load_histogram_cols.
• Use the GATHER_AUTO option to gather incrementally. This will re-analyze objects which
currently have no statistics and objects with stale statistics
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
40
Optimizing E-Business Suite Database Tier
Gather Applications Statistics
• Analyze all schemas at 10%, then for some specific objects use higher sampling x%
– Due to data skew, some tables benefit from higher sampling %
– From11g, can use value of zero (AUTO_SAMPLE_SIZE)
• Do not invalidate cursors
– While submitting concurrent request to gather statistics set “Invalidate Dependent Cursors”
parameter to No.
• Consider locking statistics for very volatile tables once a representative set is gathered.
– For example, interface tables, ap_selected_invoices, wsh_pr_workers etc.
– These intermediate tables are used in batch processes.
– Use FND_STATS.LOAD_XCLUD_TAB to skip the table.
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
41
Optimizing E-Business Suite Database Tier
Gather Dictionary and Fixed Stats
• Gather dictionary object statistics when there is a significant change in dictionary, for
example, lot of new objects created.
exec dbms_stats.gather_dictionary_stats;
• Gather fixed object statistics with reasonable load on the system
exec dbms_stats.gather_fixed_objects_stats
• Dictionary and Fixed stats should be gather manually as they are not gathered
automatically in the Release 12 upgrade by adstats.sql or any other method.
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
42
Prepare: Database Tier
Gather Dictionary and Fixed Stats
• Fixed Object Statistics should be gathered:
– After any associated platform or database upgrade that is part of the overall
Oracle E-Business Suite upgrade.
– After any SGA/PGA parameters have changed.
– After Release 12 upgrade, when there is representative activity on the system.
• Dictionary Statistics should be gathered:
– After any associated platform or DB upgrade that is part of the overall Oracle EBusiness Suite upgrade.
– After the Release 12 upgrade.
– After move to OATM
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
43
Optimizing E-Business Suite on RAC
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
44
Optimizing E-Business Suite on RAC
Verify Optimal HW/DB Configurations
• Interconnect– a Private; GigE or Infiniband
• Network Configuration– Use Jumbo frames for the interconnect
– ifconfig <adapter> mtu 9000
• Make sure that the I/O subsystem sized for performance, not just storage; write optimized
• Increase the buffer Cache by 10%
• Use Parallel Query Instance Affinity (Set PX affinity local each node)
– 11gR2 - PARALLEL_FORCE_LOCAL =true
• Monitor CPU utilization, run queue lengths and interconnect traffic
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
45
Optimizing E-Business Suite on RAC
Utilize Instance Affinity and PCP
• Follow recommendation in EBS RAC conversion MOS documents for patch and parameter
information
– (MOS ID 823587.1) 11gr2 RAC with EBS R12
• Use Affinity: Direct some workload components to specific RAC nodes to maximize scalability
by minimizing inter-node communication and synchronization work.
• Utilize Parallel Concurrent Processing (PCP) to leverage the Cluster (RAC).
• For RAC and PCP refer to (MOS ID 1359612.1)
• Starting from 12.1, you can define node affinity at the program level
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
46
Optimizing E-Business Suite on RAC
Utilize Instance Affinity and PCP
• PCP: Maintain 1 – 1 correspondence of CM nodes to RAC nodes
– Set <s_cp_twotask> per CM node to tie to a specific RAC instance.
– Do not use a load balanced TNS entry for the value of s_cp_twotask. The request may hang if the
sessions are load balanced.
– Define primary/secondary CM nodes for Failover
• From 12.1 , Define node affinity at the program level
– Concurrent | Program | Define ( See MOS ID 1129203.1)
– Starting from 12.1.3 , a new option to define node affinity at the program level – Concurrent ->
Program -> Define
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
47
Optimizing E-Business Suite on RAC
Utilize Instance Affinity and PCP
• Extend Affinity to the Applications Tier (Forms)
– Set Profile Option "Database Instance" at App or Resp level which can be tied to
two_task value or to a service
• For 12.1.3, Extend Affinity to the Applications Tier (Self-service)
– Set Profile Options “App%Agent” to web tier hosts configured for specific services
…or
– Set Profile Option "Applications Database ID" to node specific DBC file name
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
48
Optimizing E-Business
Data Management
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
49
Optimizing E-Business Data Management
High
Volume
Active
Less
Active
Low
Volume
0
Data Volume
Activity
What Do We Know About Data
1
Months
1
5
From Day 0 data starts to grow
Data comes from many sources such as
–
Transactional e.g. Oracle Payables
–
Transient data (i.e. login,
concurrent requests etc..)
–
Reference/Seed data e.g. lookup
codes
Is all this data needed or required to be
10
available online?
Years
FACT: More than 80% of Data is Inactive
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
50
Optimizing E-Business Data Management
Data Management
Optimal Data Management
Available Oracle Solutions
Reduced Hardware Cost
Reduced network impact from data
movement
Reduced backup time
Reduced upgrade time and associated
downtime
Increased Transactions Per Second Rates
(TPS)
Improved Operating Performance
Reduced Total Cost of Ownership (TCO)
Data Growth Control Methods
Archive/Purge Functionality within
E-Business Suite
Data Management Methods
Database Partitioning
Database Compression
Information Lifecycle Management
(ILM)
These methods can be used interchangeably
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
51
Optimizing E-Business Data Management
E-Business Suite: Archiving & Purging
• Where possible try to utilize the standard Oracle supplied
programs
– Purge/Archive programs provided at the module level in the E-Business
Suite
– Many documented standard programs available (260 purge and archive
programs in R12)
• Purge Portal
– Purge Portal introduced in 11i10
– Single purge/archive management console
– Purge programs can be configured, initiated and monitored
– Set the execution frequency as well view history of purge programs.
• Accessing the Purge Portal
– System Administrator > Oracle Applications Manager >Purging/Critical Activities
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
52
Optimizing E-Business Data Management
E-Business Suite: Archiving & Purging
• E-Business Suite provides Purge/Archive
solutions for the key transactional entities
Human
Resources
• The native E-Business Suite purge and archive
programs are synchronized with the data
model
– Oracle’s solution considers cross-product
Workflo
dependencies
w
– Example: We can’t simply delete data in
Oracle Payables
Alert
Payables
General
Ledger
iExpense
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
53
Optimizing E-Business Data Management
E-Business Suite: Partitioning
• When ?
– Many times it will be an implementation decision driven by huge data volume
– Can be physical DM decision to improve performance
• Situations where partitioning can help
– Large deletes could become simple truncates of a partition
– Re-population/Refreshing of large portions of data could become a “partition exchange” operation
– Very large tables & their indexes can be split into smaller segments, reducing I/O
• Partition Key
– Should be a key that most statements will have in the WHERE clause to reap the benefits of
partition pruning & elimination as well as index I/O (local index)
– Should not change often to avoid row-movement between partitions
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
54
Optimizing E-Business Data Management
E-Business Suite: Partitioning
• E-Business Suite Release 12 is certified with the Oracle Database 11g Partitioning Option
• (MOS ID 554539.1) – Database Partitioning for Oracle E-Business Suite
• The Oracle E-Business Suite uses partitioning out-of-the-box with few modules:
– 200+ tables
– Tables have a natural and logical partition key
– Majority of the runtime access path (of the standard product) is based on this natural partition
key
– Changing out of the out of the box partitioned tables is not supported.
• Many E-Business Suite tables do not have a natural partitioning key which would apply to all
customers
– Based on their own requirements, customers are free to partition the tables in a logical manner
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
55
Optimizing E-Business Data Management
E-Business Suite: Advanced Compression (ACO)
• Advanced Compression (ACO) option is an ideal choice for saving space with read-intensive
operations and can reduce the associated Disk I/O. ACO reduces storage requirement from
2-4 times.
• In our benchmarks, online workload showed up to 30% improvement with up to 6%
increase in CPU consumption
• As for the batch programs, Order-to-Cash batch runtime increased by only 0.43%, while
payroll process runtime reduced by 17% with additional 3.7% CPU usage
Oracle E-Business Suite Release 12.1 with Oracle Database 11g Advanced Compression
[MOS ID 1110648.1]
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
56
Optimizing E-Business Data Management
Hybrid Columnar Compression (HCC)
• The majority of E-business suite transactions are OLTP in nature, thus we do not recommend
using HCC for E-business suite, for the following reasons:
– Updating compressed data is very costly: Up to 14X slower
– Updating will undo space saving : Data will be uncompressed when updating an HCC
compressed table. This will result in chained rows which will affect table scans
– Coarse locking - Update of 1 row can lock thousands of rows
• Our benchmark testing indicates that HCC creates performance and space penalties in
certain conditions due to the overhead associated with uncompressing and recompressing EBusiness Suite data. EBS users should not enable Hybrid Columnar Compression in its
currently-released version.
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
57
Upgrade Best Performance Practices
Performance and Downtime
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
58
Plan, Prepare, Test and Upgrade
• As with most planned production maintenance activities, a successful upgrade is highly dependent
on how well-defined the upgrade plan is.
• Pre-production testing and validation on an comparable system prior to the actual upgrade is key
to minimizing unforeseen contingencies.
• Optimizing the upgrade process is all about making the right choices at the planning stage and
verifying these choices with a test environment before production.
Planning Your Oracle E-Business Suite (EBS) Upgrade from Release 11i to Release 12
(MOS ID 1406960.1)
Best Practices for Minimizing Oracle E-Business Suite Release 12 Upgrade Downtime
(MOS ID 1581549.1)
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
59
Test: Pre-Production Testing
Testing and Final Run Planning
•
Testing should be done on a comparable system that has the same CPU, IO and memory
capacity as the target production system
•
It is critical to do multiple rounds of testing with different settings to maximize server
utilization, while considering the following
– Memory utilization (no swapping/ excessive paging)
– CPU utilization (scale down if at 100%)
– I/O response times (scale down if averages > 20 ms)
•
When analyzing Release 12 Upgrade performance issues, the goal is prevent wasted test
iterations by maximizing the number of performance issues resolved.
•
Testing timings along with system resource utilization should be used to plan the final
upgrade with a 20% contingency factor.
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
60
Test: Pre-Production Testing
Get the Diagnostic Data
•
Start with Express Diagnosis of Oracle E-Business Suite Release 12 Upgrade Performance Issues
(MOS ID 1583752.1)
• SQL Tuning
•
•
Trace files
•
SQLT output (MOS ID: 215187.1)
•
Trace Analyzer (MOS ID: 224270.1)
•
AWR Report (MOS ID: 748642.1)
•
11g SQL Monitor Report
•
AWR SQL Report (awrsqrpt.sql)
• Database Tuning
• AWR Report (MOS ID: 748642.1)
• ADDM report (MOS ID: 250655.1)
• Active Session History (ASH)
• OS - OSWatcher (MOS ID: 301137.1) Q
Correlate AWR, system vitals and expensive SQLs. Start with the top events and top SQLs sections
in the AWR reports.
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
61
Test: Pre-Production Testing
Common Optimizations
•
Once you have identified the long running jobs and SQL, you can check My Oracle Support for
known issues and potential solutions or workarounds.
•
However, bear in mind that the fix or workaround may not necessarily fix your particular
problem.
•
If you cannot confirm that you have exactly the same issue (from the diagnostics) you may still
apply the fix, but should continue to gather diagnostics and search for a solution until the issue is
fully resolved.
•
If you have identified that a long running job has an inefficient execution plan, you could use an
SQL Profile to apply hints that will help the CBO choose a better execution plan. You will need
SQL tuning expertise to do this.
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
62
Oracle E-Business Suite Learning Stream
Keep Your Skills Current through Continuous Learning
Expert Delivered
Access to 300+ instructional videos delivered by Oracle
subject matter experts
Training Across the Entire E-Business Suite
For professionals of all experience levels looking to
regularly broaden and deepen their knowledge
Continuously Refreshed Content
Covers technical and functional aspects of Oracle EBusiness Suite solutions
Request Topics that Interest You
Subscription Service
Preview the Learning Stream
NOW!
education.oracle.com/streams/ebusinesssuite
Technical Upgrade Best Practices for Oracle E-Business Suite
Samer Barakat and Udayan Parvate
http://education.oracle.com/pls/web_prod-plq-dad/db_pages.getpage?page_id=724&get_params=streamId:61,p_loId:2687
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Oracle Confidential – Internal/Restricted/Highly Restricted
63
Questions and Answers
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Additional Resources
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
66
Target Release:
12.1 Only
References
•
R12.1 documentation roadmap (790942.1)
•
Oracle E-Business Suite Release 12.1 Info center (806593.1)
•
Database preparation guidelines for R12.1 upgrade (761570.1)
•
Recommended Performance Fixes (244040.1)
•
R12 Upgrade Sizing & Best Practices (399362.1)
•
R12.1 EBS pre-install patches Report (1448102.1)
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
67
Additional Resources
Optimizing E-Business Suite Database Tier
• 1121043.1 Collecting Diagnostic Data for Performance Issues in Oracle E-Business Suite
• 1362660.1 Enhanced Performance Diagnostics for Oracle E-Business Suite Release 12.1.3
• 301137.1 OSW Document
• 461053.1 OSWg Document
• 215187.1 SQLTXPLAIN (SQLT)
• 224270.1 Trace Analyzer (TRCA)
• 39817.1 Interpreting Raw SQL_TRACE files
• 280391.1 How to Retrieve SID Information For a Running Request
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
68
Additional Resources
Tuning the Concurrent Manager
• 1359612.1 Webcast 'E-Business Suite - RAC & Parallel Concurrent Processing
• 1367676.1 Webcast 'E-Business Suite - Concurrent Manager Performance
• 164085.1 Enhancing and Automating Oracle Applications Concurrent Processing
• 1057802.1 Best Practices for Performance for Concurrent Managers
• 1304305.1 E-Business Concurrent Processing Information Center
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
69
Additional Resources
EBS Upgrade and Platform Migration
• Best Practices for Minimizing Oracle E-Business Suite Release 12 Upgrade Downtime (MOS ID 1581549.1)
• Express Diagnosis of Oracle E-Business Suite Release 12 Upgrade Performance Issues (MOS ID 1583752.1)
• R12.1 documentation roadmap (MOS ID 790942.1)
• Oracle E-Business Suite Release 12.1 Info center (MOS ID 806593.1)
• Database preparation guidelines for R12.1 upgrade (MOS ID 761570.1)
• Patching FAQs (MOS ID 459156.1, 225165.1)
• Staged or shared APPL_TOP and distributed AD (MOS ID 734025.1, 384248.1, 236469.1)
• OAM “Patch Wizard” overview and FAQ (MOS ID 976188.1, 976688.1)
• AD Command Line Options for Release R12 (MOS ID 1078973.1)
• Recommended Performance Fixes (MOS ID 244040.1)
• R12 Upgrade Sizing & Best Practices (MOS ID 399362.1)
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
70
Additional Resources
EBS Upgrade and Platform Migration
• EBS R12.1 Upgrade related Reports
– EBS 12.1.3 Data Model Comparison Report (MOS ID 1290886.1)
– EBS ATG Seed Data Comparison Report (MOS ID 1327399.1)
– EBS File Comparison Report (MOS ID 1446430.1)
– EBS pre-install patches Report (MOS ID 1448102.1)
• White papers
– Planning Your Oracle E-Business Suite Upgrade from Release 11i to Release 12.1 (MOS ID 987516.1)
– R12 Upgrade considerations by product: Financials (MOS ID 889733.1)
– Oracle E-Business Suite Upgrades and Platform Migration (MOS ID 1377213.1)
– Oracle E-Business Suite Release 12.2 Information Center - Upgrade (MOS ID 1583158.1)
– Script to Diagnose adop and Other AD-TXK Issues in Oracle E-Business Suite Release 12.2 (MOS ID
1901242.1)
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
71
Additional Resources
E-Business Suite: Archiving & Purging
• Archive/Purge Product Information
– Product User guides – GL/AP/AR/FA/OM
– Support Notes:
• (MOS ID 138264.1) – General Ledger Archive/Purge FAQ
• (MOD ID 144431.1) – Fixed Assets Archive/Purge FAQ
• (MOS ID 136919.1) – General Ledger Archive/Purge Setup and Usage
• (MOS ID 752322.1): Reducing Oracle E-Business Suite Data Footprint
• E-Business Suite: Technology Learning
https://blogs.oracle.com/stevenChan/entry/e_business_suite_technology_learning
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
72
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Was this manual useful for you? yes no
Thank you for your participation!

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Download PDF

advertisement