Optim Solutions Update and Optimizing DB2 for z/OS Query

Two for One: Optim Solutions Update and
Optimizing DB2 for z/OS Query
Performance
Bryan F. Smith bfsmith@us.ibm.com
IBM
Tuesday, August 3, 2010: 1:30 PM-2:30 PM
Hynes, Room 103
Abstract
• Optim Solutions Update: Are you wondering how Optim Solutions
relates to your z/OS DB2 Tools investment? Do you want to see what
the lab is doing to help the z/OS DBA? Join one of our chief architects
to hear how Optim solutions can extend administrative function for
DB2 for z/OS, both today and in the future. The speaker will cover data
design, performance and query monitoring, performance and query
optimization, and key roadmap items.
• Optimizing DB2 for z/OS Query Performance: There are lots of factors
that affect query performance and no silver bullets. Investigating
performance issues includes looking at indexes, statistics, query
design, cache size, execution mode, and the list goes on. Join the
speaker as he digs into some of these areas in detail and at how
Optim Solutions can help get resolutions faster and with more
flexibility.
Agenda
Introduction to Integrated Data Management (IDM)
What can Data Studio be used for in DB2 for z/OS environments?
How Java and .NET applications can be managed like CICS/IMS applications
What results can be realized by query tuning
Data movement options
What value is there to extending the DB2 backup and recovery utilities?
Other DB2 for z/OS Tooling
What's cooking back at the lab
• Performance Management
• Configuration Management
• Maintenance Management
• Wrap-up
•
•
•
•
•
•
•
•
Data Management Must Drive Competitive
Advantage
Survey: CIO’s want to strengthen competitive advantage by
better managing enterprise data
75% of CIO’s believe they can
strengthen their competitive
advantage by better using and
managing enterprise data.
• Data management must
drive competitive
advantage
• Facilitate business alignment
78% of CIO’s want to improve
the way they use and manage
their data.
…but only 15% believe that their
data is currently comprehensively
well managed.
• Accelerate delivery of
enterprise-ready, data-driven
applications
• Free up staff to drive business
growth and optimization
Source: Accenture CIO Data Management Survey 2007. n=167 CIOs
Introducing Integrated Data Management
An integrated, modular environment to manage
enterprise application data, and optimize data-driven
applications, from requirements to retirement
• Grow the business, without growing
costs
• Develop and deploy business critical
applications faster
• Mitigate compliance risks with modeldriven data governance
• Prevent runaway infrastructure spending
• Improve performance – of work teams,
databases, applications, and business
units
Integrated Data Management = Data Studio+Optim+DB2 for z/OS Tools
Integrated Data Management
Data Architect
Application
Manager
Design
Developer
Develop
Optimize
Policies
Models
Metadata
Tester
Operate
Database
Administrator
• Deliver increasing value across the
lifecycle, from requirements to
retirement
Deploy
• Facilitate collaboration and efficiency
across roles, via shared artifacts
automation and consistent interfaces
• Increase ability to meet service level
agreements, improving problem
isolation, performance optimization,
capacity planning, and workload and
impact analysis
• Comply with data security, privacy,
and retention policies leveraging
shared policy, services, and reporting
infrastructure
Supporting Heterogeneous
Environments
InfoSphere Data Architect
Design
Optim Development Studio
Optim Data Growth
Solutions
Optim Query Workload Tuner
Optim Query Tuner
Policies
Models
Metadata
DB2 Performance Management
DB2 Performance Expert and
Extended Insight
DB2 Automation Tool
DB2 Recovery Solution
DB2 Utilities Suite
Develop
Optimize
Operate
DB2 Change Management Solution
Optim Test Data
Management
Optim Data Privacy
Solutions
Deploy
Optim pureQuery Runtime
Optim Database Administrator
DB2 Audit Management Expert
Database Encryption
Integrated Data Management User Interface
Directions
Data Studio and Optim
Eclipse UI
Develop and Deploy
Design and Development
Configuration
Object Management
Change Management
Automation Planning
Data Governance
Web UI
Operate
Monitoring
Automation Management
Reporting
Data Governance
z/OS Tools
3270
z/OS Database Administration
Rich z/OS function
Application Management
Database Administration
Performance Management
Data Governance
DB2 for z/OS Tools Portfolio
Application Management
InfoSphere Data Architect
Optim Development Studio
DB2 Path Checker
DB2 Bind Manager
DB2 SQL Performance Analyzer
DB2 Table Editor
Optim pureQuery Runtime
Utilities Management
DB2 Utilities Suite
DB2 Automation Tool
DB2 Automation Toolkit SAP Edition
DB2 Utilities Enhancement Tool
DB2 High Performance Unload
Business Intelligence
IBM DataQuant
IBM QMF
DB2 Web Query Tool
Database Administration
DB2 Administration Tool
DB2 Object Comparison Tool
DB2 Administration Toolkit SAP Edition
DB2 Storage Management Utility
Performance Management
OMEGAMON XE DB2 Performance Expert
OMEGAMON XE DB2 Performance Monitor
DB2 Buffer Pool Analyzer
Optim Query [Workload] Tuner
DB2 Query Monitor
DB2 Performance Toolkit SAP Edition
Information Integration
WebSphere Classic Data Event Publisher
WebSphere Classic Federation Server
WebSphere Classic Replication Server
WebSphere DataStage
WebSphere Replication Server
DataMirror
Backup and Recovery
Application Recovery Tool for
IMS and DB2 Databases
DB2 Archive Log Accelerator
DB2 Change Accumulation Tool
DB2 Cloning Tool
DB2 Log Analysis Tool
DB2 Object Restore Tool
DB2 Recovery Expert
Data Governance
IBM Optim Data Growth
IBM Optim Data Privacy
IBM Optim Test Data Management
DB2 Audit Management Expert
IBM Database Encryption Expert
Data Encryption for DB2 and IMS
DB2 Developer Workbench vs. Data Studio
before
IBM DB2 Developer Workbench V9.1
SQL Query Editor
SQLJ Editor
SQL Builder
XQuery Builder
SQL Routine Debugger
Java Routine Debugger
XML Editor
XML Schema Editor
Data Management
Visual Explain
Project Management
Data Studio is a full replacement of
DB2 Developer Workbench
plus much more
• DB2 for Linux, Unix, Windows v8.x, v9.x
• DB2 for z/OS v7, v8, v9
• DB2 for i5/OS v5r2, v5r3, v5r4
• Informix Dynamic Server (IDS) v9.x, v10.x, v11
now
IBM Data Studio V2.2
Integrated Query Editor – SQL + XQuery
SQLJ Editor
SQL Builder
XQuery Builder
SQL Routine Debugger
Java Routine Debugger
XML Editor
XML Schema Editor
Data Management
Visual Explain
Project Management
No
-ch
arg
e
ER Diagramming
Data Distribution Viewer
Object Management
Browse & Update Statistics
Security Access Control
Connection Management integration with Kerberos and LDAP
Data Web Services
IDS Server Support
Data Studio is no longer used as a brand
Data Studio: Data Modeling / App & DB Dev
A Consistent and Productive work environment
Create, Alter, Drop, Browse and
Filter database objects
ER Diagramming
Object Editors
Object Properties
•
Express yourself with optimal queries
•
•
•
•
•
Content assistance for database objects
Rapid interactive end-user feedback
Extensible templates
Multiple SQL statement testing
SQL assistance and XQuery assistance
Integrated Query Editor
Unleash SQL in your Java IDE
• SQL content assist
• SQL validation
Unleash SQL in your Java IDE
• Run SQL at design time without the need to compile the program
• Launch Visual Explain
Stored Procedure Support
SQL and Java Debugger
Integrated editor and debugger for SQL PL and Java (JDBC or SQLJ)
Data Management
Edit Data
Data Distribution Editor
Roles, Users, Privileges
View data distribution skews
with the Data Distribution Viewer
Privileges
By User
Manage
Roles
Privileges
By Object
IBM Data Studio
No
-ch
arg
e
D
oad
l
n
ow
da
it to
y!
IBM Optim Development Studio
An integrated database development environment
that speeds application design, development, and deployment
while increasing data access performance and manageability.
• Enhance developer productivity
•
•
•
•
Drag and drop creation of Web services
Provide a seamless SQL/Java experience
Generate a data access layer using Java objects, JSON,
or, XML
Enhance problem isolation and impact analysis, even
when using frameworks that generate the SQL
• Provide expert-equivalent performance
•
•
•
Facilitate use of JDBC and SQL data access best
practices
Improve DB2 performance, predictability, and
manageability by enabling transparent activation of static
SQL (i.e. no change to the application) for Java and
.NET
Facilitate DBA collaboration and optimization
• Enhance security
•
Eliminate SQL injection risk
“Development Studio enables us to
bridge the gap between objectoriented design and relational
database technology. As a result,
we can speed the development of
high quality applications and
improve developer productivity by
between 25 and 50 percent”
pureQuery
• How Java and .NET applications can be managed like
CICS/IMS applications
• What problems do these application platforms present to the
DBA?
• ORM frameworks obscure the SQL generated
• Dynamic SQL
• Capturing SQL for review/tune/revisement
• Converting to static SQL and the benefits of doing so
Frequently Cited Concerns
I have more and more Java workload
coming into my data server driving up
costs, but the budget is not keeping pace.
I don’t even want to allow frameworkgenerated SQL on my database. If I can’t see
it, I don’t know how it will impact me.
Java and .NET performance problems are a
real pain to resolve because I can’t even tell
what application issued the SQL.
Contemporary Application Stack
Challenges
• Simplify development, but …
•
•
•
•
Challenge problem resolution
Impact performance
Obscure impact analysis
Impede capacity planning
Application Server
DB Java Driver
Persistence Layer
3
Data Access Logic
2
EJB Query Language
Connection
Pool
Business
Logic
1
Database Server
Network
A
A
A
Introducing pureQuery
A high-performance, data access platform to simplify
developing, managing, securing, and optimizing data access
for new and existing applications.
pureQuery Components:
• Optim Development Studio
• Integrated development environment with Java and SQL support
• Improve problem isolation and impact analysis
• Simple and intuitive API
• Enables SQL access to databases or in-memory Java objects
• Facilitates best practices
• Optim pureQuery Runtime
• Flexible static SQL deployment for DB2
and
pureQuery Improves Performance, Security,
and Manageability for DB2 and
Oracle…Without Changing a Line of Code
Three steps
Static SQL value
1. Capture the SQL
• Make response time predictable
•
•
•
Use pureQuery API, generate from WebSphere
JPA, or capture while executing
Use with custom-developed, framework-based,
or packaged applications
Translation of literals to host variables (new in
2.2)
2. Bind SQL to DB2 (Client Optimization)
•
Use tooling in Data Studio Developer, WAS
console or command line
3. Choose execution mode
•
•
•
Lock in the SQL access path pre-execution
• Limit user access and reduce SQL injection
•
Grant execute privileges on the query packages rather
than access privileges on the table
• Accelerate problem resolution
•
Trace SQL execution to a specific package and the
originating source
• Improve impact analysis and capacity planning
•
Visualize application SQL and correlation metadata
• Increase system capacity
•
Drive down DB cycles
Dynamic or static
Choose at deployment time instead of
development time
"The ability to use static SQL with pureQuery is huge.
Recently, I worked with a client who could reduce CPU
usage by 7 percent thanks to this one feature.“
— David Beulke, Pragmatic Solutions Inc.
Dynamic vs. Static Execution
Dynamic SQL
Full Prepare
Dynamic SQL
Short Prepare
Static SQL
Check auth for package / plan
Check auth for package / plan
Check auth for package / plan
Parse SQL Statement
Copy skeleton from cache to local
DB2 thread storage
Extract access path from
SYSIBM.PACKAGES and
STATEMENTS
Execute SQL statements
Execute SQL statements
Check Table / View Auth
Calculate access path
Store access path in a temporary
package
Execute SQL statement
Static Execution Advantages
Feature
Dynamic SQL
(pureQuery, JDBC)
Static SQL
(pureQuery, SQLJ)
Performance
Can approach static SQL performance
with help from dynamic SQL cache.
Cache misses are costly
All SQL parsing, catalog access, done at
BIND time. Fully optimized during
execution.
Access path reliability
Unpredictable – Any prepare can get a
new access path as statistics or host
variables change
Guaranteed – locked in at BIND time All
SQL available ahead of time for analysis
by EXPLAIN.
Authorization
Privileges handled at object level. All
users or groups must have direct table
privileges – Security exposure, and
administrative burden
Privileges are package based. Only
administrator needs table access.
Users/Groups have execute authority.
Prevent non-authorized SQL execution.
Monitoring, Problem
determination
Database View is of the JDBC or CLI
package – No easy distinction of where
any SQL statement came from.
Package View of applications makes it
simple to track back to the SQL statement
location in the application
Capacity planning,
Forecasting
Difficult to summarize performance
data at program level.
Package Level Accounting gives program
view of workload to aid accurate
forecasting.
Tracking dependent
objects
No record of which objects are
referenced by a compiled SQL
statement
Object dependencies registered in
database catalog
Improving Throughput with pureQuery, a
z/OS Example
• In-house testing shows over 40%reduction in CPU costs over dynamic JDBC
using pureQuery and DB2 for z/OS (type-2)
• Read IBM Optim pureQuery Runtime for z/OS Performance
• IRWW – an OLTP workload, cache hit ratio between 70 and 85%
Preset application path never
varies at runtime
Reduced CPU per transaction means
more growth on existing resources
Java Database Access and pureQuery
Many on-ramps for new and existing
applications
Web
services
Java Persistence
API (JPA)
JPA API
pureQuery
DB2 (z/OS, i & LUW), Informix, and Oracle now
More coming
Deploying with pureQuery Runtime
Application
pureQuery Runtime
Application tier:
z/OS, Linux, UNIX, Windows
JDBC Driver
Database tier:
Static or dynamic
execution
DB2 for z/OS
DB2 for i
DB2 for Linux, UNIX, and Windows
Informix Dynamic Server
Oracle
Unique Package Names Improves PD on
DB2 for z/OS
• Most dynamic Java applications use packages SYSLNx00 making it
hard to identify specific programs
• Unique package names link SQL to Java Beans, similar to CICS
transaction names to programs.
Static
pureQuery
Java SQL
Dynamic
Java SQL
Static SQL for security – Administering
table privileges
• Dynamic SQL
• Table privileges granted directly to users(groups)/secondary authids
• Security exposure and administrative burden
GRANT SELECT
ON TABLE PAYROLL
TO DEPT_D47
NAME
• Users get no table privileges
SELECT NAME,
POSITION FROM
PAYROLL…
POSITION
SALARY
…
GRANT SELECT
ON PAYROLL
TO BIND_ADMIN
• Static SQL (pureQuery)
GRANT EXECUTE
ON PACKAGE
POSITION_REPORT
TO DEPT_D47
PACKAGE
PAYROLL
PAYROLL
BIND
NAME
POSITION
SALARY
…
Reduce Costs with zIIP and zAAP
• Using COBOL or SQL/PL stored procedures instead of executing SQL directly
• Separation of application development from data access
• DBAs retain better control over SQL including static execution
• If not written in DB2 9 for z/OS's Native SQL/PL, the stored procedure
must use general purpose processors
• pureQuery introduces alternatives
• Create Java stored procedures to run on zAAP using the pureQuery
runtime
• DBAs retain control, Data Studio helps with development, pureQuery executes
statically
• Execute SQL directly from Java application or method to run on zIIP
• Developers use Data Studio Developer to generate access layer with pureQuery,
content assist helps with best practices and SQL validation, packages SQL for easy
collaboration with DBA, pureQuery executes statically
More Visibility and Control of Application
SQL
•
•
•
•
•
Capture SQL
Share, review, and optimize SQL
Revise and validate equivalency
Bind for static execution or run dynamically
Restrict SQL to eliminate SQL injection
Capture
Review
Optimize
Revise
Restrict
Capture Application SQL: At
Development or Later
Capture
Review
Optimize
Three methods
1. Use pureQuery API
2. Use JPA and generate the
pureQuery file
3. Set captureMode=ON and
execute the program
IBM Optim
Development
Studio
Revise
Restrict
Visualize Application and SQL Metadata
Capture
Review
Optimize
Revise
Restrict
• Review the captured SQL
• View metrics about execution frequency and duration
• Share captured SQL with DBA
IBM Optim
Development
Studio
SQL Outline
Speed up problem isolation for developers –
even when using frameworks
• Capture application-SQL-data
object correlation (with or without
the source code)
• Trace SQL statements to using
code for faster problem isolation
• Enhance impact analysis
identifying application code
impacted due to database
changes
• Answer “Where used” questions
like “Where is this table used
within the application?”
• Use with modern Java
frameworks e.g. Hibernate,
Spring, iBatis, OpenJPA
IBM Optim
Development
Studio
Optimize SQL
Capture
Review
Optimize
Revise
Restrict
• Launch Visual Explain
• Copy SQL to Optim Query Workload Tuner
Query Tuning
Introduction to query tuning
Query performance causes application outage
• Customer example
• Application service drops from 3 seconds to 5 minutes
• Limited performance management expertise
• Took 3 days to identify the problem query
• 5 day application outage
• Loss of confidence in IT team
• Time and money in analysts
• Expert analysis revealed
• Relevant statistics were not being collected
• Wasted CPU resources due to default statistics collection
• Index design was very poor
• Rapid development practices drove untuned queries into production
s
e
c
n
r
ss
a
g
u
e
t
i
o
c
a
s
s
o
de
re
rd
pr
o
r
r
d
o
o
e
o
t
o
P
P
Po
W as
Introduction to query tuning
Why is query tuning such a pain?
• Development
• Lack of query tuning skills
• Use of frameworks that generate SQL
• Inadequate testing environments to drive the workload and data
scale
• Disconnect between application and data groups
• Production
•
•
•
•
Performance problems appear without warning
Aggregate information from multiple places
Complexity of analysis requires specialized skills
Need to consider entire workloads, not just single queries
Introduction to query tuning
• Why do we have to perform query and system tuning? Isn’t this why
I have DB2?
• DB2 is powerful, but it’s not all-knowing
•
•
•
•
DB2 doesn’t know what queries are going to be submitted
DB2 doesn’t know object sizes or cardinality unless statistics are gathered
DB2 doesn’t know about cross-table correlation
DB2 doesn’t know how many rows will be fetched, unless told
• The biggest reason for a non-optimal access path is because the
desired statistics are not collected
• Defining the task "query tuning" -- Developer versus DBA tasks
• Developer tasks
• Follow shop standards
• Avoid Stage 2 (and Stage 3) predicates
• DBA tasks
• Design indexing scheme
• Collect and maintain needed statistics
• Analyze access paths and influence them when needed
Introduction to query tuning
• Techniques used
• Rules-based analysis of SQL statement text
• Invoking EXPLAIN to understand the access path and cost
of the query blocks
• Invoking EXPLAIN to understand what statistics are
interesting to DB2
• Simulation of index schemes
• Other issues
• Sometimes the SQL is generated and cannot be reviewed
IBM Tooling that is available to assist
with this
• IBM Tooling that is available to assist with this
• DB2 SQL Performance Analyzer (SQL PA)
• Optim Query Workload Tuner
SQL Performance Analyzer
•
•
ISPF and batch
Enhanced Explain
•
•
•
•
•
Compare access paths
What if
Retro explain
Migrate DB2 Statistics
Provides advice on each SQL statement (helps users write better SQL)
• Warnings, alerts, guidelines and recommendations
• Performance notes
•
•
Forecasts SQL performance (response/wait times, CPU, I/O counts)
Forecasts the cost of the query, in terms of
• Charge back (monetary, in national currency)
• QUNITS (query service units)
•
Acts as a governor for Static or Dynamic SQL
• in QMF, as a preemptive exit
• in DRDA and IMS or CICS, via a Stored Procedure call
Suitable for SQL Develop, Tuning and Control
Query Workload Tuner
• Eclipse-based interface
• Gathers SQL from various sources (catalog, statement cache,
captured file)
• Visualize queries and costs (query annotation and visual explain)
• View DB2 Optimizer’s query transformations
• Analyze one query or a whole workload
• Access path
• Statistics
• Index
• Validate changes
Suitable for SQL Tuning, Statistics Collection and Index Design by DBAs
Streamlined Analysis
Define or select workload
Execute advisors
Drill into advice
Validate improvement
Gather High Cost Queries and
Workloads
DB2
Plan table
Statement table
Catalog plan or package
Statement cache
Optim Development Studio
QMF and QMF HPO
DB2 Query Monitor
OMEGAMON XE for DB2
File, text, or exported
workload
User defined category
Define or select workload
Visualize Queries and Costs to Speed
Analysis
Easily see tables,
sections, join
predicates, etc.
• Accelerate analysis, reduce downtime
• Spot human errors
• Identify where filtering should occur
Examine table
statistics and
additional information
View Optimizer Transformations
• Accelerate analysis, reduce downtime
• Spot transformations which will occur
Execute Advisors
• Statistics
• Get recommendations on
the best statistics to
capture to influence
access path selection
• Query
• Get recommendations
regarding how to rewrite
the query for better
efficiency
• Index
• Get recommendations on
additional indexes that can
reduce database scans
Analyze Access Plans
• Visualize access path
• See flow of query processing
• See indexes and operations
• See optimizer rationale
• Assess access path stability to reduce risk of performance regression
• Is the optimizer able to apply the filtering early?
• Are there indexes that support an efficient path?
• Do statistics allow distinction between the choices?
Improve Statistics Quality and Collection
• Provides advice on
• Missing statistics
• Conflicting statistics
• Out-of-date statistics
Generates
RUNSTATS control
statements
• Results
•
•
•
•
Accurate estimated costs
Better query performance
Less CPU consumption
Improved maintenance
window throughput
“Half of access path PMRs could
be resolved by statistics advisor
before calling IBM support.”
– IBM Support
Indicates
conflicting and
missing statistics
Conflicting
statistics
explanation
Improve Query Design
Guard against errors and oversights:
Further constrain query, increase index
utilization, and reduce data reads
View analysis summary
Highlights relevant
components of the
query
Filter recommendations
by severity
Query Advisor checks for
• Missing join predicate for referential
constraint
• Predicates that can be rewritten as
indexable
• Stage 2 predicates that can be
rewritten as stage 1 predicates
Recommendation
and rationale
Indexing Advice to Improve Query
Efficiency
• Improve query efficiency
• Indexing foreign keys in queries
that do not have indexes defined
• Identifying index filtering and
screening
• Support for index only access
• Indexing to avoid sorts
• Simplify use
• Consolidate indexes and provide
a single recommendation
• Enables what-if analysis
• Provides DDL to create indexes
• Run immediately or save
• Test before deployment
• Utilize virtual index capabilities
built into the DB2 engine
Visual Plan Hints for Experienced DBAs
• Why hints?
• Version to version migration
• Assumptions made by
optimizer when SQL contains
parameter markers or host
variables
• Reduce errors
• Hint editor
• Hint validation
• Hint deployment
Query tuning tools for z/OS – Unique
benefits
Optim Query Workload Tuner
DB2 SQL Performance Analyzer
• Better support for modern dev
environment and dynamic SQL
• Better support for classic z/OS env,
DBRMs, batch analysis, and z/OS
libraries
• Eclipse-based, GUI and stored
procedure-driven
• ISPF-based, Command and JCL- driven
• Integrates with OM/PE and QM GUI
interfaces, Data Studio, Optim
• Integrates with OM/PE VTAM and QM
3270 interfaces
• Provides virtual index capability for
single queries and across workloads
• Creates real indexes for “WHAT-IF”
scenarios for single queries
• Recommends rewriting queries only
when improvements result
• Has limited support to create statistics
in Query Environment Capture and
Workload Environment Capture service
support
• Strategic investment spans databases
• Offers a best practices Query Advisor
with approx. 150 rules
• Has full support for cloning statistics
• Continued z/OS-specific investment
Optimization Service Center (the fate of)
• Current OSC remains supported through DB2 9
• OSC functionality will be split among the following
offerings:
• Data Studio (no charge)
• Query Tuner (single query tuning capabilities already overlap
with Q[W]T today)
• Base function in DB2 for z/OS customers
• SQL environment capture
• Profile monitoring
Contents of Eclipse-based Query Tuning offerings
Data Studio
Optim Query Tuner
for z/OS
Optim Query Workload
Tuner for z/OS
Queries from all sources
•
•
•
Reports
•
•
•
Query Formatter
•
•
•
Access Plan Graph
•
•
•
Query Statistics Advisor
•
•
•
Query Annotation
•
•
Visual Plan Hint
•
•
Query Index Advisor
•
•
Query Advisor
•
•
Access Path Advisor
•
•
Workload Statistics Advisor
•
Workload Index Advisor
•
Workload Query Advisor
•
End of query tuning
Revise SQL Without Modifying the
Application
Capture
Review
Optimize
Revise
Restrict
Edit or delete SQL
Tooling validates equivalent SQL
Shows replaced
Eliminate SQL Injection
Capture
Review
Restrict SQL execution to
only those statements
captured
Set capturedOnly=true in
pdq.properties
pureQuery Runtime looks
for it it in the classpath
Optimize
Revise
Restrict
Optimize for WebSphere and DB2 with
pureQuery
Capture metadata from existing
applications
Jump start application design
• Generate SQL and Code from Database Objects
• Setup basic DAO Pattern
• Capture from JPA without executing
• Derive performance, costs, security and
manageability value
Enhance development productivity
Design
• Code generation, content assist
• Database aware, Java SQL Editor
Reduce HW and SW costs
• Up to 42% lower CPU/Trans
• Move workload to zIIP and zAAP
Develop
Optimize
Policies
Models
Metadata
Replace SQL without
changing the source
• Editor validates equivalency
Simplify impact analysis
• Categorize by Java, SQL, Database,
Packages, track back to line of code
Focus tuning efforts
• Find and sort by query elapsed
time from Java
Speed up problem resolution
Operate
• Trace SQL back to line of code in the
application
Deploy
Enhance performance
Prevent SQL Injection
• Lock down SQL for dynamic or static
execution
• Leverage best practices, automatically for JPA
• Use static execution, automatically for JPA
• Lock in access plans for consistent performance
Reduce security exposure
• Grant access to queries, not tables
Optim Development Studio and
pureQuery Runtime
D
oad
l
n
ow
y!
a
d
it to
Data movement options
•
UNLOAD Utility
Designed for loading back into
DB2 for z/OS
Backup
tablespace
SELECT
FROM
UNLOAD utility
•
Crossloader (LOAD from INCURSOR DD)
• Ultra fast loading of data when
the source is in a relational
table – even remote
File(s)
LOAD/UNLOAD via pipes
Perfect for remote
loading/unloading
tablespace
Local
or rem
ot e
WHERE…
LOAD utility
High Performance Unload (z/OS and LUW)
• Ideal data moving to another platform
• SQL interface enables developer usage
• Low CPU consumption
UNLOAD utility
or
Pipe
LOAD utility
tablespace
Backup
tablespace
HPU
“select c2, char c3
where…”
File(s)
Recovery Tooling
• What value is there to extending the DB2 backup and
recovery utilities?
• Safety net for a complex task
• Ability to view and make surgical changes using the DB2
recovery log
• Allows you to use DB2 9 function on a DB2 Version 8
system
Recovery management
•
•
•
•
DB2 Change Accumulation Tool
DB2 Log Analysis Tool
DB2 Object Restore Tool
DB2 Recovery Expert
• DB2 Cloning Tool
DB2 Recovery Expert for z/OS
• Wizard driven
• Object selection (Database, Table space, Table,…… Multiple objects
supported)
• Point in time selection (Time, Quiesce
point, Backup point, Log point)
• Recovery Paths / alternate recovery resources
• Analyzes all possible recovery paths for a version and attaches a relative cost to
each
• Recommends the least cost recovery path while allowing other paths to be chosen
• Recovery paths include
• Traditional restore and log apply (forward recovery)
• Generating undo operations (backward recovery)
• Recommend set of objects to recover
• Prompts the user to show related objects (next slide)
• Assists in including these objects in the recovery process
• Manage recovery via policies: Recovery Policy – Insure that I can recover an
application (set of DBMS objects) within twenty minutes to any point in time in the
last week. {Supersedes the need for a backup policy}
Data Governance
•
•
•
•
•
•
IBM Optim Data Growth
IBM Optim Data Privacy
IBM Optim Test Data Management
DB2 Audit Management Expert
IBM Database Encryption Expert
Data Encryption for DB2 and IMS
Create Production-like, Privatized Databases
Optim Test Data Manager and Data Privacy Solutions
• Increase software quality
• Use test data that accurately reflects
production systems
Design
• Accelerate release schedules
Develop
Optimize
Policies
Models
• Comparison to expected results and fast
refresh
Metadata
Tester
Operate
Deploy
• Comply with privacy regulations
• De-identify personal information with
realistic substitutions
• Protect confidential information
• Mask sensitive data
• Facilitate consistent use
Optim Test Data Management and Data Privacy Solutions
Increase software quality and comply with regulatory requirements
based on test data that accurately reflects production systems while
protecting client privacy and sensitive information.
CUSTOMERS
-- ---- ---- ---- ------- ----- ---- ---- ---- ------- ----
Production or
Clone
ORDERS
-- -- ------ -- --------- ----- -- ------ -- --------- ----- -- ------ -- --------- ----- -- ------ -- --------- ----
Extract
Extracted
Files
DETAILS
------
•
----------------
----------------
----------------
-------------------------------
----------------
Create targeted, right-sized test databases
Mask
Masked
Files
Load
Insert / Update
Compare
• Refresh, reset, and maintain test environments – quickly and easily
• Extract complete, referentially intact business objects
• Compare data to pinpoint and resolve application defects faster
•
Development
De-identify sensitive information
• Substitute confidential information with fictionalized data
• Protect private data in non-production environments
Test
• Provide contextual, application-aware, persistent data masking
•
Facilitate consistent use
• Define practices in InfoSphere Data Architect
• Use repeatedly across environments
Anonymous
Analysis
Optimize Systems for Growth
Optim Data Growth Solution
• Improve application performance
Application
Manager
• Accelerate application upgrades
Design
Optimize
Archive,
Optimize
upgrade,
retire
• Reduce the risk of application
consolidation and retirement
• Comply with retention requirements
• Find information for legal discovery
Operate
Develop
Models
Policies
Metadata
Deploy
Optim Data Growth Solution
Facilitates application growth, application upgrades, application
retirement, and retention management.
Freed
Historical
Retrieved
Archive
Archives
Historical
Data
Reference
Data
Retrieve
Reporting Data
Active
Universal Access to Application Data
Application
Application
XML
ODBC / JDBC
• Complete business object provides historical reference snapshot of business activity
• Storage device independence enables storage cost optimization
• Immutable file format enables data retention compliance
What's cooking back at the lab?
• Performance Management
The (not so new) differences between
modern distributed apps and legacy
Cobol and PL/I apps
Legacy apps
Distributed Java apps
Mostly static SQL. Each program module was mapped to one
DBRM under usually one DB2 PLAN.
Primarily dynamic SQL. No DBRMs or DB2 PLANs.
1
-> Each app module automatically identified by auth ID,
DBRM, and DB2 PLAN, and optionally CICS or IMS region
-> Program had to identify itself via DB2 client info fields.
Otherwise it would use default ids such as java.exe.
Program was running on mainframe only, no distributed
access. Very small number of apps accessing database.
Distributed apps can access database from anywhere. A ‚new
app‘ is developed in short period of time.
2
-> DBA knew each app personally ☺
-> Increasing number of different apps accessing database.
Not easy to keep an overview which app is running where.
DB2 pre-processor was well integrated into development
languages
No pre-processor used. Program simply calls JDBC methods,
or CLI functions to execute SQL statements
-> each SQL statement was identified by source line number
-> No information about location of SQL statement
Developer was coding SQL directly
Program is generating the statement text on-the-fly by
concatenating strings, or frameworks and persistence layers
like iBatis, Spring, EJB, JPA are generating the SQL
statements out of programming language constructs
-> easy to find and change SQL statement if necessary
-> can be extremely difficult to identify origin of SQL
statement. Changing SQL statement text can be impossible if
you can‘t control the generation.
3
So, what is it what we are looking for?
#1
#3
#2
Something ...
#1
#2
#3
...which tells me which application it is, and ultimatively which
business function is performing
...where I can measure what my application/user is really
experiencing (from application to database and back) and which
tells me which components are involved and where my application
is spending its time.
...which tells me where the SQL statement is coded and let‘s me
change it if necessary.
Performance Expert Extended Insight
Feature
• IBM offers a feature for its database monitor DB2 Performance Expert
for LUW called ‚Extended Insight‘ to collect exactly this information
automatically ...
• Supports standard applications (WebSphere, InfoSphere Warehouse, SAP,
DataStage, Cognos, eXtremeScale) and homegrown Java applications
setting the DB2 client information fields
• Support for static and dynamic SQL
• Works with DB2 for LUW V9.1 or higher. DB2 for LUW V9.7 will give you
additional insight into database side – where your application is spending
its time
• Easy deployment: only a few files have to be copied into location of DB2
driver on application side; no configuration on client side necessary.
• Collects statement and transaction level information with low overhead on
client side (<= 5% overhead)
• Allows to check for Service Level Agreement (SLA) violations
Database Workload Overview - Extended
Insight Feature
End-to-end database monitoring
•
Check your SLAs
•
•
•
•
•
... the most advanced way to
monitor your applications and
solution accessing data in a
database.
introduced in DB2 Performance
Expert V3.2 in 12/2008 as priced
feature with focus on WebSphere
applications
now enhanced to support out-ofthe-box: WebSphere, Cognos,
DataStage, SQL Warehouse, SAP
will also be used to monitor other
applications using JCC or CLI
database drivers (.NET support will
follow)
intuitive to use because looking at
the applications/solutions behind it
and not into DB2 resources or DB
connections
will auto-detect the application
behind the DB connection
Database Workload Details - Extended
Insight Feature
showing you the real response time of your
application and where it is spending its time
including database side – new with DB2 V9.7 FP1
seemlessly integrated with Optim Query
Tuner; identify problematic SQL and tune it
Tell me where my SQL is and let me control it
Or why JPA and other frameworks/persistency layers are then a nightmare
for a DBA
• SQL statement are not coded directly and generated on-the-fly during execution
@Entity
@Table(name="ORDER_TABLE")
public class Order {
private int id;
private String address;
private Customer customer;
@Column(name="SHIPPING_ADDRESS")
public String getAddress() {
return address;
}
public void setAddress(String address) {
this.address = address;
}
@Id
@Column(name="ORDER_ID")
public int getId() {
return id;
}
@ManyToOne()
@JoinColumn(name="CUSTOMER_ID")
public Customer getCustomer() {
return customer;
}
public void setId(int id) {
this.id = id;
}
Java annotations
define OR mapping,
so that Entity
Manager
can generate SQL
public void setCustomer(Customer customer) {
this.customer = customer;
}
}
• The developer can code additional queries – in SQL but also in its own EJB Query
Language, which is once again ‘translated’ into SQL statements on the fly
orm.xml:
Query query = em.createNamedQuery("SoldOut");
List results = query.getResultList();
<entity class=“Order">
...
<named-query name="SoldOut">
<![CDATA[ SELECT p FROM Order p
WHERE p.status == "Sold Out" ]]>
</named-query>
</entity>
How can you fix a problematic SQL statement if you can’t even find it?
Tell me where my SQL is and let me
control it How pureQuery runtime can help
pureQuery runtime allows to
capture SQL statements and convert
them into static SQL
Includes information about origin
of statement (class, method, java
package, EJB query, etc.) and stack
trace
Information can be uploaded into
performance monitor when
application SQL is converted
Works with all Java applications;
transparent to app
Database Performance Monitoring +
Tivoli Enterprise portal
• Can run stand alone, or inside Tivoli Enterprise Portal
• Integrated into ITCAM for Transactions; high application response
times can be directly analyzed with database performance monitor
Tivoli Enterprise Portal
transaction topology
How to prepare yourself for
the future …
• New functionality will not replace existing Optimization Expert,
Query Monitor, or OMEGAMON XE products, but be an addon it them …
• It will most likely have a pre-requisite for OMEGAMON XE
Performance Exert for DB2 on z/OS (OMPE)
• Long-term: It might consolidate OMPEs graphical user
interfaces into one
DB2 on z/OS, DB2 on LUW,
Oracle, IDS
DB2 on zOS monitoring
zOS DBA
Classic Interface
Solution monitoring
Data server monitoring &
administration
Operator,
Subject matter
expert (SMEs)
DBA for
DB2 on zOS,
DB2 on LUW,
IDS, ORACLE
Tivoli Enterprise Portal
Optim family
(with db monitor plugged in)
(OPM, ODDM, OCC)
What's cooking back at the lab ?
• Configuration Management
• Speeding up problem determination by analyzing what has
changed
• Simplifying implementation of and checking adherence to
best practices
• Centralizing control of client configuration
• Maintenance Management
• Managing large, complex enterprise environments by
exception
• Shifting from time-based maintenance via utilities to policybased maintenance
• Visualization of space management
The problem …
Our solution!
Manage by Exception to Lower TCO
Enable Fully-manual to Fully-automated
Management
Manage by Exception to Lower TCO
IBM Integrated
Data
Management
• Getting started
• Downloads
• IBM Data Studio
• IBM Optim
• Community
• developerWorks
• forums
• zone and space
• articles
• tutorials
• Documentation
• Support
Data Management Communities for DB2
• IDUG – the worldwide community of DB2 users
• Membership is FREE – join today! www.idug.org
• Data Management Community – share and interact with peers
around the world
IOD 2009
• www.ibm.com/software/data/management/community.html
Content Planning
• Information Champions – recognizes individuals who have
made the most outstanding contributions to the Information
Management community
• www.ibm.com/software/data/champion
Disclaimer
© Copyright IBM Corporation [current year]. All rights reserved.
U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule
Contract with IBM Corp.
THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES
ONLY. WHILE EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE
INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED “AS IS” WITHOUT WARRANTY OF
ANY KIND, EXPRESS OR IMPLIED. IN ADDITION, THIS INFORMATION IS BASED ON IBM’S CURRENT
PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM WITHOUT NOTICE. IBM
SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE
RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION. NOTHING CONTAINED IN THIS
PRESENTATION IS INTENDED TO, NOR SHALL HAVE THE EFFECT OF, CREATING ANY WARRANTIES OR
REPRESENTATIONS FROM IBM (OR ITS SUPPLIERS OR LICENSORS), OR ALTERING THE TERMS AND
CONDITIONS OF ANY AGREEMENT OR LICENSE GOVERNING THE USE OF IBM PRODUCTS AND/OR
SOFTWARE.
IBM, the IBM logo, ibm.com, DB2, Optim, Tivoli, Rocket Software, and Data Studio are trademarks or registered trademarks
of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM
trademarked terms are marked on their first occurrence in this information with a trademark symbol (® or ™), these symbols
indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such
trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is
available on the Web at “Copyright and trademark information” at www.ibm.com/legal/copytrade.shtml
Other company, product, or service names may be trademarks or service marks of others.