Oracle® Fail Safe
Release Notes
Release 3.4.1 for Microsoft Windows (32-Bit)
E10719-01
September 2007
This document describes the new features in this release, software errors fixed,
software compatibility, hardware compatibility, notes about installation and
deinstallation, and notes on configuring resources for high availability.
1 How These Notes Are Organized
The remainder of these release notes are divided into the following sections:
■
"New and Changed Features" on page 1
■
"Software Errors Fixed" on page 3
■
"Software Compatibility" on page 6
■
"Hardware Compatibility" on page 6
■
"Installation and Deinstallation" on page 7
■
"Oracle Services for MSCS" on page 8
■
"Oracle Fail Safe Manager" on page 9
■
"Oracle Database" on page 9
■
"Disk Resources" on page 10
■
"Virtual Addresses" on page 10
■
"Oracle Enterprise Manager Integration" on page 10
■
"Documentation Updated for This Release" on page 11
■
"Additional Information About Oracle Fail Safe" on page 11
■
"Documentation Errors and Omissions" on page 12
■
"Documentation Accessibility" on page 22
2 New and Changed Features
This release of Oracle Fail Safe provides the new and changed features described in the
following sections.
2.1 Support for Oracle Database 11g
This release introduces support for Oracle Database 11g, excluding the Management
Agent. See "Software Compatibility" for details about the supported version of each of
these resources.
1
2.2 Sample Database Seed Files No Longer Included in Installation
In prior releases of Oracle Fail Safe a database seed (.DFB) file was included in the Fail
Safe installation kit for each supported release of Oracle Database. Those files are no
longer included. When a sample database is created by Oracle Fail Safe the database
seed file in the Database Configuration Assistant template directory is used.
For example, in previous releases of Oracle Fail Safe, when creating a sample database
for Oracle Database 10g Release 2, the file Oracle_Fail_Safe_Sample_102.dfb
would be copied to the Database Configuration Assistant templates directory,
ORACLE_HOME\assistants\dbca\templates, and Database Configuration
Assistant would be invoked to create the sample database. With this release the
template file provided by the Oracle Database installation, Seed_Database.dfb, is
used instead.
Using the Database Configuration Assistant template seed file ensures that the
template file is compatible with the installed version of Oracle Database.
2.3 New Surrogate Process Trace Log
In prior releases, when the Oracle Fail Safe trace facility was enabled, the Fail Safe
Server (FSS) trace file would sometimes get overwritten when remote operations were
invoked from other nodes.
Section B.3, "Tracing Oracle Fail Safe Problems" in Oracle
Fail Safe Concepts and Administration Guide for a description of the trace
facility
See Also:
This problem would occur because two different processes were using the same trace
file definition, FSS_TRACE_OUTPUT. The Fail Safe Server (FsSvr.exe) is the
intended user of the log file, but the file would also be opened and written to by
ephemeral Fail Safe surrogate processes (FsSurrogate.exe) that would be invoked
by the Microsoft DCOM subsystem. When those processes were invoked they would
create a new copy of the trace file, deleting the contents created by the main Fail Safe
Server.
In this release of Oracle Fail Safe, a new registry entry has been created that enables
the surrogate process to have its own trace file specification, eliminating the chance
that the surrogate would overwrite trace files. Refer to Section 15.11, "New Surrogate
Process Trace Log," for more information on how to utilize this new feature.
2.4 Support for Oracle Management Agent
Release 3.3.4 introduces support for Oracle Management Agent. It continues to
provide support for the following Oracle resources:
■
Oracle Database
■
Oracle Intelligent Agent (release 9.2 and earlier)
See "Software Compatibility" for details about the supported version of each of these
resources. See "Configuring Oracle Management Agent for High Availability" for
details on configuring Oracle Management Agent for high availability.
2
2.5 Support for Oracle Application Server Components
Prior to release 3.3.4, to configure Oracle Application Server components for high
availability, you had to configure them as generic services. With this release, custom
support for configuring Oracle Application Server components is provided. The
following components are included:
■
Oracle Process Manager and Notification Server (OPMN)
■
Application Server Control service
■
The metadata repository, if it is in the Oracle home where Oracle Application
Server was installed
See "Configuring Oracle Application Server Components for High Availability" for
details on configuring Oracle Application Server components for high availability.
2.6 Oracle HTTP No Longer Supported
Unlike previous releases, this release of Oracle Fail Safe does not include support for
configuring Oracle HTTP Server for high availability.
3 Software Errors Fixed
This section describes software errors that have been fixed in this release of Oracle Fail
Safe.
3.1 Oracle Services for MSCS Security Setup Fails On Non-English
Systems
On systems that do not have English as the default language, if the Microsoft local
culture definition did not recognize the string "Self" as representing the current user,
then the Oracle Services for MSCS Security Setup tool (FsSecurity.exe) would fail
to correctly update the DCOM security settings on the system. When this problem
was encountered, a message would be entered into the Windows Application event
log with the text:
Failed to look up user account SELF with error: 1332
Unable to add SELF to DCOM access ACL.
The problem was due to the FsSecurity.exe application attempting to use the
string "Self" for the generic self user ID. That string may not be meaningful on
non-English cultures.
This problem can be avoided by temporarily removing the DefaultAccessPermission
ACL from the windows registry (HKEY_LOCAL_
MACHINE\SOFTWARE\Microsoft\Ole), running the MSCS Security Setup tool, and
then adding back the DefaultAccessPermission ACL. This operation must only be
executed by system administrators that understand the risks of manually editing the
Windows registry and know how to backup and recover it.
FsSecurity.exe has been changed to build the security identifier (SID) in binary
form using predefined constants rather than attempting to use the string "Self". This
change allows FsSecurity.exe to be culture independent when adding SIDs to the
DCOM DefaultAccessPermission ACL.
3
3.2 Standby Database Won't Stay Online
A database that was configured as a Data Guard standby database would not stay
online when brought online, instead, it would cycle continuously between the online
and offline states.
The problem was due to Oracle Fail Safe not properly noting the role and open mode
of the database. Different database queries are utilized for Is Alive processing,
depending on the database role (primary or standby) and open mode (mounted or
read only). Fail Safe was not always issuing the correct query when polling the
standby database, which resulted in unexpected results from the query and thus a
forced shutdown of the database.
The Fail Safe Is Alive query code has been corrected to properly note the database role
and always use the correct query when doing Is Alive polling.
3.3 Verify Cluster Fails with No Error Message Reported
Executing a Verify Cluster would result in the command terminating before
completing with no specific error message. For example, the output from the
command may look like the following:
FS-10660: NODE1 : Gathering cluster information
The clusterwide operation failed !
A pop-up box would be created that contained no text; only an "X" error image and an
"OK" button. No errors would be reported in the Windows application event log and
Oracle Fail Safe trace logs would show no errors.
This problem was caused by incorrect DCOM security settings; the remote node could
not establish a communication link back to the node running the Oracle Fail Safe
server and thus could not report any errors to the server and client. Correcting the
DCOM Access Permissions using the Microsoft Component Services administrative
tool resolved the problem.
To determine if there are problems with DCOM follow the instructions in this
Microsoft Knowledge Base article:
http://support.microsoft.com/kb/892500/en-us
Oracle Fail Safe has been enhanced to log DCOM errors in the Windows application
event log and Oracle Fail Safe trace log.
3.4 Verify Cluster Fails After Installing 3.3.4 and then Re-Installing 3.3.3
If Oracle Fail Safe Release 3.3.4 was installed, then deinstalled using the Oracle
Universal Installer, and a previous release, such as 3.3.3 was installed, a Verify Cluster
operation would fail with errors similar to the following:
FS-10665: Checking DLLs for resource provider
** WARNING : FS-10669: The resource provider DLL
D:\ORACLE\OFS333\FS\FSSVR\BIN\ was not found on node NUMBERONE
This problem occurred because the deinstallation neglected to remove new registry
entries that were introduced in release 3.3.4.
This problem can be resolved by removing the following registry entries from the
Windows registry key \HKEY_CLASSES_ROOT\OracleFailSafe\Resources:
■
Oracle10gAgent
4
■
·OracleAS
The product deinstallation procedure for release 3.3.4 has been corrected and will now
properly remove those registry keys entries. Also, the Verify Cluster command has
been changed to ignore unrecognized resource names in the registry.
3.5 Fail Safe Installation Corrupts DCOM ACLs in Microsoft Windows
Server 2003 R2
If a system had an explicit definition for the DCOM (OLE) DefaultAccessPermission
ACL, when the "Oracle Services for MSCS Security Setup" tool (FsSecurity.exe)
was executed DCOM applications could fail to open windows. The system event log
would contain error messages that said, "Invalid value for registry".
This problem was due to a new ACL format being introduced in Microsoft Windows
Server 2003 R2, which was incompatible with the format being generated by
FsSecurity.exe. The problem is described in this Microsoft article:
http://msdn2.microsoft.com/en-us/library/ms693364.aspx
This problem can be avoided by temporarily removing the DefaultAccessPermission
ACL from the windows registry (HKEY_LOCAL_
MACHINE\SOFTWARE\Microsoft\Ole), running the MSCS Security Setup tool, and
then adding back the DefaultAccessPermission ACL. This operation should only be
executed by system administrators that understand the risks of manually editing the
Windows registry and know how to backup and recover it.
FsSecurity.exe was adding an ACE for user SELF using the old style access rights
but the existing ACL contained an ACE that had the new style. In this release, ACLs
created by FsSecurity.exe will use the new, post Windows Server 2003 SP2 format.
3.6 Various Errors When Verifying Standalone Database
When attempting to verify a standalone database Oracle Fail Safe could return various
Fail Safe errors, such as: 10916, 10341,10342, 10496, 10491, 10343, 10347, 10999, 10989,
10795, 10890. When examing the Windows application event log the entries would
typically mention module FsDdbsUpi.c. For example:
The request is aborted due to internal error when calling OpenOdbsHst() in module
.\FsDdbsUpi.c, line 5932.
These errors were more prevalent on 64-bit platforms, but could also occur on 32-bit
platforms. There is no workaround for these problems. A number of corrections have
been made to the database interface module, FsDdbsUpi.c, to prevent these errors
from occurring.
3.7 Various Errors When Adding a Resource to a Group
Attempts to add a resource to a group could fail with various errors, often with a Fail
Safe 10999 error displayed and "unhandled exception" errors listed in the Windows
application event log and the Oracle Fail Safe trace files. For example, the Fail Safe
Manager may display a message similar to the following:
FS-10427: Creating database instance ORCL for Oracle Net service name ORCL
** ERROR : FS-10999: An internal programming error has occurred
** ERROR : FS-10989: The resource provider Oracle Database raised an unhandled
exception
** ERROR : FS-10778: The Oracle Database resource provider failed to configure the
5
cluster resource ORCL
** ERROR : FS-10890: Oracle Services for MSCS failed during the add operation
These errors were caused by incorrect argument passing and insufficient error
handling in the database interface module, FsDdbsUpi.c. There is no workaround
for these problems. The errors have been corrected in this release.
4 Software Compatibility
Oracle Services for MSCS must be installed on Microsoft Windows 2000 Advanced
Server or Datacenter Server. When that condition is met, then:
■
■
Oracle Fail Safe Manager is compatible with the following operating systems:
-
Microsoft Windows 2000
-
Microsoft Windows XP
-
Microsoft Windows Server 2003
-
Microsoft Windows Server 2003 R2
Oracle Fail Safe Server is compatible with the software listed in the following
table:
Software
Release or Version
Microsoft Cluster Server
(MSCS)
Version 1 or later
Oracle Database (Standard
and Enterprise editions)
Oracle9i Release 2 (9.2)
Oracle Database 10g Release 1 (10.1)
Oracle Database 10g Release 2 (10.2)
Oracle Database 11g Release 1 (11.1)
Oracle Intelligent Agent
Release 9.2.0
Oracle Management Agent
Release 10.1.0.2
Release 10.1.0.3
(A Management Agent release for Microsoft Windows only.)
Oracle Enterprise Manager
Release 9.2.0
Release 10.1.0
Oracle Application Server
Release 10.1.2
Oracle Fail Safe does not support Automatic Storage
Management. Also, Oracle Fail Safe Server and Oracle Fail Safe
Manager is not supported on Windows Vista.
Note:
5 Hardware Compatibility
Consult your hardware vendor to ensure that the hardware you intend to use with
Oracle Fail Safe is certified for use with Microsoft Cluster Server software.
6
6 Installation and Deinstallation
This section includes topics about Oracle Fail Safe installations.
For complete installation and deinstallation instructions, see Oracle Fail Safe Installation
Guide.
6.1 Do Not Run Microsoft Windows Event Viewer During Oracle Services
for MSCS Installation
If you attempt to install Oracle Services for MSCS while the Microsoft Windows Event
Viewer is running, the installation procedure may return an error indicating that it
cannot copy the fsus.dll file (fsja.dll on Japanese systems) and asking if you want to
retry, ignore, or cancel. You can continue the installation if you stop the Microsoft
Windows Event Viewer and then click Retry in the Oracle Fail Safe installation Error
window.
6.2 MSCS Cluster Administrator Displays Problems with Fail-Safe
Resource Types
Sometimes, after completing an Oracle Fail Safe installation, you see problems with
the fail-safe resource types (such as databases) in MSCS Cluster Administrator. MSCS
Cluster Administrator denotes the problem by displaying an Oslash symbol (Ø) over
the resource type name.
If this occurs, follow these steps:
1.
If you forgot to reboot the cluster nodes after installing Oracle Fail Safe, do so now.
2.
Make sure that the PATH environment variable includes the Oracle Services for
MSCS path. (In the Command Prompt window, enter PATH.) The Oracle Services
for MSCS path (ORACLE_HOME\fs\fssvr\bin) should be included. If it is not
included, add it, and then reboot the nodes on which the Oracle Services for MSCS
path is missing.
3.
Make sure that the Oracle Fail Safe resource DLL, FsResOdbs.dll, is installed in
ORACLE_HOME\fs\fssvr\bin.
If the resource DLL is not there, reinstall Oracle Fail Safe.
4.
Use Oracle Fail Safe Manager to verify the cluster (on the Troubleshooting menu,
select Verify Cluster), then reboot each cluster node, one at a time. The Verify
Cluster command automatically verifies registration of Oracle resource DLLs. You
may not need to reboot all cluster nodes. After you reboot one node, check MSCS
Cluster Administrator to see if the Oslash symbol has been removed from the
resource type names. If the Oslash symbol is gone, you do not need to reboot all
cluster nodes.
6.3 Microsoft Visual C++ 2005 SP1 Redistributable Must be Installed
This version of Oracle Fail Safe requires that the Microsoft Visual C++ 2005 runtime
libraries be installed on all nodes that will run Fail Safe. The interactive installation
will automatically run the Microsoft installer to install the required images.
7
6.4 Oracle 11g Installation Warning About Homes in PATH
If you are installing Oracle Database 11g and Oracle Fail Safe is currently installed,
then a system "path" variable warning will display. This warning can be ignored; to
execute correctly, Oracle Fail Safe must include its home directory in the system path.
Click the check box next to the warning message and proceed with the installation.
6.5 MSDTC Must Be Configured When Cluster Created
Some Oracle Fail Safe operations rely on the Microsoft Distributed Transaction
Coordinator (MSDTC) and the OracleMSCSServices server will not start if MSDTC is
not correctly configured. Follow the instructions in the following Microsoft
Knowledge Base article to install and configure MSDTC in a cluster:
http://support.microsoft.com/kb/301600
7 Oracle Services for MSCS
This section includes topics about Oracle Services for MSCS.
7.1 DCOM Security Changes on Microsoft Windows 2000
If the list of Access permissions for the default setting of Distributed Component
Object Model (DCOM) is empty, the SYSTEM and SELF accounts have implicit access
rights. In Microsoft Windows 2000, if a user or group is added to the list, the implicit
rights no longer apply; rights are granted only to explicitly named users or groups. In
Microsoft Windows NT, the SYSTEM and SELF accounts retain their implicit rights.
During the installation of Oracle Services for MSCS, the Oracle Fail Safe user account
is added to the default access permissions list. If the access list is not empty, then the
SYSTEM and SELF accounts are automatically added.
After installing Oracle Services for MSCS, if there are problems opening hyperlinks in
an HTML file or problems using Microsoft Outlook or Outlook Express, use the
dcomcnfg.exe tool to add the necessary users to the default list of Access
permissions.
See Microsoft Support articles Q274696 and 892500 for more information.
7.2 Oracle Services for MSCS and Microsoft Cluster Server Can Run Under
Different Accounts
When your cluster was first configured and Microsoft Cluster Server was first
installed, it was installed under a Microsoft Windows user account.
Oracle Services for MSCS runs as a Microsoft Windows service under a domain user
account (not the system account) that has Administrator privileges on all cluster
nodes. When you install Oracle Services for MSCS, you are instructed to provide a
user name and password combination for a user account that has the required
privileges. This account does not have to be the same account under which Microsoft
Cluster Server was installed.
When "Is Alive" polling of Oracle Fail Safe resources is performed using the Microsoft
Windows account under which Microsoft Cluster Server is running, not the account
under which Oracle Services for MSCS is running.
8
8 Oracle Fail Safe Manager
This section includes topics about Oracle Fail Safe Manager.
8.1 Oracle Fail Safe Manager Requires a User Name and Password When
Run on Microsoft Windows 2000, Microsoft Windows XP, or Microsoft
Windows Server 2003
You must supply a user name and password when you log on to Oracle Fail Safe
Manager running on a Microsoft Windows 2000, Microsoft Windows XP, or Microsoft
Windows Server 2003 system.
If you log on to Oracle Fail Safe Manager running on any other operating system, you
are not required to specify the user name and password if your user account has
appropriate privileges.
9 Oracle Database
This section includes information about Oracle databases.
9.1 Errors During Execution of Verify Standalone Database Command
In some cases (perhaps due to another program updating the file), Microsoft Windows
may determine that the initialization parameter file for the database is locked by
another user and will not allow the file to be temporarily renamed or opened for
read/write access. This can cause problems when the Verify Standalone Database
command is executed and may result in somewhat cryptic error messages being
reported. If you encounter error messages that are similar to the following when
executing the Verify Standalone Database command, check to see if you can
temporarily rename the initialization parameter file for the database without getting
an operating system error message:
FS-10890: Oracle Services for MSCS failed during the Verify Standalone operation
FS-10818: The Oracle Database resource provider failed during preparation for
configuration processing for resource TESTDB1.US.ORACLE.COM
FS-10160: Failed to verify standalone Oracle database TESTDB1.US.ORACLE.COM
FS-10611: Failed to open file d:\oracle\database\initestdb1.ora for read
0xB: An attempt was made to load a program with an incorrect format
If another application appears to have control of the file, you can resolve the problem
by restarting the cluster node that owns the disk where the file resides (be sure to
move any cluster disks that contain database files back to the node that hosts the
database after you restart).
9.2 Default Oracle Intelligent Agent Is Stopped and Restarted When the
Database Is Shut Down
The default Oracle Intelligent Agent incorrectly discovers fail-safe databases on the
node where the default Oracle Intelligent Agent is running, and maintains a
connection to the database. (The default Oracle Intelligent Agent listens on the node's
host address, and therefore should not discover fail-safe databases because they use a
virtual address.) Therefore, when a fail-safe database is taken offline in normal or
transactional mode using Oracle Fail Safe Manager, Oracle Fail Safe shuts down the
default Oracle Intelligent Agent prior to shutting down the database. Oracle Fail Safe
9
restarts the default Oracle Intelligent Agent after the database shutdown operation is
complete.
9.3 Create Sample Database
Oracle Fail Safe includes a Create Sample Database command that installs a
preconfigured sample database on a cluster disk specified by the user. The sample
database has limited functions and is intended only for testing purposes and for use
with the online Oracle Fail Safe Tutorial; it should not be used for production. To
create a database for production, use Oracle Database Configuration Assistant or
create the database manually.
9.4 User Name for Database Must Be SYS
To ensure the success of all Oracle Fail Safe release 3.2.1, 3.3.1, 3.3.2, 3.3.3, and 3.4.1
operations, the database user name must be SYS unless operating system
authentication is used. If operating system authentication is used, then Oracle Fail Safe
does not use the SYS account.
10 Disk Resources
Oracle Fail Safe allows the use of EMC GeoSpan disks. However, if you attempt to add
a resource to a group and an EMC GeoSpan disk used by the resource is not already in
that group, then Oracle Fail Safe returns the error FS-10203 and rolls back the
operation.
If this occurs, add the resource to the group that already contains the EMC GeoSpan
disk that the resource requires.
11 Virtual Addresses
If an MSCS network name contains trailing spaces and you attempt to have Oracle Fail
Safe Manager add a virtual address to a group, the operation fails and the following
error is returned:
NT-5045: The cluster network was not found
The workaround to this problem is to rename the network name using MSCS Cluster
Administrator to remove the trailing spaces.
12 Oracle Enterprise Manager Integration
This section includes information about integrating Oracle Fail Safe with Oracle
Enterprise Manager.
12.1 Partial Support of JobOut Subdirectory
Oracle Intelligent Agent release 8.1.7 deposits its jobs output files into a subdirectory
called JobOut. For highly available Intelligent Agents, the JobOut subdirectory is
under the agent's ConfigPath directory on the cluster disk. The Intelligent Agent
requires the JobOut subdirectory to run jobs.
When creating an Oracle Intelligent Agent and adding it to a group, Oracle Fail Safe
creates the JobOut subdirectory on the cluster disk. However, when verifying a group
10
with a highly available Intelligent Agent in it, Oracle Fail Safe does not verify that the
JobOut subdirectory exists. In addition, when changing a highly available Intelligent
Agent's cluster disk, Oracle Fail Safe does not create a JobOut subdirectory on the new
disk, nor does it remove the JobOut subdirectory from the old disk.
12.2 Fail-Safe Database Discovered as Standalone Database on Physical
Nodes
When you view standalone databases on physical nodes, you may see fail-safe
databases that are online on the physical node being discovered. This problem appears
on systems running Microsoft Windows NT version 4.0 Service Pack 5 or Service Pack
6a.
If there is an Intelligent Agent resource in the group, then databases in the group will
not be discovered under the physical nodes. The exception to this case is when the
database uses more than one virtual address. If the first virtual address used in the
listener.ora and tnsnames.ora files for the database is not selected for use by the agent,
then the databases in the group will be discovered under the physical nodes.
12.3 Default Intelligent Agent of Physical Node Is No Longer Restarted by
Oracle Fail Safe
Prior to Oracle Fail Safe release 3.1, the default Intelligent Agent of a physical node
was restarted during the Add Database to Group and Remove Database from Group
operations. Starting with Oracle Fail Safe release 3.1, the default Intelligent Agent of a
physical node is no longer automatically restarted. If you need an updated list of
databases under the physical nodes, restart the default agent manually.
13 Documentation Updated for This Release
See the following documentation, which was not updated for this release, but which is
included in the kit, for additional information:
■
Oracle Fail Safe Concepts and Administration Guide
■
Oracle Fail Safe Installation Guide
■
Oracle Fail Safe Error Messages
■
Oracle Fail Safe Tutorial
Specific references to release 3.3.3 in the previous list of documents are also applicable
to release 3.4.1 with the exception of references to Oracle HTTP Server. Configuring
Oracle HTTP Server for high availability is not supported in this release of Oracle Fail
Safe.
The documentation that comes with the kit is provided in HTML and PDF online
formats. Viewing the PDF files requires Adobe Acrobat Reader 3.0 or later. You can
download the newest version from the Adobe Web site at
http://www.adobe.com/prodindex/acrobat/readstep.html
14 Additional Information About Oracle Fail Safe
Refer to the following Web sites for more information about Oracle Fail Safe:
■
Oracle Fail Safe on the Oracle Technology Network
11
http://otn.oracle.com/docs/tech/windows/failsafe/index.html
Updated software compatibility information, white papers, and so on are posted
on the Oracle Technology Network Web site.
■
Oracle Enterprise Manager on the Oracle Technology Network
http://www.oracle.com/technology/documentation/oem.html
■
Oracle Support Services
http://www.oracle.com/support/
Contact your Oracle support representative for technical assistance and additional
information, or visit the Oracle Support Services Web site to find out about other
available resources.
15 Documentation Errors and Omissions
This section corrects known errors and omissions from the Oracle Fail Safe
documentation set for release 3.4.1. It includes the following topics:
■
"Configuring Oracle Management Agent for High Availability" on page 12
■
"Removing Oracle Management Agent from a Group" on page 15
■
"Configuring Oracle Application Server Components for High Availability" on
page 15
■
"Removing Oracle Application Server Components from a Group" on page 16
■
"Transparent Application Failover Support" on page 17
■
"Using the fssvr GETSECURITY Command Qualifier" on page 17
■
"Messages Omitted from Oracle Fail Safe Error Messages" on page 18
■
"Incorrect Reference to Database Parameter File Parameter" on page 21
■
"References to Microsoft Cluster Server (MSCS)" on page 22
■
"Using Highly Available Databases with Oracle Data Guard" on page 22
■
"New Surrogate Process Trace Log" on page 22
15.1 Configuring Oracle Management Agent for High Availability
You can configure Oracle Enterprise Manager 10g Grid Control to monitor databases
configured for high availability. In general, the steps required to do so are:
1.
Install the Oracle Management Agent.
2.
Create an Oracle Management Agent that listens on a virtual address.
3.
Add the Oracle Management Agent created in Step 2 to the same group as the
Oracle Database (or databases) configured for high availability.
4.
Configure the Oracle Management Agent to monitor the database or databases.
The sections that follow describe, in detail, the prerequisites and steps required to
configure Oracle Management Agent for high availability.
12
15.1.1 Prerequisites for High Availability
You must install the following software on the cluster system before you can configure
an Oracle Management Agent for high availability:
■
■
■
Microsoft Cluster Server – version 1 or later
Oracle Database – any release supported by Oracle Enterprise Manager 10g Grid
Control
Oracle Management Agent
You must install the Management Agent on each cluster node, using the same
Oracle home on each node.
■
Oracle Fail Safe – Release 3.3.1 or later
In addition, the following components must be configured:
■
■
An Oracle database instance must be configured for high availability.
An Oracle Enterprise Manager Management Server must be configured and
available for setup. The Management Server need not reside on the cluster system.
15.1.2 Step 1: Make the Management Agent Highly Available
Use Oracle Fail Safe Manager to add the new Management Agent to the fail-safe group
that contains the databases it will monitor. You add the Management Agent to the
group as a generic service. Follow these steps:
1.
From the Resources menu, click Add to Group. The Add Resource to Group
Wizard opens.
2.
In the Resource Type box, select Management Agent.
3.
In the Group Name box, select the group to which you want to add the
Management Agent. This must be the group that contains the database you want
to monitor with the Management Agent.
4.
Click Next. The Management Agent Oracle Home page opens.
5.
In the Oracle Home box, select the Oracle Home that you want to add to the
group.
6.
Click Next. The Management Agent Virtual Address page opens.
7.
In the Virtual Address box, select the desired virtual address.
8.
In the Port box, enter an open port number.
9.
Click Finish. The Finish Adding Management Agent window opens.
10. This window summarizes the properties you have set with the wizard. If the
summary information is correct, click OK. Otherwise, click Cancel and then click
Back to find the page or pages in the wizard on which you want to change entries.
When you click OK, a window opens to display the progress of this clusterwide
operation. (A clusterwide operation is one that occurs on multiple cluster nodes.)
15.1.3 Step 2: Add the Highly Available Database as a Target in Oracle Enterprise
Manager
To configure the highly available database for monitoring through the highly available
Management Agent, follow these steps:
1.
Log in to the Oracle Enterprise Manager 10g Grid Control Console.
13
2.
Click the Targets tab. The Hosts page opens.
3.
Click the Databases secondary tab. The Databases page opens.
4.
Click Add (in the upper right-hand section of the page). The Add Database Target:
Specify Host page opens.
5.
Click the flashlight icon. The Search and Select Host window opens.
6.
Select the virtual host, and then click Select. The Search and Select Host window
closes and the Host field in the Add Database to Target: Specify Host page
contains the selected virtual host.
7.
Click Continue. The Targets Discovered on Host page opens.
8.
Select the highly available database or databases that you want to monitor, then
click OK. The Database Configuration Results page opens.
9.
Click OK.
15.1.4 Step 3: Test the Highly Available Management Agent
To test that the Management Agent is highly available, run a job against the highly
available database it is monitoring, and follow these steps:
1.
Log in to the Oracle Enterprise Manager 10g Grid Control Console.
2.
Create a SQL job and submit it against the highly available database, as follows:
a.
Click the Jobs tab. The Job Activity page opens.
b.
In the Create Job box, select SQL Script, and then click Go. The Create ’SQL
Script’ Job page opens.
c.
In the Job Name box, enter TEST JOB, and in the SQL Script field, enter
SELECT * FROM ts$.
d.
In the Databases region of the page, click Add. The Add Targets page opens.
e.
Select the highly available database host name (which matches the virtual host
name), and then click Add. The Create ’SQL Script’ Job page opens.
f.
In the Host and Database Credentials portion of the page, specify the database
credentials, and then click Submit.
3.
Make sure that the submitted job completes successfully.
4.
Create another job against the same database (by following Step 1 and Step 2 in
this list), but schedule it for 10 minutes from the current time.
5.
Using Oracle Fail Safe Manager, fail over the group.
6.
After 10 minutes have passed, check to see that the second job scheduled ran
successfully.
15.1.5 Step 4: Remove Extraneous Targets from the Oracle Enterprise Manager
Environment
When you reach this step, the Oracle Enterprise Manager 10g Grid Control Console
shows three targets for the same database. During the Management Agent installation,
the installer automatically discovers Oracle components, including highly available
databases, and adds the discovered components as targets.
Because the highly available database instance exists on each cluster node, there are
two targets for the database – each monitored by a different Management Agent. In
14
addition, you create a third target when you add the database as a target for the highly
available Management Agent listening on the virtual server (in Section 15.1.3).
You can safely remove the database targets that were discovered when the
Management Agent was installed by following these steps:
1.
Log in to the Oracle Enterprise Manager 10g Grid Control Console.
2.
Click the Targets tab. The Hosts page opens.
3.
Click the physical (as opposed to virtual) host name for one of the cluster nodes.
The Host page for that physical host opens.
4.
Click the Targets locator link.
5.
Select the database on this host, and then click Remove.
6.
Repeat Step 2 though Step 5 for each cluster node.
15.2 Removing Oracle Management Agent from a Group
If you decide you no longer want to have an Oracle Management Agent configured for
high availability, remove it from the group that contains it. When you do so, the Oracle
Management Agent is deleted from the cluster.
To remove an Oracle Management Agent from a group, follow these steps:
1.
Open Oracle Fail Safe Manager.
2.
In the tree view, select the Oracle Management Agent that you want to remove
and then, on the Resources menu, select Remove from Group.
3.
The Confirm Remove from Group box opens. Click Yes in the Confirm Remove
from Group box.
15.3 Configuring Oracle Application Server Components for High
Availability
Oracle Fail Safe Manager provides a wizard to help you configure instances of the
following Oracle Application Server components for high availability:
■
■
■
Oracle Process Manager and Notification Server (OPMN)
Oracle Enterprise Manager 10g Application Server Control (Application Server
Control) service
The metadata repository, if it was installed in the same Oracle home as the other
Oracle Application Server components
If a metadata repository is in the same home where you installed Oracle Application
Server, before adding instances of Oracle Application Server components to a group,
you must do the following:
■
■
Add at least one virtual address to the group.
Ensure that the database files are on cluster disks. If a failover occurs, any cluster
node that is a possible owner of the instance must be able to access the database
files.
You need the following information to complete the wizard. Examples are provided in
parentheses.
■
Oracle home where Oracle Application Server is installed (OAS)
15
■
If the metadata repository was installed in the Oracle home where Oracle
Application Server is installed, you also need the following information:
-
Database instance name (OFS5)
-
Database name (OFS5DB)
-
Database parameter file (T:\OFSDB\PARA\initofs5.ora)
-
Database SYS account password if the database has a password file
To configure Oracle Application Server components for high availability, follow these
steps:
1.
Install Oracle Application Server as described in Oracle Application Server
Installation Guide (10g Release 2 (10.1.2) for Microsoft Windows).
2.
Open Oracle Fail Safe Manager.
3.
On the tree view, select the standalone Oracle Application Server instance that you
want to add to a group.
4.
From the Resources menu, select Add to Group. The Add Resource to Group
Wizard opens.
5.
In the Resource Type box, select Oracle Application Server.
6.
In the Group Name box, select the group to which you want to add Oracle
Application Server.
7.
Click Next.
8.
Complete the remaining wizard pages. If you need assistance, click Help on the
wizard page.
Note the following:
■
■
■
Unlike a standalone database instance, a standalone Oracle Application Server
instance is represented in the tree view on each cluster node. You can select the
standalone instance that you want to add to a group from any one of the cluster
nodes.
A standalone Oracle Application Server instance is represented in the tree view by
the name of the home into which Oracle Application Server was installed. After it
is configured into the group, it is represented by the resources that compose it:
OPMN and Application Server Control. Oracle Fail Safe brings these resources
online after it has configured them on each cluster node.
To make an Oracle Application Server metadata repository that is not installed in
the same home as Oracle Application Server highly available, you add it to the
group to which you intend to add the other Oracle Application Server
components. It is important that you add it to the group before you add the other
Oracle Application Server components because Oracle Fail Safe sets up a
dependency between OPMN and the metadata repository when it configures
OPMN into the group.
15.4 Removing Oracle Application Server Components from a Group
If you decide you no longer want to have Oracle Application Server components
configured for high availability, remove them from the group that contains them.
Because the OPMN server has a dependency on the Oracle Application Server Control
service, you must remove the Oracle Application Server Control service from the
16
group before you remove the OPMN server. In addition, if the metadata repository
was added to the group when you configured the other Oracle Application Server
components into the group, it must be removed after the Oracle Application Server
Control service and the OPMN server are removed because of dependencies it has on
the other components.
To remove Oracle Application Server components from a group:
1.
Open Oracle Fail Safe Manager.
2.
In the tree view, select the Oracle Application Server Control service that you want
to remove and then, on the Resources menu, select Remove from Group.
3.
The Confirm Remove Resource from Group box opens. Click Yes in the Confirm
Remove from Group box. A window opens to show you the progress of the
operation.
4.
When the operation to remove the Oracle Application Server Control service
completes, select the OPMN server that you want to remove and then, on the
Resources menu, select Remove from Group.
5.
The Confirm Remove Resource from Group box opens. Click Yes in the Confirm
Remove from Group box. A window opens to show you the progress of the
operation.
6.
When the operation to remove the OPMN server completes, select the metadata
repository (if it was added when you configured the Oracle Application Server
Control service into the group) and then, on the Resources menu, select Remove
from Group.
7.
The Confirm Remove Resource from Group box opens. Click Yes in the Confirm
Remove from Group box. A window opens to show you the progress of the
operation.
After you remove the Oracle Application Server components from a fail-safe group,
they remain on the cluster as standalone resources. Note the following:
■
■
Unlike a standalone database instance, a standalone Oracle Application Server
instance is represented in the tree view on each cluster node.
A standalone Oracle Application Server instance is represented in the tree view by
the name of the home into which Oracle Application Server was installed.
15.5 Transparent Application Failover Support
The description of transparent application failover in Oracle Fail Safe Concepts and
Administration Guide includes a note that states: The transparent application failover
feature is supported for Oracle Database Enterprise Edition only. Beginning with
Oracle Database 10g Release 10.1.0.3, the transparent application failover feature is
also supported for Oracle Database 10g Standard Edition.
15.6 Using the fssvr GETSECURITY Command Qualifier
Oracle Fail Safe provides the fssvr command qualifier, /GETSECURITY. The
/GETSECURITY qualifier displays security information about the system where the
command is run. Oracle recommends that you run the fssvr command with this
qualifier (on each cluster node) to help diagnose FS-1075n errors (where n is a value
between 0 and 7, inclusive).
The command and its associated output should be similar to the following:
17
fssvr /getsecurity
Looking up user account information for OracleMSCSServices.
The user account must be a domain user acount with local Administrator
privileges. The user account must also have the 'Log on as batch job'
privilege.
User account specified for OracleMSCSServices is NEDCDOMAIN\cluadmin
User account specified has local Administrator privileges
User account has the 'Log on as batch job' privilege
Looking up user account information for Cluster Service. The user account
must be a domain user account with local Administrator privileges. The user
account must also have the 'Log on as batch job' privilege.
User account specified for Cluster Service is NEDCDOMAIN\cluadmin
User account specified has local Administrator privileges
User account has the 'Log on as batch job' privilege
Checking to see if DCOM is enabled.
DCOM is enabled.
DCOM must be enabled.
15.7 Messages Omitted from Oracle Fail Safe Error Messages
The following error messages are omitted from Oracle Fail Safe Error Messages:
FS-10254 Unable to configure the Oracle Database 10g TNS listener due to an old
listener on system.
Cause: The Oracle Database 10g TNS listener could not be configured because
there is a TNS listener from a prior release of Oracle Database on the system.
Action: Upgrade the existing TNS listener to the Oracle Database 10g TNS
listener, as follows:
1.
Stop the existing TNS listener.
2.
Move the entries for the default listener in the existing listener.ora file to the
listener.ora file in the Oracle Database 10g home.
FS-10404 The database uses a nonclustered disk in one of the system parameters.
Value of parameter is %s.
Cause: One of the following system parameters specifies a nonclustered disk:
udump, cdump, bdump, db_recovery_file_dest, or audit_file_dest.
Action: Either make sure that the specified directories exist on all the nodes or
change the system parameters so that clustered disks are used.
FS-10417 Bringing the resource %s online.
Cause: Bringing the specified resource online.
Action: This is an informational message. No action is needed.
FS-10686 A nonclustered disk is specified in the system parameter. Do you want to
continue?
Cause: The cdump, bdump, udump, db_recovery_file_dest, or audit_file_dest
destination specified by the parameter file uses a local disk drive.
Action: Make sure that the specified directories exist on the remote node before
continuing, then click Yes. Click No to cancel the operation.
18
FS-10687 Underlying Microsoft Windows service %s already exists on node %s. Do
you want to delete it?
Cause: The underlying Microsoft Windows service you are about to create already
exists on the system.
Action: Make sure that the underlying Microsoft Windows service is a service that
can be removed, then click Yes. Click No to cancel the operation.
FS-10688 Underlying Microsoft Windows service %s already exists on node %s.
Retry operation after removing the service.
Cause: The underlying Microsoft Windows service you are about to create already
exists on the system.
Action: Retry the operation after removing the specified Microsoft Windows
service.
FS-10735 Access to a cluster disk has failed: %s : error=%d.
Cause: The underlying cluster call has failed.
Action: Review the cluster system for failures. Oracle Fail Safe may continue its
operation, depending on the circumstances.
FS-10736 The database depends on a disk in "Cluster Group". The group is
unavailable on this node and cannot be moved.
Cause: The disk specified is in the Cluster Group and is running on a
different node from the node where the database resides. The Cluster Group
cannot be moved as it would result in the Oracle Fail Safe server being terminated.
Action: Move the Cluster Group to the node where the database is running.
FS-11205 Checking for configuration directory.
Cause: This is an informational message.
Action: No action is needed.
FS-11436 The Oracle Management Agent cannot be initialized.
Cause: The Oracle Management Agent cannot be initialized.
Action: Check the secondary error.
FS-11437 Failed to determine owner node for the Oracle Management Agent.
Cause: Cannot determine the owner node for the Oracle Management Agent.
Action: Check the secondary error.
FS-11438 Failed to prepare to configure the Oracle Management Agent.
Cause: Cannot prepare to configure the Oracle Management Agent.
Action: Check the secondary error.
FS-11439 Failed to configure the Oracle Management Agent for the cluster resource.
Cause: Cannot configure the Oracle Management Agent for the cluster resource.
Action: Check the secondary error.
FS-11440 Failed to create a new Oracle Management Agent.
Cause: Cannot create a new Oracle Management Agent.
Action: Check the secondary error.
FS-11441 Failed to remove the Oracle Management Agent from the group.
19
Cause: Cannot remove the Oracle Management Agent from the group.
Action: Check the secondary error.
FS-11442 Port numbers are not properly specified.
Cause: The port numbers are not properly specified for the current operation.
Action: Retry the operation with the proper port numbers specified.
FS-11443 The emctl.bat file cannot not be found.
Cause: The emctl.bat file cannot be found.
Action: Check the Oracle Management Agent installation for the emctl.bat file. If
not found, reinstall the Oracle Management Agent.
FS-11444 The Oracle Management Agent image cannot be found.
Cause: The Oracle Management Agent image cannot be found.
Action: Check the Oracle Management Agent installation for the Agent
executable. If not found, then reinstall the Oracle Management Agent.
FS-11445 The configuration directory specified for the new Oracle Management
Agent already exists. Do you want to continue?
Cause: The directory specified for the configuration directory already exists.
Action: : Click Yes in the error message window if you want Oracle Services for
MSCS to continue the operation and overwrite the configuration directory. Click
No if you want to terminate the operation.
FS-11446 The operation has been terminated.
Cause: You have selected to terminate the current operation.
Action: This is an informational message. No action is needed.
FS-11447 The set data operation for the Oracle Management Agent failed.
Cause: The set data operation for the Oracle Management Agent failed.
Action: Check the secondary error.
FS-11448 The get data operation for the Oracle Management Agent failed.
Cause: The set data operation for the Oracle Management Agent has failed.
Action: Check the secondary error.
FS-11449 New Oracle Management Agent service %s was not created.
Cause: The command to create a new Oracle Management Agent failed.
Action: Make sure that the Oracle Management Agent is properly installed in the
specified Oracle home.
FS-11450 Listener %s has dependency on %s.
Cause: The listener service image is on a cluster disk. The user specified that
Oracle Fail Safe not move the cluster disk into a disk group.
Action: Rerun the operation and allow Oracle Fail Safe to move the disk resource
into the specified group.
FS-11451 Failed to set the disk dependency for listener %s.
Cause: Cannot set the disk dependency for the specified listener.
Action: Check the secondary error.
20
FS-11452 Creating Oracle Application Server cluster resources.
Cause: Cluster resources for Oracle Application Server are being created.
Action: This is an informational message. No action is needed.
FS-11453 Creating a cluster resource for OPMN Service %s.
Cause: Creating the OPMN Service resource.
Action: This is an informational message. No action is needed.
FS-11454 Creating a cluster resource for Oracle Application Server Control Service
%s.
Cause: Creating Oracle Application Server Control resource.
Action: This is an informational message. No action is needed.
FS-11455 Bringing resources for Oracle Application Server online.
Cause: Bringing resources for Oracle Application Server online.
Action: This is an informational message. No action is needed.
FS-11456 Error configuring the Oracle Application Server services.
Cause: Error configuring the Oracle Application Server services.
Action: Check the secondary error.
FS-11457 Microsoft Windows service %s is not found. Removing %s resource from
cluster.
Cause: The node did not have Oracle Application Server configured with the
specified Microsoft Windows service.
Action: This is an informational message. No action is needed.
FS-11458 The Oracle home for resource %s is on a cluster disk.
Cause: The Oracle home for the specified resource resides on a cluster disk. The
verify operation will not continue when this type of configuration exists.
Action: If it was not intended for the Oracle home to be on a cluster disk, uninstall
and reinstall the product properly. Otherwise, no action is needed.
FS-11459 This operation may take a few minutes.
Cause: This operation may take a few minutes.
Action: This is an informational message. No action is needed.
FS-11460 The Management Agent Windows service %s does not exist on node %s.
Do you want to create it?
Cause: The verify operation found that the Management Agent Windows service
does not exist for the group on the specified node.
Action: Click Yes in the error message window if you want Oracle Services for
MSCS to create the Management Agent Windows service. Click No if you want to
try to fix the problem yourself. If you click No, the verify operation will continue.
15.8 Incorrect Reference to Database Parameter File Parameter
Section 7.3.3.3.1 of Oracle Fail Safe Concepts and Administration Guide contains a
reference to a REMOTE_LOGIN_PASSWORD parameter for the database parameter
file. There is no such parameter. The reference should be to the REMOTE_LOGIN_
PASSWORDFILE parameter.
21
15.9 References to Microsoft Cluster Server (MSCS)
Prior to the introduction of Microsoft Windows 2000, Microsoft Corporation referred
to Microsoft Cluster Service as Microsoft Cluster Server. Because Oracle Fail Safe
works with both Microsoft Cluster Service and Microsoft Cluster Server, both products
are referred to as Microsoft Cluster Server in the Oracle Fail Safe documentation.
15.10 Using Highly Available Databases with Oracle Data Guard
While Oracle Fail Safe provides high availability to single-instance Oracle databases,
Oracle Data Guard provides disaster tolerance. For example, Oracle Fail Safe can
ensure nearly continuous high availability for a given system, but does not protect
against a disaster that incapacitates the site where that system resides. Similarly, while
Oracle Data Guard provides excellent disaster recovery features, the time required to
switch operations from the primary site to a physically separate site can range from
several minutes to hours. By combining Oracle Fail Safe with Oracle Data Guard, your
databases can be highly available and disaster tolerant.
If you have an Oracle Support contract, you can find information about using Oracle
Data Guard with Oracle Fail Safe, by logging into OracleMetaLink and searching for
note 259902.1 at
http://metalink.oracle.com
15.11 New Surrogate Process Trace Log
The following entry must be added to Table B-1 of the Oracle Fail Safe Concepts and
Administration Guide:
Value
String
Description
FSU_TRACE_OUTPUT
A path and file name
Specifies the path and file
name for the file to which you
want tracing information
about the Oracle Services for
MSCS surrogate to be written.
For example: C:\fsu_
tracelog.log
Note that the FSU_TRACE_OUTPUT file is always appended to and never overwritten.
This means the file will continually grow until the file is deleted, or until the FSU_
TRACE_OUPUT registry entry is deleted or redefined. Oracle recommends that the file
be monitored to ensure that it does not grow too large and that tracing be enabled only
for short periods of time.
16 Documentation Accessibility
Our goal is to make Oracle products, services, and supporting documentation
accessible, with good usability, to the disabled community. To that end, our
documentation includes features that make information available to users of assistive
technology. This documentation is available in HTML format, and contains markup to
facilitate access by the disabled community. Accessibility standards will continue to
evolve over time, and Oracle is actively engaged with other market-leading
technology vendors to address technical obstacles so that our documentation can be
accessible to all of our customers. For more information, visit the Oracle Accessibility
Program Web site at http://www.oracle.com/accessibility/.
22
Accessibility of Code Examples in Documentation
Screen readers may not always correctly read the code examples in this document. The
conventions for writing code require that closing braces should appear on an
otherwise empty line; however, some screen readers may not always read a line of text
that consists solely of a bracket or brace.
Accessibility of Links to External Web Sites in Documentation
This documentation may contain links to Web sites of other companies or
organizations that Oracle does not own or control. Oracle neither evaluates nor makes
any representations regarding the accessibility of these Web sites.
TTY Access to Oracle Support Services
Oracle provides dedicated Text Telephone (TTY) access to Oracle Support Services
within the United States of America 24 hours a day, 7 days a week. For TTY support,
call 800.446.2398. Outside the United States, call +1.407.458.2479.
Oracle Fail Safe Release Notes, Release 3.4.1 for Microsoft Windows (32-Bit)
E10719-01
Copyright © 2007, Oracle. All rights reserved.
The Programs (which include both the software and documentation) contain proprietary information; they are provided under a license agreement
containing restrictions on use and disclosure and are also protected by copyright, patent, and other intellectual and industrial property laws. Reverse
engineering, disassembly, or decompilation of the Programs, except to the extent required to obtain interoperability with other independently created
software or as specified by law, is prohibited.
The information contained in this document is subject to change without notice. If you find any problems in the documentation, please report them
to us in writing. This document is not warranted to be error-free. Except as may be expressly permitted in your license agreement for these Programs,
no part of these Programs may be reproduced or transmitted in any form or by any means, electronic or mechanical, for any purpose.
If the Programs are delivered to the United States Government or anyone licensing or using the Programs on behalf of the United States Government,
the following notice is applicable:
U.S. GOVERNMENT RIGHTS Programs, software, databases, and related documentation and technical data delivered to U.S. Government
customers are "commercial computer software" or "commercial technical data" pursuant to the applicable Federal Acquisition Regulation and
agency-specific supplemental regulations. As such, use, duplication, disclosure, modification, and adaptation of the Programs, including
documentation and technical data, shall be subject to the licensing restrictions set forth in the applicable Oracle license agreement, and, to the
extent applicable, the additional rights set forth in FAR 52.227-19, Commercial Computer Software--Restricted Rights (June 1987). Oracle USA,
Inc., 500 Oracle Parkway, Redwood City, CA 94065.
The Programs are not intended for use in any nuclear, aviation, mass transit, medical, or other inherently dangerous applications. It shall be the
licensee's responsibility to take all appropriate fail-safe, backup, redundancy and other measures to ensure the safe use of such applications if the
Programs are used for such purposes, and we disclaim liability for any damages caused by such use of the Programs.
Oracle, JD Edwards, PeopleSoft, and Siebel are registered trademarks of Oracle Corporation and/or its affiliates. Other names may be trademarks
of their respective owners.
The Programs may provide links to Web sites and access to content, products, and services from third parties. Oracle is not responsible for the
availability of, or any content provided on, third-party Web sites. You bear all risks associated with the use of such content. If you choose to purchase
any products or services from a third party, the relationship is directly between you and the third party. Oracle is not responsible for: (a) the quality
of third-party products or services; or (b) fulfilling any of the terms of the agreement with the third party, including delivery of products or services
and warranty obligations related to purchased products or services. Oracle is not responsible for any loss or damage of any sort that you may incur
from dealing with any third party.
23
24
Download PDF
Similar pages