Using the DataOrchestrator™ ODS

Using the DataOrchestrator™ ODS
Reissued Manual as of July 9, 2013
This is a new edition of the Using the DataOrchestrator™ ODS manual. This edition replaces the previous
edition dated January 10, 2012, and incorporates the changes delivered with SU62183.74-485
DataOrchestrator ODS Summer 2013.
The Primary Changes Made
Section
Pages
Changes Made
Defining Parameters
62
Updated to resolve AnswerNet change request 60687.85 for the
Batch Commit Size field on the DataOrch Parameters (DOPA) form.
Defining Settings for
Refreshing Data
160
Updated in accordance with AnswerNet change request 62789.41 to
skip foreign key creation for incremental refreshes.
In addition, all references to the “Datatel Daemon” were changed to reference the “Colleague Daemon.”
Ellucian Colleague
Using the DataOrchestrator™ ODS
Release 18
July 9, 2013
For corrections and clarifications to this manual, see AnswerNet page 5696
Banner®, Colleague®, PowerCAMPUS®, Luminis® and Datatel® are trademarks of Ellucian or its affiliates and are registered in the U.S. and
other countries. Ellucian, Advance, DegreeWorks, fsaATLAS, Course Signals, SmartCall, Recruiter, MOX, ILP, and WCMS are trademarks of
Ellucian or its affiliates. Other names may be trademarks of their respective owners.
©2007–2013 Ellucian. All rights reserved. The unauthorized possession, use, reproduction, distribution, display or disclosure of this material or
the information contained herein is prohibited.
Contains confidential and proprietary information of Ellucian and its subsidiaries. Use of these materials is limited to Ellucian licensees, and is
subject to the terms and conditions of one or more written license agreements between Ellucian and the licensee in question.
In preparing and providing this publication, Ellucian is not rendering legal, accounting, or other similar professional services. Ellucian makes no
claims that an institution's use of this publication or the software for which it is provided will guarantee compliance with applicable federal or state
laws, rules, or regulations. Each organization should seek legal, accounting and other similar professional services from competent providers of
the organization’s own choosing.
Prepared by: Ellucian
4375 Fair Lakes Court
Fairfax, Virginia 22033
United States of America
Table of Contents
11
Introduction
13
About This Manual
13
14
14
16
17
19
In This Chapter
Who Should Read This Manual
Terms Used in This Manual
What This Manual Covers
How This Manual is Organized
Where to Find More Information
21
About the DataOrchestrator ODS
21
22
24
24
25
26
29
In This Chapter
Understanding the DataOrchestrator ODS
What Operational Data Stores Cannot Do
Facts about Operational Data Stores
Envision Data Mapping on an ODS Target Database
Architecture of the DataOrchestrator ODS
Hardware and Software Configuration for the
DataOrchestrator ODS
31
Overview and Installation
33
Overview
33
34
35
36
36
37
38
40
42
42
In This Chapter
Installation Overview for the DataOrchestrator ODS
Prerequisites for the DataOrchestrator ODS
Where to Build the Operational Data Stores
SQL Server
Oracle
Preparing the ODS Target Database
SQL Server Script for Creating the ODS Target Database
Adding the IDX Filegroup
Procedure for Adding the IDX Filegroup
45
Installing the DataOrchestrator ODS
45
46
46
In This Chapter
Before You Begin
Adding the Optional Module
Using the DataOrchestrator™ ODS, July 9, 2013
5
Table of Contents
47
51
51
6
Setting Up the ODS Target Database Server and Installing
Software Updates
Installing the Oracle JDBC Driver
Procedure for Installing the Oracle JDBC Driver
53
Setting Up DataOrchestrator ODS
Parameters
55
Overview of Using the DataOrchestrator
ODS
55
56
In This Chapter
Workflow for Using the DataOrchestrator ODS
59
Defining DataOrchestrator ODS
Parameters
59
59
60
60
64
In This Chapter
Form Used
Defining Parameters
Noteworthy Fields on the DOPA Form
Procedure for Defining DataOrchestrator ODS Parameters
65
Setting Up Targets
67
Defining a Target
67
68
70
71
72
In This Part
Forms Used
Defining a Target
Overview of the DataOrch Target (DOTA) Form
Noteworthy Fields on the DOTA Form
77
Defining Additional Target Parameters
77
77
78
78
80
In This Chapter
Form Used
Defining Additional Target Parameters
Noteworthy Fields on the DOTP Form
Procedure for Defining Additional Target Parameters
81
Defining Source Transforms
81
81
In This Chapter
Forms Used
Using the DataOrchestrator™ ODS, July 9, 2013
Table of Contents
82
83
84
88
89
92
92
94
Defining Source Transforms
Using the DataOrch Source Transform (DOST) Form
Noteworthy Fields on the DOST Form
Defining Filter Criteria
Noteworthy Fields on the DOFC Form
Viewing Generated Target Transforms
Noteworthy Fields on the DOGT Form
Procedure for Defining Source Transforms
95
Defining Target Transforms
95
96
98
99
100
105
106
111
112
114
115
115
118
118
121
122
124
125
127
127
129
129
131
131
134
134
137
In This Chapter
Forms Used
Defining Target Transforms
Using the DataOrch Target Transform (DOTT) Form
Noteworthy Fields on the DOTT Form
Defining Target Transform Columns
Noteworthy Fields on the DOTC Form
Defining Properties for a Target Transform Column
Noteworthy Fields on the DOPR Form
Defining Operations for Transform Columns
The Field Extract (DOFE) Form
Noteworthy Fields on the DOFE Form
The Multivalue Operation (DOMV) Form
Noteworthy Fields on the DOMV Form
The Pointer Reference (DOFR) Form
Noteworthy Fields on the DOFR Form
The Null Value Replacement (DONV) Form
Noteworthy Fields on the DONV Form
The String Concatenation (DOCA) Form
Noteworthy Fields on the DOCA Form
The Substring Selection (DOSS) Form
Noteworthy Fields on the DOSS Form
The Validation Code LookUp (DOVC) Form
Noteworthy Fields on the DOVC Form
The Expression Entry (DOEE) Form
Noteworthy Fields on the DOEE Form
Procedure for Defining Target Transforms
139
Defining and Creating SQL Views
139
140
141
142
In This Chapter
Forms Used
Defining SQL Views
Noteworthy Fields on the DOTV Form
Using the DataOrchestrator™ ODS, July 9, 2013
7
Table of Contents
8
143
144
147
148
149
Defining the SQL Select Statement for an SQL View
Noteworthy Fields on the DOVS Form
Procedure for Defining SQL Views
Creating SQL Views
Procedure for Creating SQL Views Using the DOTV Form
151
Procedure for Defining a Target
151
152
In This Chapter
Procedure for Defining a Target
157
Refreshing ODS Data
159
Defining and Running an ODS Refresh
159
159
160
162
167
168
170
In This Chapter
Forms Used
Defining Settings for Refreshing Data
Noteworthy Fields on the DORE Form
Defining Additional Parameters for the Refresh
Noteworthy Fields on the DORP Form
Procedure for Defining a Refresh
173
Calculating Stored Computed Columns
173
174
175
176
177
179
In This Chapter
Forms Used
Calculating Stored Computed Columns
Procedure for Activating Stored Computed Columns
Procedure for Calculating Stored Computed Columns
Procedure to Update Flags in the STUDENT.TERMS.CC
File
181
Viewing Reports and Refresh History
183
Viewing Errors for a Refresh
183
183
184
185
187
In This Chapter
Form Used
Viewing the Error Analysis Report for a Refresh
Noteworthy Fields on the DOEA Form
Procedure for Running the Error Analysis Report
189
Viewing Target Transform Data for a Target
189
In This Chapter
Using the DataOrchestrator™ ODS, July 9, 2013
Table of Contents
189
190
191
192
Form Used
Viewing the Transform Summary Report for a Target
Noteworthy Fields on the DOTS Form
Procedure for Running the Transform Summary Report
193
Viewing the History of a Refresh
193
193
194
196
In This Chapter
Form Used
Viewing the History of a Refresh
Procedure for Viewing the History of a Refresh
197
Maintaining the DataOrchestrator
ODS
199
Maintaining Transforms
199
199
200
201
203
204
205
206
207
208
In This Chapter
Form Used
Maintaining DataOrchestrator ODS Transforms
Noteworthy Fields on the DOMA Form
Procedure for Deleting Transforms
Procedure for Copying Transforms
Procedure for Renaming Transforms
Procedure for Copying and Renaming Transforms
Procedure for Calculating Foreign Keys
Procedure for Regenerating Source Transforms
209
Maintaining SQL Views
209
209
210
211
213
214
215
216
In This Chapter
Form Used
Maintaining SQL Views
Noteworthy Fields on the DOVM Form
Procedure for Deleting SQL Views
Procedure for Copying SQL Views
Procedure for Renaming SQL Views
Procedure for Copying and Renaming SQL Views
217
Copying Targets to Another Target
217
217
218
219
In This Chapter
Form Used
Copying One or More Targets to Another Target
Noteworthy Fields on the DOTY Form
Using the DataOrchestrator™ ODS, July 9, 2013
9
Table of Contents
220
10
Procedure for Copying One or More Targets to Another
Target
221
Deleting DataOrchestrator ODS Objects
221
221
222
223
224
In This Chapter
Form Used
Deleting DataOrchestrator ODS Objects
Noteworthy Fields on the DOOD Form
Procedure for Deleting DataOrchestrator ODS Objects
225
Appendices
227
Checklist for Setting Up the
DataOrchestrator ODS
227
228
In This Appendix
Checklist for Setting Up the DataOrchestrator ODS
231
Frequently Asked Questions
231
232
In This Appendix
Frequently Asked Questions about the DataOrchestrator ODS
237
Troubleshooting the DataOrchestrator
ODS
237
238
In This Appendix
DataOrchestrator ODS Issues
Using the DataOrchestrator™ ODS, July 9, 2013
Using the DataOrchestrator™ ODS
Introduction
Introduction
About This Manual
In This Chapter
This chapter includes information about the users who should read this
manual, terms used in this manual, the product that is covered in this manual,
the organization of the topics in this manual, and additional resources for
more information.
Table 1 lists the topics in this chapter.
Table 1: Topics in This Chapter
Topic
Page
Who Should Read This Manual
14
Terms Used in This Manual
14
What This Manual Covers
16
How This Manual is Organized
17
Where to Find More Information
19
Using the DataOrchestrator™ ODS, July 9, 2013
13
Introduction: About This Manual
Who Should Read This Manual
This manual is for system administrators and system programmers who set up
and support reporting for your institution. No specific knowledge of the
structure of source Envision files is assumed.
Terms Used in This Manual
The following are the terms and descriptions that are important to
understanding and using the concepts presented in this manual. These terms
are used throughout this manual:
Operational Data Store (ODS). An ODS is a repository of data that can deal
with a specific area of knowledge and provides easy access to the data for
reporting and analysis. It is a snapshot of data captured at a point in time.
Source Database. This refers to the Colleague environment from which you
want to create an ODS.
Target Database. This refers to the database where you want to place an
ODS.
Target. A target contains all of the necessary information about an ODS
target database including:
 Connection information.
 The set of source and target transforms that should be run to create and
update the tables in the ODS.
 Current update status information of the target.
Source Transform. A source transform allows you to specify the data to
move to an ODS target database by selecting a set of fields and/or computed
columns from a single Colleague file. The selected data will be structured on
the target database in one or more tables in the same way as the standard SQL
representation for that data in a Colleague environment. Source transforms do
not allow you to make any data transformations when the data is populated.
Target Transform. A target transform allows you to specify the format of the
data in a table on the ODS target database, the Colleague source data elements
that will populate each column, as well as any data transformation operations
to be run when the data is populated. Target transforms are a powerful tool for
the creation of ODS tables that are optimized for reporting purposes, and
provide greater flexibility in defining how the data is structured on the ODS.
14
Using the DataOrchestrator™ ODS, July 9, 2013
Terms Used in This Manual
Refresh. The refresh is the specification for an executable process that
refreshes a subset of the transforms associated with a target.
 Full Refresh. A full refresh means that all records of the source file for the
transform you choose will be processed through the transform and updated
on the ODS target database.
 Incremental Refresh. An incremental refresh means that only those
records of the source file that changed since the last refresh will be
processed through the transform.
Reporting Data Access Server (RDAS). A DMI Listener role defined for
use with the DataOrchestrator ODS. This DMI Listener role includes a set of
transactions for bulk data management.
SQL View. An SQL view combines data from various physical tables where
it is stored so that the result appears to be data from a single table. As a result,
a report or query is easier to build because the user creating the report does
not need to know the tables where data elements are stored or their join
relationships.
Using the DataOrchestrator™ ODS, July 9, 2013
15
Introduction: About This Manual
What This Manual Covers
This manual discusses the DataOrchestrator ODS™, which provides the
ability to create and maintain operational data stores, which are individual
data stores comprised of a defined subset of information from Colleague.
The information in the operational data stores can be kept up-to-date through
periodic refreshes of current data from the Colleague source database. The
operational data stores can be used as a source for reporting against this
Colleague data and can also be used as a source for data extracts for
propagation into data warehouse systems.
The chapters in this manual contain the information and procedures for
installing the software for the DataOrchestrator ODS, as well as implementing
and using the DataOrchestrator ODS.
In this manual you will find high-level installation and setup information, and
references to more detailed installation documentation. This documentation
provides information on how to perform the following tasks for implementing
the DataOrchestrator ODS and operational data stores.
 Install the following software components:
• DataOrchestrator ODS, Envision, and DMI software updates.
• An additional DMI Listener for the DataOrchestrator ODS.
• JDBC driver for an Oracle target database.
 Define the settings on which the DataOrchestrator ODS depends, including
the set of source and target transforms for an ODS.
 Export and refresh data to the operational data stores.
16
Using the DataOrchestrator™ ODS, July 9, 2013
How This Manual is Organized
How This Manual is Organized
Table 2 shows how this manual is organized.
Table 2: Organization of This Manual
Part
Part 1
Chapter Title
Summary
About This Manual
Includes information about who should read this
manual, terms used in the manual, what this manual
covers, how it is organized, and where to find
information on related topics.
About the DataOrchestrator
ODS
Discusses what the DataOrchestrator ODS is and
how the DataOrchestrator ODS can enhance your
institution’s reporting.
Overview
Provides an overview of the installation process and
the prerequisites for implementing the
DataOrchestrator ODS. Also included is information
on preparing the database where you want to build
operational data stores.
Installing the
DataOrchestrator ODS
Provides information about adding the optional
module for the DataOrchestrator ODS, retrieving the
Ellucian software updates, performing the installation
steps, and installing the Oracle JDBC driver.
Overview of Using the
DataOrchestrator ODS
Provides a high-level overview of the suggested
workflow for using the DataOrchestrator ODS.
Setting Up
DataOrchestrator ODS
Parameters
Defining DataOrchestrator
ODS Parameters
Provides information on how to set up the parameters
for the DataOrchestrator ODS.
Part 4
Defining a Target
Provides an overview on setting up and modifying a
target to export and transform Colleague source data
to an operational data store using the
DataOrchestrator ODS.
Defining Additional Target
Parameters
Describes how to define the additional parameters for
a target.
Defining Source
Transforms
Provides an explanation of how to define source
transforms for a target.
Defining Target Transforms
Provides an explanation of how to define target
transforms.
Defining and Creating SQL
Views
Describes how to define, create, and maintain SQL
views for a target.
Procedure for Defining a
Target
Provides the procedure for defining a target.
Introduction
Part 2
Overview and
Installation
Part 3
Setting Up Targets
Using the DataOrchestrator™ ODS, July 9, 2013
17
Introduction: About This Manual
Table 2: Organization of This Manual (cont’d)
Part
Part 5
Chapter Title
Defining and Running an
ODS Refresh
Provides information on how to set up a refresh for
exporting Colleague source data using the
DataOrchestrator ODS, and how to refresh data.
Calculating Stored
Computed Columns
Includes information on how to activate, update, and
export stored computed columns that are used as
source data elements for operational data stores.
Viewing Errors for a
Refresh
Discusses how to view the error analysis report for a
previously run refresh.
Viewing Target Transform
Data for a Target
Discusses how to view information for target
transforms for a specified target.
Viewing the History of a
Refresh
Discusses how to view the settings and the export
statistics for a previously run refresh.
Maintaining Transforms
Provides information on how to copy existing target
transforms, delete transforms, and rename
transforms. Also details how to regenerate source
transforms, and create foreign keys for transforms.
Maintaining SQL Views
Provides information on how to copy existing SQL
views, delete views, and rename SQL views.
Copying Targets to Another
Target
Gives information on how to copy all transforms and
SQL views from one or more targets to a single
destination target.
Deleting DataOrchestrator
ODS Objects
Includes information on how to delete existing
refreshes, targets, or refresh history records from the
source Colleague database.
Checklist for Setting Up the
DataOrchestrator ODS
Provides a quick checklist of the activities to perform
to set up the DataOrchestrator ODS.
Frequently Asked
Questions
Provides answers to frequently asked questions
when using the DataOrchestrator ODS and the data
models and views for reporting.
Troubleshooting the
DataOrchestrator ODS
Provides suggestions for items to check or steps to
take if you encounter issues in using the
DataOrchestrator ODS.
Refreshing ODS Data
Part 6
Viewing Reports and
Refresh History
Part 7
Maintaining the
DataOrchestrator ODS
Appendices
18
Summary
Using the DataOrchestrator™ ODS, July 9, 2013
Where to Find More Information
Where to Find More Information
Table 3 lists additional resources for finding more information.
Table 3: Additional Resources
Type of Information
Resource
Information on using the
DataOrchestrator ODS data models for
reporting.
Reporting from the DataOrchestrator ODS Data Models
Information on the mapping of
Colleague data to an ODS target
database when using DataOrchestrator
source transforms. (See page 24.)
Mapping Envision Files for SQL Server and Oracle
Installing a newly licensed optional
module and retrieving software
updates. (See page 46.)
Updating Colleague Software
Creating a new DMI_DAS. (See
page 47.)
Installation Procedures for Colleague Release 18
Setting up batch processes to run at
scheduled intervals. (See page 175.)
Envision Runtime Administration
How to activate and calculate stored
computed columns. (See page 175.)
Stored Computed Columns
Assistance with implementing
SQL-based reporting solutions
Ellucian Consulting Services. Contact Services Scheduling:
Technical Support
Ellucian Technical Support:
1-800-328-2835
1-800-328-2835
Using the DataOrchestrator™ ODS, July 9, 2013
19
Introduction: About This Manual
20
Using the DataOrchestrator™ ODS, July 9, 2013
Introduction
About the DataOrchestrator ODS
In This Chapter
This chapter provides information that you should understand about the
DataOrchestrator ODS before you use the product to create and use
operational data stores for your institution. It addresses the following
questions:
 What is the DataOrchestrator ODS, and what value does it bring to
reporting?
 What are the limitations of the operational data stores?
 How can you maximize your use of the operational data stores?
Also included is information about Envision data mapping on an ODS target
database. Table 4 lists the topics in this chapter.
Table 4: Topics in This Chapter
Topic
Page
Understanding the DataOrchestrator ODS
22
Facts about Operational Data Stores
24
Envision Data Mapping on an ODS Target Database
25
Architecture of the DataOrchestrator ODS
26
Hardware and Software Configuration for the DataOrchestrator ODS
29
Using the DataOrchestrator™ ODS, July 9, 2013
21
Introduction: About the DataOrchestrator ODS
Understanding the DataOrchestrator
ODS
The DataOrchestrator ODS was developed to help provide increased
reporting capabilities to your institution’s staff, managers, and executives to
support their institutional effectiveness strategies and long-range planning
efforts.
The DataOrchestrator ODS is a powerful tool that allows you to export and
transform data from Colleague applications to create operational data stores
that encompass specific areas of knowledge.
Ease of access and usability, as well as fast processing of data, are some of the
advantages of operational data stores. Also, operational data stores are easy to
use and convenient because they are not restricted to the same server or
location as the source database. Data can be exported to a target SQL-based
(SQL Server or Oracle) database.
Note: The DataOrchestrator ODS does not support exporting from a
Colleague SQL Server platform to an Oracle database as this
configuration is not used by Ellucian clients.
The following are some of the benefits derived from the DataOrchestrator
ODS:
 Operational data stores, as discussed in this manual, enhance your reporting
capability. By utilizing an operational data store strategy, Ellucian provides
a better means for you to report across Colleague applications while using a
variety of reporting tools appropriate for your institution.
 When extracting data from Colleague, you can transform the data using
flexible functions so that the data is available in a form that best suits your
reporting purposes.
 Specific subsets of Colleague data can be created in the ODS target
database to optimize reporting. This means that you no longer have to
search through thousands of elements in Colleague to select information for
reporting. Instead, you are able to simply utilize the DataOrchestrator ODS
to select your own specific files and fields for each operational data store
for your reporting purposes.
 For the decision makers at your institution, who need consistent and
accurate data on a daily basis, operational data stores provide a snapshot of
information, or operational picture, from a particular point in time. For
example, all the reports created from an operational data store today will
22
Using the DataOrchestrator™ ODS, July 9, 2013
Understanding the DataOrchestrator ODS





have consistent information from one report to another, irrespective of who
created the report.
An operational data store can be used to combine data from multiple
sources for reporting purposes. You can incorporate data from your nonColleague systems into an operational data store created using the
DataOrchestrator ODS, producing an integrated reporting platform.
You have complete flexibility in determining how often operational data
stores should be refreshed. A decision may be reached between the
technical staff and the various professional staff at your institution to
determine how fresh the data needs to be for reporting from each
operational data store.
You have the ability to do full or incremental refreshes of data, and you can
also set up the process handler to automatically refresh this data.
Operational data stores improve performance of the transactional database
by reducing the load on that database because information for reporting is
derived from snapshots of data on another database server.
Your institution may have specific needs that might necessitate adding or
removing files or data elements from an operational data store. The
DataOrchestrator ODS allows you to modify the set of files and fields that
are exported to an operational data store and how this data is transformed,
resulting in a customized reporting solution for your institution.
Using the DataOrchestrator™ ODS, July 9, 2013
23
Introduction: About the DataOrchestrator ODS
What Operational Data Stores Cannot Do
In addition to understanding what an operational data store can do for your
institution, it is also important to understand what it is not designed to do.
An operational data store is not a data warehouse; it is an independent
database containing subsets of application data. The information in a data
warehouse is highly optimized and aggregated for analytic reporting, and
contains a complete historical record of institutional data over time. However,
an operational data store retains the level of detail of the data structures in the
transactional database and does not contain historical data beyond what is
available in the transactional database. In addition, the data in an operational
data store is not intended to be used for transactional purposes, such as up-tothe-minute reporting. Operational data store information is taken at a point in
time, and the information may not be the most current.
Facts about Operational Data Stores
To further understand the nature of operational data stores and to maximize
their utility, consider the following facts:
 An operational data store includes static data, not dynamic or live data. The
data is captured at the time the data is exported to the operational data store.
Thus, it is a snapshot of data at a point in time. The operational data store is
updated with new data each time the refresh is run. Therefore, the resulting
reports will consistently reflect the data existing at the time of the
operational data store’s update.
 The latest transactional data will be selected from the database whenever
you export data to the operational data store.
 You can set up the Process Handler to export data to an operational data
store automatically on a preset schedule.
Note: Ellucian recommends that any operational data store that
contains stored computed columns be set up to automatically run the
following processes:
• Update GPA SCC Flags (UGSF) if using the STUDENT.TERMS.CC file
• Update Stored Computed Column (USCC)
This ensures that the stored computed columns will be updated when
the ODS target database is updated. Refer to Envision Runtime
Administration for further information on how to set a batch process to
run at scheduled intervals.
24
Using the DataOrchestrator™ ODS, July 9, 2013
Understanding the DataOrchestrator ODS
Envision Data Mapping on an ODS Target Database
Data mapping on an ODS target database is different for source transforms
and target transforms. For source transforms, the mapping of Colleague data
on an ODS target database is identical to the mapping of Colleague data on an
R18 SQL Server or Oracle environment. For further information on this
mapping, see Mapping Envision Files for SQL Server and Oracle, available
on the Ellucian website. However, the DataOrchestrator ODS will create on
the target database only those tables and columns that are necessary to store
the Envision data elements selected for inclusion in an operational data store.
In contrast, target transforms allow a much greater flexibility in the structure
of the data on the ODS target database. The number of columns of each target
table, as well as their names and data types, can be defined in the way that
best fits your institution’s reporting requirements.
Using the DataOrchestrator™ ODS, July 9, 2013
25
Introduction: About the DataOrchestrator ODS
Architecture of the DataOrchestrator
ODS
A high-level overview of the system architecture of the DataOrchestrator
ODS is shown in Figure 1 on page 27. The main elements of this architecture
are:
 Colleague Application Environment. The top of Figure 1 displays the
Colleague application environment. This environment is the source of data
for an operational data store and controls refreshes of that data to the
operational data store. A key element in this environment is the Envision
batch process displayed on the right, which controls the execution of target
transforms.
 DMI_DAS. The middle of Figure 1 displays the DMI communication
layer. This is used by the Colleague application environment to
communicate with both the source (environment) database and the target
database where the operational data store is created. The three DMI
transactions shown in this layer are specifically designed to support bulk
extracting and loading of data for the DataOrchestrator ODS.
 Source and Target Database Servers. The source server (on the bottom
left) contains the Colleague environment database. The target server (on the
bottom right) contains the ODS target database.
26
Using the DataOrchestrator™ ODS, July 9, 2013
Architecture of the DataOrchestrator ODS
Figure 1: DataOrchestrator ODS Architecture
The directional arrows in Figure 1 show how control and data flows between
these elements during the refresh of a target transform. The process for
executing a transform is as follows:
 In the Colleague application environment, the batch process creates and
sends a DAS Bulk Export transaction from the information in the target
transform. Using the information specified in this transaction, the
DMI_DAS creates a query statement to extract the data necessary for the
transaction from the source database. This query statement is in the native
query language of the source database (UniQuery for UniData, and SQL for
Oracle or Microsoft SQL Server). The query statement runs, using the bulk
extract function of the environment database. The results are stored to a
delimited flat file.
 In the Colleague application environment, the batch process then uses a
DAS Streaming Data Transfer transaction to move the delimited flat file
from the source server to the target server.
 In the Colleague application environment, the batch process then uses a
DAS Bulk Load transaction that utilizes the target database’s native bulk
load function (BULK INSERT statement for SQL Server, and SQL*Loader
for Oracle) to load the data in the delimited flat file to the appropriate table
on the target database.
Using the DataOrchestrator™ ODS, July 9, 2013
27
Introduction: About the DataOrchestrator ODS
This process is the core of the design of the DataOrchestrator ODS. This
leverages the bulk extract and load functions of your native database platform,
while maintaining the Colleague database-independent architecture where the
DMI_DAS layer provides all database platform-specific processing.
28
Using the DataOrchestrator™ ODS, July 9, 2013
Hardware and Software Configuration for the DataOrchestrator ODS
Hardware and Software Configuration
for the DataOrchestrator ODS
A high-level overview of the configuration of the hardware and software
elements required in a DataOrchestrator ODS installation is shown in
Figure 2. This hardware and software infrastructure supports the architecture
described in “Architecture of the DataOrchestrator ODS” on page 26.
As Figure 2 shows, there must be a DMI_DAS Listener located on the source
database server, and there must also be a DMI_DAS Listener with the RDAS
role located on the target database server. For more information on the DMI
RDAS Listener, see “Setting Up the ODS Target Database Server and
Installing Software Updates” beginning on page 47.
Figure 2: DataOrchestrator ODS Hardware and Software Configuration
Using the DataOrchestrator™ ODS, July 9, 2013
29
Introduction: About the DataOrchestrator ODS
30
Using the DataOrchestrator™ ODS, July 9, 2013
Using the DataOrchestrator™ ODS
Overview and Installation
Overview and Installation
Overview
In This Chapter
This chapter provides you with an overview of the setup requirements and
installation steps for the DataOrchestrator ODS. In addition, you will find the
prerequisites for the DataOrchestrator ODS, as well as considerations for
building your resulting operational data stores and for preparing the target
database.
Table 5 lists the topics in this chapter.
Table 5: Topics in This Chapter
Topic
Page
Installation Overview for the DataOrchestrator ODS
34
Prerequisites for the DataOrchestrator ODS
35
Where to Build the Operational Data Stores
36
Preparing the ODS Target Database
38
Using the DataOrchestrator™ ODS, July 9, 2013
33
Overview and Installation: Overview
Installation Overview for the
DataOrchestrator ODS
This section provides a high-level overview of the setup requirements and
installation procedures for the DataOrchestrator ODS.
Consult Table 6 to be sure that your software environment is ready for
working with the DataOrchestrator ODS.
Table 6: Setup Requirements and Installation Procedures
Setup Requirement/
Installation Procedure
Reference
Check that your institution has the
prerequisites needed for the
DataOrchestrator ODS.
“Prerequisites for the
DataOrchestrator ODS” on
page 35.
Decide where to build the operational data
stores.
“Where to Build the Operational
Data Stores” beginning on page 36.
Prepare the target database where the
operational data stores will be built.
“Preparing the ODS Target
Database” on page 38.
For SQL Server, you can use a script to
create your ODS target database.
“SQL Server Script for Creating the
ODS Target Database” on page 40
If your institution is using a SQL Server
target database, you must be sure that the
filegroup named “IDX” is present in your
target database.
“Adding the IDX Filegroup” on
page 42
Add the optional module for the
DataOrchestrator ODS.
“Adding the Optional Module”
beginning on page 46.
Retrieve the required software updates.
“Setting Up the ODS Target
Database Server and Installing
Software Updates” beginning on
page 47.
Install and configure the target database
DMI_DAS, update DAS roles, and install
software updates.
“Setting Up the ODS Target
Database Server and Installing
Software Updates” on page 47.
If the target database is Oracle, install the
Oracle JDBC driver on the ODS target
database.
“Installing the Oracle JDBC Driver”
on page 51.
If the source environment is UniData and
the target database is Oracle, also install the
Oracle JDBC driver on the source
environment.
34
Using the DataOrchestrator™ ODS, July 9, 2013
Prerequisites for the DataOrchestrator ODS
Prerequisites for the DataOrchestrator
ODS
Table 7 lists the prerequisites that your system must meet before you install
the software updates for the DataOrchestrator ODS.
Table 7: Prerequisites for the DataOrchestrator ODS
Prerequisite
Comment
Colleague R18
Make sure Colleague is current on all
software updates. If you need help,
contact Ellucian Technical Support.
UniData 7.1
For UniData, you must have the source
Colleague database on UniData version
7.1 or later. For information about
installing UniData, see the installation
procedures provided by IBM.
SQL Server 2005, SQL Server 2008, or
SQL Server 2008 R2
You must have SQL Server 2005, SQL
Server 2008, or SQL Server 2008 R2.
Other releases are not supported.
Oracle Database 11g, 11g Release 2, or
10g Release 2
You must have Oracle Database 11g,
11g Release 2, or 10g Release 2. Other
releases are not supported.
DMI with Data Access Server (DAS)
Use the DMI_DAS installed in the
Colleague application environment on
the source (Colleague) database server.
Ports used by the DMI Listeners must
be open between the Colleague and
ODS database servers.
Open ports are required so that the
Colleague database server DMI_DAS
can communicate with the ODS
database server DMI RDAS.
The port used by the Colleague daemon
on the ODS target database server
must be open to the client PC using SA
Valet in order to install software
updates.
The port for the daemon must be open
in order to manage and update the DMI
RDAS through SA Valet.
Using the DataOrchestrator™ ODS, July 9, 2013
35
Overview and Installation: Overview
Where to Build the Operational Data
Stores
The most important thing to think about is how you will use the operational
data stores you create. How do you intend to report from them? Will you use
Crystal Reports, Safari ReportWriter, Excel, SQL Server, or a data warehouse
solution? The following information is intended to help you think through
these questions before beginning. Knowing which reporting tool you intend to
use can save you time and effort by eliminating the need to do setup work
more than once.
Note: Do not use an existing Colleague database as a target
database for an operational data store. If you try to do this, you will
receive an error message when you run the refresh.
Note: This section assumes a high level of general technical
knowledge; no detailed explanations of the technology are provided.
This section is intended as a quick look at what will be required.
SQL Server
Consider the following points for the target database:
 Database considerations:
• You must have SQL Server 2005 or SQL Server 2008.
• You will need to create the SQL Server database in which you want to
build the operational data stores. The process that exports data will create
the tables, but cannot create the database itself.
 Connection considerations:
• Ports used by the DMI Listeners must be open between the Colleague and
ODS servers.
• The port used by the Colleague daemon on the ODS server must be open
to the client PC using SA Valet in order to install software updates.
 Third-party tool considerations:
• None
36
Using the DataOrchestrator™ ODS, July 9, 2013
Where to Build the Operational Data Stores
Oracle
Consider the following points for the target database:
 Database considerations:
• You must have Oracle Database 10g Release 2 or later.
• You will need to create the Oracle instance in which you want to build the
operational data stores. The process that exports data will create the tables,
but cannot create the instance itself.
Note: The DataOrchestrator ODS does not support exporting from a
Colleague SQL Server platform to an Oracle database as this
configuration is not used by Ellucian clients.


Connection considerations:
• Ports used by the DMI Listeners must be open between the Colleague and
ODS servers.
• The port used by the Colleague daemon on the ODS server must be open
to the client PC using SA Valet in order to install software updates.
Third-party tool considerations:
• None
Using the DataOrchestrator™ ODS, July 9, 2013
37
Overview and Installation: Overview
Preparing the ODS Target Database
ALERT! You should ensure that the target database to which you
are exporting Colleague data is adequately secured using the
native security controls for your database platform. Some files
and fields may contain data that might be considered sensitive in
your institution. Access to the ODS target database containing
this data should be restricted accordingly.
Ellucian also highly recommends that you secure access to the
DataOrch Target (DOTA) and DataOrch Refresh (DORE) forms.
These forms provide a user with the ability to export any data in
Colleague to an external database.
Select the database where you want to build the operational data stores. The
operational data stores can be stored in any existing or new target SQL-based
(SQL Server or Oracle) database.
Note: You do not have to create any tables. The DataOrchestrator
ODS software creates the tables in the target database as part of the
export.


Make sure a valid target database exists and note the name of that target
database.
Make sure that a database user exists for the DataOrchestrator ODS to use
for accessing the target database. This user must have full administrative
privileges to create database tables and to create and edit data in the target
database.
If you are using SQL Server as the target database, this user must have been
created with SQL Server Authentication (not Windows Authentication). In
addition, you must set the authentication of your SQL Server to “SQL
Server and Windows.”
To do this, access Microsoft SQL Server Management Studio. Select the
SQL Server and Windows Authentication mode option under Server
authentication on the Security tab of the Server Properties window of
your SQL Server instance, as shown in Figure 3 on page 39.
Note: If you are using the Blackboard Analytics data warehouse
solution, you must use a SQL Server target database.
38
Using the DataOrchestrator™ ODS, July 9, 2013
Preparing the ODS Target Database
Figure 3: Server Properties Window in Microsoft SQL Server Management
Studio
Using the DataOrchestrator™ ODS, July 9, 2013
39
Overview and Installation: Overview
SQL Server Script for Creating the ODS Target
Database
For SQL Server, you can use a script to create your ODS target database.
Figure 4 on page 41 shows a sample script for using the CREATE
DATABASE statement to create a local ODS target database in SQL Server.
Note the following:
 This script uses directory paths D:\sqlsrvr\data and E:\sqlsrvr\txlog, and
assumes a target database name of testods. You must replace these paths
with directory structures that exist on your SQL Server file system and
replace testods with your preferred database name.
 The script places the database files on a separate drive (D:) from the
transaction log (E:). This is expected to provide performance benefits,
because the SQL server will write to the transaction log at the same time it
writes to the database.
 The collation sequence defines the alphanumeric sorting and case
sensitivity of data fields in the database.
SQL_Latin1_General_CP1_CI_AS is used as it is the default collation
sequence for SQL Server databases and allows the ODS database to be
compatible with all Ellucian reporting solutions.
Technical Tip: If you copy the script in Figure 4 on page 41 from the
PDF of this manual, you may need to first paste it into a text editor
(such as Windows Notepad) to strip out any hidden characters in the
PDF text, and then copy it from the text editor to the SQL application.
If you copy directly from the PDF to the SQL application, the script may
not run properly. However, if you cut and paste as suggested, you may
remove valid carriage returns. After pasting the text in the query
window, compare your script to Figure 4 on page 41 and enter
carriage returns, if necessary.
40
Using the DataOrchestrator™ ODS, July 9, 2013
Preparing the ODS Target Database
Figure 4: Sample CREATE DATABASE Script for Creating an ODS Target
Database in SQL Server
CREATE DATABASE testods
ON PRIMARY
( NAME = testods_dat01,
FILENAME = 'D:\sqlsvr\data\testods_dat01.mdf',
SIZE = 500MB,
MAXSIZE = UNLIMITED,
FILEGROWTH = 10% ),
FILEGROUP IDX
( NAME = testods_idx01,
FILENAME = 'D:\sqlsvr\data\testods_idx01.ndf',
SIZE = 250MB,
MAXSIZE = UNLIMITED,
FILEGROWTH = 10% )
LOG ON
( NAME = 'testods_log01',
FILENAME = 'E:\sqlsvr\txlog\testods_log01.ldf',
SIZE = 500MB,
MAXSIZE = UNLIMITED,
FILEGROWTH = 10% )
SQL_Latin1_General_CP1_CI_AS
GO
Using the DataOrchestrator™ ODS, July 9, 2013
41
Overview and Installation: Overview
Adding the IDX Filegroup
Note: If you used the script described in “SQL Server Script for
Creating the ODS Target Database” on page 40, you can skip the rest
of this chapter. The IDX filegroup was created in your target database.
If your institution is using a SQL Server target database, you must be sure that
the filegroup named “IDX” is present in your target database (in addition to
the default/primary filegroup that you use). If the IDX filegroup does not exist
on your target database, create a filegroup with this name and assign a file to
it. The IDX filegroup is used for the indices.
Procedure for Adding the IDX Filegroup
To add the IDX filegroup and assign a file to it, complete the following steps.
Step 1. Right-click on the name of the ODS target database in Management Studio,
and select Properties to display the Properties page.
Step 2. Click the Filegroups page. Click Add and enter IDX in the Name field of
the first available open line as shown in Figure 5 on page 43, and then click
OK to close the Properties page.
42
Using the DataOrchestrator™ ODS, July 9, 2013
Preparing the ODS Target Database
Figure 5: Filegroups Page
Step 3. Open the Properties page again for the database. Click the Files page.
Step 4. Click Add, and then enter the logical name for the new data file in the Logical
Name field of the first available line.
Using the DataOrchestrator™ ODS, July 9, 2013
43
Overview and Installation: Overview
Step 5. Click the value shown in the Filegroup field. A dropdown list of available
Filegroups is displayed. Select IDX as shown in Figure 6.
Figure 6: Files Page
Step 6. Enter an initial space allocation for the new file in the Initial Size (MB)
column.
Step 7. Check the Autogrowth column and adjust it as needed, as the default of 1 MB
is probably not sufficient.
Step 8. Check the Path column to confirm that this is the path where you want the
index file created.
Step 9. Click OK to save your updates.
44
Using the DataOrchestrator™ ODS, July 9, 2013
Overview and Installation
Installing the DataOrchestrator
ODS
In This Chapter
This chapter provides information about adding the optional module, setting
up the ODS target database server, and performing additional installation
steps.
Table 8 lists the topics in this chapter.
Table 8: Topics in This Chapter
Topic
Page
Adding the Optional Module
46
Setting Up the ODS Target Database Server and Installing Software
Updates
47
Installing the Oracle JDBC Driver
51
Using the DataOrchestrator™ ODS, July 9, 2013
45
Overview and Installation: Installing the DataOrchestrator ODS
Before You Begin
Table 9 lists the tasks that must be complete before you can continue with the
procedures in this chapter.
Table 9: Before You Begin
Task
Reference
License the DODS optional module from
Ellucian.
Ellucian client sales.
Make sure you have the prerequisites for
the DataOrchestrator ODS.
“Prerequisites for the
DataOrchestrator ODS” on
page 35.
Know where to build your operational data
stores.
“Where to Build the Operational
Data Stores” on page 36.
Have the target database prepared.
“Preparing the ODS Target
Database” on page 38.
Adding the Optional Module
To add the DODS optional module for the DataOrchestrator ODS, see
Updating Colleague Software, available from the Documentation section of
the Ellucian website, for information on installing newly licensed optional
modules.
46
Using the DataOrchestrator™ ODS, July 9, 2013
Setting Up the ODS Target Database Server and Installing Software Updates
Setting Up the ODS Target Database
Server and Installing Software Updates
To install the DataOrchestrator ODS, perform the following steps to set up the
ODS target database and retrieve and install software updates.
Step 1. Install the standard Colleague daemon on the ODS target database server.
Step 2. Use SA Valet to create a new DMI Listener on the ODS target database server.
See Installation Procedures for Colleague Release 18.0, available from the
Ellucian website. See “Procedure for Installing a New DMI Listener.”
When creating the new DMI Listener, in the DMI Listener Roles Selection
window, click DBAS and RDAS.
Technical Tip: When you create the new Listener, it will inherit the
original DAS values of the existing Listener. Be sure to review the Java
memory, so that the performance parameters are acceptable for your
servers.
Step 3. Use SA Valet to retrieve the necessary software updates to your local product
repository. See Updating Colleague Software (available for downloading
from the Documentation section of the Ellucian website) for information on
retrieving and installing software updates.
Step 4. Install the DMI software update in your environment. This will update all
existing DMI Listeners in your Colleague environment and your Colleague
local product repository.
Step 5. Install the Envision software updates in your Colleague environment.
Using the DataOrchestrator™ ODS, July 9, 2013
47
Overview and Installation: Installing the DataOrchestrator ODS
Step 6. Create the ODS target database.
a. For SQL Server, you can use a script to create your ODS target database.
For more information, see “SQL Server Script for Creating the ODS
Target Database” beginning on page 40. If you do not use this script, you
must be sure that the filegroup named “IDX” is present in your target
database. For more information, see “Adding the IDX Filegroup”
beginning on page 42.
b. For Oracle, use the Database Creation Assistant (DBCA) to create your
ODS target database. Set the database character set as follows:
• Default database character set: WE8ISO8859P1
• Default national character set: AL16UTF16 - Unicode UTF-16 Universal
character set
Step 7. Create a subdirectory to be used by the DMI RDAS Listener on the target
database server to store temporary files for the Bulk Load transaction. The
following permissions are needed for this directory:
 UNIX Oracle target database. The user who starts up the target DMI
RDAS Listener must have read/write permissions.
 SQL Server target database.
• The Windows SYSTEM user must have read/write permissions.
Note: If the Listener is ever started manually (instead of via SA Valet
as a Windows service), then the user who starts the Listener must also
have read/write permissions.
• The DMI RDAS Listener logs in with SQL Server authentication, so the
SQL Server process account must have read/write permissions for the
BULK INSERT command.
This directory must have sufficient space allocated for these temporary files.
The space needed for temporary files can be significant depending on how
you define the target. Ellucian recommends that you monitor the space usage
in this directory as your refresh runs.
Step 8. If you are using a UniData or Oracle source environment, skip to Step 9. For
SQL Server only, ensure that the user who will be running the
DataOrchestrator ODS refresh has sufficient permissions on the source
database to create SQL views.
If this is a SQL server source environment, you will skip Step 9 through
Step 13. You are done with installation.
48
Using the DataOrchestrator™ ODS, July 9, 2013
Setting Up the ODS Target Database Server and Installing Software Updates
Step 9. Create a subdirectory on the source database server for use by the DMI_DAS
Listener. This subdirectory will be used to store temporary files from the Bulk
Extract transaction. The following permissions are needed for this directory:
 UNIX UniData source.
• The user logged in to Colleague must have read/write permissions.
• The user who starts the source DMI_DAS must have read/write
permissions.
 Windows UniData source.
• The user logged in to Colleague must have read/write permissions.
• The SYSTEM user must have read/write permissions.
Note: If the Listener is ever started manually (instead of via SA Valet
as a Windows service), then the user who starts the Listener must also
have read/write permissions.

UNIX Oracle source.
• Oracle needs read/write permissions.
• The user who starts the source DMI_DAS must have read/write
permissions.
This directory must have sufficient space allocated for these temporary files.
The space needed for temporary files can be significant depending on how
you define the target. Ellucian recommends that you monitor the space usage
in this directory as your refresh runs.
Step 10. Log in to Colleague as the DMI administrative user (for example, dmiadmin).
Step 11. In the UT application, access the DMI Pre-Authenticated Server (DMCC)
form. Enter the DMI Admin password.
Using the DataOrchestrator™ ODS, July 9, 2013
49
Overview and Installation: Installing the DataOrchestrator ODS
Step 12. On the DMCC form, enter information for the application server and the
Colleague database server.
Note: If your institution's Colleague database is UniData, the
application server and the Colleague database server are the same
server, so you need to enter only one row of information.
a. In the Server Name field, enter the name the server is known by in the
DMI Registry environment. This is usually the DNS name.
b. In the Enabled field, enter Yes for enabled.
c. In the Domain Name field, enter the fully qualified domain name for this
server.
d. In the IP Address field, enter the IP address of the server. (Do not reuse the
DNS name.)
Step 13. Save and exit from the DMCC form.
50
Using the DataOrchestrator™ ODS, July 9, 2013
Installing the Oracle JDBC Driver
Installing the Oracle JDBC Driver
Note: You must be a registered user of the Oracle website to
download the Oracle JDBC driver.
If your target database is Oracle, you need to install the Oracle JDBC
driver on the ODS target database. In addition, if you have a UniData
source environment and your target database is Oracle, then you also
must install the Oracle JDBC driver on your source environment.
Procedure for Installing the Oracle JDBC Driver
Step 1. Download the JDBC driver from the Oracle website. See AnswerNet
document 4527 for download instructions and the current supported version
of the driver.
Step 2. Save the file to your server.
Step 3. Use FTP to transfer the file to the machine where DMI is installed and place it
in the lib folder under your home DMI folder. For example, if the DMI folder
is /ellucian/coll18/production/ods_das, then place the file under the /ellucian/
coll18/production/ods_das/lib folder.
Step 4. Stop and restart the DMI Listener.
Using the DataOrchestrator™ ODS, July 9, 2013
51
Overview and Installation: Installing the DataOrchestrator ODS
52
Using the DataOrchestrator™ ODS, July 9, 2013
Using the DataOrchestrator™ ODS
Setting Up DataOrchestrator ODS Parameters
Setting Up DataOrchestrator ODS Parameters
Overview of Using the
DataOrchestrator ODS
In This Chapter
This chapter provides a high-level overview of the suggested workflow for
using the DataOrchestrator ODS. The purpose of the overview is to give you a
an understanding of the steps involved in setting up and maintaining the
DataOrchestrator ODS.
Table 10 lists the topic covered in this chapter.
Table 10: Topic in This Chapter
Topic
Workflow for Using the DataOrchestrator ODS
Using the DataOrchestrator™ ODS, July 9, 2013
Page
56
55
Setting Up DataOrchestrator ODS Parameters: Overview of Using the DataOrchestrator ODS
Workflow for Using the
DataOrchestrator ODS
Table 11 shows the steps to set up, refresh, and maintain the DataOrchestrator
ODS and operational data stores.
Table 11: Steps to Set Up, Refresh, and Maintain the DataOrchestrator ODS
and Operational Data Stores
Step
Form
Reference
1. Define parameters for the
DataOrchestrator ODS.
DataOrch
Parameters
(DOPA)
“Defining DataOrchestrator
ODS Parameters” on page 59.
2. Create a target, which
defines the configuration of
an operational data store on
a target database. This
includes defining source
and target transforms to
refresh on the target
database.
DataOrch
Target (DOTA)
“Defining a Target” beginning
on page 67 (all chapters in
Part 4).
Use detail
forms as
needed.
(Optional) Define and create
SQL views as shown.
3. Create a Refresh ID with an
associated target. Define
parameters to control how
the DataOrchestrator ODS
refreshes a target database
with data from Colleague
source files.
DataOrch
Refresh
(DORE)
“Defining and Running an
ODS Refresh” on page 159.
Colleague
Studio (Entity
editor >
Attributes >
Calculation)
“Calculating Stored Computed
Columns” on page 173.
(Optional) Create SQL
views.
4. Note: If your institution does
not use stored computed
columns, skip this step.
Activate and calculate
stored computed columns
for any ODS target
database that contains
stored computed columns.
If your ODS target database
uses the
STUDENT.TERMS.CC file,
update the flags as shown.
56
Update Stored
Computed
Column
(USCC)
Update GPA
SCC Flags
(UGSF)
Using the DataOrchestrator™ ODS, July 9, 2013
Workflow for Using the DataOrchestrator ODS
Table 11: Steps to Set Up, Refresh, and Maintain the DataOrchestrator ODS
and Operational Data Stores (cont’d)
Step
Form
Reference
5. Run an error analysis report
for any errors encountered
when exporting data from a
Colleague source file to an
ODS target database.
DataOrch
Error Analysis
(DOEA)
“Viewing Errors for a Refresh”
on page 183.
6. (Optional) Run the
Transform Summary report
to view information for target
transforms associated with
a target.
DataOrch
Transform
Summary
(DOTS)
“Viewing Target Transform
Data for a Target” on
page 189.
7. (Optional) View historical
information for previously
run refreshes for a specific
target.
DataOrch
Refresh
History
(DORH)
“Viewing the History of a
Refresh” on page 193.
8. (Optional) Copy existing
transforms from one target
to another, delete
transforms, rename target
transforms, and copy and
rename transforms. Also,
regenerate source
transforms, and calculate
foreign keys for transforms.
DataOrch
Transform
Main (DOMA)
“Maintaining Transforms” on
page 199.
9. (Optional) Copy existing
SQL views from one target
to another, delete SQL
views, rename SQL views,
and copy and rename SQL
views.
DataOrch View
Maintenance
(DOVM)
“Maintaining SQL Views” on
page 209.
10.(Optional) Copy all
transforms and SQL views
from one or more targets to
a single destination target.
DataOrch
Target Copy
(DOTY)
“Copying Targets to Another
Target” on page 217
11.(Optional) Delete existing
refreshes, targets, or
refresh history records from
the source Colleague
database.
DataOrch
Object Delete
(DOOD)
“Deleting DataOrchestrator
ODS Objects” on page 221.
Using the DataOrchestrator™ ODS, July 9, 2013
57
Setting Up DataOrchestrator ODS Parameters: Overview of Using the DataOrchestrator ODS
58
Using the DataOrchestrator™ ODS, July 9, 2013
Setting Up DataOrchestrator ODS Parameters
Defining DataOrchestrator ODS
Parameters
In This Chapter
This chapter describes how to set up the parameters for the DataOrchestrator
ODS.
Table 12 lists the topics covered in this chapter.
Table 12: Topics in This Chapter
Topic
Page
Form Used
59
Defining Parameters
60
Procedure for Defining DataOrchestrator ODS Parameters
64
Form Used
Table 13 lists and describes the form used in this chapter.
Table 13: Form Used in This Chapter
Form
DataOrch Parameters (DOPA)
Using the DataOrchestrator™ ODS, July 9, 2013
Purpose
Enter setup information for the
DataOrchestrator ODS.
59
Setting Up DataOrchestrator ODS Parameters: Defining DataOrchestrator ODS Parameters
Defining Parameters
Use the DataOrch Parameters (DOPA) form to define parameters that the
DataOrchestrator ODS will use when performing refreshes.
Figure 7: DataOrch Parameters (DOPA) Form
Noteworthy Fields on the DOPA Form
The fields described in this section are important for setting up parameters for
the DataOrchestrator ODS.
Source DMI_DAS Listener Name
Note: If you are using a SQL Server Colleague environment, leave
this field blank.
Use the Source DMI_DAS Listener Name field to enter the name of the DMI
Listener on the source database server.
60
Using the DataOrchestrator™ ODS, July 9, 2013
Defining Parameters
Source DMI Listener Host Name
Note: If you are using a SQL Server Colleague environment, this field
will be blank.
The Source DMI Listener Host Name field displays the TCP/IP address or the
DNS alias name of the computer where the DMI Listener is installed.
Source DMI Listener Port
Note: If you are using a SQL Server Colleague environment, this field
will be blank.
The Source DMI Listener Port field displays the port number associated with
the DMI Listener.
Termination Error Threshold
Use the Termination Error Threshold field to specify the default termination
error threshold for all transforms. When this threshold is reached, data load
will stop to the ODS target database table for a specific transform.
You can use the error analysis report generated by the DataOrch Error
Analysis (DOEA) form to identify the source of errors and correct the source
data or modify the transform. For more information on this report, see
“Viewing the Error Analysis Report for a Refresh” on page 184.
Depending on your target database type, this setting defines the following:
 SQL Server. This value specifies the number of errors allowed per batch
commit of a transform.
 Oracle. This value specifies the number of errors allowed for the entire
data load of a transform.
Using the DataOrchestrator™ ODS, July 9, 2013
61
Setting Up DataOrchestrator ODS Parameters: Defining DataOrchestrator ODS Parameters
Batch Commit Size
Use the Batch Commit Size field to enter the batch commit size for the data
load into the ODS target database. This is the number of rows to write and
commit to the ODS target database for each batch load. Ellucian recommends
that you leave the default set at 10,000.
Note: A refresh exports data to the ODS target database in batches.
This setting allows you to control the size of these batches.
Max Errors Saved
Use the Max Errors Saved field to enter the maximum number of error rows
to return and store for a transform. An error row is a database row that could
not be inserted into the target database, but which also doesn't stop the
database from loading subsequent rows. There may be more errors than this
number encountered by the refresh, but this is the maximum number stored by
the DataOrchestrator ODS for later analysis.
Override Source User ID
Use the Override Source User ID field to enter a valid override user ID set up
by your institution for accessing the source database. This ID must have full
access to the source database. If defined, this ID will be used instead of the ID
of the user running the refresh to access the source database.
Note: If you enter an Override Source User ID in this field, then you
must also enter the password for this user ID in the Override Source
User Pwd field.
Override Source User Pwd
Use the Override Source User Pwd field to enter the valid override password
for accessing the source database that corresponds to the user ID entered in
the Override Source User ID field.
Note: If you enter a password in this field, then you must also enter
the user ID in the Override Source User ID field.
62
Using the DataOrchestrator™ ODS, July 9, 2013
Defining Parameters
Source Drive Name
For a Windows source environment, use the Source Drive Name field to enter
the default source drive name of the temporary file path. If you are running on
any other source environment, leave this field blank. This value specifies
which drive on the source Colleague database server will store the temporary
files created by the refresh.
Source Temporary File Path
Note: If you are using a SQL Server source environment, leave this
field blank.
Use the Source Temporary File Path field to enter the default directory path on
the source Colleague database server where the temporary files, created by the
refresh, will be stored. Enter each element in the path on a separate line and
omit any slashes (/).
This directory must have sufficient space allocated for these temporary files.
The space needed for temporary files can be significant depending on how
you define the target. Ellucian recommends that you monitor the space usage
in this directory as your refresh runs.
The following permissions are needed for this directory:
 UNIX UniData. For the directory where the files are being written:
• The user logged in to Colleague must have read/write permissions.
• The user who starts the source DMI_DAS must have read/write
permissions.
 Windows UniData. For the directory where the files are being written:
• The user logged in to Colleague must have read/write permissions.
• The SYSTEM user must have read/write permissions.
Note: If the Listener is ever started manually (instead of via SA Valet
as a Windows service), then the user who starts the Listener must also
have read/write permissions.

UNIX Oracle. For the directory where the files are being written:
• Oracle needs read/write permissions.
• The user who starts the source DMI_DAS must have read/write
permissions.
Using the DataOrchestrator™ ODS, July 9, 2013
63
Setting Up DataOrchestrator ODS Parameters: Defining DataOrchestrator ODS Parameters
The temporary files in this directory may contain data that might be
considered sensitive at your institution. Access to the directory should be
restricted accordingly.
Procedure for Defining DataOrchestrator ODS
Parameters
Step 1. From the Envision Runtime (UT) application, access the DataOrch
Parameters (DOPA) form.
Step 2. Enter the parameters for the DataOrchestrator ODS. Refer to online help for
more information.
Step 3. Save and exit from the DOPA form.
64
Using the DataOrchestrator™ ODS, July 9, 2013
Using the DataOrchestrator™ ODS
Setting Up Targets
Setting Up Targets
Defining a Target
In This Part
This Part describes how to create and modify a target, which defines the
configuration of an operational data store on a target database.
Table 14 lists the topics covered in this Part.
Table 14: Topics in This Part
Topic
Page
Forms Used
68
Defining a Target
70
Defining Additional Target Parameters
77
Defining Source Transforms
81
Defining Target Transforms
95
Defining Target Transform Columns
105
Defining Operations for Transform Columns
114
Defining and Creating SQL Views
139
Procedure for Defining a Target
151
Using the DataOrchestrator™ ODS, July 9, 2013
67
Setting Up Targets: Defining a Target
Forms Used
Table 15 shows the forms used in this Part.
Table 15: Forms Used to Define a Target
Form
Purpose
DataOrch Target (DOTA)
Create and maintain a target, including the
source and target transforms to refresh on
a target database. Also view data on the
status of this target.
Additional Target Parameters
(DOTP)
Enter additional parameters for a target.
DataOrch Filter Criteria (DOFC)
Enter a MIOSEL expression to limit the
records of a file processed through a
transform. Also, enter any file suite
instances to be included when the
transform is refreshed.
DataOrch Target Views (DOTV)
Define SQL views for a target and view
information about their statuses.
DataOrch View Spec (DOVS)
Define the select statement, column
documentation, and materialized view
parameters for an SQL view.
For Source Transforms only
DataOrch Source Transform (DOST)
Define the fields and computed columns of
a file to refresh on the target database.
Generated Target Transforms
(DOGT)
View the target transforms that were
generated to implement a source
transform.
For Target Transforms only
68
DataOrch Target Transform (DOTT)
Define a target transform that extracts data
from Colleague and creates a table
containing that data on the target database.
Transform Columns (DOTC)
Specify the output columns for a target
transform. Each target column is defined by
an input field and an optional transform
operation that produces the value to store
in the corresponding target database
column.
Column Properties (DOPR)
Define properties for a transform column.
Using the DataOrchestrator™ ODS, July 9, 2013
Forms Used
Table 15: Forms Used to Define a Target (cont’d)
Form
Purpose
For Target Transform Columns only - Optional Operations
Field Extract (DOFE)
Define a field extract operation. This
returns subfields of the data in the source
field.
Multivalue Operation (DOMV)
Define a multivalue operation. Either the
Multivalue Position or the Multivalue
Aggregation operation can be performed
on a source field.
Pointer Reference (DOFR)
Define a pointer reference operation. This
references a field in a file, which is pointed
to by a value in the source field.
Null Value Replacement (DONV)
Define a null value replacement operation.
Either the Null Replacement or the Null
Test operation can be performed on a
source field.
String Concatenation (DOCA)
Define specifications for a concatenation
operation for a transform column.
Substring Selection (DOSS)
Define a substring selection operation. Set
up a starting position of a substring for the
source field and the length of the substring.
Validation Code LookUp (DOVC)
Define a validation code lookup operation.
Enter a validation code table name that
corresponds to the value in the source field
and choose which field to return from the
validation code.
Expression Entry (DOEE)
Define an expression to use to return a
value for a transform column
Using the DataOrchestrator™ ODS, July 9, 2013
69
Setting Up Targets: Defining a Target
Defining a Target
Use the DataOrch Target (DOTA) form as the starting point to create or
modify a target. Depending on what you want to include in the target, you will
also need to access additional forms. Figure 8 shows the additional forms that
are available when defining a target.
Figure 8: Forms Available for Defining a Target
Define Additional Target Parameters
DOTP
(define additional parameters for target)
Define SQL Views (Optional)
DOTV
(define SQL views)
DOTA
DOVS
(define select statement, column documentation, and
materialized view parameters for an SQL view)
Define Source Transform
DOFC (limit records from the file) (optional)
DOST
(select fields and
computed columns
of a file to refresh)
DOGT (view target transforms generated from
the source transform)
Define Target Transform
DOFC (limit records from the file) (optional)
DOTT
(define a transform
that extracts data from
Colleague)
DOTC (select output columns for a transform)
DOPR (define properties for a column)
Define Operation (optional)
DOFE (field extract operation)
DOMV (multivalue operation)
DOFR (pointer reference operation)
DONV (null value replacement operation)
DOCA (string concatenation operation)
DOSS (substring selection operation)
DOVC (validation code lookup operation)
DOEE (expression entry operation)
70
Using the DataOrchestrator™ ODS, July 9, 2013
Defining a Target
Overview of the DataOrch Target (DOTA) Form
The DataOrch Target (DOTA) form is used to define and maintain targets, and
to view information about the status of these targets. On the DOTA form, you
can enter source and target transforms that you want to refresh to a target
database.
Source transforms allow you to specify the data to move to a target ODS
database by selecting a set of fields and/or computed columns from a single
Colleague file. The selected data will be structured on the target database in
the same way as the standard SQL representation for that data in a Colleague
environment. Source transforms do not allow you to make any data
transformations when the data is moved. Detail to the DataOrch Source
Transform (DOST) form to include fields and computed columns that will be
sent for a transform.
Target transforms allow you to specify the format of the data on the target
ODS database, as well as any data transformation operations to be run when
the data is moved. Target transforms are a powerful tool for the creation of
ODS tables that are optimized for reporting purposes, and provide greater
flexibility in defining how the data is structured on the ODS. Detail to the
DataOrch Target Transform (DOTT) form to specify the input fields and
computed columns for a transform, and to define how these data elements will
be transformed when written to the target ODS database.
You must also detail from the DOTA form to the Additional Target
Parameters (DOTP) form to define the Target Drive Name and Target
Temporary File Path for the temporary files created by this process.
In addition, you can detail from the DOTA form to the DataOrch Target Views
(DOTV) form to define the SQL views associated to this target.
After data has been refreshed to this target database, you can view statistics
for the ODS that show its overall status.
Using the DataOrchestrator™ ODS, July 9, 2013
71
Setting Up Targets: Defining a Target
Figure 9: DataOrch Target (DOTA) Form
Noteworthy Fields on the DOTA Form
The fields described in this section are particularly important for
understanding as well as defining and modifying a target.
Status
The Status field displays the status of the target after the most recent refresh.
Oldest Transform Refresh and Latest Transform Refresh
These fields tell you the refresh dates and times for both the oldest and the
most recently updated transforms associated with the operational data store.
This allows you to see easily the date range of the transforms and determine
whether you should take any action based on these dates. For example, if the
Oldest Transform Refresh field has an older date than expected, this may alert
you that one or more of your transforms is not being updated on a regular
basis.
72
Using the DataOrchestrator™ ODS, July 9, 2013
Defining a Target
Description
Use the Description field to enter the text description to be displayed for this
target.
Database Type
Use the Database Type field to enter the type of database where the
operational data store will reside. Select the database type from the dropdown list. You can select SQL Server or Oracle.
Database Host Name
Use the Database Host Name field to enter the IP address of the computer on
which the target database resides. You may have an existing DNS alias name
created for the computer where the target database resides. Ellucian
recommends that you use that DNS alias name in the Database Host Name
field for ease of use.
Database Port
Use the Database Port field to enter the port number of the server on which
the target database resides.
Leave this field blank if your Oracle or Microsoft SQL Server database
listener service is using the default port number; otherwise, you must enter a
port number in this field.
The database port number is the database listener service port number.
 For Oracle the default is either 1521 or 1526.
 For Microsoft SQL Server the default is 1433.
Note: This is not the DMI Listener port number.
Using the DataOrchestrator™ ODS, July 9, 2013
73
Setting Up Targets: Defining a Target
Database Name/TNS
Use the Database Name/TNS field to enter the target database name for SQL
Server or the Transparent Network Substrate (TNS) name for Oracle.
This is the name of the target database to which you want to connect.
 For a Microsoft SQL Server target database, enter the database name.
 For an Oracle target database, enter the TNS.
Schema Name
Use the Schema Name field to enter the schema name for an Oracle target
database.
The schema name is an “owner” string that is prefixed to every target table to
uniquely identify the table in a database instance. By default, all tables created
on the target database will be created with the schema name of the user who is
logged in. To create tables in another schema on the target database, specify
the name of the schema in this field.
For a Microsoft SQL Server database, the only schema name that Ellucian
supports is “dbo”. Therefore, if you selected “SQL Server” in the Database
Type field, the Schema Name field is populated with this value and is inquiry
only.
Database User ID
Use the Database User ID field to enter a valid user ID for accessing the target
database. This ID must have full access to the target database.
Password
Use the Password field to enter the password for accessing the target database.
Enter a valid password for accessing the target database, corresponding to the
user ID entered in the Database User ID field. When you enter the password,
the value is displayed as asterisks.
74
Using the DataOrchestrator™ ODS, July 9, 2013
Defining a Target
Temporary File Path
Detail on the Temporary File Path field to access the Additional Target
Parameters (DOTP) form to enter the name of the directory that is used for
temporary files created on the ODS target database during a refresh. For more
information, see “Defining Additional Target Parameters” beginning on
page 77.
SQL Views
To view and define SQL views, detail on the SQL Views field to access the
DataOrch Target Views (DOTV) form.
From the DOTV form, you can detail to the DataOrch View Spec (DOVS)
form to define the select statement, column documentation and materialized
view parameters for an SQL view. For more information, see “Defining and
Creating SQL Views” beginning on page 139.
Source Transforms
Use the Source Transforms table to enter a file name to be included in the
ODS target database. File names entered here can be refreshed to an ODS
target database by using the DataOrch Refresh (DORE) form. You cannot
enter application-hierarchy files here (for example, ST.VALCODES,
CORE.VALCODES, etc.). Instead, specify the file suite template file name
(for example, VALCODES, etc.).
Note: The file entered cannot be a logical view. The file must be a
physical file in Colleague.
If you want to select fields to be included, detail on the specific source
transform to access the DataOrch Source Transform (DOST) form.
If you specify a file containing stored computed columns in this field, you
must run the Update Stored Computed Column (USCC) process to ensure that
the stored computed columns are refreshed before the DORE process is run.
For more information, see “Defining Source Transforms” beginning on
page 81.
Note: Deleting a file from the Source Transforms table does not
automatically delete the corresponding table on the target database.
This must be done manually by the database administrator.
Using the DataOrchestrator™ ODS, July 9, 2013
75
Setting Up Targets: Defining a Target
Target Transforms
Use the Target Transforms table to enter a name for a target transform to be
included in the target database.
The target transform name entered here will be used as the name of the
corresponding target table when this transform is refreshed using the
DataOrch Refresh (DORE) form.
To define the input fields and column transform operations for the transform,
detail on the specific target transform to access the DataOrch Target
Transform (DOTT) form. For more information, see “Defining Target
Transforms” beginning on page 95.
Note: Deleting a name from the Target Transforms table does not
automatically delete the corresponding table on the target database.
This must be done manually by the database administrator.
Refresh Date/Time
The Refresh Date/Time columns display the date and time when the most
recent refresh started for this target for source and/or target transforms.
Status
The Status column displays the status of the most recent refresh for this target
for source and/or target transforms.
Columns
For source transforms, the Columns column displays the number of columns
included in the source transform. This number is a sum of the columns from
the generated target transforms for this source transform (not including the
primary key, which is inferred).
For target transforms, the Columns field displays the number of columns
included in the target transform, including the primary key.
76
Using the DataOrchestrator™ ODS, July 9, 2013
Setting Up Targets
Defining Additional Target
Parameters
In This Chapter
This chapter describes how to set up the additional target parameters for the
DataOrch Target (DOTA) form.
Table 16 lists the topics covered in this chapter.
Table 16: Topics in This Chapter
Topic
Page
Form Used
77
Defining Additional Target Parameters
78
Form Used
Table 17 lists and describes the form used in this chapter.
Table 17: Form Used in This Chapter
Form
Additional Target Parameters (DOTP)
Using the DataOrchestrator™ ODS, July 9, 2013
Purpose
Enter additional parameters for a target.
77
Setting Up Targets: Defining Additional Target Parameters
Defining Additional Target Parameters
Detail on the Temporary File Path field from the DataOrch (DOTA) form.
This allows you to access the Additional Target Parameters (DOTP) form to
define the drive name and temporary file path for a target.
Figure 10: Additional Target Parameters (DOTP) Form
Noteworthy Fields on the DOTP Form
The fields described in this section are particularly important for defining
parameters for a target.
Target Drive Name
Use the Target Drive Name field to enter the target drive name of the
temporary file path for a Windows server. If you are running on any other
source environment, this field should be left blank.
This field specifies which drive on the ODS target database server will store
the temporary files created by the refresh.
78
Using the DataOrchestrator™ ODS, July 9, 2013
Defining Additional Target Parameters
Target Temporary File Path
Use the Target Temporary File Path to enter the directory path on the ODS
target database server where the temporary files created by the refresh will be
stored. Enter each element in the path on a separate line and omit any slashes
(/).
This directory must have sufficient space allocated for these temporary files.
The space needed for temporary files can be significant depending on how
you define the target. Ellucian recommends that you monitor the space usage
in this directory as your refresh runs.
The following permissions are needed for this directory:
 UNIX Oracle. For the directory where the files are transferred to and will
be bulk loaded from:
• The user who starts up the source DMI_DAS must have read/write
permissions.
 Windows SQL Server. For the directory where the files are created (via
the Microsoft SQL Server bcp utility) and will be bulk loaded from:
• The SYSTEM user must have read/write permissions.
Note: If the Listener is ever started manually (instead of via SA Valet
as a Windows service), then the user who starts the Listener must also
have read/write permissions.
• The DMI_DAS logs in with SQL Server authentication, so the SQL Server
process account must have read/write permissions for the BULK INSERT
command.
The temporary files in this directory may contain data that might be
considered sensitive at your institution. Access to the directory should be
restricted accordingly.
Using the DataOrchestrator™ ODS, July 9, 2013
79
Setting Up Targets: Defining Additional Target Parameters
Procedure for Defining Additional Target Parameters
Step 1. See “Procedure for Defining a Target” beginning on page 151.
80
Using the DataOrchestrator™ ODS, July 9, 2013
Setting Up Targets
Defining Source Transforms
In This Chapter
This chapter describes how to set up the source transforms for the DataOrch
Target (DOTA) form.
Table 18 lists the topics covered in this chapter.
Table 18: Topics in This Chapter
Topic
Page
Forms Used
81
Defining Source Transforms
82
Using the DataOrch Source Transform (DOST) Form
83
Defining Filter Criteria
88
Viewing Generated Target Transforms
92
Forms Used
Table 19 lists and describes the forms used in this chapter.
Table 19: Forms Used in This Chapter
Form
Purpose
DataOrch Source Transform (DOST)
Define the fields and computed columns
of a file to refresh on the target
database.
DataOrch Filter Criteria (DOFC)
Enter a MIOSEL expression to limit the
records of a file processed through a
transform. Also, enter any file suite
instances to be included when the
transform is refreshed.
Generated Target Transforms (DOGT)
View the target transforms that were
generated to implement a source
transform.
Using the DataOrchestrator™ ODS, July 9, 2013
81
Setting Up Targets: Defining Source Transforms
Defining Source Transforms
To define a source transform for a target, use the following forms:
 DataOrch Source Transform (DOST)
 Filter Criteria (DOFC)
To view the target transforms generated from a source transform, detail to the
Generated Target Transforms (DOGT) form from the DOST form.
82
Using the DataOrchestrator™ ODS, July 9, 2013
Defining Source Transforms
Using the DataOrch Source Transform (DOST) Form
Detail on a Source Transform on the DataOrch Target (DOTA) form to access
the DOST form (or access the form directly). Use the DOST form to select the
fields and computed columns of a file that will be refreshed to the target
database when you use the DataOrch Refresh (DORE) form. You can also
detail to the DataOrch Filter Criteria (DOFC) form to specify filters, including
file suite instances, to limit the records of the input file of the transform. In
addition, you can view data about when the source transform was last
refreshed.
You also have the option to:
 Manipulate the As of Date field for an incremental update to control the
content of the update. The As of Date field must have a value to perform an
incremental update, whether or not the transform has an Incremental Date
Field.
 Set the refresh to truncate input fields for string-type columns to the length
specified in the transform.
 Specify an error threshold for the transform at which the data export will
stop.
The source transform will be run by generating a set of target transforms to
create and populate data for each target database table needed to hold the
specified input fields of the source file. To view the target transforms
generated from this source transform, detail to the Generated Target
Transforms (DOGT) form.
When target transforms are generated, their output fields are built with the
data types and sizes of the referenced Colleague fields. However, if data types
or sizes of any referenced Colleague fields change, the target transforms need
to be recreated. To do this, you must use the Regenerate Source Transforms
operation on the DataOrch Transform Maint (DOMA) form. For more
information, see “Maintaining Transforms” beginning on page 199.
Using the DataOrchestrator™ ODS, July 9, 2013
83
Setting Up Targets: Defining Source Transforms
Figure 11: DataOrch Source Transform (DOST) Form
Noteworthy Fields on the DOST Form
The fields described in this section are particularly important when selecting
the fields and computed columns of a file that will be refreshed to the target
database.
Select All Fields (Y/N)
Use the Select all Fields field to indicate whether you want to select all
available fields for the source transform. Enter Yes to select all available
fields (which clears any previous entries in the Include Fields table). Enter
No to choose specific fields in the Include Fields table or to select no fields.
84
Using the DataOrchestrator™ ODS, July 9, 2013
Defining Source Transforms
Include Fields
Use the Include Fields list to select which fields will be refreshed for the file
you chose. The fewer fields you select, the better the refresh performance will
be and the less chance errors will be encountered.
Technical Tip: If you use a wildcard LookUp for your fields, you can
select multiple elements from the LookUp resolution form.
If you modify the Include Fields list, then target tables will be recreated
during the next refresh. To confirm that you want this to occur, you see the
following alert:
Field list change will cause the target table to be
recreated. Proceed (Y/N).
Click Yes to proceed or No to return to the Include Fields list and leave the list
unchanged. If you click Yes, the As of Date will be deleted to force a full
refresh.
Include Computed Columns
Use the Include Computed Columns list to select which computed columns
will be refreshed for the file you chose. Computed columns are recalculated
during the refresh, so the fewer computed columns you select, the better the
refresh performance will be and the less chance errors will be encountered.
Technical Tip: If you use a wildcard LookUp for your computed
columns, you can select multiple elements from the LookUp resolution
form. The LookUp list is restricted to only single-valued computed
columns that have been generated for database use on the Define
Computed Column (DCC) form or in Colleague Studio in the Entity
editor > Attributes > Generate wizard.
You cannot enter multivalued computed columns as source data
elements for a transform.
If you modify the Include Computed Columns list, then target tables will be
recreated during the next refresh. To confirm that you want this to occur, you
see the following alert:
Field list change will cause the target table to be
recreated. Proceed (Y/N).
Using the DataOrchestrator™ ODS, July 9, 2013
85
Setting Up Targets: Defining Source Transforms
Click Yes to proceed or No to return to the Include Computed Column list and
leave the list unchanged. If you click Yes, the As of Date will be deleted to
force a full refresh.
Note: For a multi-part key file, each CDD element making up the key
is included in this list as a separate CDD element.
Filters/File Suite Instances
Detail on the Filters/File Suite Instances field to access the DataOrch Filter
Criteria (DOFC) form in order to enter filter criteria, including file suite
instances, for this source transform. Colleague uses this criteria to specify
what records from the input file will be processed by the source transform
when it is run.
Incremental Date Field
The Incremental Date Field displays the MIO-managed change date field for
the input file you select for the source transform. There are two ways to
control what is selected for an incremental update:
 The Incremental Date Field together with the As of Date.
 The Incremental Filter Criteria.
Note: Not all files will have a MIO-managed change date field. If this
is the case for this file, you see the following message in this field: NO
MIO CHANGE DATE FIELD.
As of Date
The As of Date field affects only incremental refreshes. If the transform has
an Incremental Date Field, then the records updated on or after this will be in
the next incremental refresh. You can also use the As Of Date field in the
expression specified for the incremental filter criteria to control what data is
included during the next incremental refresh. For more information on using
incremental filter criteria, see “Defining Filter Criteria” on page 88.
If the transform has an Incremental Date field, then all records from the
source file that have a MIO-maintained change date on or after this As of Date
will be processed by the transform in the next incremental refresh.
The refresh updates this date with the current date when a transform is
refreshed to the target database and no errors are encountered. This As of Date
will not be modified if any source file records cannot be written to the ODS
86
Using the DataOrchestrator™ ODS, July 9, 2013
Defining Source Transforms
target database due to data errors. This ensures that the refresh will include
these records in a future refresh when the data errors are corrected in the
source file.
You can manually change the As of Date to affect which records are selected
during an incremental refresh. You may want to do this if you do not want the
refresh to attempt to send previously identified error records in future
incremental refreshes. However, this will mean that not all records from the
source environment will exist on the target database.
Truncate Strings (Y/N)
The Truncate Strings field allows you to indicate whether or not you want the
refresh to truncate input fields for string-type columns in this transform if the
data exceeds the length specified in Colleague for the source field.
Note: Truncation applies only to string-type columns. Also, truncation
does not apply to the primary key of a transform.
Termination Error Threshold
Use the Termination Error Threshold field to enter the error threshold for the
selected transform. When the threshold is reached, data load will stop for the
transform to the ODS target database table.
Depending on your target database type, this setting defines the following:
 SQL Server. The number of errors allowed per batch commit, as specified
on the DataOrch Parameters (DOPA) form.
 Oracle. The number of errors allowed for the entire data load of this
transform.
This setting overrides the default in the Termination Error Threshold field on
the DataOrch Parameters (DOPA) form.
View Generated Transforms
Each DataOrchestrator source transform will create and update one or more
tables on the ODS target database. Each source transform will be
implemented by a set of target transforms, each of which will create and
modify one of these target tables. You can detail to the Generated Target
Transforms (DOGT) form to view the set of target transforms generated from
this source transform.
Using the DataOrchestrator™ ODS, July 9, 2013
87
Setting Up Targets: Defining Source Transforms
Defining Filter Criteria
Detail to the Filter Criteria (DOFC) form to enter and verify MIOSEL
expressions to use against the input file of a transform to limit the set of
records from the file that will be processed through the transform. When
saving from the form, the MIOSEL statements (containing the entered
expressions against the input file for the transform) will be run to verify their
syntax.
The expressions will be used during a full refresh as:
SELECT [file name] [filter criteria to run]
The expressions will be used during an incremental refresh as:
SELECT [file name] [filter criteria to run] AND
( [ incremental date field GE “as of date” ] OR
( [incremental filter criteria to run] ))
For example:
SELECT PERSON WITH SPOUSE AND ( PERSON.CHANGE.DATE
GE "12/30/08" OR ( LAST.NAME EQ “Smith” ))
The expressions will be used during an incremental refresh when a transform
does not have an Incremental Date Field as:
SELECT [filename][filter criteria to run] AND
[incremental filter
criteria to run]
For example:
SELECT PERSON WITH SPOUSE AND LAST.NAME EQ "Smith"
If the input file is a file suite template, you can use the DOFC form to filter the
instances of that file suite to include in the refresh. You can also indicate
which of these instances to update on an ongoing basis, and which contain
static data that will not change and therefore do not need to be updated as part
of the refresh.
Note: If you add a new file suite instance, but enter No in the Update
Instance field, the file suite instance will not be included when this
transform is refreshed. You must run the refresh at least once with the
Update Instance field set to “Yes”.
88
Using the DataOrchestrator™ ODS, July 9, 2013
Defining Source Transforms
Figure 12: DataOrch Filter Criteria (DOFC) Form
Note: Figure 12 shows an example of a filter criteria for a source
transform. However, you can also use a filter criteria for a target
transform.
Noteworthy Fields on the DOFC Form
The fields described in this section are particularly important when limiting
the set of records from the file that will be processed through a transform.
File Name
The File Name field displays the name of the file to which the filter criteria
will be applied.
Using the DataOrchestrator™ ODS, July 9, 2013
89
Setting Up Targets: Defining Source Transforms
Filter Criteria to Run
Use the Filter Criteria to Run field to add or modify a MIOSEL expression
you want to use to filter the input records for the transform. The expression
must start with the selection clause keyword WITH.
The following keywords are also available to use as a filter criteria to be
replaced by a value when the refresh is run:
 TODAY(). This returns the current date when the refresh is run. For
example:
ACPG.START.DATE LE TODAY()
 ASOFDATE(). This returns the As of Date specified for the transform.
Note: You cannot enter an SQL statement as the filter criteria.
If you modify this field, the next refresh of this transform will be a full
refresh.
Note: If "Null Filter" is displayed and you copy this target template
transform (using the DOMA or DOTY form), no change is made to the
Filter Criteria to Run field in the transform to which you are copying.
Incremental Filter Criteria to Run
Use the Incremental Filter Criteria to Run field to construct a MIOSEL
expression you want to use to filter the input records for the transform for an
incremental refresh. If the transform has an Incremental Date Field, then this
expression will be added to the Incremental Date Field filter for an
incremental refresh with the OR keyword. Otherwise, the Incremental Filter
Criteria expression alone will be added to the overall filter with the AND
keyword. The expression must start with the selection clause keyword WITH
and cannot have any other WITH keywords in its expression (use AND or OR
instead).
90
Using the DataOrchestrator™ ODS, July 9, 2013
Defining Source Transforms
The following keywords are also available to use as a filter criteria to be
replaced by a value when the refresh is run:
 TODAY(). This returns the current date when the refresh is run. For
example:
ACPG.START.DATE LE TODAY()
 ASOFDATE(). This returns the As of Date specified for the transform.
Note: If "Null Filter" is displayed and you copy this target template
transform (using the DOMA or DOTY form), no change is made to the
Incremental Filter Criteria to Run field in the transform to which you are
copying.
File Suite Instances
Use the File Suite Instances list to enter the file suite instance IDs to include
when the transform is refreshed.
Update Instance (Y/N)
Use the Update Instance (Y/N) field to indicate whether this instance needs to
be updated when the transform is refreshed. Enter Yes to include the
instance; otherwise, enter No. If you enter No, the instance is updated only if
the columns selected from the file suite template for the transform change. In
this case, the data for all file suite instances will be deleted and recreated.
Note: If you are adding a new file suite instance, but enter No in this
field, the file suite instance will not be included when this transform is
refreshed. You must run the refresh at least once with this field set to
Yes.
Using the DataOrchestrator™ ODS, July 9, 2013
91
Setting Up Targets: Defining Source Transforms
Viewing Generated Target Transforms
Detail on the View Generated Transforms field from the DOST form to access
the Generated Target Transforms (DOGT) form. The DOGT form allows you
to view target transforms that have been generated to implement a specific
source transform. Each of the target transforms represents a table on the target
database.
Figure 13: Generated Target Transforms (DOGT) Form
Noteworthy Fields on the DOGT Form
The fields described in this section are particularly important when viewing
target transforms that have been generated to implement a specific source
transform.
Generated Target Transforms
The Generated Target Transforms column displays the name of a generated
target transform that is associated with a particular source transform. This
name represents a table that will be created on the target database.
92
Using the DataOrchestrator™ ODS, July 9, 2013
Defining Source Transforms
Status
The Status column displays the status of the most recent refresh of the
transform.
Start Date and Start TIme
These columns display the start date and time of the most recent refresh of the
target transform.
Total Rows
The Total Rows column displays the count of the total number of rows written
to the target database during the most recent run of the target transform.
Deleted
The Deleted column displays the count of the number of rows deleted on the
target database during the most recent run of the target transform.
Error
The Error column displays the count of the number of transform output rows
that could not be loaded into the target database due to data or execution
errors during the most recent run of the target transform.
End Date and End Time
These columns display the end date and time of the most recent refresh of the
target transform.
Using the DataOrchestrator™ ODS, July 9, 2013
93
Setting Up Targets: Defining Source Transforms
Procedure for Defining Source Transforms
See “Procedure for Defining a Target” beginning on page 151.
94
Using the DataOrchestrator™ ODS, July 9, 2013
Setting Up Targets
Defining Target Transforms
In This Chapter
This chapter describes how to set up the target transforms for the DataOrch
Target (DOTA) form.
Table 20 lists the topics covered in this chapter.
Table 20: Topics in This Chapter
Topic
Page
Forms Used
96
Defining Target Transforms
98
Using the DataOrch Target Transform (DOTT) Form
99
Defining Target Transform Columns
105
Defining Properties for a Target Transform Column
111
Defining Operations for Transform Columns
114
Using the DataOrchestrator™ ODS, July 9, 2013
95
Setting Up Targets: Defining Target Transforms
Forms Used
Table 21 lists and describes the forms used in this chapter.
Table 21: Forms Used in This Chapter
Form
Purpose
DataOrch Target Transform (DOTT)
Define a target transform that extracts
data from Colleague and creates a table
containing that data on the target
database.
DataOrch Filter Criteria (DOFC)
Enter a MIOSEL expression to limit the
records of a file processed through a
transform. Also, enter any file suite
instances to be included when the
transform is refreshed.
Transform Columns (DOTC)
Specify the output columns for a target
transform. Each target column is defined
by an input field and an optional
transform operation that produces the
value to store in the corresponding
target database column.
Column Properties (DOPR)
Define properties for a transform
column.
For Target Transform Columns only - Optional Operations
96
Field Extract (DOFE)
Define a field extract operation. This
returns subfields of the data in the
source field.
Multivalue Operation (DOMV)
Define a multivalue operation. Either the
Multivalue Position or the Multivalue
Aggregation operation can be performed
on a source field.
Pointer Reference (DOFR)
Define a pointer reference operation.
This references a field in a file, which is
pointed to by a value in the source field.
Null Value Replacement (DONV)
Define a null value replacement
operation. Either the Null Replacement
or the Null Test operation can be
performed on a source field.
String Concatenation (DOCA)
Define specifications for a concatenation
operation for a transform column.
Substring Selection (DOSS)
Define a substring selection operation.
Set up a starting position of a substring
for the source field and the length of the
substring.
Using the DataOrchestrator™ ODS, July 9, 2013
Forms Used
Table 21: Forms Used in This Chapter(cont’d)
Form
Purpose
Validation Code LookUp (DOVC)
Define a validation code lookup
operation. Enter a validation code table
name that corresponds to the value in
the source field and choose which field
to return from the validation code.
Expression Entry (DOEE)
Define an expression to use to return a
value for a transform column
Using the DataOrchestrator™ ODS, July 9, 2013
97
Setting Up Targets: Defining Target Transforms
Defining Target Transforms
To define a target transform for a target, use the following forms:
 DataOrch Target Transform (DOTT)
 Filter Criteria (DOFC)
 Transform Columns (DOTC)
You can also choose to detail from the DOTC form to define the following for
a target transform column:
 Define properties for the column on the Columns Properties (DOPR) form.
 Define an operation for the column using one of the following forms:
• Field Extract (DOFE)
• Multivalue Operation (DOMV)
• Pointer Reference (DOFR)
• Null Value Replacement (DONV)
• String Concatenation (DOCA)
• Substring Selection (DOSS)
• Validation Code LookUp (DOVC)
• Expression Entry (DOEE)
98
Using the DataOrchestrator™ ODS, July 9, 2013
Defining Target Transforms
Using the DataOrch Target Transform (DOTT) Form
Detail on a Target Transform on the DataOrch Target (DOTA) form to access
the DOTT form (or access the form directly). Use the DOTT form to define a
target transform that extracts data from Colleague and creates a table
containing that data on the target database. You can also view information
about when a specific target transform was last refreshed. To view and modify
the columns associated with this target transform, detail to the Transform
Columns (DOTC) form. In addition, you can detail to the DataOrch Filter
Criteria (DOFC) form to specify criteria to filter the records of the source file
of the transform.
You also have the option to:
 Manipulate the As of Date field for an incremental update to control the
content of the update. The As of Date field must have a value to perform an
incremental update, whether or not the transform has an Incremental Date
Field.
 Indicate that the transform needs to be refreshed as a full update.
 Set the refresh to truncate input fields for string-type columns in the
transform.
 Specify an error threshold for the transform at which the data export will
stop.
 Enter a description of the transform.
Using the DataOrchestrator™ ODS, July 9, 2013
99
Setting Up Targets: Defining Target Transforms
Figure 14: DataOrch Target Transform (DOTT) Form
Noteworthy Fields on the DOTT Form
The fields described in this section are particularly important when defining a
target transform that extracts data from Colleague and creates a table
containing that data on the target database.
Status
The Status field displays the status of the most recent refresh of the transform.
Start Date/Time
These fields display the start date and time of the most recent refresh of the
target transform.
End Date/Time
These fields display the end date and time of the most recent refresh of the
target transform.
100
Using the DataOrchestrator™ ODS, July 9, 2013
Defining Target Transforms
Refresh ID
The Refresh ID field displays the ID of the most recent refresh that processed
the target transform.
Total Rows
The Total Rows field displays the count of the total number of rows written to
the target database during the most recent run of the target transform.
Deleted
The Deleted field displays the count of the number of rows deleted from the
target database during the most recent run of the target transform.
Error
The Error field displays the count of the number of transform output rows on
which an error occurred during the most recent run of the target transform.
Source File
The Source File field displays the source file on which the target transform is
based. The source file selected will be used to control which fields and
computed columns you will be allowed to select when creating the target
transform. Also, the source file’s incremental change date field will control
the incremental updates of the target transform.
Note: The file entered cannot be a logical view. The file must be a
physical file in Colleague.
Column Definitions
Detail on the Column Definitions field to access the Transform Columns
(DOTC) form in order to view and modify the columns associated with the
target transform.
Using the DataOrchestrator™ ODS, July 9, 2013
101
Setting Up Targets: Defining Target Transforms
Multivalued (Y/N)
This field indicates whether or not the transform is multivalued. The
transform is considered multivalued if it contains one or more multivalued
columns. The ODS target table created by a multivalued transform will
include an additional column named “POS” and can contain multiple rows per
Colleague primary key.
Filters/File Suite Instances
Detail on the Filters/File Suite Instances field to access the DataOrch Filter
Criteria (DOFC) form in order to enter filter criteria, including file suite
instances, for the target transform. Colleague uses this criteria to specify what
records from the source file are processed by the target transform when it is
run. For more information on the DOFC form, see “Defining Filter Criteria”
on page 88.
Incremental Date Field
The Incremental Date Field displays the MIO-managed change date field for
the source file you selected for the target transform. There are two ways to
control what is selected for an incremental update:
 The Incremental Date Field together with the As of Date
 The Incremental Filter Criteria
Note: Not all source files will have a MIO-managed change date field.
If this is the case for this source file, you see the following message in
this field: NO MIO CHANGE DATE FIELD.
As of Date
The As of Date field affects only incremental refreshes. If the transform has
an Incremental Date Field, then the records updated on or after this will be in
the next incremental refresh. You can also use the As Of Date field in the
expression specified for the incremental filter criteria to control what data is
included during the next incremental refresh. For more information on using
incremental filter criteria, see “Defining Filter Criteria” on page 88.
If the transform has an Incremental Date Field, then all records from the
source file that have a MIO-maintained change date on or after this As of Date
will be processed by the transform in the next incremental refresh.
102
Using the DataOrchestrator™ ODS, July 9, 2013
Defining Target Transforms
The refresh updates this date with the current date when a transform is
refreshed to the target database and no errors are encountered. This As of Date
will not be modified if any source file records cannot be written to the ODS
target database due to data errors. This ensures that the refresh will include
these records in a future refresh when the data errors are corrected in the
source file.
You can manually change the As of Date to affect which records are selected
during an incremental refresh. You may want to do this if you do not want the
refresh to attempt to send previously identified error records in future
incremental refreshes. However, this will mean that not all records from the
source environment will exist on the target database.
Refresh as Full Only (Y/N)
In the Refresh as Full Only (Y/N) field, indicate whether or not this transform
can be updated only as full when refreshed using the DataOrch Refresh
(DORE) process. For example, a target transform should not be refreshed
incrementally in the following instances:
 The transform contains computed columns dependent on data from files
other than the transform’s source file that change on a regular basis.
 The transform has pointer-reference operations and the referenced data
values change on a regular basis.
In both these instances, the inclusion of the source records is not triggered in
an incremental refresh. For this reason, you would enter Yes for the
transform to update only as full during a refresh.
Note: If this field is set to Yes, the target transform will be refreshed
as a full update regardless of the setting on the DataOrch Refresh
(DORE) form in the Update Type or Override Update Type fields.
Truncate Strings (Y/N)
Use the Truncate Strings (Y/N) field to indicate whether you want the refresh
to truncate input fields for string type columns in the transform if the data
exceeds the specified length for the column.
Truncation applies only to string type columns. Also, truncation does not
apply to the primary key of a transform.
Using the DataOrchestrator™ ODS, July 9, 2013
103
Setting Up Targets: Defining Target Transforms
Termination Error Threshold
Use the Termination Error Threshold field to specify the error threshold for
the transform. When the threshold is reached, data load will stop for the
transform to the ODS target database table.
Depending on your target database type, this setting defines the following:
 SQL Server. The number of errors allowed per batch commit, as specified
on the DataOrch Parameters (DOPA) form.
 Oracle. The number of errors allowed for the data load of the transform.
This setting overrides the default in the Termination Error Threshold field on
the DataOrch Parameters (DOPA) form.
Description
Enter or view a description for this target transform.
104
Using the DataOrchestrator™ ODS, July 9, 2013
Defining Target Transforms
Defining Target Transform Columns
Detail from the DOTT form to the Transform Columns (DOTC) form to
specify the output columns for a target transform. Each target column is
defined by an input field and an optional transform operation that will
produce the output value to be stored in the corresponding ODS target
database column.
The available input fields that can be used are fields and computed columns
from the source file of this target transform. Other target columns of this
transform can also be used as source fields, in order to nest or combine
column operations. However, to be used as a source field for a column, other
transform columns must precede the column being defined in the transform
column list.
When you create a target transform, be aware of the following cardinality
rules that apply to columns marked “Yes” in the Export (Y/N) field:
 Only one unassociated multivalue column can be included in a transform
unless its data type has been changed to a single value by a transform
operation.
 Multiple associated data elements may be included only if they are from the
same association.
 Unassociated multivalues and associated multivalues cannot be included in
the same target transform.
When you save from this form, these rules will be checked. If the rules are not
followed in the transform, you will receive an error message.
You can also detail from this form to the Column Properties (DOPR) form to
specify properties for a column. If you define properties on the DOPR form,
then the DOTC form will display an “X” in the Addl field.
Using the DataOrchestrator™ ODS, July 9, 2013
105
Setting Up Targets: Defining Target Transforms
Figure 15: Transform Columns (DOTC) Form
Noteworthy Fields on the DOTC Form
The fields described in this section are particularly important when specifying
the output columns for a target transform.
Source File
The Source File field displays the source file on which the target transform is
based. The source file will be used as the reference from which fields and
computed columns can be chosen when creating the target transform. Also,
the source file’s incremental change date field will control the incremental
updates of the target transform.
Target Key Column
The Target Key Column field displays the name of the target primary key
column to be created on the target database for a target transform.
106
Using the DataOrchestrator™ ODS, July 9, 2013
Defining Target Transforms
Source Key Field
The Source Key Field field displays the source key field to which the target
primary key column is mapped.
Key Size
The Key Size field displays the size of the target key column to be created on
the target database.
Key Target Data Type
The Key Target Data Type field displays the data type of the target key
column to be created on the target database.
Export Key (Y/N)
The Export Key (Y/N) field displays the export flag for the Target Key
Column. The Target Key Column will always have “Yes” as its export flag.
This column will always be exported to the target database.
Target Column
Use the Target Column field to enter the name of the column to be created in
the target table for this transform in the ODS target database.
To add a timestamp column, enter DO_REFRESH_TIMESTAMP, or use
LookUp to select that value. The timestamp column indicates when the
refresh for this target was started, not when the rows were created in the target
database.
Using the DataOrchestrator™ ODS, July 9, 2013
107
Setting Up Targets: Defining Target Transforms
Source Field
Use the Source Field field to enter the source field or computed column from
the source file that is the input to the target column. In order to nest or
combine operations, the source field can also be another target column, which
must be defined in the transform column list preceding its use as a source
field.
Note: The computed columns available are restricted to only singlevalued computed columns that have been generated for database use
on the Define Computed Column (DCC) form or in Colleague Studio in
the Entity editor > Attributes > Generate wizard.
You cannot enter multivalued computed columns as source data
elements for a transform.
Size
Use the Size field to enter the size of the target column to be created on the
target database.
Target Data Type
The Target Data Type field displays the Envision data type that describes the
column to be created on the target database. The data type may be modified
based on the transform operation that is selected for the target column.
Operation
Use the Operation field to choose an operation to perform in order to generate
an output value for the target column. When you choose an operation, you
will detail automatically to the appropriate form to allow you to enter
specifications to use with that operation. There are ten operations available:
 Field Extract. Colleague opens the Field Extract (DOFE) form. This form
allows you to extract a subset of the fields of a source delimited string by
specifying the field delimiter and the number of fields to extract from a
starting field.
 Multivalue Aggregation. (For multivalue target data types only.)
Colleague opens the Multivalue Operation (DOMV) form. This form
allows you to select from different aggregation operations including: Sum,
108
Using the DataOrchestrator™ ODS, July 9, 2013
Defining Target Transforms








Count, Min, and Max. If you select this operation, the column’s data type is
changed to single-value.
Multivalue Position. (For multivalue target data types only.) Colleague
opens the Multivalue Operation (DOMV) form. This form allows you to
choose a specific position or the last position in a multivalued field to
become a column in the target transform. If you select this operation, the
column’s data type is changed to single-value.
Pointer Reference. Colleague opens the Pointer Reference (DOFR) form.
This form allows you to specify a pointer field as the source field for the
column and use this operation to select data fields from another Envision
file using the pointer field as the link. Multivalued target columns are
restricted to referencing only single-valued fields from another file.
Null Replacement. Colleague opens the Null Value Replacement (DONV)
form. This form allows you to specify a value in the If Null Return field
that will be returned if the source field value for the column is null. The
data type in this field must match the data type of the source field.
Null Test. Colleague opens the Null Value Replacement (DONV) form.
This form allows you to specify a value for both the If Null Return and If
Not Null Return fields to create a Null Test Operation to choose the value in
the column based on whether the source field value is null or not. The data
types of the two fields must match.
String Concatenation. Colleague opens the String Concatenation (DOCA)
form. This form allows you to create a concatenation operation using
quoted strings, single-valued string type fields from the source file, or
single-valued string type target columns from the associated transform
defined before this column. All arguments must be single-valued source
data elements or literals.
Substring. Colleague opens the Substring Selection (DOSS) form. This
form allows you to define a starting position and length of a substring for
the source field.
Validation Code LookUp. Colleague opens the Validation Code LookUp
(DOVC) form. This form allows you to enter a validation code table name
that corresponds to the value in the source field and choose which field to
return from the validation code.
Expression Entry. Colleague opens the Expression Entry (DOEE) form.
This form allows you to enter an expression to use to return a value for this
column. Operations can be nested.
Using the DataOrchestrator™ ODS, July 9, 2013
109
Setting Up Targets: Defining Target Transforms
Addl
The Additional field displays an “X” if properties were defined for the
column on the Column Properties (DOPR) form. These properties include:
 Creating an index.
 Entering an output format string.
 Specifying foreign keys.
Export (Y/N)
Use the Export (Y/N) field to export the target column to the target database.
To export the target column, enter Yes; otherwise, enter No.
You would typically enter “No” if a target column is used in another
operation, but the column’s data is not needed on the target database. Columns
that are marked “No” will not be created on the target database and will not
affect the cardinality of the target transform.
110
Using the DataOrchestrator™ ODS, July 9, 2013
Defining Target Transforms
Defining Properties for a Target Transform Column
Detail from the DOTC form to the Column Properties (DOPR) form to define
properties for a transform column, including the following:
 An index for the column.
 The column’s output format string.
 Foreign keys for the column.
Note: You cannot modify the output format string for the key field of a
transform.
Figure 16: Column Properties (DOPR) Form
Using the DataOrchestrator™ ODS, July 9, 2013
111
Setting Up Targets: Defining Target Transforms
Noteworthy Fields on the DOPR Form
The fields described in this section are particularly important when
defining properties for a target transform column.
Target ID
The Target ID field displays the ID of the target containing the column.
Target Transform
The Target Transform field displays the name of the target transform
containing the column.
Target Column
The Target Column field displays the name of the target transform column.
Create Index (Y/N)
Use the Create Index (Y/N) field to indicate whether you want the refresh to
create an index in the ODS target database for the transform column. Enter
Yes to create an index for this column; otherwise, enter No or leave this
field blank.
Note: Indexes are created in the ODS target database during a
refresh.
Output Format String
Use the Output Format String field to enter a column’s output format string,
which corresponds to the Conversion String allowable in Envision. The
format defined for the column determines the type of column on the ODS
target database.
112
Using the DataOrchestrator™ ODS, July 9, 2013
Defining Target Transforms
Related Transform
In the Related Transform field, enter transform names to relate to the column.
A foreign key will be created in the ODS target database from the column to
the table created for the related transform.
Note: To detect foreign key relationships automatically, choose the
“Calculate Foreign Keys” operation in the Operation field on the
DataOrch Transform Maint (DOMA) form.
Create Foreign Key (Y/N)
Use the Create Foreign Key (Y/N) field to indicate whether you want the
refresh to create a foreign key in the ODS target database to point to the
corresponding related transform in the Related Transform field.
Enter Yes to create a foreign key for the column. Enter No if you do not
want a foreign key to be created.
Description
Use the Description field to enter or view a description for this target
transform column.
Using the DataOrchestrator™ ODS, July 9, 2013
113
Setting Up Targets: Defining Target Transforms
Defining Operations for Transform Columns
Detail from the DOTC form to define an operation for a target transform
column using one of the following forms:
 Field Extract (DOFE)
 Multivalue Operation (DOMV)
 Pointer Reference (DOFR)
 Null Value Replacement (DONV)
 String Concatenation (DOCA)
 Substring Selection (DOSS)
 Validation Code LookUp (DOVC)
 Expression Entry (DOEE)
114
Using the DataOrchestrator™ ODS, July 9, 2013
Defining Target Transforms
The Field Extract (DOFE) Form
Detail from the DOTC form to use this form to define a field extract column
operation. Use the field extract operation to return one or more subfields of
the data in the source field where the subfields are delimited by the passed
character. This form also provides a description field to enter a description of
the column.
Figure 17: Field Extract (DOFE) Form
Noteworthy Fields on the DOFE Form
The fields described in this section are particularly important when defining a
field extract column operation.
Target ID
The Target ID field displays the ID of the target associated with the target
transform.
Using the DataOrchestrator™ ODS, July 9, 2013
115
Setting Up Targets: Defining Target Transforms
Target Transform
The Target Transform field displays the name of the current target transform.
Target Column
The Target Column field displays the name of the target column that will have
its value determined by the result of the operation.
Source File
The Source File field displays the source file on which the definition of the
target transform is based.
Source Field
The Source Field field displays the source field to which the target column is
mapped. This is the field against which the field extraction operation will be
applied. The source field can also be another target column, as long as it
precedes this column in the transform’s column list.
Delimiter
The Delimiter field allows you to enter the delimiter to use as a separator for
the source field’s data. If you leave this field blank, Colleague uses a single
space as the delimiter. Only one-character entries are accepted, with the
exception of the @VM and @SM expressions, which represent value mark
and sub-value mark delimiters, respectively.
For example, an asterisk is used as the delimiter to get the first element of a
multi-part key.
Start Field Number
Use the Start Field Number field to enter the starting field number, from the
fields as delineated by the delimiter, at which to start extracting fields for the
field extract.
116
Using the DataOrchestrator™ ODS, July 9, 2013
Defining Target Transforms
Number of Fields to Extract
Use the Number of Fields to Extract field to enter the number of fields to
extract, beginning with the start field number.
Description
Use the Description field to enter a description for the target transform
column.
Using the DataOrchestrator™ ODS, July 9, 2013
117
Setting Up Targets: Defining Target Transforms
The Multivalue Operation (DOMV) Form
Detail from the DOTC form to use this form to define a multivalue transform
operation. Either the Multivalue Position or the Multivalue Aggregation
operations can be performed on the source field. The result of either operation
is a single value. This form also provides a description field to enter a
+description of the target column.
Figure 18: Multivalue Operation (DOMV) Form
Noteworthy Fields on the DOMV Form
The fields described in this section are particularly important when defining
or viewing a multivalue transform operation.
Target ID
The Target ID field displays the ID of the target associated with the target
transform.
118
Using the DataOrchestrator™ ODS, July 9, 2013
Defining Target Transforms
Target Transform
The Target Transform field displays the name of the current target transform.
Target Column
The Target Column field displays the name of the target column from the
transform that will have its value determined by the result of the operation.
Source File
The Source File field displays the source file on which the target transform is
based.
Source Field
The Source Field field displays the multivalued source field to which the
target column is mapped. This is the field to which the aggregation operation
will be applied. The source field can also be another target column, which
needs to be defined before being used as a source field.
Position in List
The Position in List field displays the position in a multivalued list contained
in the source field that the column will reference. You can enter a positive
integer or the keyword “last” in this field to select a specific value in the
multivalued source field that the column will reference. The result of the
operation will be a single value.
Aggregation Operation
Use the Aggregation Operation field to select from a list of available
aggregation operations to perform on the source field. The result of the
operation will be a single value.
 Count. A count of the number of values in the list.
 Min. The minimum or lowest value in the list.
 Max. The maximum or highest value in the list.
 Sum. The sum of all values in the list.
Using the DataOrchestrator™ ODS, July 9, 2013
119
Setting Up Targets: Defining Target Transforms
Description
Use the Description field to enter a description for the target transform
column.
120
Using the DataOrchestrator™ ODS, July 9, 2013
Defining Target Transforms
The Pointer Reference (DOFR) Form
Detail from the DOTC form to use this form to define a pointer reference
column operation. This allows a reference to a field of an Envision file other
than the source file defined for the target transform. The pointer reference
operation uses the value in the Source Field field as a pointer to a record of
this second file, and returns the value of the field named in the operation.
Multivalued source fields are restricted to referencing only single-valued
fields from another file.
This form also provides a description field to enter a description of the
column.
Note: Only fields from another file can be selected. Also, computed
columns from another file are not available for selection.
Figure 19: Pointer Reference (DOFR) Form
Using the DataOrchestrator™ ODS, July 9, 2013
121
Setting Up Targets: Defining Target Transforms
Noteworthy Fields on the DOFR Form
The fields described in this section are particularly important when defining a
pointer reference column operation.
Target ID
The Target ID field displays the ID of the target associated with the target
transform.
Target Transform
The Target Transform field displays the name of the current target transform.
Target Column
The Target Column field displays the name of the target column from the
transform that will have its value determined by the result of the operation.
Source File
The Source File field displays the source file for the overall target transform
of which the column is a component. This file contains the source field, unless
the source file is itself another column defined in the target transform.
Source Field
The Source Field field displays the source field that the target column will use
as a pointer to a secondary file. The source field can also be another target
column from the transform as long as it is defined before being used as a
source field.
122
Using the DataOrchestrator™ ODS, July 9, 2013
Defining Target Transforms
File
Use the File field to enter the file that contains the field referenced by the
selected pointer.
Note: If the Source File is a file suite template file (for example,
GLS.FYR, CS.ACYR, etc.), then you can specify a different file suite
template file of the same type (for example, .FYR, .ACYR, etc.) in this
field. When a refresh is run, the file suite instance string for the file
specified in this field will be replaced with the file suite instance being
processed.
Pointer
The Pointer field displays the key to the secondary file. The value of the
pointer field selected is used to look up the referenced record in the secondary
file.
Field
Use the Field field to enter the field in the secondary file that is being returned
by the operation.
Description
Use the Description field to enter a description for the target transform
column.
Using the DataOrchestrator™ ODS, July 9, 2013
123
Setting Up Targets: Defining Target Transforms
The Null Value Replacement (DONV) Form
Detail from the DOTC form to use this form to define a null value
replacement operation.
If you selected the Null Replacement operation for this column on the
Transform Columns (DOTC) form, you can specify a value in the If Null
Return field that will be returned if the source field value for the column is
null. However, the data type in this field must match the data type of the
source field.
If you selected the Null Test operation for this column on the DOTC form,
you can specify a value for both the If Null Return and If Not Null Return
fields to create a Null Test Operation to choose the value in the column based
on whether the source field value is null or not. However, the data types of the
two fields must match. For example, to output a column in a transform
indicating whether a Faculty member has a special status, you can specify the
Null Test operation with an input field of FAC.SPECIAL.STATUS and enter
No in the If Null Return field and Yes in the If Not Null Return field. In
addition, you can enter a description of the target column.
Figure 20: Null Value Replacement (DONV) Form
124
Using the DataOrchestrator™ ODS, July 9, 2013
Defining Target Transforms
Noteworthy Fields on the DONV Form
The fields described in this section are particularly important when defining a
a null value replacement operation.
Target ID
The Target ID field displays the ID of the target associated with the target
transform.
Target Transform
The Target Transform field displays the name of the current target transform.
Target Column
The Target Column field displays the name of the target column from the
transform that will have its value determined by the result of the operation.
Source File
The Source File field displays the name of the source file on which the
definition of the target transform is based.
Source Field
The Source Field field displays the source field to which the target column is
mapped. This is the field against which the operation will be applied. The
source field can also be another target column, as long as it precedes the
column in the transform’s column list.
If Null Return
Use the If Null Return field to enter a value to replace the source field data
when that data is null. You can also enter quoted strings, numeric values,
fields from the source file, or target columns from the associated transform
defined before the column.
Using the DataOrchestrator™ ODS, July 9, 2013
125
Setting Up Targets: Defining Target Transforms
If Not Null Return
Use the If Not Null Return field to enter a value to replace the source field
data when that data is not null. You can also enter quoted strings, numeric
values, fields from the source file, and target columns from the associated
transform defined before the column.
Description
Use the Description field to enter a description for the target column.
126
Using the DataOrchestrator™ ODS, July 9, 2013
Defining Target Transforms
The String Concatenation (DOCA) Form
Detail from the DOTC form to use this form to define the specifications for a
concatenation operation. All arguments must be single-valued source data
elements or literals. You can also enter a description of the target column.
Figure 21: String Concatenation (DOCA) Form
Noteworthy Fields on the DOCA Form
The fields described in this section are particularly important when defining a
a string concatenation operation.
Target ID
The Target ID field displays the ID of the target associated with the target
transform.
Target Transform
The Target Transform field displays the name of the current target transform.
Using the DataOrchestrator™ ODS, July 9, 2013
127
Setting Up Targets: Defining Target Transforms
Target Column
The Target Column field displays the name of the target column from the
transform that will have its value determined by the result of the operation.
Source File
The Source File field displays the name of the source file on which the
definition of the target transform is based.
Source Field
The Source Field field displays the source field to which the target column is
mapped. This is the field against which the operation will be applied. The
source field can also be another target column, as long as it precedes the
column in the transform’s column list.
Concatenation Operand List
Use the Concatenation Operand List field to enter quoted strings, singlevalued string type fields from the source file, or single-valued string type
target columns from the associated transform defined before the column.
Description
Use the Description field to enter a description for the target column.
128
Using the DataOrchestrator™ ODS, July 9, 2013
Defining Target Transforms
The Substring Selection (DOSS) Form
Detail from the DOTC form to use this form to define the specifications for a
substring operation. You can also enter a description of the target column.
Figure 22: Substring Selection (DOSS) Form
Noteworthy Fields on the DOSS Form
The fields described in this section are particularly important when defining a
a substring operation.
Target ID
The Target ID field displays the ID of the target associated with the target
transform.
Target Transform
The Target Transform field displays the name of the current target transform.
Using the DataOrchestrator™ ODS, July 9, 2013
129
Setting Up Targets: Defining Target Transforms
Target Column
The Target Column field displays the name of the target column from the
transform that will have its value determined by the result of the operation.
Source File
The Source File field displays the name of the source file on which the
definition of the target transform is based.
Source Field
The Source Field field displays the source field to which the target column is
mapped. This is the field against which the operation will be applied. The
source field can also be another target column, as long as it precedes this
column in the transform’s column list.
Starting Position
Use the Starting Position field to enter or view the starting position of the
substring for the source field.
Substring Length
Use the Substring Length field to enter the length of the string to be returned
from the substring of the source field, based on the Starting Position field.
Description
Use the Description field to enter a description for the target column.
130
Using the DataOrchestrator™ ODS, July 9, 2013
Defining Target Transforms
The Validation Code LookUp (DOVC) Form
Detail from the DOTC form to use this form to define the specifications for a
validation code lookup operation. This form allows you to enter a validation
code table name that corresponds to the value in the source field and choose
which field to return from the validation code. You can also enter a
description of the target column.
Figure 23: Validation Code LookUp (DOVC) Form
Noteworthy Fields on the DOVC Form
The fields described in this section are particularly important when defining a
a validation code lookup operation.
Target ID
The Target ID field displays the ID of the target associated with the target
transform.
Using the DataOrchestrator™ ODS, July 9, 2013
131
Setting Up Targets: Defining Target Transforms
Target Transform
The Target Transform field displays the name of the current target transform.
Target Column
The Target Column field displays the name of the target column from the
transform that will have its value determined by the result of the operation.
Source File
The Source File field displays the name of the source file on which the
definition of the target transform is based.
Source Field
The Source Field field displays the source field to which the target column is
mapped. This is the field against which the operation will be applied. The
source field can also be another target column, as long as it precedes the
column in the transform’s column list.
Validation Code Name
Use the Validation Code Name field to enter the validation code table name
that corresponds to the value in the source field. You can select a validation
code from any installed application validation code table. This operation will
match the value in the input field with the appropriate code from the
validation code table specified in the Validation Code Name field.
Return Field Name
Use the Return Field Name field to select which field you want to return from
the validation code for this column. This identifies the information associated
with the matching validation code value that the transform will return as the
value of the operation. Typically, you will enter description to return
the text description of the validation code value.
132
Using the DataOrchestrator™ ODS, July 9, 2013
Defining Target Transforms
Description
Use the Description field to enter a description for the target column.
Using the DataOrchestrator™ ODS, July 9, 2013
133
Setting Up Targets: Defining Target Transforms
The Expression Entry (DOEE) Form
Detail from the DOTC form to use this form to define the specifications for an
operation manually. You can also enter a description of the target column.
When you save from the DOEE form, the expression is run against its source
file to verify the syntax. If the expression fails, then the Export (Y/N) field on
the Transform Columns (DOTC) form is set to “No” for the corresponding
column to prevent an error from being encountered when the transform is run.
Figure 24: Expression Entry (DOEE) Form
Noteworthy Fields on the DOEE Form
The fields described in this section are particularly important when defining a
an expression entry operation.
Target ID
The Target ID field displays the ID of the target associated with the target
transform.
134
Using the DataOrchestrator™ ODS, July 9, 2013
Defining Target Transforms
Target Transform
The Target Transform field displays the name of the current target transform.
Target Column
The Target Column field displays the name of the target column from the
transform that will have its value determined by the result of the operation.
Source File
The Source File field displays the name of the source file on which the
definition of the target transform is based.
Source Field
The Source Field field displays the source field selected for the column.
Expression
Use the Expression field to enter the expression to use to return a value for the
column.
Note: If you use a literal string argument, the argument must be
quoted.
The following operation names and arguments are available.
 Field Extract.
SubFieldExtract(FieldName,“Delimiter”,StartFieldNum,NumFieldsTo
Extract). For example:
SubFieldExtract(INSTITUTIONS_ATTEND_ID,"*",1,1)
 Multivalue Aggregation. ListAggregate(FieldName,“Function”). For
example:
ListAggregate(SEC.WITHDRAWN.STUDENTS,"COUNT")
 Multivalue Position.
ListElementByPos(FieldName,PositionInMultiValue). For example:
ListElementByPos(APP.RECRUIT.PURPOSES,3)
Using the DataOrchestrator™ ODS, July 9, 2013
135
Setting Up Targets: Defining Target Transforms






Pointer Reference.
PointerRef(SourceFieldName,“TargetFileName”,“TargetFieldName”). For
example:
PointerRef(INV.TERM,"TERMS","TERM.DESC")
Null Replacement. ReplaceNull(SourceFieldName,AlternateSourceField/
String). For example:
ReplaceNull(ARD.REVERSAL.AMT,0)
If Null. IfNull(SourceFieldName,NullReturnField/
String,NotNullReturnField/String). For example:
IfNull (INV.SPONSOR.PAYMENT, "MC","SB")
String Concatenation. ConCat(SourceFieldName/
String1,SourceFieldName/String2,...SourceFieldName/StringX). For
example:
ConCat(STPR_STUDENT,"*",STPR_ACAD_LEVEL)
Substring. SubString(SourceFieldName,StartPosition,Length). For
example:
SubString (CITY, 1, 10)
Validation Code LookUp.
AssocLookupByPtr(“ValcodeName”,“TargetFile”,“LookupField”,
LookupValue,“ReturnField”). For example:
AssocLookupByPtr("GRADUATION.TYPES",
"CORE.VALCODES","VAL.INTERNAL.CODE",
INSTA.GRAD.TYPE,"VAL.EXTERNAL.REPRESENTATION")
Note: Operations can be nested; operations can be used anywhere
that a Field Name is referenced in these syntax descriptions.
Note: You can use only fields from the source file in an expression;
you cannot use previously defined fields of the transform.
Description
Use the Description field to enter a description for the target column.
136
Using the DataOrchestrator™ ODS, July 9, 2013
Defining Target Transforms
Procedure for Defining Target Transforms
See “Procedure for Defining a Target” beginning on page 151.
Using the DataOrchestrator™ ODS, July 9, 2013
137
Setting Up Targets: Defining Target Transforms
138
Using the DataOrchestrator™ ODS, July 9, 2013
Setting Up Targets
Defining and Creating SQL Views
In This Chapter
This chapter describes how to define, create, and maintain SQL views for a
target. SQL views are an optional feature that you can utilize, and they can be
particularly helpful for your reporting users. Views simplify accessing
information in your operational data store and provide predefined queries that
can be shared by all reporting users.
You also have the option to indicate if an SQL view should be created as a
materialized view. This means that the SQL statement that defines the view is
run and all the resulting data rows are stored in a database table. This may
significantly improve the response time for queries using the SQL view.
However, this also means that the data content of the view must be refreshed
whenever the tables providing data for the view are updated. You can either
directly update the data content for a materialized view, or set up a
materialized view to refresh automatically whenever any of the transforms on
which it depends are updated.
Setting up SQL views is a two-step process. You must first define the view,
then create the view on the ODS target database. This allows you to control
when views are created on the ODS target database.
Table 22 lists the topics covered in this chapter.
Table 22: Topics in This Chapter
Topic
Page
Forms Used
140
Defining SQL Views
141
Defining the SQL Select Statement for an SQL View
143
Procedure for Defining SQL Views
147
Creating SQL Views
148
Procedure for Creating SQL Views Using the DOTV Form
149
Using the DataOrchestrator™ ODS, July 9, 2013
139
Setting Up Targets: Defining and Creating SQL Views
Forms Used
Table 23 lists and describes the forms used in this chapter.
Table 23: Forms Used in This Chapter
Form
140
Purpose
DataOrch Target Views (DOTV)
Define, create, and maintain SQL views
for a target, and view information about
the statuses of these SQL views.
DataOrch View Spec (DOVS)
Modify the select statement used to
create an SQL view, modify the column
documentation for the SQL view, or
define or modify the materialized view
parameters for the SQL view. Test the
SQL view select statement on the ODS
target database.
Using the DataOrchestrator™ ODS, July 9, 2013
Defining SQL Views
Defining SQL Views
Detail on the SQL Views field from the DataOrch (DOTA) form. This allows
you to access the DataOrch Target Views (DOTV) form where you can view
or define the SQL views associated to a target.
When you detail from the DOTA form, you can only define SQL views; you
cannot create them, so the following fields are not accessible:
 View Creation Selection
 Create on Save (Y/N)
Note: When you are ready to create views for a target, you have two
options for creating them. For more details, see “Creating SQL Views”
on page 148.
Figure 25: DataOrch Target Views (DOTV) Form
Status information is available for the SQL view, including when the SQL
view was last modified and the last date it was successfully created on the
ODS target database.
Using the DataOrchestrator™ ODS, July 9, 2013
141
Setting Up Targets: Defining and Creating SQL Views
Noteworthy Fields on the DOTV Form
The fields described in this section are particularly important for defining
SQL views associated to a target.
SQL View Name
In the SQL View Name field, enter a name for the SQL view that you want to
associate to the target. The name you enter here will be used as the name of
the corresponding SQL view when the view is created on the ODS target
database.
Detail from this field to access the DataOrch View Spec (DOVS) form if you
want to define or modify the SQL select statement, column documentation, or
materialized view parameters for an SQL view.
Note: Deleting a name from the SQL View Name list does not
automatically delete the corresponding SQL view on the target
database. This must be done manually by the system administrator.
Modify Date
The Modify Date field shows the date of the last modification of the SQL
view.
Success Date
The Success Date field shows the date of the last successful creation of the
view on the ODS target database. If the SQL view is modified, or an error is
encountered when creating it on the target database, the Success Date will be
reset to a null value.
Material View
The Material View field displays “Yes” if the SQL select statement defined on
the DataOrch View Specs (DOVS) form will produce a table on the target
database. If this field is blank, the SQL select statement will produce a view
on the target database. You can detail on the SQL View Name field to change
this setting.
142
Using the DataOrchestrator™ ODS, July 9, 2013
Defining SQL Views
Defining the SQL Select Statement for an SQL View
To view and define the SQL statement, column documentation, or the
materialized view parameters for an SQL view, detail on an SQL View Name
from the DOTV form to the DataOrch View Spec (DOVS) form.
Use the DOVS form to perform the following functions:
 View or modify the select statement used to create the SQL view on the
ODS target database.
 View or modify the description.
 View or modify the column documentation for the SQL view.
 View or set the option indicating that a view column or set of view columns
make up the primary key of the SQL view.
Note: This information is used to create the primary key on the
database table for a materialized view. For SQL views that are not
marked as materialized, this is informational only.



Test the SQL view select statement on the ODS target database.
Set the option indicating the view is a materialized view.
View or modify the list of ODS target transforms on which a materialized
view depends.
Figure 26: DataOrch View Spec (DOVS)
Using the DataOrchestrator™ ODS, July 9, 2013
143
Setting Up Targets: Defining and Creating SQL Views
Status information is also available for the SQL view, including when the
SQL view was last modified and the last date it was successfully created on
the ODS target database.
Note: There can be two versions of an SQL view, one with Oraclespecific SQL syntax and one with SQL Server syntax. In the header of
the DOVS form, the version is displayed in the Database Type field.
This is set up in the Database Type field on the DataOrch Target
(DOTA) form for the associated target.
Noteworthy Fields on the DOVS Form
The fields described in this section are particularly important for defining
SQL select statements and column documentation for an SQL view.
SQL Select Statement
Use the SQL Select Statement field to enter an SQL select statement used to
create the SQL view on the ODS target database. For example:
SELECT ID, LAST_NAME, BIRTH_DATE FROM PERSON WHERE
LAST_NAME EQ “Smith”
The &schema& keyword is also available for use in the SQL Select
Statement. For example:
&schema&.TABLE_NAME will be replaced with the schema name (dbo for
SQL Server) of the target database when the view is created.
Note: SELECT must be the first word of the SQL Select Statement.
144
Using the DataOrchestrator™ ODS, July 9, 2013
Defining SQL Views
Key
In the Key field, enter Yes to identify a column as being part of a
materialized view’s key. By entering “Yes”, this column will be included in
the primary key for the table created for the materialized view. If more than
one column is marked “Yes”, then a multi-column key will be created on this
table, and the order of the columns in the key will be in the same order in
which they are listed on the DOVS form.
Note: The set of columns identified as the primary key must contain
unique values, or the creation of the key will fail when the materialized
view is refreshed.
Description
In the Description field, enter a description for the SQL view.
SQL View Columns
In the SQL View Columns field, enter the column names in the SQL select
statement. For materialized SQL views, the SQL View Column entries that
have the Key field set to “Yes” are used to create the primary key for the table
on the database. These field names must exist in the SQL view or the creation
of the materialized SQL view will fail. Otherwise, the column names entered
here are for your institution's documentation purposes only. No column names
will be added to the SQL select statement. Also, the SQL select statement will
not be checked to verify the column names exist in the select statement.
Description
Enter a description for the SQL view column.
Create Materialized View (Y/N)
In the Create Materialized View (Y/N) field, enter “Yes” to create a table on
the target database from the SQL select statement. Creating a table from the
SQL select statement allows you to create indexes and foreign keys on the
resulting target table. Enter “No” if you want to create a view on the target
database from the SQL select statement.
Using the DataOrchestrator™ ODS, July 9, 2013
145
Setting Up Targets: Defining and Creating SQL Views
Dependent Transforms
In the Dependent Transforms field, enter the transforms on which this
materialized view is dependent.
Dependent transforms are required when defining a materialized view. If the
SQL select statement references a transform, then enter the transform’s name
in this list.
Note: When you run a refresh using the DataOrch Refresh (DORE)
form, if the refresh processes a transform on which a materialized view
is dependent, then the materialized view is automatically updated.
This ensures that the most recent data from the transform is included
in the materialized view.
Test SQL View Now
Detail on the Test SQL View Now field to test the SQL view on the ODS
target database. The select statement will be validated by creating a temporary
view named DATAORCH_TEST_VIEW on the ODS target database, which
will be deleted after the view is created.
Note: A select statement must be entered in the SQL Select
Statement field in order to attempt to test the SQL view.
146
Using the DataOrchestrator™ ODS, July 9, 2013
Defining SQL Views
Procedure for Defining SQL Views
Step 1. Access the DOTV form by detailing on the SQL Views field from the
DataOrch (DOTA) form.
Step 2. Define the SQL View Names on the DOTV form. For more details, see online
help.
Step 3. If you want to define or modify the select statement, column documentation,
and/or materialized view parameters for the SQL view, detail on the SQL
View Name field to access the DataOrch View Spec (DOVS) form.
a. Enter the SQL select statement that defines the view and the column
documentation on the DOVS form. For more details, see online help.
b. If you want to create a materialized view, enter Yes in the Create
Materialized View (Y/N) field, and then enter the transforms on which the
materialized view is dependent in the Dependent Transforms field. For
more details, see online help.
c. If you want to test the SQL view on the ODS target database, detail on the
Test SQL View Now field.
d. When you are finished viewing the results of the test, save from the DOVS
form.
Step 4. Save from DOTV form to return to the DOTA form.
Using the DataOrchestrator™ ODS, July 9, 2013
147
Setting Up Targets: Defining and Creating SQL Views
Creating SQL Views
After you have defined your SQL views for a target and saved from the
DOTV form, you must save from the DOTA form before you can create your
SQL views. This saves all of your SQL view definitions for the target.
When you are ready to create the SQL views on the ODS target database, you
have two options for creating the views:
 Use the DataOrch Refresh (DORE) form. Creating views on this form
allows you an easy way to create all the views by simply running a refresh
of the target. For more details, refer to “Defining and Running an ODS
Refresh” beginning on page 159.
 Access the DOTV form from the Quick Access field. If you access the
DOTV form from the Quick Access field (not by detailing from the DOTA
form), you can create the SQL views. This form allows you more flexibility
in controlling which views you want to create on your ODS target database.
Note: In addition to these two options for creating views, any SQL
view defined as a materialized view will be refreshed when any of the
target transforms defined in the Dependent Transforms list on the
DOVS form is refreshed.
148
Using the DataOrchestrator™ ODS, July 9, 2013
Creating SQL Views
Procedure for Creating SQL Views Using the DOTV
Form
Use the following steps to create SQL views using the DOTV form.
Step 1. Access the DOTV form by entering DOTV in the Quick Access field. At the
DataOrch Target LookUp prompt, enter the target for which you want to
create SQL views.
The following fields are now accessible:
 View Creation Selection
 Create on Save (Y/N)
Figure 27: DataOrch Target Views (DOTV) Form
Using the DataOrchestrator™ ODS, July 9, 2013
149
Setting Up Targets: Defining and Creating SQL Views
Step 2. (Optional) Choose an option from the View Creation Selection drop-down
list. The following options are available:
 All Views. This option updates the Create on Save (Y/N) field to Yes for
all SQL views.
 Unsuccessful Views. This option updates the Create on Save (Y/N) field to
Yes for all SQL views without a corresponding Success Date.
 Deselect Views. This option updates the Create on Save (Y/N) field to No
for all SQL views.
Step 3. Alternately, you can use the Create on Save (Y/N) field to select views to
create. Enter Yes to create one or more SQL views on the ODS target
database when saving from this form; otherwise, enter No.
Note: If an SQL view does not receive errors during creation, then its
corresponding Success Date field will be updated to the current date.
Otherwise, the Success Date field for the SQL view will be cleared.
Step 4. Save your entries on the DOTV form.
When you save from the DOTV form, SQL views are created if their
associated Create on Save (Y/N) field is set to Yes, and if no errors are
encountered.
150
Using the DataOrchestrator™ ODS, July 9, 2013
Setting Up Targets
Procedure for Defining a Target
In This Chapter
This chapter describes the procedure for defining a target.
Table 24 lists the topics covered in this Chapter.
Table 24: Topics in This Chapter
Topic
Procedure for Defining a Target
Using the DataOrchestrator™ ODS, July 9, 2013
Page
152
151
Setting Up Targets: Procedure for Defining a Target
Procedure for Defining a Target
ALERT! Ellucian highly recommends that you secure access to
the DOTA form.
Perform the steps below to define or modify a target.
Note: Do not use an existing Colleague database as a target
database for an operational data store. If you try to do this, you will
receive an error message when you run the refresh.
Step 1. Access the DataOrch Target (DOTA) form.
Step 2. Enter the information for this target. See online help for further information.
ALERT! The target database must be refreshed by only one
target. If two targets refresh the same target database, the
operational data store status information will be inaccurate, the
data written to that database can be corrupted, and the refresh
may encounter errors.
Step 3. (Required) On the DOTA form, detail on the Temporary File Path field to
access the Additional Target Parameters (DOTP) form.
a. (Target Windows server only). Enter the target drive name of the
temporary file path.
b. Enter the directory path on the ODS target database server where the
temporary files, created by the refresh, will be stored.
ALERT! This directory must have sufficient space allocated for
these temporary files. The space needed for temporary files can
be significant depending on how you define the target. Ellucian
recommends that you monitor the space usage in this directory
as your refresh runs.
Be sure that access to this directory is in accordance with the
online help for the Temporary File Path field on DOTP. Also, the
temporary files in this directory may contain data that might be
considered sensitive at your institution. Access to the directory
should be restricted accordingly.
152
Using the DataOrchestrator™ ODS, July 9, 2013
In This Chapter
c. Save and exit from the DOTP form.
Step 4. On the DOTA form, to add a source transform to this target, enter the file
name in the Source Transforms table. Source transforms allow you to specify
the data to move to an ODS target database by selecting a set of fields and/or
computed columns from a single Colleague file.
File names entered here can be refreshed to an ODS target database by using
the DataOrch Refresh (DORE) form. If you specify a file containing stored
computed columns in this field, you must run the Update Stored Computed
Column (USCC) process to ensure that the stored computed columns are
refreshed before the DORE process is run. For more information on using
stored computed columns with the DataOrchestrator ODS, see Support
Solution 6877. (If the target also contains the STUDENT.TERMS.CC file, the
Update GPA SCC Flags [UGSF] process should be run before the USCC
process.)
Note: Deleting a file from the Source Transforms table does not
automatically delete the corresponding table on the target database.
This must be done manually by the system administrator.
Step 5. If you are adding a source transform, Colleague automatically opens the
DataOrch Source Transform (DOST) form and can specify which fields and/
or computed columns of the file to include in this source transform. To view
or modify an existing transform, detail on the specific source transform to
access the DOST form. See online help for further information.
a. (Optional) On the DOST form, detail on the Filters/File Suite Instances
field to access the Filter Criteria (DOFC) form in order to enter, view, or
modify filter criteria for this source transform. Colleague uses this criteria
to specify what records from the input file will be processed by the source
transform when it is run. See online help for further information. Save and
exit from the DOFC form to return to the DOST form.
b. (Optional) On the DOST form, if you want to see the set of tables that will
be created on the ODS target database, detail on the View Generated
Transforms field to access the Generated Target Transforms (DOGT) form
to view the target transforms generated from this source transform. Save
and exit from the DOGT form to return to the DOST form.
c. Save and exit from the DOST form.
Using the DataOrchestrator™ ODS, July 9, 2013
153
Setting Up Targets: Procedure for Defining a Target
Step 6. On the DOTA form, to add a target transform to this target, enter a name for
the transform in the Target Transforms table. The target transform name you
enter will be used as the name of the corresponding target table when this
transform is refreshed using the DORE process.
Note: Deleting a name from the Target Transforms table does not
automatically delete the corresponding table on the target database.
This must be done manually by the system administrator.
Target transforms allow you to specify the format of the data on the target
database, as well as any data transformation operation to run when the data is
moved.
Step 7. If you are adding a target transform, Colleague automatically opens the
DataOrch Target Transform (DOTT) form to define the transform. To view or
modify an existing transform, detail on the specific target transform to access
the DOTT form. See online help for further information.
a. On the DOTT form, detail on the Column Definitions field to access the
Transform Columns (DOTC) form. Use the DOTC form to add, view, or
modify the output columns for the transform. Each column is defined by
an input field (and an optional transformation operation) that produces the
output value for the corresponding target database column. See online help
for further information.
If you specify a stored computed column as an input field, you must run
the Update Stored Computed Column (USCC) process to ensure that the
stored computed columns are refreshed before the DORE process is run.
For more information on using stored computed columns with the
DataOrchestrator ODS, see Support Solution 6877. (If you specify a stored
computed column from the STUDENT.TERMS.CC file, the Update GPA
SCC Flags [UGSF] process should be run before the USCC process.)
b. (Optional) On the DOTC form, detail on a target column name to access
the Column Properties (DOPR) form in order to specify properties for this
column. See online help for further information. If you detailed, save and
exit from the DOPR form to return to the DOTC form.
c. (Optional) On the DOTC form, you can choose a transformation operation
to perform in order to generate the output value for the target database
154
Using the DataOrchestrator™ ODS, July 9, 2013
In This Chapter
column. To do this, select an operation from the Operation field dropdown list. The following options are available:
 Field Extract. Details to the Field Extract (DOFE) form.
 Multivalue Aggregation. Details to the Multivalue Operation
(DOMV) form.
 Multivalue Position. Details to the Multivalue Operation (DOMV)
form.
 Pointer Reference. Details to the Pointer Reference (DOFR) form.
 Null Replacement. Details to the Null Value Replacement (DONV)
form.
 Null Test. Details to the Null Value Replacement (DONV) form.
 String Concatenation. Details to the String Concatenation (DOCA)
form.
 Substring. Details to the Substring Selection (DOSS) form.
 Validation Code LookUp. Details to the Validation Code LookUp
(DOVC) form.
 Expression Entry. Details to the Expression Entry (DOEE) form.
See online help for further information on these forms.
d. If you detailed from the Operation field on DOTC to use any of the forms
mentioned in the list above, save and exit from the form to return to the
DOTC form.
e. Save and exit from the DOTC form to return to the DOTT form.
f. (Optional) On the DOTT form, detail on the Filters/File Suite Instances
field to access the DOFC form in order to enter, view, or modify filter
criteria for this target transform. Colleague uses this criteria to specify
what records from the source file are processed by the target transform
when it is run. See online help for further information. Save and exit from
the DOFC form to return to the DOTT form.
Step 8. If you have finished defining target transforms, save and exit from the DOTT
form to return to the DOTA form.
Using the DataOrchestrator™ ODS, July 9, 2013
155
Setting Up Targets: Procedure for Defining a Target
Step 9. (Optional) On the DOTA form, detail on the SQL View field to define SQL
views.
a. Define the SQL View Names for the views on the DataOrch Target Views
(DOTV) form.
b. Detail on the SQL View Name field to define the select statement and
column documentation for a view on the DataOrch View Spec (DOVS)
form.
c. Optionally, on the DOVS form, you can indicate that the SQL view should
be created as a materialized view by entering Yes in the Create
Materialized View field, and then adding all transforms referenced in the
SQL view definition to the Dependent Transforms list. You should also
enter Yes in the Key field for the view columns defining the primary key
for the materialized view, as this will greatly improve the performance of
queries against the materialized view.
d. Optionally, you can detail on the Test SQL View Now field to test the SQL
view on the ODS target database, then save to return to the DOVS form.
e. Save and exit from both the DOVS and DOTV forms
Step 10. If you have finished defining the target, save and exit from the DOTA form.
Step 11. To export data to the target database, run the DataOrch Refresh (DORE)
process. If you want, you can also create all SQL views for the target when
running the refresh. For more information, see “Defining and Running an
ODS Refresh” beginning on page 159.
Alternately, you can create the SQL views without running a refresh by
accessing the DOTV form from the Quick Access menu. For more
information, see “Procedure for Creating SQL Views Using the DOTV Form”
on page 149.
156
Using the DataOrchestrator™ ODS, July 9, 2013
Using the DataOrchestrator™ ODS
Refreshing ODS Data
Refreshing ODS Data
Defining and Running an ODS
Refresh
In This Chapter
This chapter describes how to define the parameters that control the way the
DataOrchestrator ODS refreshes a target database with information from
Colleague source files.
Table 25 lists the topics covered in this chapter.
Table 25: Topics in This Chapter
Topic
Page
Forms Used
159
Defining Settings for Refreshing Data
160
Defining Additional Parameters for the Refresh
167
Procedure for Defining a Refresh
170
Forms Used
Table 26 shows the forms used in this chapter.
Table 26: Forms Used to Set Up Refreshes for ODS Target Databases
Form
Purpose
DataOrch Refresh (DORE)
Specify or update settings that control
refreshes of a target database. Run a
refresh that updates the specified target
database.
Additional Refresh Parameters
(DORP)
Enter additional parameters for a refresh.
Using the DataOrchestrator™ ODS, July 9, 2013
159
Refreshing ODS Data: Defining and Running an ODS Refresh
Defining Settings for Refreshing Data
Use the DataOrch Refresh (DORE) form to define the settings to be used
when doing an initial export, or when updating data from the Colleague
source environment to an ODS target database. You can also use the DORE
form to run an export of data to this ODS target database and to view the
statistics that resulted from the previous export. Each Refresh ID that you set
up is associated with one specific target.
When running a refresh, you can specify a default Update Type (Full or
Incremental). A Full update processes all records of the source file for the
transform. An Incremental update processes only those records of the source
file that either:
 Changed since the last refresh if the transform has an Incremental Date
Field, or
 Match the Incremental Filter Criteria.
If you choose Incremental, only the non-computed column data elements in
the source file will be checked for updates since the last refresh. This means
that if the transform references data elements from other Colleague files
(either directly or through the use of computed columns), no dependency
checking is done on these elements. Updates to their values will not
automatically trigger the inclusion of these input elements in the Incremental
update. In addition, foreign key creation is skipped during an incremental
refresh.
You can also choose to include specific source or target transforms in the
refresh, or you can include all source and/or target transforms.
If you enter specific transforms (in the Include Source Transforms and/or
Include Target Transforms tables), you can choose to override the default
Update Type for a transform to either Full or Incremental.
The DataOrch Target (DOTA), DataOrch Source Transform (DOST), and
Transform Columns (DOTC) forms allow you to select which fields and
computed columns will be refreshed for each transform. The fewer fields (and
particularly the fewer computed columns) that are referenced for each
transform, the better the refresh performance will be. However, if any of these
output fields change for a transform, then a full refresh will be done for the
transform, and the target table will be dropped and recreated. Depending on
the transform, this could significantly impact the time needed for the
subsequent refresh.
160
Using the DataOrchestrator™ ODS, July 9, 2013
Defining Settings for Refreshing Data
You can also specify whether, when this refresh is run, you want to create all
of the SQL views associated to the target.
Note: When you run a refresh, if the refresh processes a transform on
which a materialized view is dependent, then the materialized view is
automatically updated. This ensures that the most recent data from the
transform is included in the materialized view.
To improve performance of the refresh, you can specify the number of
separate concurrent Envision processes to use for a refresh. Using more
concurrent Envision processes to run a refresh will result in better
performance for the refresh, but will use resources on the application server.
This may also impact performance of other Colleague processes.
In addition, you can specify whether you want a refresh to run immediately, or
choose just to save the entered settings for this target.
Technical Tip: When the refresh of an ODS target database takes
place, all information stored in Colleague about the target, the
transforms and SQL views being refreshed, and the refresh are written
to the target database. This information can be queried on the target
database using SQL-based tools.
Technical Tip: When you run a full refresh (after the initial refresh),
each ODS target database table is cleared only after the data extract
and file transfer occurs for the corresponding transform, so that users
are minimally impacted.
You can view the refresh status on the DORE form. The following statuses
may be displayed in the header:
 InProc. A transform is currently being refreshed.
 Success. The transform was successfully refreshed without any errors.
 Warning. The transform or refresh received errors during processing.
 Fail. The transform or refresh received fatal errors during processing.
Note: A maximum of five transforms are allowed to fail before the
refresh will be stopped automatically.
You can also detail from the DORE form to access the Additional Refresh
Parameters (DORP) form.
Using the DataOrchestrator™ ODS, July 9, 2013
161
Refreshing ODS Data: Defining and Running an ODS Refresh
Figure 28: DataOrch Refresh (DORE) Form
When you first enter the DORE form, Colleague displays the DataOrch
Refresh LookUp prompt where you can enter one of the following:
 An ID for a refresh that you want to add and set up.
 An ID for an existing refresh for which you want to run a refresh and/or
modify the settings.
Noteworthy Fields on the DORE Form
The fields described in this section are particularly important when defining
the settings for an ODS refresh.
Target ID
Use the Target ID field to enter one of the following for a new Refresh ID:
 The ID for a target that you want to create and set up.
 The ID for an existing target that you want to associate with this new
Refresh ID.
Note: After you specify a Target ID for this Refresh ID and then save,
you cannot change the Target ID.
162
Using the DataOrchestrator™ ODS, July 9, 2013
Defining Settings for Refreshing Data
Also, use this field to view or modify the target information that is associated
with an existing Refresh ID. To do this, detail to the DataOrch Target (DOTA)
form.
For more information on creating or modifying a target, see “Defining a
Target” beginning on page 67.
RDAS Listener
Use the RDAS Listener field to enter the name of the DMI Listener with the
RDAS role on the ODS target database server. This Listener will be used to
refresh data from the source Colleague environment to the ODS target
database.
DMI Listener Host Name
The DMI Listener Host Name field displays the TCP/IP address or DNS alias
name of the computer where the DMI Listener is installed. This is the DMI
Listener with the RDAS role on the ODS target database server.
Port
The Port field displays the port number associated to the DMI Listener.
Update Type
When running a refresh, you can specify whether all records of the input files
for the listed transforms should be processed (Full), or only those records of
the input files that have changed since the last refresh (Incremental).
Transform Selection Option
Use the Transform Selection Option field to select one of the following
options for this refresh:
 All source transforms. All source transforms from the associated target
will be refreshed, using the default in the Update Type field. In addition,
Using the DataOrchestrator™ ODS, July 9, 2013
163
Refreshing ODS Data: Defining and Running an ODS Refresh
you can enter specific target transforms to be refreshed in the Include
Target Transforms table (and override the default Update Type).
Note: Choosing this option clears previous entries in the Include
Source Transforms table and the associated Override Update Type
table. The tables become inquiry only.

All target transforms. All target transforms from the associated target will
be refreshed, using the default in the Update Type field. You can also enter
specific source transforms to be refreshed in the Include Source Transforms
table (and override the default Update Type).
Note: Choosing this option clears previous entries in the Include
Target Transforms table and the associated Override Update Type
table. The tables become inquiry only.

All transforms. All source and target transforms from the associated target
will be refreshed, using the default defined in the Update Type field.
Note: Choosing this option clears previous entries in the Include
Source Transforms and the Include Target Transforms tables, as well
as the associated Override Update Type tables. The tables become
inquiry only.

Specific transforms. You can enter specific source and/or target
transforms to be refreshed in the Include Source Transforms and/or Include
Target Transforms tables (and override the default Update Types).
Include Source Transforms/Include Target Transforms
If you want to include only specific transforms in the refresh, use the Include
Source Transforms and/or the Include Target Transforms tables to list those
transforms. Use LookUp to select one or more transforms to process from a
list of transforms associated with this target.
Override Update Type
If you enter transforms in the Include Source Transforms and/or Include
Target Transform tables, you can also choose to override the default Update
Type for a specific transform and control whether all records will be updated
or only those records changed since the last update.
164
Using the DataOrchestrator™ ODS, July 9, 2013
Defining Settings for Refreshing Data
However, if a target transform is set to Refresh as Full Only on the DataOrch
Target Transform (DOTT) form, the target transform will be refreshed as a
full update regardless of the setting in this field.
Check Connectivity
To test connectivity, detail on the Check Connectivity field to log in to the
specified target database in order to verify the connection.
Note: To test connectivity, the DataOrch Target (DOTA) form must
already be set up with the DMI Listener Host Name, DMI Listener Port,
and the associated target information. If this information has not been
entered, the following error is displayed: Incomplete Setup.
Create SQL Views (Y/N)
Use the Create SQL Views (Y/N) field to indicate whether or not you want to
create all of the SQL views associated to the target, including any
materialized views. Enter Yes to create the SQL views; otherwise, enter No.
Note: This setting allows you an easy way to create all the views for
a target when you run the refresh. However, you do not need to create
all the views each time you refresh a target if none of the definitions for
the views have changed.
You also have the option of using the DOTV form instead to create
views. The DOTV form allows you more flexibility in controlling which
views you want to create on your ODS target database. For more
information, see “Defining and Creating SQL Views” beginning on
page 139.
Temporary File Path
Detail on the Temporary File Path field to access the Additional Refresh
Parameters (DORP) form to enter the source drive name and temporary file
path for this refresh.
Run Stored Procedure
In the Run Stored Procedure field, enter Yes to run the stored procedure for
this refresh ID.
Using the DataOrchestrator™ ODS, July 9, 2013
165
Refreshing ODS Data: Defining and Running an ODS Refresh
After the refresh runs, indicate whether you want to run the stored procedure
that was created for this refresh ID. Enter Yes to run the stored procedure;
otherwise, enter No.
Note: You can create a stored procedure in the target database that
can be run after a refresh is complete. This stored procedure must be
named for the refresh ID, with _SP appended to the end. For example,
if the refresh ID is D01_REFRESH, the stored procedure must be
named D01_REFRESH_SP.
The stored procedure has access to the status information for the
DataOrchestrator ODS, which allows you to create a stored procedure that
contains more complex processing. The status information for the entire
refresh is stored in the DO_REFRESH table, and the refresh status for
individual transforms is stored in the DO_TARGET_SPEC table on the target
database.
Run Refresh (Y/N)
Use the Run Refresh (Y/N) field to indicate whether or not you want to
refresh the selected target database when you save your entries on the DORE
form. Enter Yes to run a refresh, or enter No if you want only to save the
refresh settings.
Number of Concurrent Threads
Specify the number of concurrent Envision processes to use when running
transforms to export data to the target database. Using more threads will
usually result in better performance for the refresh, but will use resources on
the Colleague database server. This may impact performance of other
Colleague processes.
ALERT! Using multiple threads for your refresh can significantly
affect the system resources available to your Colleague
environment. Ellucian recommends that you carefully monitor
the system resource usage of your Colleague database server
while running your ODS refresh and adjust the scheduling of
your refresh and the number of threads you assign it accordingly.
Note: When more threads are used, not only are more system
resources used as well, but also more UniData licenses, and both of
these can affect the performance of other processes.
166
Using the DataOrchestrator™ ODS, July 9, 2013
Defining Additional Parameters for the Refresh
Defining Additional Parameters for the
Refresh
Optionally, you can detail from the DataOrch Refresh (DORE) form by using
the Temporary File Path field to go to the Additional Refresh Parameters
(DORP) form. The DORP form allows you to define a Source Drive Name
and Source Temporary File Path that overrides the corresponding settings on
the DataOrch Parameters (DOPA) form for this refresh.
Technical Tip: The reason you may want to override the
corresponding settings on DOPA is because this allows you to limit
access to the source temporary file path for this refresh.
Figure 29: Additional Refresh Parameters (DORP) Form
Using the DataOrchestrator™ ODS, July 9, 2013
167
Refreshing ODS Data: Defining and Running an ODS Refresh
Noteworthy Fields on the DORP Form
The fields described in this section are particularly important when defining
additional parameters for a refresh.
Refresh ID
The Refresh ID field displays the ID of the refresh associated with the
parameters.
Source Drive Name
Note: If you are using a SQL Server source database environment,
leave this field blank.
If you use a Windows source environment and use a source database other
than SQL Server, use the Source Drive Name field to enter the source drive
name of the temporary file path. This value specifies which drive on the
source Colleague database server will store the temporary files created by the
refresh.
Note: This value will override the default Source Drive Name on the
DataOrch Parameters (DOPA) form.
Source Temporary File Path
Note: If you are using a Windows SQL Server source environment,
leave this field blank.
Use the Source Temporary File Path field to enter the directory path on the
source Colleague database server where the temporary files, created by the
refresh, will be stored. Enter each element in the path on a separate line and
omit any slashes (/).
This directory must have sufficient space allocated for these temporary files.
The space needed for temporary files can be significant depending on how
you define the target. Ellucian recommends that you monitor the space usage
in this directory as your refresh runs.
168
Using the DataOrchestrator™ ODS, July 9, 2013
Defining Additional Parameters for the Refresh
The following permissions are needed for this directory:
 UNIX UniData. For the directory where the files are being written:
• The user logged in to Colleague must have read/write permissions.
• The user who starts the source DMI_DAS must have read/write
permissions.
 Windows UniData. For the directory where the files are being written:
• The user logged in to Colleague must have read/write permissions.
• The SYSTEM user must have read/write permissions.
Note: If the Listener is ever started manually (instead of through SA
Valet as a Windows service), then the user who starts the Listener
must also have read/write permissions.

UNIX Oracle. For the directory where the files are being written:
• Oracle needs read/write permissions.
• The user who starts the source DMI_DAS must have read/write
permissions.
The temporary files in this directory may contain data that might be
considered sensitive at your institution. Access to the directory should be
restricted accordingly.
Note: The path you enter in this field will override the default Source
Temporary File Path on the DataOrch Parameters (DOPA) form.
Using the DataOrchestrator™ ODS, July 9, 2013
169
Refreshing ODS Data: Defining and Running an ODS Refresh
Procedure for Defining a Refresh
ALERT! You should ensure that the target database to which you
are exporting Colleague data is adequately secured using the
native security controls for your database platform. Some files
and fields may contain data that might be considered sensitive in
your institution. Access to the target database containing this
data should be restricted accordingly.
Ellucian also highly recommends that you secure access to the
DORE form. This form provides a user with the ability to export
any data in Colleague to an external database.
Perform the steps below to enter settings for a refresh for a specific target, and
to run the refresh, if desired.
Step 1. Access the DataOrch Refresh (DORE) form.
Step 2. At the DataOrch Refresh LookUp prompt, either add a new Refresh ID or
enter an existing one.
Step 3. If you entered an existing Refresh ID, the Target ID is displayed and cannot
be changed. If you are adding a new Refresh ID, enter the ID of a new or
existing target in the Target ID field.
Note: After you specify a Target ID for this Refresh ID and then save,
you cannot change this association.
If you add a new Target ID, you will automatically detail to the DataOrch
Target (DOTA) form so that you can define the new target. For an existing
target, you can manually detail to the DOTA form to view or update the target
information. For information on defining or updating a target, see “Defining a
Target” on page 67.
For information about the fields on the DOTA form, see online help. When
you are finished on the DOTA form, save to return to the DORE form.
Step 4. On the DORE form, enter or modify the settings for refreshing this target. For
more information on these fields, see online help.
170
Using the DataOrchestrator™ ODS, July 9, 2013
Defining Additional Parameters for the Refresh
Step 5. (Optional) To test connectivity, detail on the Check Connectivity field to log
in to the specified target database in order to verify the connection. When you
are finished checking connectivity, save to return to the DORE form.
Step 6. (Optional) On the DORE form, you can detail on the Temporary File Path
field to access the Additional Refresh Parameters (DORP) form to override
the source drive name and temporary file path defined on the DOPA form for
this refresh. When you are finished entering parameters, save and exit from
the DORP form to return to the DORE form.
ALERT! This directory must have sufficient space allocated for
these temporary files. The space needed for temporary files can
be significant depending on how you define the target. Ellucian
recommends that you monitor the space usage in this directory
as your refresh runs.
Be sure that access to this directory is in accordance with the
online help for the Temporary File Path field on the DORP form.
Also, the temporary files in this directory may contain data that
might be considered sensitive at your institution. Access to the
directory should be restricted accordingly.
Step 7. (Optional) In the Create SQL Views (Y/N) field, indicate whether you want to
create all of the SQL views associated to the target. Enter Yes to create the
SQL views; otherwise, enter No.
Step 8. (Optional) After the refresh runs, indicate whether you want to run the stored
procedure that was created for this refresh ID. Enter Yes in the Run Stored
Procedure field to run the stored procedure; otherwise, enter No.
Step 9. In the Run Refresh (Y/N) field, enter Yes if you want to refresh the selected
target when you save your entries on the DORE form. Enter No if you want
only to save the refresh settings.
Step 10. In the Number of Concurrent Threads field, specify the number of threads you
want to use for exporting the files to the target. Remember that using more
concurrent Envision processes will usually result in better performance for the
refresh, but will use resources on the application server. This may impact
performance of other Colleague processes.
Step 11. Save from the DORE form.
Using the DataOrchestrator™ ODS, July 9, 2013
171
Refreshing ODS Data: Defining and Running an ODS Refresh
Step 12. If you are running a refresh, complete the Peripheral Settings and Process
Handler forms according to your preferences.
If you entered “Yes” in the Run Refresh (Y/N) field, the DataOrchestrator
ODS refreshes the target database and reports errors on the DataOrch Error
Analysis report. For more information on this report, see “Viewing Errors for
a Refresh” on page 183.
172
Using the DataOrchestrator™ ODS, July 9, 2013
Refreshing ODS Data
Calculating Stored Computed
Columns
In This Chapter
Note: To use the Blackboard Analytics data warehouse solution with
the DataOrchestrator ODS, you must run the processes in this
chapter.
If your institution does not use any targets containing stored computed
columns, you can skip this chapter. Stored computed column files are
indicated by having .CC as the last three characters of the file name.
Stored computed columns are delivered inactive and must be
activated.
This chapter describes how to activate and calculate stored computed columns
for any ODS target database that contains stored computed columns.
Table 27 lists the topics covered in this chapter.
Table 27: Topics in This Chapter
Topic
Page
Forms Used
174
Calculating Stored Computed Columns
175
Procedure for Activating Stored Computed Columns
176
Procedure for Calculating Stored Computed Columns
177
Procedure to Update Flags in the STUDENT.TERMS.CC File
179
Using the DataOrchestrator™ ODS, July 9, 2013
173
Refreshing ODS Data: Calculating Stored Computed Columns
Forms Used
Table 28 lists and describes the forms used in this chapter.
Table 28: Forms Used in This Chapter
Form
Purpose
Update Stored Computed Column
(USCC)
Calculate stored computed columns for
an ODS target database.
Update GPA SCC Flags (UGSF)
Update flags in the
STUDENT.TERMS.CC file for the
Cumulative GPA by Term.
Note: Use the USCC process for the
initial calculation of stored computed
columns in this file. For subsequent
updates, Ellucian recommends running
the UGSF process before the USCC
process.
174
Using the DataOrchestrator™ ODS, July 9, 2013
Calculating Stored Computed Columns
Calculating Stored Computed Columns
To calculate stored computed columns referenced by a target, you first
activate all stored computed columns using Colleague Studio, and then use
the Update Stored Computed Column (USCC) process in Colleague to
populate the stored computed columns in the source files. No information is
available for these stored computed columns until both processes are run
against the source files.
You need to activate a given stored computed column in Colleague Studio
only once. However, Ellucian recommends that you run the USCC process
every time you run the DataOrch Refresh (DORE) process to export data to an
ODS target database containing stored computed columns.
Note: If you run the DORE process without running the USCC
process, the ODS target database will contain stored computed
column data only for the most recent time the USCC process was run.
How often you run the USCC process depends on how often your stored
computed columns need to be recalculated. You may also consider how
frequently reports are run. Based on that you can run the USCC process
before running reports to get the most up-to-date data.
Technical Tip: If your target contains the STUDENT.TERMS.CC file,
use the USCC process for the initial calculation of stored computed
columns in this file. For subsequent updates, Ellucian recommends
that you use the UGSF process (before using the USCC process) to
update the STUDENT.TERMS.CC file.
Note: For any ODS target database that contains stored computed
columns, Ellucian recommends that you set up the process handler to
automatically run the USCC process. (If the target also contains the
STUDENT.TERMS.CC file, the UGSF process should be set up to run
automatically before the USCC process.)
This ensures that the stored computed columns will be updated when
the ODS target database is updated. Refer to Envision Runtime
Administration for further information on how to set up a batch process
to run at scheduled intervals.
Using the DataOrchestrator™ ODS, July 9, 2013
175
Refreshing ODS Data: Calculating Stored Computed Columns
Procedure for Activating Stored Computed Columns
To activate stored computed columns, follow the steps below. You need to
activate a given stored computed column only once.
Step 1. To activate stored computed columns, access Colleague Studio using the
following path: Entity editor > Attributes > Calculation.
Step 2. Select the stored computed column that you want to make active.
For example, the Blackboard Analytics data warehouse solution contains the
following stored computed columns that you must make active:
 APPLCC.READMIT.FLAG (APPLICATIONS.CC file)
 STTRCC.TERM.GPA (STUDENT.TERMS.CC file)
 STTRCC.TRGR.CUM.GPA (STUDENT.TERMS.CC file)
Step 3. In the Recalculation method field, select Batch to make this stored
computed column active. Save your work.
Step 4. Select any other stored computed column that you want to make active and
repeat Step 3, or exit from Colleague Studio.
176
Using the DataOrchestrator™ ODS, July 9, 2013
Calculating Stored Computed Columns
Procedure for Calculating Stored Computed
Columns
To calculate stored computed columns, follow the steps below in Colleague.
The following example shows how to calculate stored computed columns if
your ODS target database contains the APPLICATIONS.CC and
STUDENT.TERMS.CC files.
Step 1. Access the Update Stored Computed Column (USCC) form in the appropriate
application. If you are in Colleague Student, for example, the following
message is displayed:
Scanning ST application for stored computed
columns...
The USCC form then lists the names of all ST files with active stored
computed columns, as shown in Figure 30.
Figure 30: Update Stored Computed Column (USCC) Form
Using the DataOrchestrator™ ODS, July 9, 2013
177
Refreshing ODS Data: Calculating Stored Computed Columns
Step 2. In the Files to Recalculate field, remove all files from the list except those
associated with the ODS target database. For example, the
APPLICATIONS.CC and STUDENT.TERMS.CC files are necessary for the
Blackboard Analytics data warehouse solution.
Figure 31: The USCC Form After Removing All Files Except Needed Stored
Computed Columns
Step 3. For the initial update of these files, enter Yes in the Force Recalculation
field to calculate all stored computed columns.
For subsequent updates of other files containing stored computed columns,
enter Yes in the Force Recalculation field to recalculate all stored computed
columns, or enter No to recalculate only those that have changed.
Note: For subsequent updates of the STUDENT.TERMS.CC file,
Ellucian recommends that you run the UGSF process before the
USCC process, and do not force recalculation on this file. This is
because, for many Colleague environments, the USCC process may
take an extended period to run. For further details on the UGSF
process, see “Procedure to Update Flags in the
STUDENT.TERMS.CC File” on page 179.
178
Using the DataOrchestrator™ ODS, July 9, 2013
Calculating Stored Computed Columns
Step 4. Save your work and exit from the form. For more details on the activating
computed columns and the USCC process, refer to Stored Computed
Columns.
Procedure to Update Flags in the
STUDENT.TERMS.CC File
Note: If your ODS target database does not use the
STUDENT.TERMS.CC file or you are performing the initial calculation
of the STUDENT.TERMS.CC file, you can skip this procedure.
Step 1. Access the Update GPA SCC Flags (UGSF) form in Colleague Student.
Figure 32: Update GPA SCC Flags (UGSF) Form
Step 2. In the Update change flags for Cumulative GPA by Term? field, enter Yes
to set the change flags in the STUDENT.TERMS.CC file when you save from
UGSF.
Using the DataOrchestrator™ ODS, July 9, 2013
179
Refreshing ODS Data: Calculating Stored Computed Columns
This allows the correct Cumulative GPA by Term values to be recalculated
when you run the USCC process without a forced recalculation of these stored
computed columns of all records in the file.
These flags are set on records in the STUDENT.TERMS.CC file in the
following case: a student’s GPA changes for a given term (thereby affecting
the Cumulative GPA by Term for all subsequent terms). Flags are set on all
terms following the changed term.
Step 3. Save your work and exit from the form.
180
Using the DataOrchestrator™ ODS, July 9, 2013
Using the DataOrchestrator™ ODS
Viewing Reports and Refresh History
Viewing Reports and Refresh History
Viewing Errors for a Refresh
In This Chapter
This chapter describes how to run an error analysis report for the errors
encountered when exporting data from a Colleague source file to an ODS
target database.
Table 29 lists the topics covered in this chapter.
Table 29: Topics in This Chapter
Topic
Page
Form Used
183
Viewing the Error Analysis Report for a Refresh
184
Procedure for Running the Error Analysis Report
187
Form Used
Table 30 lists and describes the form used in this chapter.
Table 30: Form Used in This Chapter
Form
DataOrch Error Analysis (DOEA)
Using the DataOrchestrator™ ODS, July 9, 2013
Purpose
View the errors received for transforms
after running a refresh.
183
Viewing Reports and Refresh History: Viewing Errors for a Refresh
Viewing the Error Analysis Report for a
Refresh
Use the DataOrch Error Analysis (DOEA) form after running a refresh to
view the errors encountered for transforms. The error information displayed is
for those data rows of a transform that were not written to the ODS target
database.
You can display errors for all transforms associated with a refresh, or you can
limit the number of errors displayed by selecting only specific transforms.
You can also choose to show a report that contains only summary information
about the errors in each transform, or you can include detailed information
about each recorded error. In addition, you can choose to sort error rows per
transform by error type or by source file record ID.
The number of errors shown for a transform can be affected by the Max Errors
Saved field on the DataOrch Parameters (DOPA) form. This field is used to
limit the number of errors reported for transforms. Only this number of errors
will be included for a specific transform in this report.
Technical Tip: The Error Analysis Report for the refresh includes
warning messages for each transform that results in an empty target
ODS table. These warnings are included because these transforms do
not result in useful information in the operational data store and may
indicate either an error in the design of the transform, or that the
transform can be omitted from the refresh.
Technical Tip: To help you correct data errors in the source
environment, you may want to use the Colleague Data Scanner Utility
(CDSU) process. For more information, see Preparing for Release 18,
available on the Ellucian website.
184
Using the DataOrchestrator™ ODS, July 9, 2013
Viewing the Error Analysis Report for a Refresh
Figure 33: DataOrch Error Analysis (DOEA) Form
Noteworthy Fields on the DOEA Form
The fields described in this section are important when setting up the error
analysis report to run.
Refresh ID
The Refresh ID field displays the ID of the refresh.
Target ID
The Target ID field displays the ID of the associated target.
Status
The Status field displays the status of the refresh.
Using the DataOrchestrator™ ODS, July 9, 2013
185
Viewing Reports and Refresh History: Viewing Errors for a Refresh
Last Start Date/Time
The Last Start Date/Time field displays the date and time when the most
recent refresh started.
Last End Date/Time
The Last End Date/Time field displays the date and time when the most recent
refresh ended.
Limit to Transforms
Use the Limit to Transforms field to enter the transforms associated with this
refresh for which you want to view data or refresh errors. You can enter only
those transforms from the target that have data or refresh errors.
Note: If you leave this field blank, then all transforms for this refresh
that have data or refresh errors will be included.
Rows with Errors
The Rows with Errors field displays the number of rows with errors.
Show Detailed Errors (Y/N)
Use the Show Detailed Errors (Y/N) field to enter Yes to view error
summary information and to display each error row for the transforms
selected. Enter No to view only error summary information for each
transform.
Sort by Error Type (Y/N)
Use the Sort by Error Type (Y/N) field to enter Yes to sort error rows per
transform by error type. Enter No to sort error rows per transform by source
file record ID.
186
Using the DataOrchestrator™ ODS, July 9, 2013
Viewing the Error Analysis Report for a Refresh
Procedure for Running the Error Analysis Report
Use the following steps to run the Error Analysis Report.
Step 1. Access the Error Analysis (DOEA) form. At the DataOrch Refresh LookUp
prompt, enter the refresh for which you want to view error analysis.
Step 2. On the DOEA form, enter or modify the settings for running the report. For
more information on these fields, see online help.
Step 3. Save your entries on the DOEA form.
Step 4. Complete the Peripheral Settings and Process Handler forms according to
your preferences.
Step 5. When you are finished viewing, exit from the report.
Using the DataOrchestrator™ ODS, July 9, 2013
187
Viewing Reports and Refresh History: Viewing Errors for a Refresh
188
Using the DataOrchestrator™ ODS, July 9, 2013
Viewing Reports and Refresh History
Viewing Target Transform Data
for a Target
In This Chapter
This chapter describes how to run the Transform Summary report in order to
view the information for target transforms for a specified target.
Table 31 lists the topics covered in this chapter.
Table 31: Topics in This Chapter
Topic
Page
Form Used
189
Viewing the Transform Summary Report for a Target
190
Procedure for Running the Transform Summary Report
192
Form Used
Table 32 lists and describes the form used in this chapter.
Table 32: Form Used in This Chapter
Form
DataOrch Transform Summary (DOTS)
Using the DataOrchestrator™ ODS, July 9, 2013
Purpose
View the information for target
transforms associated with a target.
189
Viewing Reports and Refresh History: Viewing Target Transform Data for a Target
Viewing the Transform Summary
Report for a Target
Use the DataOrch Transform Summary (DOTS) form to view a report of the
target transforms for a specific target. This report includes data associated
with a target transform, and also information about each column associated to
the transform.
You can choose whether or not to include the description of the transform and
the descriptions of the transform’s columns.
Note: This report can be used only for target transforms. You cannot
enter source transforms.
Figure 34: DataOrch Transform Summary (DOTS) Form
190
Using the DataOrchestrator™ ODS, July 9, 2013
Viewing the Transform Summary Report for a Target
Noteworthy Fields on the DOTS Form
The fields described in this section are important when setting up the
Transform Summary report to run.
Target ID
The Target ID field displays the ID of the selected target.
Target Transforms
Use the Target Transforms field to enter target transform names, associated to
the Target ID, to include in the report.
Include Transform Description (Y/N)
Use the Include Transform Description (Y/N) field to indicate whether to
include the description of the transform in the report. Enter Yes to include
the description, or enter No to exclude the description.
Include Column Descriptions (Y/N)
Use the Include Column Descriptions (Y/N) field to indicate whether to
include the columns’ descriptions for the transform in the report. Enter Yes
to include the descriptions, or enter No to exclude the descriptions.
Using the DataOrchestrator™ ODS, July 9, 2013
191
Viewing Reports and Refresh History: Viewing Target Transform Data for a Target
Procedure for Running the Transform Summary
Report
Use the following steps to run the Transform Summary report.
Step 1. Access the DataOrch Transform Summary (DOTS) form. At the DataOrch
Target LookUp prompt, enter the ID of the target for which you want to view
transform information.
Step 2. On the DOTS form, enter or modify the settings for running the report. For
more information on these fields, see online help.
Step 3. Save your entries on the DOTS form.
Step 4. Complete the Peripheral Settings and Process Handler forms according to
your preferences.
Step 5. When you are finished viewing, exit from the report.
192
Using the DataOrchestrator™ ODS, July 9, 2013
Viewing Reports and Refresh History
Viewing the History of a Refresh
In This Chapter
This chapter describes how to view the historical information for a refresh,
including the settings, status, and statistics of previously run refreshes for a
specific target.
Table 33 lists the topics covered in this chapter.
Table 33: Topics in This Chapter
Topic
Page
Form Used
193
Viewing the History of a Refresh
194
Procedure for Viewing the History of a Refresh
196
Form Used
Table 34 lists and describes the form used in this chapter.
Table 34: Form Used in This Chapter
Form
DataOrch Refresh History (DORH)
Using the DataOrchestrator™ ODS, July 9, 2013
Purpose
View the settings, status, and statistics
of a previously run refresh for a specific
target.
193
Viewing Reports and Refresh History: Viewing the History of a Refresh
Viewing the History of a Refresh
The DataOrch Refresh History (DORH) form is used to view historical
information about previously run refreshes. This form shows the refresh
status, settings, and statistics of a previously run refresh for a specific ODS
target database. The settings and statistics include the following:
Settings for the refresh:
 Refresh ID
 Target ID
 Database name/TNS
 DMI Listener host name
 DMI Listener port number
 Number of concurrent threads
For each target transform run as part of the refresh, the following information
is included:
 Target transform and source file names.
 The Update Type for the transform. This would indicate the Override
Update Type if the default Update Type was overridden for a transform.
 Status of the transform, indicating whether it was successfully refreshed or
not.
 Date/time the target transform started and finished processing.
 Transform information for the refresh:
 Total number of rows written for the transform.
 Number of rows deleted on the target database.
 Number of errors encountered per transform.
For each materialized SQL view that is run as part of the refresh, the
following information is included:
 Materialized SQL view name. Note that an entry for a materialized SQL
view is indicated by the words “Materialized View” in the Source File field.
 An Update Type of “Full.” (Materialized SQL views cannot be
incrementally updated).
 The status of the refresh of the materialized SQL view, indicating whether
it was successfully refreshed or not.
 Date/time the materialized SQL view started and finished processing.
194
Using the DataOrchestrator™ ODS, July 9, 2013
Viewing the History of a Refresh
Figure 35: DataOrch Refresh History (DORH) Form
You can also view the refresh status on the DORH form. The following
statuses may be displayed.
 InProc. The transform is currently being refreshed.
 Success. The transform was successfully refreshed without any errors.
 Warning. The transform or refresh received errors during processing.
 Fail. A transform or refresh received fatal errors during processing.
Using the DataOrchestrator™ ODS, July 9, 2013
195
Viewing Reports and Refresh History: Viewing the History of a Refresh
Procedure for Viewing the History of a Refresh
Use the following step to view the history of a refresh.
Step 1. Access the DataOrch Refresh History (DORH) form. At the DataOrch
Refresh History LookUp prompt, enter the ID of the history record you want
to view.
Technical Tip: The history record ID consists of three parts:
1. The Refresh ID.
2. The date, in internal format, when the user ran the refresh.
3. The exact time, in internal format, when the user ran the refresh.
For example, DO.REFRESH.ABC_14016_52885. The order of the
Refresh IDs are sorted by date and time. The most recent refresh
appears first.
Step 2. When you are finished viewing, exit from the DORH form.
196
Using the DataOrchestrator™ ODS, July 9, 2013
Using the DataOrchestrator™ ODS
Maintaining the DataOrchestrator ODS
Maintaining the DataOrchestrator ODS
Maintaining Transforms
In This Chapter
This chapter describes how to maintain DataOrchestrator ODS transforms.
You can copy existing transforms, delete transforms as needed, and rename
target transforms. You can also regenerate source transforms, and calculate
foreign keys for both source and target transforms.
Table 35 lists the topics covered in this chapter.
Table 35: Topics in This Chapter
Topic
Page
Form Used
199
Maintaining DataOrchestrator ODS Transforms
200
Procedure for Deleting Transforms
203
Procedure for Copying Transforms
204
Procedure for Renaming Transforms
205
Procedure for Copying and Renaming Transforms
206
Procedure for Calculating Foreign Keys
207
Procedure for Regenerating Source Transforms
208
Form Used
Table 36 lists and describes the form used in this chapter.
Table 36: Form Used in This Chapter
Form
DataOrch Transform Maint (DOMA)
Using the DataOrchestrator™ ODS, July 9, 2013
Purpose
Maintains DataOrchestrator ODS
transforms.
199
Maintaining the DataOrchestrator ODS: Maintaining Transforms
Maintaining DataOrchestrator ODS
Transforms
Use the DataOrch Transform Maint (DOMA) form to perform the following
functions:
 Delete transforms from a target.
 Copy transforms from one target to another target.
 Rename target transforms.
 Copy transforms from a target, and then rename them to a different target or
to the same target.
 Calculate foreign keys for transforms.
 Regenerate source transforms.
Figure 36: Example of Deleting a Transform using the DataOrch Transform
Maint (DOMA) Form
200
Using the DataOrchestrator™ ODS, July 9, 2013
Maintaining DataOrchestrator ODS Transforms
Noteworthy Fields on the DOMA Form
The fields described in this section are important when maintaining
transforms using the DataOrchestrator ODS.
Operation
Use the Operation field to choose an operation to perform on a target. Select
one of the following operations:
 Delete. Delete transforms from a target.
 Copy. Copy transforms from one target to another.
 Rename. Rename target transforms associated with a target. Source
transforms cannot be renamed.
 Calculate Foreign Keys. Generate foreign key metadata for transforms.
Foreign key metadata will be created in the following cases:
• If the input field for a column is a pointer to a Colleague file, then a foreign
key is created to any transform with that file as its source file.
• A foreign key is created for each multivalued transform to all singlevalued transforms with the same source file.
 Regenerate Source Transforms. Regenerate source transforms associated
with a target. This will regenerate the set of target transforms to create and
populate data for each target database table needed to hold the specified
input fields of the source file. You will need to regenerate source transforms
if any Colleague metadata changes for the fields you export in a source
transform.
From Target ID
Use the From Target ID field to enter the name of the target from which to
select associated transforms to process.
Transform List
Use the Transform List field to enter the transforms to process that are
associated with the target selected in the From Target ID field.
Rename To
Use the Rename To field to enter a new name for a target transform on the
Transform List. Source transforms cannot be renamed.
Using the DataOrchestrator™ ODS, July 9, 2013
201
Maintaining the DataOrchestrator ODS: Maintaining Transforms
To Target ID
If you are copying to another Target ID, you must enter the name of the target
to which you want to copy transforms. For a DataOrchestrator ODS target,
you can enter the same target name as you entered in the From Target ID field.
You would do this if you want to copy transforms from a target onto the same
target with a new name. This allows you to create new transforms from
existing transforms and modify them as needed.
202
Using the DataOrchestrator™ ODS, July 9, 2013
Maintaining DataOrchestrator ODS Transforms
Procedure for Deleting Transforms
To delete transforms from a target, use the following steps.
Step 1. Access the DataOrch Transform Maint (DOMA) form.
Step 2. In the Operation field, select Delete.
Step 3. In the From Target ID field, enter the ID of the target from which you want to
delete transforms.
Step 4. In the Transform List field, enter the transforms you want to delete.
Step 5. Save from the DOMA form to delete the transforms you selected.
Using the DataOrchestrator™ ODS, July 9, 2013
203
Maintaining the DataOrchestrator ODS: Maintaining Transforms
Procedure for Copying Transforms
To copy transforms from one target to another, use the following steps.
Step 1. Access the DataOrch Transform Maint (DOMA) form.
Step 2. In the Operation field, select Copy from the drop-down list.
Note: If you copy to an existing transform, you will overwrite the filter
of the transform being copied to with the filter of the transform being
copied from. Also, if you copy to an existing transform that has any of
the same column names, and the column sizes are larger in the
existing transform, the larger size is used.
Step 3. In the From Target ID field, enter the ID of the target from which you want to
copy transforms.
Step 4. In the Transform List field, enter the transforms you want to copy.
Step 5. In the To Target ID field, enter the ID of the target to which you want to copy
the transforms.
Step 6. Save from the DOMA form to copy the transforms you selected.
204
Using the DataOrchestrator™ ODS, July 9, 2013
Maintaining DataOrchestrator ODS Transforms
Procedure for Renaming Transforms
To rename transforms, use the following steps.
Step 1. Access the DataOrch Transform Maint (DOMA) form.
Step 2. In the Operation field, select Rename.
Step 3. In the From Target ID field, enter the ID of the target for which you want to
rename transforms.
Step 4. In the Transform List field, enter the transforms you want to rename.
Step 5. In the Rename To field, enter the new name for each transform you listed.
Step 6. Save from the DOMA form to rename the transforms you selected.
Using the DataOrchestrator™ ODS, July 9, 2013
205
Maintaining the DataOrchestrator ODS: Maintaining Transforms
Procedure for Copying and Renaming Transforms
To copy and rename transforms from a target to another target, or to the same
target, use the following steps.
Step 1. Access the DataOrch Transform Maint (DOMA) form.
Step 2. In the Operation field, select Copy from the drop-down list.
Note: If you copy to an existing transform, you will overwrite the filter
of the transform being copied to with the filter of the transform being
copied from. Also, if you copy to an existing transform that has any of
the same column names, and the column sizes are larger in the
existing transform, the larger size is used.
Step 3. In the From Target ID field, enter the ID of the target from which you want to
copy transforms.
Step 4. In the Transform List field, enter the transforms you want to copy.
Step 5. In the Rename To field, enter the new name for each transform you listed.
Step 6. In the To Target ID field, enter the ID of the target to which you want to copy
the transforms.
Step 7. Save from the DOMA form to copy and rename the transforms you selected.
206
Using the DataOrchestrator™ ODS, July 9, 2013
Maintaining DataOrchestrator ODS Transforms
Procedure for Calculating Foreign Keys
To calculate foreign keys for source and target transforms, use the following
steps.
Step 1. Access the DataOrch Transform Maint (DOMA) form.
Step 2. In the Operation field, select Calculate Foreign Keys from the drop-down
list.
Step 3. In the From Target ID field, enter the ID of the target from which you want to
calculate foreign keys for transforms.
Step 4. In the Transform List field, enter the transforms for which you want to
calculate foreign keys.
Step 5. Save from the DOMA form to calculate foreign keys for the transforms you
selected.
If you want to view or modify foreign key metadata for target transforms,
follow these steps:
a. Access the DataOrch Target (DOTT) form.
b. Detail from the Column Definitions field to access the Transform Columns
(DOTC) form.
c. Detail from a target column name to the Column Properties (DOPR) form.
For more information, see “Defining Target Transforms” beginning on
page 95.
Note: The foreign key metadata for source transforms cannot be
modified after it has been calculated.
Using the DataOrchestrator™ ODS, July 9, 2013
207
Maintaining the DataOrchestrator ODS: Maintaining Transforms
Procedure for Regenerating Source Transforms
To regenerate source transforms, use the following steps.
Step 1. Access the DataOrch Transform Maint (DOMA) form.
Step 2. In the Operation field, select Regen Source Transforms from the drop-down
list.
Step 3. In the From Target ID field, enter the ID of the target from which you want to
regenerate source transforms.
Step 4. In the Transform List field, enter the source transforms you want to
regenerate.
Step 5. Save from the DOMA form to regenerate the source transforms you selected.
208
Using the DataOrchestrator™ ODS, July 9, 2013
Maintaining the DataOrchestrator ODS
Maintaining SQL Views
In This Chapter
This chapter describes how to maintain SQL views. You can copy existing
SQL views, delete views as needed, and rename SQL views.
Table 37 lists the topics covered in this chapter.
Table 37: Topics in This Chapter
Topic
Page
Form Used
209
Maintaining SQL Views
210
Procedure for Deleting SQL Views
213
Procedure for Copying SQL Views
214
Procedure for Renaming SQL Views
215
Procedure for Copying and Renaming SQL Views
216
Form Used
Table 38 lists and describes the form used in this chapter.
Table 38: Form Used in This Chapter
Form
DataOrch View Maintenance (DOVM)
Using the DataOrchestrator™ ODS, July 9, 2013
Purpose
Maintains SQL views.
209
Maintaining the DataOrchestrator ODS: Maintaining SQL Views
Maintaining SQL Views
Use the DataOrch View Maintenance (DOVM) form to perform the following
functions:
 Delete SQL views from a target.
 Copy SQL views from one target to another target.
 Rename SQL views.
 Copy SQL views from a target, and then rename them to a different target
or to the same target.
Note: This process modifies only the SQL view metadata. No
processing will occur on the ODS target database.
Note that when you when you copy an SQL view to a target and an SQL view
with the same name exists on that target, the information in the target SQL
view is updated as follows:
 The SQL statement associated with view is replaced with that of the SQL
view being copied.
 If the SQL view being copied is a materialized SQL view, the materialized
SQL view parameters, including the Create Materialized View flag and the
Dependent Transforms list are overwritten. However, if the SQL view
being copied is not a materialized SQL view, the existing data in the target
SQL view is not changed.
 The SQL view’s column information for the view being copied will be
integrated into the target SQL view column information. Any new column
entries will be added to the SQL View Columns list (on the DataOrch View
Spec [DOVS] form) and information for existing columns will be
overwritten. The only exception to this is the Key field. If this field is set to
“Yes” in the existing SQL view, it will not be overwritten with a value of
“No” from the SQL view being copied.
210
Using the DataOrchestrator™ ODS, July 9, 2013
Maintaining SQL Views
Figure 37: Example of Deleting an SQL View using the DataOrch View
Maintenance (DOVM) Form
Noteworthy Fields on the DOVM Form
The fields described in this section are important when deleting, copying, or
renaming SQL views using the DataOrchestrator ODS.
Operation
Use the Operation field to choose an operation to perform on a target. Select
one of the following operations:
 Delete. Delete SQL views from a target.
 Copy. Copy SQL views from one target to another.
 Rename. Rename SQL views associated with a target.
Note: Deleting an SQL view from a target does not automatically
delete the corresponding SQL view on the ODS target database. This
must be done manually by the system administrator.
Using the DataOrchestrator™ ODS, July 9, 2013
211
Maintaining the DataOrchestrator ODS: Maintaining SQL Views
From Target ID
Use the From Target ID field to enter the name of the target from which to
select associated SQL views to process.
SQL Views
Use the SQL Views list to enter the SQL views to process that are associated
with the target selected in the From Target ID field.
Rename To
Use the Rename To field to enter a new name for an SQL view on the SQL
Views list.
To Target ID
If you are copying to another Target ID, you must enter the name of the target
to which you want to copy SQL views. You can enter the same target name as
you entered in the From Target ID field to copy and rename SQL views to the
same target. You would do this if you want to copy SQL views from a target
onto the same target with a new name. This allows you to create new SQL
views from existing SQL views and modify them as needed.
212
Using the DataOrchestrator™ ODS, July 9, 2013
Maintaining SQL Views
Procedure for Deleting SQL Views
To delete SQL views from a target, use the following steps.
Step 1. Access the DataOrch View Maintenance (DOVM) form.
Step 2. In the Operation field, select Delete.
Step 3. In the From Target ID field, enter the ID of the target from which you want to
delete SQL views.
Step 4. In the SQL Views list, enter the SQL views you want to delete.
Step 5. Save from the DOVM form to delete the SQL views you selected.
Using the DataOrchestrator™ ODS, July 9, 2013
213
Maintaining the DataOrchestrator ODS: Maintaining SQL Views
Procedure for Copying SQL Views
To copy SQL views from one target to another, use the following steps.
Step 1. Access the DataOrch View Maintenance (DOVM) form.
Step 2. In the Operation field, select Copy from the drop-down list.
Step 3. In the From Target ID field, enter the ID of the target from which you want to
copy SQL views.
Step 4. In the SQL Views list, enter the SQL views you want to copy.
Step 5. In the To Target ID field, enter the ID of the target to which you want to copy
the SQL views.
Step 6. Save from the DOVM form to copy the SQL views you selected.
214
Using the DataOrchestrator™ ODS, July 9, 2013
Maintaining SQL Views
Procedure for Renaming SQL Views
To rename SQL views, use the following steps.
Step 1. Access the DataOrch View Maintenance (DOVM) form.
Step 2. In the Operation field, select Rename.
Step 3. In the From Target ID field, enter the ID of the target for which you want to
rename SQL views.
Step 4. In the SQL Views list, enter the SQL views you want to rename.
Step 5. In the Rename To field, enter the new name for each SQL view you listed.
Step 6. Save from the DOVM form to rename the SQL views you selected.
Using the DataOrchestrator™ ODS, July 9, 2013
215
Maintaining the DataOrchestrator ODS: Maintaining SQL Views
Procedure for Copying and Renaming SQL Views
To copy and rename SQL views from a target to another target, or to the same
target, use the following steps.
Step 1. Access the DataOrch View Maintenance (DOVM) form.
Step 2. In the Operation field, select Copy from the drop-down list.
Step 3. In the From Target ID field, enter the ID of the target from which you want to
copy SQL views.
Step 4. In the SQL Views list, enter the SQL views you want to copy.
Step 5. In the Rename To field, enter the new name for each SQL view you listed.
Step 6. In the To Target ID field, enter the ID of the target to which you want to copy
the SQL view.
Step 7. Save from the DOVM form to copy and rename the SQL views you selected.
216
Using the DataOrchestrator™ ODS, July 9, 2013
Maintaining the DataOrchestrator ODS
Copying Targets to Another
Target
In This Chapter
This chapter describes how to copy all transforms and SQL views from one or
more targets to a single destination target.
Table 39 lists the topics covered in this chapter.
Table 39: Topics in This Chapter
Topic
Page
Form Used
217
Copying One or More Targets to Another Target
218
Procedure for Copying One or More Targets to Another Target
220
Form Used
Table 40 lists and describes the form used in this chapter.
Table 40: Form Used in This Chapter
Form
DataOrch Target Copy (DOTY)
Using the DataOrchestrator™ ODS, July 9, 2013
Purpose
Copy transforms and SQL views from
one or more targets to another single
target.
217
Maintaining the DataOrchestrator ODS: Copying Targets to Another Target
Copying One or More Targets to
Another Target
Use the DataOrch Target Copy (DOTY) form to copy all transforms and SQL
views from one or more targets to a single destination target. This form
provides the ability to install all target templates with different versions all at
once. The target IDs listed in the Copy from Target IDs field will be sorted
alphabetically before processing to ensure that the target templates are
installed in the correct order.
Note: This form copies all transforms and SQL views from the targets
listed in the Copy from Target IDs field. To select specific transforms
and SQL views from a target, use the DataOrch Transform Maint
(DOMA) form and the DataOrch View Maintenance (DOVM) form.
For further details on how individual transforms and SQL views will be
copied, see the following information for the DOMA and DOVM forms:
• “Maintaining Transforms” beginning on page 199
• “Maintaining SQL Views” beginning on page 209
Figure 38: DataOrch Target Copy (DOTY) Form
218
Using the DataOrchestrator™ ODS, July 9, 2013
Copying One or More Targets to Another Target
Noteworthy Fields on the DOTY Form
The fields described in this section are important when copying all transforms
and SQL views from one or more targets to a single destination target.
Copy from Target IDs
Use the Copy from Target IDs field to enter one or more targets whose
transforms and SQL views you want to copy to the destination target.
Copy to Target ID
Use the Copy to Target ID field to enter the target ID to which you want to
add the transforms and SQL views from the targets you listed in the Copy
from Target IDs field.
Note: Target templates are not allowed, as they cannot be modified.
Also, the destination target cannot be a target that is listed in the Copy
from Target IDs field.
Using the DataOrchestrator™ ODS, July 9, 2013
219
Maintaining the DataOrchestrator ODS: Copying Targets to Another Target
Procedure for Copying One or More Targets to
Another Target
Use the following steps to copy all transforms and SQL views from one or
more targets to a single destination target.
Step 1. Access the DataOrch Target Copy (DOTY) form.
Step 2. On the DOTY form, enter one or more targets from which you want to copy
all transforms and SQL views. Also, enter the destination target. For more
information on these fields, see online help.
Step 3. Save from the DOTY form to copy the transforms and SQL views from the
targets you selected to the destination target.
220
Using the DataOrchestrator™ ODS, July 9, 2013
Maintaining the DataOrchestrator ODS
Deleting DataOrchestrator ODS
Objects
In This Chapter
This chapter describes how to delete DataOrchestrator ODS objects. You can
delete existing refreshes, targets, or refresh history records from the source
Colleague database.
Table 41 lists the topics covered in this chapter.
Table 41: Topics in This Chapter
Topic
Page
Form Used
221
Deleting DataOrchestrator ODS Objects
222
Procedure for Deleting DataOrchestrator ODS Objects
224
Form Used
Table 42 lists and describes the form used in this chapter.
Table 42: Form Used in This Chapter
Form
DataOrch Object Delete (DOOD)
Using the DataOrchestrator™ ODS, July 9, 2013
Purpose
Delete DataOrchestrator ODS objects.
221
Maintaining the DataOrchestrator ODS: Deleting DataOrchestrator ODS Objects
Deleting DataOrchestrator ODS Objects
Use the DataOrch Object Delete (DOOD) form to delete DataOrchestrator
ODS objects. You can delete existing refreshes, targets, or refresh history
records from the source Colleague database.
Note: If you delete a target, you will also delete all of the target’s
associated transforms, transform columns, and transform error
records.
Figure 39: DataOrch Object Delete (DOOD) Form
222
Using the DataOrchestrator™ ODS, July 9, 2013
Deleting DataOrchestrator ODS Objects
Noteworthy Fields on the DOOD Form
The fields described in this section are important when deleting objects using
the DataOrchestrator ODS.
Object Type
Use the Object Type field to choose the DataOrchestrator ODS object type for
which you want to delete IDs.
Note: The object type you select determines which IDs you can enter
in the IDs to Delete field.
IDs to Delete
Use the IDs to Delete field to enter the IDs of the records to delete for the
object type you selected. When you save, these IDs will be deleted from the
source Colleague database.
Using the DataOrchestrator™ ODS, July 9, 2013
223
Maintaining the DataOrchestrator ODS: Deleting DataOrchestrator ODS Objects
Procedure for Deleting DataOrchestrator ODS
Objects
Use the following steps to delete an object.
Step 1. Access the DataOrch Object Delete (DOOD) form.
Step 2. On the DOOD form, enter the settings for deleting objects. For more
information on these fields, see online help.
Step 3. Save from the DOOD form to delete the objects you selected.
224
Using the DataOrchestrator™ ODS, July 9, 2013
Using the DataOrchestrator™ ODS
Appendices
Appendices
Checklist for Setting Up the
DataOrchestrator ODS
In This Appendix
This appendix provides a quick checklist of the activities that the system
administrator and system programmer perform to set up and maintain the
DataOrchestrator ODS.
Using the DataOrchestrator™ ODS, July 9, 2013
227
Appendices: Checklist for Setting Up the DataOrchestrator ODS
Checklist for Setting Up the
DataOrchestrator ODS
Table 1: Checklist for Setting Up the DataOrchestrator ODS
Step
User
Description
Reference
1.
System
administrator
Check that your institution has the prerequisites
needed for the DataOrchestrator ODS.
“Prerequisites for the
DataOrchestrator ODS” on
page 35.
2.
System
administrator
Decide where to build the operational data stores.
“Where to Build the
Operational Data Stores”
beginning on page 36.
3.
System
administrator
Prepare the target database where the operational
data stores will be built.
“Preparing the ODS Target
Database” beginning on
page 38.
4.
System
administrator
Note: If your institution does not use SQL Server,
skip this step.
“SQL Server Script for
Creating the ODS Target
Database” on page 40.
For SQL Server, you can use a script to create your
ODS target database.
5.
System
administrator
Note: If your institution does not use SQL Server,
skip this step.
“Adding the IDX Filegroup” on
page 42.
If you are using a SQL Server target database, the
filegroup “IDX” must be in your target database.
6.
System
administrator
Add the DODS optional module for the
DataOrchestrator ODS.
“Adding the Optional Module”
on page 46.
7.
System
administrator
Create a new DMI Listener on the ODS target
database server and select the roles DBAS and
RDAS for the Listener. Use the Colleague release
system to retrieve and load the appropriate software
updates.
“Setting Up the ODS Target
Database Server and
Installing Software Updates”
on page 47.
See Updating Colleague
Software for information on
retrieving and installing
software updates.
8.
System
administrator
Note: If your institution does not use Oracle, skip
this step.
“Installing the Oracle JDBC
Driver” on page 51.
If the target database is Oracle, install the Oracle
JDBC driver on the ODS target database. If your
source environment is UniData and your target
database is Oracle, you also must install the Oracle
JDBC driver on your source environment.
228
Using the DataOrchestrator™ ODS, July 9, 2013
Checklist for Setting Up the DataOrchestrator ODS
Table 1: Checklist for Setting Up the DataOrchestrator ODS (cont’d)
Step
User
Description
Reference
9.
System
administrator
Define parameters for the DataOrchestrator ODS.
“Defining DataOrchestrator
ODS Parameters” beginning
on page 59.
10.
System
administrator
Create a target, which defines the configuration of
an operational data store on a target database. This
includes defining source and target transforms to
refresh on the target database.
“Defining a Target” beginning
on page 67 (all chapters in
Part 4).
(Optional) Define and create SQL views as shown.
11.
System
administrator
Create a Refresh ID with an associated target.
Define parameters to control how the
DataOrchestrator ODS refreshes a target database
with data from Colleague source files.
“Defining and Running an
ODS Refresh” beginning on
page 159.
12.
System
administrator
Note: If your institution does not use stored
computed columns, skip this step.
“Calculating Stored Computed
Columns” beginning on
page 173.
Activate and calculate stored computed columns for
any target database that contains stored computed
columns. If your ODS target database uses the
STUDENT.TERMS.CC file, update the flags as
shown.
Note: Ellucian recommends that any operational
data store that contains stored computed columns
be set up to automatically run the following
processes:
Refer to Envision Runtime
Administration for information
on how to set a batch process
to run at scheduled intervals.
• Update GPA SCC Flags (UGSF) if using the
STUDENT.TERMS.CC file
• Update Stored Computed Column (USCC)
This ensures that the stored computed columns will
be updated when the ODS target database is
updated.
13.
System
administrator
Run a refresh that updates the specified ODS target
database. To run the refresh, access the DataOrch
Refresh (DORE) form and enter Yes in the Run
Refresh (Y/N) field.
“Defining and Running an
ODS Refresh” beginning on
page 159.
(Optional) Create all SQL views for the target when
you run the refresh by entering Yes in the Create
SQL Views (Y/N) field.
Save from the DORE form to run the refresh.
14.
System
administrator
and system
programmer
Run an error analysis report for any errors
encountered when exporting data from a Colleague
source file to an ODS target database.
Using the DataOrchestrator™ ODS, July 9, 2013
“Viewing Errors for a Refresh”
beginning on page 183.
229
Appendices: Checklist for Setting Up the DataOrchestrator ODS
Table 1: Checklist for Setting Up the DataOrchestrator ODS (cont’d)
Step
User
Description
Reference
15.
System
administrator
and system
programmer
(Optional) Run a Transform Summary report in
order to view information for target transforms for a
specified target.
“Viewing Target Transform
Data for a Target” beginning
on page 189.
16.
System
administrator
and system
programmer
(Optional) View historical information for previously
run refreshes for a specific target.
“Viewing the History of a
Refresh” beginning on
page 193.
17.
System
administrator
and system
programmer
(Optional) Copy existing target transforms, delete
transforms, and rename transforms. Regenerate
source transforms, and create foreign keys for
source and target transforms.
“Maintaining Transforms”
beginning on page 199.
18.
System
administrator
and system
programmer
(Optional) Copy existing SQL views, delete views,
and rename SQL views.
“Maintaining SQL Views”
beginning on page 209.
19.
System
administrator
and system
programmer
(Optional) Copy all transforms and SQL views from
one or more targets to a single destination target.
“Copying Targets to Another
Target” beginning on
page 217.
20.
System
administrator
and system
programmer
(Optional) Delete existing refreshes, targets, or
refresh history records from the source Colleague
database.
“Deleting DataOrchestrator
ODS Objects” beginning on
page 221.
230
Using the DataOrchestrator™ ODS, July 9, 2013
Appendices
Frequently Asked Questions
In This Appendix
This appendix describes provides answers to frequently asked questions when
using the DataOrchestrator ODS and the data models and views for reporting.
Note: For more information on the DataOrchestrator ODS and the
data models and views, reference AnswerNet page 5983,
“DataOrchestrator Support Solutions.”
Table 2 lists the topics covered in this appendix.
Table 2: Topics in this Chapter
Topic
Frequently Asked Questions about the DataOrchestrator ODS
Using the DataOrchestrator™ ODS, July 9, 2013
Page
232
231
Appendices: Frequently Asked Questions
Frequently Asked Questions about the
DataOrchestrator ODS
Listed below are questions and answers for frequently asked questions about
the DataOrchestrator ODS and the data models and views associated with the
DataOrchestrator ODS.
Q What port will I need to open up in my firewall on the source database
server?
A The port on which the database is running on the source database server
should allow incoming requests.
Q What ports will I need to open up in my firewall on the target database
server?
A The following ports on the target database server should allow incoming
requests:


The port where the Colleague daemon is installed (usually port
9000).
The port you have chosen for the RDAS listener.
Q When I set up the RDAS Listener on the target database server, do I set up
a new local product repository on that server?
A No. You do not need to set up a new local product repository on the target
database server. The only prerequisite for installing the new listener on the
target database server is that you have installed a Colleague daemon on the
target database server.
Q I have a SQL Server target database. When testing connectivity on the
DataOrch Refresh (DORE) form, I receive the following message. What
does this mean?:
Unable to log in to target database.
Failed Login to ods_test on sdw2k3qasql1 by
abcdef: Fatal: 18456(14):
Login failed for user 'abcdef'.
232
Using the DataOrchestrator™ ODS, July 9, 2013
Frequently Asked Questions about the DataOrchestrator ODS
A Check the target SQL Server database instance where ods_test is set up to
be sure a user named “abcdef” is defined with SQL Server authentication.
Also, be sure the security for that SQL Server instance is set up to allow
for both Windows and SQL Server authentication.
Q What permissions should I give users on the target SQL Server database?
A Give server roles of public and bulkadmin to the user. Give database role
membership of public, db_ddladmin, db_datareader, and db_datawriter for
that particular database. If you do not want to give the user that level of
access, you can define an override source database login on the secured
form DataOrch Parameters (DOPA). During a refresh, that override login
will be used to access the source database instead of the user’s own login.
Q When checking connectivity on the DORE form, I receive “DMI_OPEN”
errors. How do I resolve these?
A First, check to be sure all listeners for the environment are running. If some
are not running, be sure to start them, and then look for the following:


On the DORE form, be sure the RDAS listener installed on your
target database server was selected.
If your source database is UniData or Oracle, then go to the
DataOrch Parameters (DOPA) form and be sure the DMI_DAS
listener is the DMI_DAS listener on your Colleague database server.
Q I created a new source transform using the PERSON file. The tables
subsequently created on the SQL target database were not what I expected.
How can I know which tables will be created and can I choose the tables,
or is this based solely on the columns/fields that I select from the PERSON
file?
For example, our institution does not use the MILITARY_INFO table, but
because I selected the PERSON file, it was created. How could I skip this
table?
Note: This question could be posed for any file and association name;
PERSON and MILITARY_INFO are examples.
A To understand tables created on the SQL target database, refer to Mapping
Envision Files for SQL Server and Oracle, available on the Ellucian
website. This mapping pertains to source transforms only which create the
Using the DataOrchestrator™ ODS, July 9, 2013
233
Appendices: Frequently Asked Questions
Release 18 SQL schema table structure (that is, all single-valued columns
in one table, all unassociated multivalues in the filename_LS table, and all
associated columns in a table named after the association).
The tables created also depend on which fields you choose on the
DataOrch Source Transform (DOST) form when setting up your source
transform. If you choose all the fields for the PERSON file, for example,
you will have all 15 tables that make up the PERSON file in your target
database.
If you want to omit tables from the target database, select specific columns
from the PERSON file. For example, to prevent the MILITARY_ASSOC
table from being created on the target database, do not select any of the
following columns on the DOST form, which are all part of the
MILITARY_ASSOC association:





MILITARY.STATUSES
MILITARY.BRANCHES
MILITARY.GOVT.BENEFITS
MILITARY.START.DATES
MILITARY.END.DATES
Also, you can view which tables will be created on the target database for
a specific source transform by detailing on the View Generated Transforms
field on the DOST form.
Q When running a refresh, I receive the following message:
User error 10015: - unable to authenticate your
login.
A This message occurs because a pre-authenticated server is used for the file
transfer in the DataOrchestrator ODS refresh. This means that on the DMI
Pre-Authenticated Server (DMCC) form, you must have your application
server and your Colleague database server defined. If you are a UniData
client who has not distributed the DAS, then your application server and
Colleague database server are the same, so you will need to enter only one
line.
If you have the DMCC form set up and are still experiencing issues, then
follow these steps:
234
Using the DataOrchestrator™ ODS, July 9, 2013
Frequently Asked Questions about the DataOrchestrator ODS
Step 1. Start the following listeners with logging turned on:
 The listener defined on the DORE form.
 The listener installed on the DataOrch Parameters (DOPA) form.
Step 2. Rerun the Check Connectivity function on the DORE form to receive the
error again, then search for an entry in the dmi.log file such as the following:
Unable to authenticate a client login from
(hp.datatel.com,127.0.0.1)
Use the DNS name and the IP address in this entry to define the preauthenticated server on the DMCC form.
Step 3. Restart the listeners again with logging turned off.
Q When running a refresh, I receive the following error:
Unable to log in to target database.
Failed Login to <db_name> on <server_name> by
<user>: Premature end of Stream: server won't
transmit TDS header
A This is probably due to an incorrect database port specified on the DOTA
form. Access the DOTA form and verify that the correct database port is
specified.
Q When running a refresh, I receive the following error:
The file separator detected (“/”) is not the same
separator in the file path “...”
A Look for the following:


In SA Valet, make sure the listener on the target ODS database that is
specified on DOTA has both the RDAS and DBAS roles.
If your source database is UniData or Oracle, then go to the DOPA
form and make sure the correct source listener is specified in the
Source DMI_DAS Listener Name field.
Using the DataOrchestrator™ ODS, July 9, 2013
235
Appendices: Frequently Asked Questions
236
Using the DataOrchestrator™ ODS, July 9, 2013
Appendices
Troubleshooting the
DataOrchestrator ODS
In This Appendix
This appendix provides suggestions for items to check or steps to take if you
encounter issues in using the DataOrchestrator ODS.
Note: For the most up-to-date information about the troubleshooting
issues listed in this appendix, as well as any additional issues reported
since the publication of this manual, see AnswerNet page 6063. If you
do not have access to AnswerNet at your institution, consult with your
system administrator.
Using the DataOrchestrator™ ODS, July 9, 2013
237
Table 3 lists the issues, possible causes, and resolutions for troubleshooting.
Table 3: Troubleshooting Issues
1.
Issue
Possible Cause
Resolution
Receive one of the following errors on the
DataOrch Refresh (DORE) form when
checking connectivity or running a refresh:
The source directory path on the
DataOrch Parameters (DOPA) form
could be incorrect or the permission
on the directory could be incorrect.
Verify the path on the DOPA form is a valid path on the
source database server. Also, verify the permissions to
the directory correspond with the installation instructions
on setting up the source database server temporary
directory.
UniData source:
Error - encountered during Data
Extract
Cannot open D:\...\DOR... file.
Oracle source:
Using the DataOrchestrator™ ODS, July 9, 2013
Error - encountered during Data
Extract
ORA-20014: EXTRACT_DATA: Error
extracting data. Message: ORA20002: OPEN_FILE:
INVALID_OPERATION: File could
not be opened as requested
Appendices: Troubleshooting the DataOrchestrator ODS
238
DataOrchestrator ODS Issues
Using the DataOrchestrator™ ODS, July 9, 2013
Table 3: Troubleshooting Issues (cont’d)
Issue
2.
Receive one of the following errors on the
DORE form when checking connectivity or
running a refresh:
SQL Server source:
Possible Cause
The target directory path on the
Additional Target Parameters
(DOTP) form could be incorrect or
the permission on the directory
could be incorrect.
Resolution
Verify the path on the DOTP form is a valid path on the
target database server. Also verify the permissions to
the directory correspond with the installation instructions
on setting up the target database server temporary
directory.
Error - encountered during Data
Extract:
SQLState = S1000, NativeError =
0Error = [Microsoft][ODBC SQL
Server Driver]Unable to open BCP
host data-file
UniData and Oracle source:
Error encountered during file
transfer
D:\thisIsNotADirectory\... (The
system cannot find the path
specified)
DataOrchestrator ODS Issues
239
Issue
3.
Receive one of the following errors on the
DORE form when checking connectivity or
running a refresh.
UniSession Exception-39207:
Error [39207] occurred on
server. Possible client-side
licensing failure.:
Unable to connect to the
registry database.
OR
Unable to login to target
database. Registry error-30008:
Unable to access the registry
database for the request.
Possible Cause
UniData licenses on the source
database have run out.
Resolution
Specify fewer threads to use on the DORE form, or run
the refresh at a time when fewer users are logged in to
Colleague.
Using the DataOrchestrator™ ODS, July 9, 2013
Appendices: Troubleshooting the DataOrchestrator ODS
240
Table 3: Troubleshooting Issues (cont’d)
Using the DataOrchestrator™ ODS, July 9, 2013
Table 3: Troubleshooting Issues (cont’d)
Issue
4.
Receive one of the following errors when
running a refresh.
Possible Cause
Resolution
The computed column does not
exist in the source database.
The computed column must be able to be generated to
the database using the Define Computed Columns
(DCC) form. It may be that the column needs to be
regenerated in order for the DataOrchestrator to use it.
The target RDAS listener does not
also have the DBAS role it needs, or
the listener specified on the DOPA
form is incorrect.
Verify that the target RDAS listener has both the DBAS
and RDAS roles according to the installation
instructions. Verify that the listener name on the DOPA
form is the correct source database listener.
The DOPA form is not defined for a
UniData or Oracle source
environment.
Verify that the parameters on the DOPA form are
correct.
Error - encountered during Data
Extract for table:
Fatal: 4121(16): Cannot find
either column "dbo" or the userdefined function or aggregate
computedColumn, or the name is
ambiguous.
OR
Error - encountered during Data
Extract for table:
computedColumn is not a column
in table
5.
Receive an error such as the following
when running a refresh:
6.
Receive an error such as the following
when running a refresh:
No source listener host name
specified
OR
No source extract file path
specified
241
DataOrchestrator ODS Issues
The file separator detected
("/") is not the same separator
in the file path "..."
Issue
7.
Receive an error such as the following
when running a refresh:
Possible Cause
Resolution
The transform is taking over an hour
for a UniData environment to
process.
The UniData RPC connection default timeout needs to
be increased. The UniData unirpcservices file needs the
udapi_server timeout increased from 3600 to 14400.
The setting requires unirpcd to be restarted in order to
take effect.
Could not RPC execute command
For example:
/usr/ud71/unishared/unirpc/
unirpcservices:
defcs /cm/tools/ud71_10/bin/udapi_server
* TCP/IP 0 14400
8.
Receive an error such as the following
when running a refresh with a SQL Server
target database:
Using the DataOrchestrator™ ODS, July 9, 2013
A user named "abcdef" is not able to
connect to the target SQL Server
database with SQL Server
authentication.
Check the target SQL Server database instance where
ods_test is set up to be sure that a user named "abcdef"
is defined with SQL Server authentication. Also, be sure
the security for that SQL Server instance is set up to
allow for both Windows and SQL Server authentication.
The refresh process is not able to
connect to either the source
DMI_DAS listener or target RDAS
listener.
Check that all listeners for the environment are running.
If some are not running, start them, and then look for the
following:
Unable to log in to target
database. Failed Login to
ods_test on sdw2k3qasql1 by
abcdef: Fatal: 18456(14): Login
failed for user 'abcdef'
9.
When checking connectivity on the DORE
form, the following error is received:
DMI_OPEN
• On the DORE form, be sure that the RDAS listener
installed on your target database server was
selected.
• If your source database is UniData or Oracle, then
access the DOPA form and be sure the DMI_DAS
listener is the DMI_DAS listener on your Colleague
database server.
Appendices: Troubleshooting the DataOrchestrator ODS
242
Table 3: Troubleshooting Issues (cont’d)
Using the DataOrchestrator™ ODS, July 9, 2013
Table 3: Troubleshooting Issues (cont’d)
Issue
10.
Receive an error such as the following:
User error 10015: - unable to
authenticate your login.
Possible Cause
Resolution
This message occurs because a
pre-authenticated server is used for
the file transfer in the
DataOrchestrator ODS refresh.
On the DMI Pre-Authenticated Server (DMCC) form,
you must have your application server and your
Colleague database server defined. If your institution
uses UniData and has not distributed the DAS, then
your application server and Colleague database server
are the same, so you will need to enter only one line.
If you have the DMCC form set up and are still
experiencing issues, then follow these steps:
Step 1: Start the following listeners with logging turned
on:
• The listener defined on the DORE form.
• The listener installed on the DOPA form.
Step 2: Rerun the Check Connectivity function on the
DORE form to receive the error again, then search for
an entry in the dmi.log file such as the following:
"Unable to authenticate a client login
from (hp.datatel.com,127.0.0.1)"
Use the DNS name and the IP address in this entry to
define the pre-authenticated server on the DMCC form.
11.
Receive an error such as the following:
Unable to log in to target
database. Failed Login to
<db_name> on <server_name> by
<user>: Premature end of Stream:
server won't transmit TDS header
This is probably due to an incorrect
database port specified on the
DOTA form.
Access the DOTA form and verify that the correct
database port is specified.
243
DataOrchestrator ODS Issues
Step 3: Restart the listeners again with logging turned
off.
Issue
12.
Receive an error such as the following:
No suitable driver.
This error occurs when using the following
forms to create or test SQL views on an
Oracle target database:
• DataOrch Target Views (DOTV)
Possible Cause
There is no Oracle JDBC driver
installed in the UniData source
environment.
If you have a UniData source
environment and your target
database is Oracle, then you also
must install the Oracle JDBC driver
on your source environment.
Resolution
Install the Oracle JDBC driver using the instructions in
“Installing the Oracle JDBC Driver” on page 51, but on
the UniData source environment. Then restart the
source environment DMI_DAS for the change to take
effect.
• DataOrch View Spec (DOVS)
Using the DataOrchestrator™ ODS, July 9, 2013
13.
The refresh will “hang” in a SQL Server
source environment.
There may be multiple versions of
the bcp.exe file. This occurs when
multiple versions of SQL Server
have been installed on the ODS
target database server, which is
used for the data extract.
Make sure the path environment variable for the ODS
target database server is pointing to the latest version of
bcp. If you update the path, you must restart the target
RDAS listener for the change to take effect.
14.
Receive the following error:
The Source Temporary File Path
entered on the DOPA form is
incorrect.
The Source Temporary File Path is set up on the DOPA
form. Check that this path exists, and that the SYSTEM
user and/or the user running DMI has full read/write
access to the folder. Also, try deleting and re-entering
the entire path. Stray spaces may cause incorrect paths
and are not easy to see in UI forms.
ERROR, cannot read source file
<<file path>>
Comment: The full error may look something like this:
Error encountered during file transfer
SSFRQ: ERROR, cannot read source file
<<file path>>
15.
Receive the following error in a UniData
environment:
In BP/_S_RPC_EXEC at line 2
In BP/_S_RPC_EXEC at line 2 too
many names in LIST
The transform being refreshed has
over 150 columns specified. For
more information, see Support
Solution 4304: UniQuery LIST field
limit.
Specify 150 columns or less for the transform. You can
either be more selective about the columns included for
the transform, or split the columns into multiple
transforms.
Appendices: Troubleshooting the DataOrchestrator ODS
244
Table 3: Troubleshooting Issues (cont’d)
Using the DataOrchestrator™ ODS, July 9, 2013
Table 3: Troubleshooting Issues (cont’d)
Issue
16.
Receive the following type of error:
Violation of PRIMARY KEY
constraint
17.
Receive the following errors in Colleague
SQL Server environment on the DORE
form when checking connectivity or running
a refresh:
Resolution
This is usually a specific data issue
where duplicate keys exist for one or
more files/tables. The source of the
problem may not always be
apparent, because the extract and
transformation process splits source
files into many different tables on the
target.
See Support Solution 6438: ODS: “Violation of
PRIMARY KEY constraint” SQL error.
This error can occur during
execution of the bcp statement that
extracts data from the Colleague
SQL Server database. To confirm
this, turn logging on in the DAS and
then rerun the refresh and view the
log. In the log, find the SBXTQ
incoming and outgoing transactions
which look something like this:
Use the following Microsoft resource to update the
server name returned by the SQL statement SELECT
@@SERVERNAME:
incoming:
DMIþ2.0þDBIQþþþsql17746142
01þþþþþþþþþþþSBXTQþ9þ0þf:\
datatel\coll18\test\ODS_LI
STENER\ods_temp\þ~|~þDOR70
64.datþSYSDEFSþSELECT
SYSDEFS SAMPLE 5 RETURNING
DELIM @IDþSBXTQ.END
outgoing:
DMIþ1.4þDBISþþþsql17746142
01þþýj3þ14821þ54787þþþþþþþ
SERRSþ7þ0þþþSQLState =
08001, NativeError =
53Error = [Microsoft][SQL
Native Client]Named Pipes
http://msdn.microsoft.com/en-us/library/ms187944.aspx
245
DataOrchestrator ODS Issues
SQLState = 08001, NativeError =
53Error = [Microsoft][SQL Native
Client]Named Pipes Provider:
Could not open a connection to
SQL Server [53]. SQLState =
HYT00, NativeError = 0Error =
[Microsoft][SQL Native
Client]Login timeout
expiredSQLState = 08001,
NativeError = 53Error =
[Microsoft][SQL Native Client]An
error has occurred while
establishing a connection to the
server. When connecting to SQL
Server 2005, this failure may be
caused by the fact that under
the default settings, SQL Server
does not allow remote
connections.
Possible Cause
Issue
(continued)
Possible Cause
Provider: Could not open a
connection to SQL Server
[53]. SQLState = HYT00,
NativeError = 0Error =
[Microsoft][SQL Native
Client]Login timeout
expiredSQLState = 08001,
NativeError = 53Error =
[Microsoft][SQL Native
Client]An error has
occurred while establishing
a connection to the
server.When connecting to
SQL Server 2005, this
failure may be caused by
the fact that under the
default settings SQL Server
does not allow remote
connections.þSERRS.END
Using the DataOrchestrator™ ODS, July 9, 2013
If you see an SERRS response, this
means the bcp statement that the
RDAS ran failed. To figure out the
server name to use in the bcp
statement, the RDAS runs the
following SQL statement to get the
server name:
SELECT @@SERVERNAME
You can run this in Management
Studio. If the server name is
incorrect, use the Microsoft link
given in the Resolution on page 245
to update the server name.
Resolution
Appendices: Troubleshooting the DataOrchestrator ODS
246
Table 3: Troubleshooting Issues (cont’d)
Using the DataOrchestrator™ ODS, July 9, 2013
Table 3: Troubleshooting Issues (cont’d)
Issue
(continued)
Possible Cause
Resolution
The @@SERVERNAME might be
incorrect from changing a server's
name. If you change the name, but
do not update the
@@SERVERNAME in SQL Server,
then you will likely run into this issue
during a DataOrchestrator ODS
refresh.
To see if the issue is due to a
mismatch, run the following on the
source database server and check
to see if it returns the correct server:
SELECT @@SERVERNAME
18.
19.
Receive a message in SQL Server
Colleague source environment similar to
the following:
Receive the following error on the DataOrch
Refresh (DORE) form when checking
connectivity or running a refresh:
The user does not have access to
the source database, or does not
have the required rights (bulkadmin).
Verify that the user has access to the source database
and has the required rights.
Error - encountered during source database
login
If an override login for the source is
defined on the DataOrch
Parameters (DOPA) form, that login
is invalid or does not have the
required rights.
If an override login is defined on the DOPA form, make
sure that login has access to the source database and
has the required rights.
247
DataOrchestrator ODS Issues
If your SQL Server Colleague source environment
username or password includes one of these
characters, change your password so that none of these
characters is used.
SQLState = 28000, NativeError =
18456Error = [Microsoft][SQL Native
Client][SQL Server]Login failed for user.
The SQL Server bcp command
treats certain characters as
parameter/command terminators.
Verify your source environment
username and password do not
include any of the following
characters: [] {}() , ; ? * ! @
Appendices: Troubleshooting the DataOrchestrator ODS
248
Using the DataOrchestrator™ ODS, July 9, 2013
Download PDF

advertising