CONCISE GUIDE TO DATA INTERCHANGE
TUESDAY, 15 APRIL 2014
Version: 1.00
Reference: Data Interchange © 2014 Australian Energy Market Operator Ltd (AEMO). All rights reserved.
Concise Guide to Data Interchange
Important Notice
AEMO has prepared this Concise Guide to Data Interchange (Guide) to provide guidance on the use of the Data
Interchange software under the National Gas or Electricity Rules (Rules), as at the date of publication.
No reliance or warranty
This Guide does not constitute legal or business advice, and should not be relied on as a substitute for obtaining detailed
advice about the National Gas or Electricity Law, the Rules or any other applicable laws, procedures or policies. While
AEMO has made every effort to ensure the quality of the information in this Guide, neither AEMO, nor any of its
employees, agents and consultants make any representation or warranty as to the accuracy, reliability, completeness,
currency or suitability for particular purposes of that information.
Limitation of liability
To the maximum extent permitted by law, AEMO and its advisers, consultants and other contributors to this Guide (or
their respective associated companies, businesses, partners, directors, officers or employees) are not liable (whether
by reason of negligence or otherwise) for any errors, omissions, defects or misrepresentations in this document, or for
any loss or damage suffered by persons who use or rely on the information in it.
Copyright
Copyright 2014 Australian Energy Market Operator Limited. The material in this publication may be used in accordance
with the copyright permissions on AEMO’s website.
Trademark notices
Microsoft is a trademark of Microsoft Corporation in the United States and/or other countries.
Oracle and Java are registered trademarks of Sun Microsystems, Inc. in the U.S. and other countries.
Documents made obsolete
The release of this document changes any version of the Concise Guide to Data Interchange.
Distribution
Available to the public.
Prepared by
PMO Documentation Team
Last update: 15/04/2014 2:45 PM
Notes
No notes
Further information
For further information, please visit www.aemo.com.au or contact:
AEMO Information and Support Hub
Phone: 1300 AEMO 00 (1300 236 600) and follow the prompts.
Email: supporthub@aemo.com.au
© 2014 AEMO
i
Concise Guide to Data Interchange
Contents
1 Introduction
1
1.1 Purpose
1
1.2 Audience
1
1.3 How to use this guide
1
1.4 What's in this guide
2
1.5 Related resources
2
1.6 Accessing the software and guides
3
2 About Data Interchange Software
4
2.1 What the Data Interchange software is for
4
2.2 Who can use the Data Interchange software
4
2.3 User rights access
5
2.3.1 Data Subscription
5
2.3.2 Data Delivery
5
2.3.3 Performance Monitor
5
2.4 How do you use Data Interchange software
5
2.5 Requirements
6
3 Components
7
4 Implementation
10
4.1 Hardware specifications
10
4.2 Data volumes
10
4.3 Supported database platforms
10
4.4 Before commencing installation
11
4.5 Setting up a standard Data Interchange environment
11
5 Maintenance
14
5.1 File data feed
14
5.2 Monitoring
15
5.2.1 Database state
15
5.2.2 Operational checks
15
5.3 Upgrades
15
5.3.1 Tests for successful upgrade
16
6 Needing Help
17
6.1 AEMO's Information and Support Hub
17
© 2014 AEMO
ii
Concise Guide to Data Interchange
6.1.1 Contacting AEMO's Information and Support Hub
17
6.1.2 Information to provide AEMO
17
6.2 Feedback
18
7 References
19
7.1 AEMO's website
19
8 Index
20
© 2014 AEMO
iii
Concise Guide to Data Interchange
Figures
D
DBA
Figure 1: standard Data Interchange
implementation
Database administrator
12
Tables
DBMS
Participant’s database management system
conforming to the MMS Data Model.
DI
Data Interchange
Table 1: Data Interchange resources
2
Table 2: Data Interchange components
7
DR
Disaster recovery
E
Table 3: steps to set up a standard Data
Interchange environment
12
Glossary
These abbreviations, symbols, and special
terms assist the reader’s understanding of
the terms used in this document. For
definitions of these terms, the reader should
always refer to the applicable market Rules.
EMMS
Wholesale Electricity Market Management System;
software, hardware, network and related processes.
energy market systems web portal
Single web portal interface to access AEMO's IT
systems.
F
FTP
File transfer protocol
L
Loader
A
pdrLoader
AEMO
Australian Energy Market Operator
AES
Advanced Encryption Standard
M
MarketNet
AEMO’s private network available to participants
having a participant ID.
AEST
Australian Eastern Standard Time
MMD Data Model
The definition of the interface to participants of
data published by AEMO. It includes database
tables, indexes, and primary keys.
B
Batcher
pdrBatcher
N
NEM
C
National Electricity Market
CSV
Comma-separated values; a file format for
exchanging data.
NER
National Electricity Rules
NGR
National Gas Rules
© 2014 AEMO
iv
Concise Guide to Data Interchange
P
PA
Participant Administrator; manages participant
organisations user access and security.
Participant ID
Registered participant identifier
PDR
Participant Data Replication
pdrBatcher
Participant Data Replication Batcher
pdrLoader
Participant Data Replication Loader
Pre-production
Test and training environment, typically showing
much less activity, if any.
Production
Live environment, actively reflecting the currently
available data.
R
Replication Manager
Manages the configuration settings and monitors
the status of the replication of data from AEMO’s
systems to the participant’s DBMS.
Rules
The National Electricity or Gas Rules.
S
Standard Data Interchange environment
A single data feed, from one Participant ID folders
on the participant file server, to a single
participant database. For example, you can have
your pre-production environment connected to the
participant file server pre-production environment
and your production environment connected to the
participant file server production environment.
Under no circumstance point your pre-production
environment to the AEMO production
environment or vice versa.
Z
ZIP
The file compression format used for exchanging
data with AEMO.
© 2014 AEMO
v
Chapter 1 Introduction
1 Introduction
In this chapter:
1.1 Purpose
1
1.2 Audience
1
1.3 How to use this guide
1
1.4 What's in this guide
2
1.5 Related resources
2
1.6 Accessing the software and guides
3
1.1 Purpose
This is a concise guide assisting participants to understand AEMO's Data Interchange
software. It describes how to set up a standard Data Interchange environment to replicate
data between AEMO's energy market systems and participants' local DBMS conforming to
the MMS Data Model. The standard environment is AEMO's recommended configuration.
This document is written in plain language for ease of reading. Where there is a discrepancy
between the Rules, and information or a term in this document, the Rules take precedence.
1.2 Audience
This guide is relevant to registered participants who implement and maintain interfaces to
AEMO's energy market systems. IT knowledge, including networking, database management,
batch file management, and disaster recovery is required to understand this guide.
1.3 How to use this guide
This is a concise guide only, assisting your understanding of the Data Interchange software
components and describing how to set up a standard Data Interchange environment.
© 2014 AEMO
1
Concise Guide to Data Interchange
Other useful references, including other configuration options, and assistance with Data
Interchange issues are available on the Data Interchange web page:
l
l
l
Data Interchange Guide: A comprehensive guide to Data Interchange that includes
options to consider before set up, different configuration options, and data recovery
options.
Guide to Troubleshooting Data Interchange: Assistance with Data Interchange software
issues.
For more details about individual Data Interchange software components, see "Related
resources" below.
Text in this format, indicates a direct hyperlink with further details of the resource listed in.
1.4 What's in this guide
l
l
l
l
l
l
Chapter 2 "About Data Interchange Software" on page 4 provides and overview of the
Data Interchange software, what it is for, who can you use, and how to use it.
Chapter 3 "Components" on page 7 explains the different components that make up Data
Interchange.
Chapter 4 "Implementation" on page 10 explains how to set up a standard
implementation of Data Interchange.
Chapter 5 "Maintenance" on page 14 explains maintenance issues such as receiving data
from AEMO, Monitoring your DBMS, and performing software upgrades.
Chapter 6 "Needing Help" on page 17 provides information to assist participants with IT
related issues and provides guidance for requesting assistance from AEMO.
Chapter 7 "References" on page 19 provides a list of resources referenced throughout this
document.
1.5 Related resources
Table 1 below describes the Data Interchange documentation set. Guides are periodically
updated, please ensure you are reading the latest version.
Table 1: Data Interchange resources
Name
Description
Guide to Setting up a standard Data
Interchange environment
A one page guide with the steps required to setup a standard
DI environment.
Data Interchange Guide
A comprehensive guide to Data Interchange that includes
options to consider before set up, different configuration
© 2014 AEMO
2
Chapter 1 Introduction
Name
Description
options, and data recovery options.
Data Subscription Services User Guide
Describes the web-based application in the energy market
systems web portal allowing participants to:
n
n
n
n
n
View a catalogue of all available files.
Subscribe to the files containing the data you need to
populate your local DBMS.
Identify the files you are currently subscribed to.
Unsubscribe from files no longer required.
Obtain the MMS Data Model scripts for building and
maintaining your local DBMS.
Participant Data Replication Batcher User
Guide (pdrBatcher)
The pdrBatcher is responsible for moving files between remote
and local folders.
Participant Data Replication Loader User
Guide (pdrLoader)
The pdrLoader parses AEMO-supplied .CSV files into a local
DBMS conforming to the MMS Data Model.
Replication Manager User Guide
Manages the configuration settings and monitors the status of
the replication of data from AEMO’s systems to a participant’s
DBMS conforming to the MMS Data Model.
Guide to Troubleshooting Data Interchange
Provides assistance with troubleshooting Data Interchange
issues.
MMS Data Model Report
The MMS Data Model is the definition of the interface to
participants of data published by AEMO. A DBMS conforming
to the MMS Data Model can contain a local copy of all current
participant-specific data recorded in AEMO's database. The
MMS Data Model includes database tables, indexes, and
primary keys.
MMS Data Model Package Summary
MMS Data Model Table to File to Report
Relationships workbook
1.6 Accessing the software and guides
Data Interchange software and associated documentation are available from the following
locations:
l
l
l
Releases directory on the participant file share: FTP to 146.178.211.25 > Data
Interchange, pdrBatcher, pdrLoader, or Replication Manager.
Data Subscription web application in the energy market systems web portal:
o
Production: https://portal.prod.nemnet.net.au
o
Pre-production: https://portal.preprod.nemnet.net.au
Data Interchange web page on AEMO’s website.
© 2014 AEMO
3
Chapter 2 About Data Interchange Software
2 About Data Interchange Software
This chapter provides and overview of the Data Interchange software, what it is for, who can
you use, and how to use it.
In this chapter:
2.1 What the Data Interchange software is for
4
2.2 Who can use the Data Interchange software
4
2.3 User rights access
5
2.4 How do you use Data Interchange software
5
2.5 Requirements
6
2.1 What the Data Interchange software is for
Data Interchange is a set of cooperating applications to replicate data between AEMO’s
Energy Market Systems and participants' local DBMS conforming to the MMS Data Model,
including management and monitoring.
Data Interchange has two core functions:
1. AEMO-side reporting applications that generate structured .CSV files into the
participant file server.
2. Participant-side software to replicate data from the participant file server to
participants' local DBMS.
2.2 Who can use the Data Interchange software
The Data Interchange software is useful for:
l
l
l
Participants wishing to replicate data between AEMO’s energy market systems and
their DBMS conforming to MMS Data Model.
Participants sharing data between participant IDs; for more details about Data Sharing,
see Guide to Information Systems.
Participants having staff with good IT skills, including networking, database
management, batch file management, and disaster recovery.
© 2014 AEMO
4
Concise Guide to Data Interchange
2.3 User rights access
Your company's participant administrator (PA) grants you permission to use the Data
Interchange software components in AEMO's web portal. The entities required for access to
each component are listed below. Where a participant user has user rights assigned by more
than one participant, they interactively choose the participant they represent, using the Set
Participant option.
For more details about participant administration and user rights access, see Guide to User
Rights Management.
2.3.1 Data Subscription
l
EMMS - Data Subscription - Maintain and View Data
l
EMMS - Data Subscription - View Data
2.3.2 Data Delivery
l
EMMS - Data Interchange
2.3.3 Performance Monitor
l
EMMS - Data Interchange - Performance Monitor
2.4 How do you use Data Interchange software
AEMO recommends participants set up a standard configuration of Data Interchange, which
is a single data feed, from one Participant ID folders on the participant file server, to a single
participant database. For example, you can have your pre-production environment connected
to the participant file server pre-production environment and your production environment
connected to the participant file server production environment.
Under no circumstance point your pre-production environment to the AEMO production
environment or vice versa.
AEMO provides working software to participants and each participant is responsible for
setting up their own systems and satisfying their requirements for data, including deciding if
the complexity of using Data Interchange is a better business decision than using the simpler
web portal applications that require only a web browser.
© 2014 AEMO
5
Chapter 2 About Data Interchange Software
The core processes of Data Interchange are:
1. Participants decide what data they wish to receive using the Data Subscription web
application.
2. The data is generated in structured .CSV files into the participant file server.
3. Participant-side Data Interchange software replicates the data from the participant file
server to participants' local DBMS.
4. Participants manage their local DBMS and the data feed.
2.5 Requirements
Successfully implementing and managing Data Interchange requires good IT skills, including
networking, database management, batch file management, and disaster recovery.
To use Data Interchange requires:
l
l
l
l
l
A user name, password, and user rights, provided by your company's participant
administrator (PA), providing access to the Data Interchange web applications in the
energy market systems web portal, see "How do you use Data Interchange software" on
previous page.
A DBMS supporting the MMS Data Model. The MMS Data Model supports versions of
Oracle and Microsoft SQL Server.
Java runtime engine suitable for the target DBMS. The distribution file from AEMO
contains supported JDBC drivers for Oracle and SQL Server.
The Replication Manager software runs on Windows OS only, other Data Interchange
software components run on both Windows and Unix-like operating systems.
For accessing the energy market systems web portal, Microsoft Internet Explorer
version 7 or later, although the recommended version is Microsoft Internet Explorer 8.
© 2014 AEMO
6
Chapter 3 Components
3 Components
Table 2 below describes the components that make up Data Interchange.
Table 2: Data Interchange components
Component
Description
Baseline data
Intended for initial loading into an participant's empty DBMS, to provide key
standing data immediately prior to commencing the routine data feed.
Participants can fully populate the tables with baseline data by copying each file in
the \MarketData\Baseline folder into the pdrLoader input folder for the matching
source environment, for either production or pre-production.
Data Delivery
The web application in the energy market systems web portal allowing registered
participants to:
n
n
n
n
Data Subscription
The web application in the energy market systems web portal allowing registered
participants to:
n
n
n
Data Interchange
bundles
View the Data Interchange Status Dashboard
Make a file delivery query
Re-request files
View your organisation's data sharing arrangements.
Obtain the latest MMS Data Model scripts for building and maintaining their
DBMS
Subscribe and unsubscribe to the report files available in the participant file
server
View the entire list of available files. Note: AEMO recommends subscribing to
all available files and selectively loading them into your DBMS as required.
This way the file is created and available if required in the future
(approximately 6 months of history is retained online). Unsubscribing from
files makes them unavailable in the future.
All the components necessary for a new installation of Data Interchange are bundled
together into compressed files; each bundle is for a specific DBMS and version of the
MMS Data Model.
The Data Interchange bundles are available from:
n
n
The Data Interchange web page on AEMO's website.
The Releases directory: FTP to 146.178.211.25 > Data Interchange.
Historical data
AEMO has monthly and annual historical data available on DVD for subscription.
This historical data is intended for historical analysis, kept and processed separately
from the current data flows. For more details about obtaining history data and the
cost involved, contact AEMO's Information and Support Hub.
MarketNet
All participants must connect to the participant file server using AEMO’s private data
network—MarketNet, which uses the FTP protocol.
The pdrBatcher defaults to use the FTP protocol, simplifying the set-up to access the
participant file server.
© 2014 AEMO
7
Concise Guide to Data Interchange
Component
Description
MMS Data Model
Establishes the target tables in a participant's DBMS conforming to the MMS Data
Model, including database tables, indexes, and constraints. It is currently delivered
for Oracle and SQL Server.
MMS data sets provided from AEMO’s pre-production are different from AEMO’s
production because the environments are operated independently.
The MMS Data Model is usually updated with each release for example, new tables or
new columns. AEMO recommends participants keep their MMS Data Model up-todate with new releases to avoid any data loss.
Participant Data
Replication Batcher
(pdrBatcher)
Responsible for transferring files to and from AEMO's participant file server to the
participant's local Data Interchange folders. It can be installed in any environment
supporting Java.
pdrBatcher is a batch application, therefore does not have a graphical user interface.
Configuration of the pdrBatcher is done using the .properties file, which is only
read when the pdrBatcher starts up, so any changes require a restart of the
application.
pdrBatcher recovers from network failure to allow for recovery of missing data.
Participant Data
Replication Loader
(pdrLoader)
Responsible for loading files to and from participant's local Data Interchange folders
to the participant's DBMS. It can be installed in any environment supporting Java.
pdrLoader polls the participant’s local Data Interchange folders and processes data
from files into the participant’s DBMS conforming to the MMS Data Model.
pdrLoader is a batch application, therefore does not have a graphical user interface.
Configuration of the pdrLoader is done using the .properties file, which is only
read when the pdrLoader starts up, so any changes require a restart of the
application.
pdrLoader has mechanisms to identify and recover missing data.
Participant file server
The publishing point from AEMO systems to participant systems, with each
participant allocated an account and access to private and public areas. Each
participant is responsible for interfacing with the participant file server.
If uncollected, files are moved to the archive folder after a couple of days. If your Data
Interchange environment is configured properly it automatically retrieves the missing
files from the archive. Files are kept in the archive for approximately six months.
AEMO’s production and pre-production environments are independently operated, so
each environment has its own IP address for its participant file server.
Participant's DBMS
Contains the data received from AEMO into tables matching the MMS Data Model.
AEMO’s production and pre-production systems are independently operated, so
ensure the data from one is not confused with the other.
When maintaining the database, ensure compatibility is maintained with the MMS
Data Model. Not keeping up-to-date with new releases will compromise the data
© 2014 AEMO
8
Chapter 3 Components
Component
Description
integrity.
Performance Monitor
The web application in the energy market systems web portal allowing registered
participants to view a graphical interface of the current Data Interchange Performance
Monitor environment run and managed by AEMO. Participants not receiving data, or
experiencing data delays can contrast the data delivery performance of their Data
Interchange environment to AEMO's Data Interchange performance environment.
Replication Manager
An interactive user interface allowing participants to configure and interrogate the
pdrLoader. The software is installed locally on the system administrator’s desktop
and must run on a Windows operating system but it can access configuration data
from any supported DBMS regardless of the store’s operating system.
© 2014 AEMO
9
Chapter 4 Implementation
4 Implementation
This chapter explains implementing Data Interchange so all the parts cooperate and are set
up correctly. This section is equivalent to a Quick Start guide and explains some basic
considerations and the order to install the components. For more detailed information, see
the Data Interchange Guide.
In this chapter:
4.1 Hardware specifications
10
4.2 Data volumes
10
4.3 Supported database platforms
10
4.4 Before commencing installation
11
4.5 Setting up a standard Data Interchange environment
11
4.1 Hardware specifications
Things to consider include redundancy, number of subscriptions, in-house business
requirements, and level of business impact of delay in availability of data, and so on.
The sizing of any server depends on factors such as how much data to keep online, how many
concurrent users, what requirement for performance, current and anticipated data volumes,
and what data to download. AEMO recommends a capacity planning exercise to establish the
appropriately-specified hardware platforms.
The software runs on both 32-bit and 64-bit systems.
4.2 Data volumes
AEMO has the following estimates:
l
l
No. of files: 2500 files per day (subscription to all suites and reports).
Data volumes: 1.2 GB monthly of compressed public .CSV files, with private data over
and above this.
4.3 Supported database platforms
Versions of Oracle or SQL Server, see Data Interchange for the latest supported versions.
© 2014 AEMO
10
Concise Guide to Data Interchange
4.4 Before commencing installation
Before commencing to set up your Data Interchange environment, ensure you have the
following available:
l
Appropriately skilled IT staff.
l
A network security design.
l
All Data Interchange components for your DBMS, see "Components" on page 7.
l
A database administrator or a service provider with a database administrator.
l
Your participant ID.
l
The IP address for the participant file server, see Guide to Information Systems.
l
Your login and password to the participant file server.
l
Your login, password, and connection details for your DBMS.
l
Your login and password to the Data Subscription web application.
4.5 Setting up a standard Data Interchange environment
This section explains how to set up a standard implementation of Data Interchange (see
Figure 1 on the facing page). AEMO recommend setting up your first Data Interchange
environment in pre-production. Once you have a successful pre-production installation (used
for testing), you can repeat the installation in production to receive live market data. Be
careful to always keep the pre-production and production data entirely separate. .
A standard Data Interchange environment is a single data feed, from one Participant ID
folders on the participant file server, to a single participant database. For example, you can
have your pre-production environment connected to the participant file server preproduction environment and your production environment connected to the participant file
server production environment.
Under no circumstance point your pre-production environment to the AEMO production
environment or vice versa. All participants are encouraged to use the pre-production
environment to test implementation and to train their participant users, before attempting
any changes or operations on the production systems.
AEMO’s production and pre-production environments are independently operated, so the set
of components shown in the Figure 1 on the facing page are duplicated for production and
pre-production.
© 2014 AEMO
11
Chapter 4 Implementation
Figure 1: standard Data Interchange implementation
To set up a standard Data Interchange environment, follow the steps in Table 3 below.
Table 3: steps to set up a standard Data Interchange environment
Step Do
Reference
1
Download and extract a DI bundle conforming to the MMS
Data Model (Oracle or MS SQL Server).
Data Interchange web page on
AEMO's website.
2
Set up Data Subscription:
Data Subscription Services User
Interface Guide
n
3
Configure report subscriptions using the Data
Subscription web application. AEMO recommends
subscribing to the full MMS Data Model data set.
Create a DBMS instance to conform to the MMS Data Model:
n
n
Install the DBMS.
Install the latest MMS Data Model structure (target
tables). Do not make any changes to the MMS Data
Model structure.
MMS Data Model GUI Installer Guide
MMS Data Model Report
4
Install and configure the Participant Data Replication Loader
(pdrLoader).
Participant Data Replication Loader
User Guide
Participant Data Replication Loader
GUI Installer Guide
README.txt
5
Run the pdrLoader application:
Participant Data Replication Loader
User Guide
n
n
In Windows, double-click pdrLoader.bat file in the
Lib folder or start the Windows service.
In unix-like environments, execute the pdrLoader.sh
file in the Lib folder.
6
Install the MMS Data Model configuration for pdrLoader.
MMS Data Model pdrLoader
Configuration Implementation Note
7
Install Replication Manager:
Replication Manager Installation
Guide.
n
Extract and install the Replication Manager software
on a local PC running Microsoft Windows to monitor
and control the operation of the pdrLoader.
© 2014 AEMO
12
Concise Guide to Data Interchange
Step Do
Reference
8
Historical data is available on CD
from AEMO’s information and support
hub, phone: 1300 AEMO 00 (1300 236
600), option 2 email:
supporthub@aemo.com.au.
Populate your DBMS with data:
n
n
Load the baseline data available from the participant
file server \MarketData\Baseline to your pdrLoader
Reports folder (baseline data excludes historical
data).
Load the optional historical data.
9
Install the Participant Data Replication Batcher (pdrBatcher).
Participant Data Replication Batcher
User Guide
10
Run the pdrBatcher application:
Participant Data Replication Batcher
User Guide
n
n
In Windows, double-click pdrBatcher.bat file in the
Lib folder or start the Windows service.
In unix-like environments, execute the
pdrBatcher.sh file in the Lib folder.
© 2014 AEMO
13
Chapter 5 Maintenance
5 Maintenance
This chapter explains maintenance issues such as receiving data from AEMO, Monitoring
your DBMS, and performing software upgrades.
In this chapter:
5.1 File data feed
14
5.2 Monitoring
15
5.3 Upgrades
15
5.1 File data feed
To receive data from AEMO, do both of the following:
l
l
Subscribe to your required files in the Data Subscription web application. AEMO
recommends subscribing to all available files and selectively loading them into your
DBMS as required. This way the file is created and available if required in the future
(for approximatel 6 months). Unsubscribing from files makes them unavailable in the
future.
Ensure the chosen files are set to be loaded by pdrLoader into your DBMS (using
Replication Manager or direct SQL statements).
A file feed can be turned off by either:
l
l
Unsubscribing from the file using the Data Subscription web application. If you
unsubscribe from a file, it is no longer delivered to your folder on the participant file
server. The file is not produced, so it is not possible to retrieve it at a later date.
Setting the pdrLoader file load to inactive in your DBMS, using Replication Manager or
direct SQL statements. If you set the pdrLoader file load to inactive, the file is still
generated and transferred but its content is not loaded to your DBMS. AEMO still
maintains a copy of that file in its online archive for up to 6 months, so the file can be
re-requested for transfer at a later date.
© 2014 AEMO
14
Concise Guide to Data Interchange
5.2 Monitoring
5.2.1 Database state
To tell if the database is up-to-date, do any one or more of the following:
l
l
l
l
Inspect the PDR_MANIFEST_LOG table directly. For detailed table descriptions, see
"Database Tables" in the Participant Data Replication Loader User Guide.
Query the PDR_REPORT_AUDIT_SUMMARY table for a regularly received report
(such as DISPATCHIS). For more details about the PDR tables, see the Participant Data
Replication Loader User Guide.
Query the MMS Data Model tables you expect to have current data (based upon your
current subscriptions).
Use the Replication Manager to show all known data discrepancies in this database.
Replication Manager only refers to the PDR tables, with the primary screen looking at
the audit table.
5.2.2 Operational checks
AEMO recommends that participants monitor their Data Interchange environment to ensure
smooth operation. Key monitoring points include:
l
Build up of files in your participant Data Interchange folders.
l
Space monitoring at the database and file system levels.
l
Log file inspection of the pdrBatcher and pdrLoader processes.
l
Timeliness of specific tables in your DBMS.
l
Running queries against the PDR tables.
For assistance with any issues, see the Guide to Troubleshooting Data Interchange.
5.3 Upgrades
Participants are required to implement upgrades to their Data Interchange systems in line
with each release. AEMO supports only the latest and previous versions of Data Interchange
components. To ensure no loss of data, AEMO recommends participants schedule resources
to maintain their environment and ensure it is current with AEMO Data Interchange
software releases.
© 2014 AEMO
15
Chapter 5 Maintenance
For each new release, do the following:
l
Subscribe to any new files relevant to your business, or choose the latest file set for the
MMS Data Model in the Data Subscription web application.
l
Download and apply the scripts to upgrade your DBMS.
l
Ensure the reports are activated to load to the database.
l
Confirm the latest files are being delivered and loaded correctly.
l
Unsubscribe from legacy files.
Details about the changes affecting participants in new Data Interchange software are
detailed in the Technical Specification delivered to participants before each release.
5.3.1 Tests for successful upgrade
To check a configuration update is applied, identify a new table that is supposed to be added
for the new version of the MMS Data Model and run a query like:
select count(1) from PDR_REPORT_RECORDS where DESTINATION_
TABLE='<NameOfNewTable>';
If the query returns 0 rows, then the MMS Data Model pdrLoader configuration has not run
successfully.
To check that a report is activated to load to the database, either:
l
l
Right click in the Replication Manager front screen and check the desired report. You
can activate reports on this screen too.
Check the IS_ACTIVE status in table PDR_REPORT_TYPE_CONFIG for the desired
report, using SQL. You can activate a report by setting the status, using SQL, too.
© 2014 AEMO
16
Chapter 6 Needing Help
6 Needing Help
6.1 AEMO's Information and Support Hub
6.1.1 Contacting AEMO's Information and Support Hub
Assistance is requested through AEMO's Information and Support Hub using one of the
following methods:
l
Phone: 1300 AEMO 00 (1300 226 600) and follow the prompts.
For non-urgent issues, normal coverage is 8:00 AM to 6:00 PM on weekdays,
Australian Eastern Standard Time (AEST).
l
l
Email: supporthub@aemo.com.au
The Customer Portal, http://helpdesk.preprod.nemnet.net.au/nemhelplite/ allows you
to log your own requests for assistance. For access credentials, see your organisation's
IT security contact or participant administrator.
Please note that AEMO recommends participants call AEMO's Information and Support Hub
for all urgent issues, whether or not you have logged a call in the Customer Portal.
6.1.2 Information to provide AEMO
Please provide the following information when requesting assistance from AEMO:
l
Your name
l
Organisation name
l
Participant ID
l
System or application name
l
Environment: production or pre-production
l
Problem description
l
Screenshots
For AEMO software-related issues please also provide:
l
Version of software
l
Properties or log files
l
Replication Manager support dump and instance name (if Data Interchange problem)
© 2014 AEMO
17
Concise Guide to Data Interchange
6.2 Feedback
To suggest improvements to this guide, please contact the AEMO Information and Support
Hub.
© 2014 AEMO
18
Chapter 7 References
7 References
The resources listed in this section contain related information that may assist you.
7.1 AEMO's website
l
l
l
l
l
l
"Data Interchange", the web page containing all software and documentation relating to the
Data Interchange components: http://www.aemo.com.au/About-theIndustry/Information-Systems/Data-Interchange (Home > About the Industry >
Information Systems > Data Interchange).
Guide to AEMO CSV Data Format Standard, http://www.aemo.com.au/About-theIndustry/Information-Systems/Using-Energy-Market-Information-Systems (Home >
About the Industry > Information Systems > Using Energy Market Information
Systems).
Guide to Information Systems, http://aemo.com.au/About-the-Industry/InformationSystems Home > About the Industry > Information Systems).
Guide to Troubleshooting Data Interchange, provides assistance with troubleshooting Data
Interchange software issues: http://www.aemo.com.au/About-theIndustry/Information-Systems/Data-Interchange#DI Guides (Home > About the
Industry > Information Systems > Data Interchange).
Guide to User Rights Management, http://www.aemo.com.au/About-theIndustry/Information-Systems/Using-Energy-Market-Information-Systems (Home >
About the Industry > Information Systems).
"IT Assistance", information to assist participants with IT related issues:
http://www.aemo.com.au/About-the-Industry/Information-Systems/IT-Assistance
(Home > About the Industry > Information Systems > IT Assistance).
© 2014 AEMO
19
Concise Guide to Data Interchange
P
8 Index
Participant's DBMS 8
B
Participant Data Replication Batcher (pdrBatcher) 8
Participant Data Replication Loader (pdrLoader) 8
Baseline data 7
Before commencing installation 11
Participant file server 8
pdrBatcher 8
C
pdrLoader 8
Components 7
Performance Monitor 9
D
R
Data Delivery 7
Related resources 2
Data Interchange bundles 7
Replication Manager 9
Data Interchange components 7
S
Data Interchange resources 2
standard Data Interchange implementation 12
Data Subscription 7
Data volumes 10
steps to set up a standard Data Interchange
environment 12
Database state 15
Supported database platforms 10
F
Feedback 18
T
Tests for successful upgrade 16
File data feed 14
U
H
Upgrades 15
Hardware specifications 10
Historical data 7
I
Implementation 10
Information and Support Hub 17
M
MarketNet 7
MMS Data Model 8
Monitoring 15
O
Operational checks 15
© 2014 AEMO
20
Download PDF
Similar pages