SpectraProc User Guide

SpectraProc User Guide

Spectral Research Solutions

User Guide

Version:

Date:

Status:

Author:

File:

Pages:

Classification:

Distribution:

0.2

16.06.2006

Draft

A. Hueni

\UserGuide_V02.doc

43

SpectraProc

SpectraProc

User Guide

History

Version Date

0.1

0.11

0.2

03.04.2006

06.04.2006

16.06.2006

Author

A. Hueni

A. Hueni

A. Hueni

Remark

First draft

Modified MySQL installation

Modified Tutorial for new example set

Page 2 of 43

A. Hueni \UserGuide_V02.doc Version 0.2 / 16.06.2006

SpectraProc

User Guide

Page 3 of 43

Table of Contents

1 Introduction .................................................................................................... 5

2 Installation and Configuration ...................................................................... 6

2.1

Database ................................................................................................................................ 6

2.1.1

Installing MySQL............................................................................................................. 6

2.1.2

Installing MySQL-Administrator and Query Browser ...................................................... 6

2.1.3

Setting up the Spectral Database Schema, Tables and Users ...................................... 6

2.1.4

Network Access .............................................................................................................. 6

2.2

SpectraProc Application ......................................................................................................... 6

3 Design of Sampling Experiments ................................................................. 8

3.1

Overview ................................................................................................................................. 8

3.2

Hierarchical Structure ............................................................................................................. 8

3.3

Directory Structure and Spectral Files .................................................................................... 8

4 Operation ...................................................................................................... 10

4.1

Overview ............................................................................................................................... 10

4.1.1

Dataflow ........................................................................................................................ 10

4.1.2

File System Interfaces .................................................................................................. 10

4.1.3

SpectraProc Operation ................................................................................................. 11

4.1.4

Spectral Processing Concept ....................................................................................... 11

4.1.5

User Interface ............................................................................................................... 12

4.2

Creating a new Study............................................................................................................ 13

4.3

Deleting an existing Study .................................................................................................... 14

4.4

Selecting Studies .................................................................................................................. 14

4.5

Loading Spectra.................................................................................................................... 14

4.6

Processing Chain Settings.................................................................................................... 15

4.6.1

Waveband Filtering....................................................................................................... 15

4.6.2

Smoothing..................................................................................................................... 16

4.6.3

Sensor Synthesizing / Downsampling........................................................................... 17

4.6.3.1

4.5.3.1 Ratio Sensors ............................................................................................................. 17

4.6.3.2

Gaussian Sensors ................................................................................................................... 18

4.6.3.3

Supplied Sensors.................................................................................................................... 19

4.6.4

Derivative Calculation ................................................................................................... 20

4.6.5

Feature Space Transformation..................................................................................... 21

4.6.5.1

DI (DGVI) ............................................................................................................................. 21

4.6.5.2

NTBI...................................................................................................................................... 21

4.6.5.3

PCT ........................................................................................................................................ 21

4.6.5.4

Creation, Modification and Deletion of Feature Space Definitions ....................................... 21

4.6.5.5

Feature Space Selection ......................................................................................................... 23

4.7

File Export............................................................................................................................. 23

4.8

Spectral Mixtures .................................................................................................................. 25

4.8.1

Endmember Designation .............................................................................................. 25

4.8.2

Abundance Settings...................................................................................................... 26

4.9

Library Building ..................................................................................................................... 27

A. Hueni \UserGuide_V02.doc Version 0.2 / 16.06.2006

SpectraProc

User Guide

Page 4 of 43

4.10

Analysis................................................................................................................................. 27

4.10.1

Separability ................................................................................................................... 27

4.10.2

Discriminant Analysis.................................................................................................... 28

4.10.3

Principal Component Analysis ...................................................................................... 29

4.10.4

Unmixing....................................................................................................................... 30

4.11

Adding new Sensors ............................................................................................................. 30

4.12

Linking to a GIS System ....................................................................................................... 33

4.12.1

Setting up the Spectral DB as a Data Source............................................................... 33

4.12.2

Establishing a Database Connection in ArcCatalog ..................................................... 33

4.12.3

Adding Table Data to a Map ......................................................................................... 35

5 Tutorial.......................................................................................................... 37

5.1

Overview ............................................................................................................................... 37

5.2

Examine the Folder and File Structure ................................................................................. 37

5.3

Creating a new Study and Loading the Spectra ................................................................... 37

5.4

Get to Know Your Data ......................................................................................................... 37

5.5

Discriminant Analysis in PCT Feature Space ....................................................................... 41

6 References.................................................................................................... 43

A. Hueni \UserGuide_V02.doc Version 0.2 / 16.06.2006

SpectraProc

User Guide

Page 5 of 43

1 Introduction

Field spectroradiometry has become increasingly popular in the last few years. The technology has advantages over conventional techniques, allowing the non destructive sampling of objects and possibly enabling the user to gain critical information more quickly and cheaply. As a result, many scientists are now actively researching applications of hyperspectral sensing. The operation of the instruments tends to be relatively easy and data are collected quickly. However, the interpretation of these data is not so simple. The main issue when dealing with hyperspectral data is their dimensionality which is the result of sampling a wide spectral range in very narrow bands. This is in itself a problem because the influence of noise on narrow channels is much higher than on traditional broadband channels. Hyperspectral data are more complex than previous multispectral data and different approaches for data handling and information extraction are needed (Vane and Goetz,

1988; Landgrebe, 1997).

Hyperspectral data are essentially multivariate data, consisting of hundreds or even thousands of variables. It has been shown that more bands do not automatically imply better results. Although the separability of classes does increase with growing dimensionality, the classification accuracy does not follow this trend endlessly but will decrease at a certain point. This is called the Hughes Phenomenon and is caused by the ever increasing number of samples needed to build sound statistics if the dimensionality grows (Landgrebe, 1997). In practice this means that more samples must be collected to ensure successful statistical analyses. It is necessary, therefore, to collect a large number of spectral data files, each containing a hyperspectral spectrum. The sheer number of files and variables can become overwhelming. Interestingly, very few studies concerned with hyperspectral data have ever mentioned how the data had been organised and stored.

A further issue that is rarely addressed is the reusability of the data. Reference data is usually compiled in so called spectral libraries. The majority of the publicly available spectral libraries are distributed as physical files. This has drawbacks such as low flexibility and low query performance

(Bojinski et al., 2003). Another drawback of most libraries is their restriction in the number of spectra per class. In many cases, only one reference spectrum is supplied. This reduces any statistical analysis to first order statistics. The use of average values may be useful in some circumstances, however, Landgrebe (1997) noted that the reduction of data to mean values results in a loss of information. Second order statistics contain vital information about the distribution of data in spectral or feature space and should therefore be included in spectral data collections.

The time and effort that is spent in collecting spectral data, combined with the characteristically large number of files, makes it clear that spectral data should be well organised. Otherwise valuable data can be lost or loses its value because of missing metadata. Considering the above, it seems logical to employ a database to store spectral data in a suitable form. A further characteristic of hyperspectral data is the data redundancy. It has been shown that neighbouring wavebands have a high degree of correlation (Thenkabail et al., 2004). This redundancy is created by oversampling, i.e. the spectral signal is sampled at small enough steps to describe very narrow features that could be discriminating (Shaw and Manolakis, 2002). The redundancy and general noisiness of the data usually mean that certain pre-processing must be carried out before any useful analysis can be performed.

SpectraProc bundled with the associated spectral database is a solution for the efficient storage and pre-processing of field spectroradiometer data. The system has been successfully used in studies concerned with New Zealand native vegetation, soil properties and pastures.

A. Hueni \UserGuide_V02.doc Version 0.2 / 16.06.2006

SpectraProc

User Guide

2 Installation and Configuration

Page 6 of 43

2.1 Database

2.1.1 Installing MySQL

□ Download the installation wizard ZIP file for Windows from www.mysql.com

or use the

ZIP file mysql-4.1.14-win32.zip supplied on the installation CD

□ Install the above ZIP file by running it

□ Use standard configuration

□ In the configuration section select: o

Install as windows service o

Root password: ‘root’ (or any other password, but make sure you can remember it)

2.1.2 Installing MySQL-Administrator and Query Browser

□ Download the windows install MSI files from www.mysql.com

or use the ZIP files mysql-administrator-1.1.2-win.msi and mysql-query-browser-1.1.14-win.msi supplied on the installation CD

□ Install both applications by running the above MSI files

2.1.3 Setting up the Spectral Database Schema, Tables and Users

□ Start the Query Browser, enter ‘localhost’ as Server Host, ‘root’ for User Name, ‘root’ for Password (or your own password if you have chosen so), ‘spectral_db’ for Default

Schema. Click ‘Yes’ when asked if you want to create this new schema.

□ Run MySQL Administrator

□ Go to the User Administration tab and create a new user. User name and password are both ‘SpectraProc’. Edit the Schema privileges by adding SELECT, INSERT, UPDATE

AND DELETE to the assigned privileges. Assign Resources: max_questions: 0, max_updates: 0, max_connections: 0 (This allows unlimited access. If you want to limit the access select positive values)

□ For the new user, add ‘localhost’ to the list of hosts from which the user SpectraProc can connect (click on the icon under User Accounts to do this).

□ Go to Restore and load the supplied Spectral_DB_XXXXXXX.SQL file. Select ‘spectral_db’ as the target schema. This will create all tables and populate them

2.1.4 Network Access

The SpectraProc application can access the database over the network if the following configurations are performed:

□ The IP addresses of all machines from which the user ‘SpectraProc’ can connect are added to the ‘SpectraProc’ privileges. Alternatively a range of IP addresses can be configured using a netmask (see MySQL online documentation for details) or a value of ‘%’ will allow access from any possible machine.

□ The port of the MySQL server (3306 by default) is not blocked by Microsoft Windows

2.2 SpectraProc Application

Copy the SpectraProc directory supplied on the installation CD to C:\Program Files\. Open the directory after copying it and open spectra_config.txt. Edit the configuration file as described below.

A. Hueni \UserGuide_V02.doc Version 0.2 / 16.06.2006

SpectraProc

User Guide

Page 7 of 43

□ Server: if the database is running on a machine different to the machine this copy of

SpectraProc is installed on change ‘localhost’ to the IP address of the database server.

If the database is access over the network also see 2.1.4.

□ Database: if the database has been given a different name during the restore operation change the name of the database.

□ Default filter settings: these are the waveband filter ranges that are used as default values in the waveband filter setup (see 4.6.1). More filter ranges can be added to the configuration file by adding a new line starting with ‘filter_br’ for each range.

□ Output drive and directory: specify the output drive and the output directory. All file output is written to this directory. This must be a valid (i.e. existing) pathname.

# Configuration settings for spectraProc

# database settings server localhost user SpectraProc password SpectraProc database spectral_db

# default filter settings filter_br 1350-1440 filter_br 1790-1980 filter_br 2360-2500

# specification of file output drive and directory output_drive C: output_dir \Data\MPhil Remote Sensing\SpectraProc_output

Hint: access to SpectraProc is made easier by placing a shortcut to the executable on the desktop.

A. Hueni \UserGuide_V02.doc Version 0.2 / 16.06.2006

SpectraProc

User Guide

3 Design of Sampling Experiments

Page 8 of 43

3.1 Overview

The data collected during sampling campaigns must be organised in a structured way in order to allow the automated import into the spectral database. Section 3.2 explains the background of the structure used and section 3.3 gives a practical example of a directory and file structure that should be adopted for data collection.

Preferably sampling experiments should be designed to reflect the structure used by SpectraProc.

Alternatively, existing data can be arranged to meet the requirements.

3.2 Hierarchical Structure

A hierarchical data structure that reflects the real world and the setup of sampling campaigns for vegetation is used. This structure is derived from the following conditions:

1. Reflectances of several different species are captured

2. In order to describe the in-species variation, several specimens of a species are sampled

3. The variability of the specimens is described by several measurements per specimen

The spatial extent where a specimen is sampled is termed a sample site, thus a species contains a number of sample sites. The sites are numbered in the order of sampling. At each site, several readings are taken to capture the variation exhibited by the specimen in question. A site therefore contains a number of spectra. This leads to a hierarchical directory structure (Figure 1).

Figure 1: Hierarchical directory structure

Although the term species is used it essentially represents the different classes found in a study.

These classes can either be assigned due to already existing classification systems for e.g. plants or minerals. In other cases a hypothesis might exist that a group of objects can be separated into classes. If so the setup of the experiment should mirror this hypothesis. If no such assumption exists all objects can be put into the same class (i.e. species) and the identification of classes could then be carried out by e.g. cluster analysis.

3.3 Directory Structure and Spectral Files

The hierarchical structure is used for the directory structure that holds the spectral files. Ideally, this directory structure is setup when designing the experiment.

Figure 2 shows an example of a directory structure. The main directory ‘Vegetation_example’ holds all species directories of the study. This main directory is the folder that needs to be specified when creating a study in the database (see 4.2). The names given to the species directories are stored in the database and are included in file output if ‘Folder name’ is specified as desired name type (see

4.7 for more information on file export).

The species directories contain the site directories. These can be named freely and the sites will be automatically numbered when read into the database. Thus, naming a site ‘1’ or ‘s1’ is perfectly acceptable.

A. Hueni \UserGuide_V02.doc Version 0.2 / 16.06.2006

SpectraProc

User Guide

Page 9 of 43

The site directories contain all spectral files collected at this site. They are auto-numbered by the

ASD capturing software.

Figure 2: Example of a directory structures holding spectral files

A. Hueni \UserGuide_V02.doc Version 0.2 / 16.06.2006

SpectraProc

User Guide

4 Operation

Page 10 of 43

4.1 Overview

4.1.1 Dataflow

A typical dataflow is illustrated in Figure 3. An ASD FieldspecPro spectroradiometer is used to capture the radiance of field objects. A GPS connected to the field laptop records the spatial position of the field object. Reflectance and metadata are automatically saved on the field laptop in a binary file for every reading taken.

These binary files are transferred to a laboratory computer where they are read by SpectraProc and stored in the relevant tables in the spectral database.

GPS

Spatial data

Reflectance Binary file

Reflectance

& metadata

Radiance

Field object

ASD

Spectroradiometer

Field laptop Lab computer

Spectral database

Figure 3: Dataflow and involved hardware

4.1.2 File System Interfaces

SpectraProc provides input and output interfaces to the file system (see Figure 4). Input file formats are: ASD binary file as produced by the ASD FieldSpecPro Spectroradiometer, ENVI Z-Profiles that are signatures extracted from hyperspectral imagery in ENVI and sensor specifications in a proprietary, tabulator separated format. ASD files can be imported into the database as part of a study or loaded into memory for classification against a study dataset. ENVI Z-Profiles can be loaded for classification only. Sensor specification files are a way of defining new sensors in the database.

Output can be written in three data formats: (1) CSV (Comma Separated Values) for import into various 3 rd

party applications like spreadsheets or statistic packages, (2) ENVI Spectral Library for import into ENVI and subsequent use for e.g. signature matching and (3) ARFF which is a special format used by WEKA. WEKA is a collection of machine learning algorithms for data mining tasks

(University of Waikato, 2005).

ASD

Binary

File

SpectraProc

CSV

File

ENVI

Z-

Profile

File

ENVI

Spectral

Library

File

Sensor

Specifications

ARFF

File

Spectral DB

Figure 4: File System Interfaces

A. Hueni \UserGuide_V02.doc Version 0.2 / 16.06.2006

SpectraProc

User Guide

Page 11 of 43

4.1.3 SpectraProc Operation

SpectraProc is study based, i.e. spectral data is grouped into studies. Advisably, a new study is created to hold data for every new sampling experiment. The spectral data files of studies must be structured according to the hierarchical structure introduced in 3. This structuring allows the automated loading of spectra into the database.

Typically, the operations carried out during the lifetime of a study are:

1. Creation of a new study

2. Loading of spectra

3. Repeated pre-processing and analysis

Optionally, in ongoing studies, new data can be added to the database by simply selecting the loading operation again. Only new spectra will be added to the database in this case.

4.1.4 Spectral Processing Concept

The spectral database only stores the raw spectral data. Further processing of the spectra is performed at runtime and the results are held in memory. Once a spectrum is loaded from the database it is put through a cascade of operations as shown in Figure 5.

The result of every stage is saved in a separate data structure in memory. This allows easy file export of spectral data at any processing step.

Consequently the structure of this processing chain implicates that a change of processing parameters of any stage will influence the results of the subsequent stages. A complete set of pre-processing parameters thus describes the processing from raw to feature space transformed data.

All processing settings are stored in the database. Repeated processing of the data with the same settings will always result in the same final data set.

Load spectrum

Waveband filtering

Smoothing

Synthesizing

Spectral DB

Raw data

Waveband filtered data

Smoothed data

Derivative calculation

Synthesized data

Derived data

Feature

Space

Transformation

Data in Feature Space

Figure 5: Spectral data processing cascade

A. Hueni \UserGuide_V02.doc Version 0.2 / 16.06.2006

SpectraProc

User Guide

Page 12 of 43

4.1.5 User Interface

The graphical user interface (GUI) is based on the structure of the processing cascade (see Figure

6). The left side of the main window consists of controls for the selection of the study and the main settings for smoothing filter, synthesizing, derivative calculation, feature space transformation and classifier discriminant function. Processing details are entered in pop up windows which are explained in the relevant sections. The text output panel in the middle of the main window is used to display processing and error information.

The listbox on top of the text output panel is used to display spectra files that are loaded directly into memory. The ‘Indiv. Classify’ button under it classifies the selected, individually loaded spectra against the current library.

The library status box on the top right of the screen indicates whether statistical information has been compiled for the current pre-processing settings.

Study selection

List of directly loaded spectra

Classification button for directly loaded spectra

Library status field

Smoothing filter

Smoothing filter detail settings button

Sensor

Derivative order

Derivative calculation method

Derivative calculation detail settings button

Feature space type

Feature space

Classifier discriminant function

Figure 6: Main window elements

Processing message window

A. Hueni \UserGuide_V02.doc Version 0.2 / 16.06.2006

SpectraProc

User Guide

Page 13 of 43

4.2 Creating a new Study

To create a new study, select ‘Database‘->‘Create New Study’ (Figure 7). This brings up the new study dialog (Figure 8).

Figure 7: Create New Study menu entry

Enter a study name of 45 character maximum and not containing special characters that can not be part of any Windows file name like: \ / : * ? < > “ |.

The study description is optional and can contain any string up to 200 characters long.

The species directory path is a Windows pathname pointing to the directory that contains all species folders of this study. To set the path either type it in manually or select the ‘Browse’ button that brings up a directory tree (Figure 9) and select the appropriate directory.

The number of samples per species for library inclusion can be set to any positive number. If a species has less than the specified number of samples it will not be included in the statistics calculations when building libraries.

Once a new study is created it will automatically be selected as the current study.

Figure 8: New study dialog

Figure 9: Directory tree dialog

A. Hueni \UserGuide_V02.doc Version 0.2 / 16.06.2006

SpectraProc

User Guide

Page 14 of 43

4.3 Deleting an existing Study

To remove a study and all associated data from the database select this study as the current study, then select ‘Database’->’Delete Current Study’ (Figure 10). Before any data is deleted a warning will be displayed (Figure 11). To delete all data of the current study click OK, to abort the operation click

‘Cancel’.

The message window will list the involved operations of the deletion.

Figure 10: Delete Current Study menu entry

Figure 11: Deletion warning box

4.4 Selecting Studies

To select a study to work with, select the appropriate study from the drop down list in main window

(Figure 12).

Figure 12: Selection of current study

4.5 Loading Spectra

To load the spectra of a study into the database make sure the correct study is selected. Please refer to 4.3 on the process of selecting the current study. Then select ‘Database‘->‘Import Spectra into current Study’ (Figure 13). A popup window will appear indicating the loading progress. Additionally, the loaded species are listed in the message window.

Figure 13: Import Spectra menu entry

A. Hueni \UserGuide_V02.doc Version 0.2 / 16.06.2006

SpectraProc

User Guide

Page 15 of 43

4.6 Processing Chain Settings

4.6.1 Waveband Filtering

Waveband filtering is the process of removing unwanted regions of the spectrum. All subsequent processing operations will then ignore these regions.

A typical example is the removal of water bands. These water band regions are usually found around 1350-1440 nm, 1790-1980 nm and 2360-2500 nm (Figure 14). If contact probes are used the noise can be minimal because the atmosphere is effectively non existent.

Raw Reflectance Curve of Pittosporum eugenioides Filtered Reflectance Curve of Pittosporum eugenioides

1.00E+00

9.00E-01

8.00E-01

7.00E-01

6.00E-01

5.00E-01

4.00E-01

3.00E-01

2.00E-01

1.00E-01

0.00E+00

300 500 700 900 1100 1300 1500 1700 1900 2100 2300 2500

Wavelength [nm]

1.00E+00

9.00E-01

8.00E-01

7.00E-01

6.00E-01

5.00E-01

4.00E-01

3.00E-01

2.00E-01

1.00E-01

0.00E+00

300 500 700 900 1100 1300 1500 1700 1900 2100 2300 2500

Wavelength [nm]

Figure 14: An example of pre and post filtering of water noise bands

The waveband filter regions are freely configurable for each study. To modify the current filter settings select ‘Chain Settings‘->‘Waveband Filter Setup’ (Figure 15).

Figure 15: Waveband Filter Setup menu entry

Figure 16: Waveband Filter Setup dialog (no regions defined)

The Waveband Filter Setup dialog lists the waveband filter ranges in the listbox on the left side

Figure 16). To add a new filter region click ‘New’. In the pop-up window enter the lower and upper wavelengths in nanometres of the region (Figure 17). After clicking on OK in this dialog, the newly created filter range is shown in the Waveband Filter Setup dialog (Figure 18).

A. Hueni \UserGuide_V02.doc Version 0.2 / 16.06.2006

SpectraProc

User Guide

Figure 17: Filter band definition dialog

Page 16 of 43

Figure 18: A new waveband filter range added to the list of filter ranges

To modify an existing range, select the range to be modified in the listbox and click ‘Modify’. To delete an existing range, select the range and click ‘Delete’.

To add the default ranges as defined in the configuration file click on ‘Add default ranges’ (see 2.2 on how to configure the default filter ranges).

4.6.2 Smoothing

Smoothing applies a smoothing function to the spectra. The default smoothing filter type is ‘None’ meaning no smoothing is carried out. The smoothing type is selected using the drop down list box in the main window (Figure 19). Two different Savitzky-Golay implementations are offered in this list: Moving Window (MW) and Fast-Fourier-Transformation (FFT). The output is exactly the same, however, due to internal overhead the MW method is currently the faster of the two.

Figure 19: Smoothing filter type drop down list box

The processing details of the smoothing filter are set by clicking on the ‘Edit’ button below the list box. This brings up a dialog where the polynomial order and the filter window size can be changed

(Figure 20). The filter size is automatically incremented resp. decremented in steps of two to keep the filter size and uneven number, i.e. the number of points on the left and right of the point to be smoothed are equal. The polynomial order sets the order of the curve that is used in the least squares approximation. Note that the Savitzky-Golay coefficients for quadratic and cubic functions are the same and the same applies to orders 4 and 5 (Savitzky and Golay, 1964).

Figure 20: Smoothing Filter Details dialog

A. Hueni \UserGuide_V02.doc Version 0.2 / 16.06.2006

SpectraProc

User Guide

Page 17 of 43

0.2

0.1

0

0

0.6

0.5

0.4

0.3

1

0.9

0.8

0.7

The result is automatically filtered to remove artefacts that appear at the start and end of every valid waveband segment (Figure 21). The new valid segment sizes are calculated by:

λ

u smoothed

= λ

u

pos

_

filter

_

size

λ

l smoothed

= λ

l

neg

_

filter

_

size

where

λ

l

,

λ

u

= lower and upper segment wavelengths

Thus every segment looses information of filter_size – 1.

Smoothed Reflectance Curve of Pittosporum eugenioides showing smoothing artefacts

Smoothed Reflectance Curve of Pittosporum eugenioides after the removal of smoothing artefacts

500 1000 1500

Wavelength [nm]

2000 2500 3000

0.6

0.5

0.4

0.3

0.2

0.1

0

0

1

0.9

0.8

0.7

500 1000 1500

Wavelength [nm]

2000 2500

Figure 21: A smoothed signature of Pittosporum eugenoides before and after the removal of smoothing artefacts

Generally, bigger filter sizes remove more noise while higher polynomial orders fit the original data values better.

4.6.3 Sensor Synthesizing / Downsampling

The sensor synthesizing/downsampling is effectively a data reduction operation. The response of the sensor selected as the current sensor is calculated from ASD data.

The synthesizing of other sensor responses using ASD data is useful due to several reasons:

1. Reduction of dimensionality

2. Direct comparison of airborne/spaceborne sensor and ground data

3. Implicit smoothing of the data

4. Prediction and assessment of the usefulness of a certain sensor

Two classes of sensors are supported: Ratio and Gaussian.

4.6.3.1 4.5.3.1 Ratio Sensors

The sensor element function of these sensors is modelled by a number of known coefficients, thus the synthesizing operation is simply a convolution of a defined wavelength region using these coefficients.

An example of such ratios is shown for Landsat7 TM band 1 (Figure 22).

3000

A. Hueni \UserGuide_V02.doc Version 0.2 / 16.06.2006

SpectraProc

User Guide

Page 18 of 43

Ratios for Landsat Band 1

1.2

1

0.8

0.6

0.4

0.2

0

Wavelength (nm)

Figure 22: Ratios for Landsat7 TM band 1

The convolution is calculated by:

r j

=

uw

_

j c i i

=

lw

_

j

r i uw

_

j c i i

=

lw

_

j

where

r

= the synthesized reflectance value of the j-th synthesized band

j c i

= the coefficient for wavelength i

r i

= reflectance value of i-th ASD band

lw

_

j

= lower wavelength of the j-th band

uw

_

j

= upper wavelength of the j-th band

4.6.3.2 Gaussian Sensors

The sensor element response function of these sensors is best approximated by a Gaussian function. The sensor elements are technically defined by the middle wavelength and the full width at half the maximum (FWHM) (see Figure 23).

Gaussian function for mu=0 and sigma=2

0.2

0.18

0.16

0.14

0.12

FWHM

0.1

0.08

0.06

0.04

0.02

0

-10 -8 -6 -4 -2 0 2 4 6 8 10

Figure 23: Gaussian curve illustrating the FWHM measure

A. Hueni \UserGuide_V02.doc Version 0.2 / 16.06.2006

SpectraProc

User Guide

Page 19 of 43

The coefficients used for the convolution operation are given by the Gaussian function:

c i

=

f

(

wavelength

_

band

_

i

)

=

1

e

1

2

wavelength

_

band

_

i

wavelenth

_

band

_

σ

j

2

2

π

σ where

wavelength

_

band

_

i

= wavelength of the i-th ASD band

wavelength

_

band

_

j

= wavelength of the j-th band of the sensor to be synthesized

c

= the i-th coefficient for the convolution operation

i

The band convolution is calculated by

r j

=

µ

range

+

c i i

= µ −

range

r i

µ +

range

c i i

= µ −

range

where

r

= the synthesized reflectance value of the j-th synthesized band

j c i

= the coefficient determined by the Gaussian function for the wavelength of the i-th ASD band

r i

= reflectance value of i-th ASD band

range

= defines the range of values to be used for the band convolution symmetrically to the middle

ASD waveband. The middle ASD waveband is the one closest to the average wavelength of the j-th synthesized band.

The nature of the Gaussian function effectively carries out a smoothing of the data. The range of values used for the convolution is set to 3 times the standard deviation:

range

= 3 ⋅ σ

, i.e.

99.74% of all contributing values are used (Papula, 1994). This range is practically not usable for all bands because some wavelengths may have been filtered previously, segmenting the spectrum. In these situations, the range is symmetrically reduced to avoid filtered regions.

4.6.3.3 Supplied Sensors

The sensors currently supplied with the software are:

Name

Hyperion

Type

Gaussian

Description

Sensor as flown on EO-1

ASD

Opt_Hyperion

Downsampling_by_10

Landsat7

ACS_210 Red_NIR

Ratio

Gaussian The Hyperion sensor according to the blueprint, i.e. all sen-

Ratio

Effects a 1:1 copy of ASD data, i.e. no data reduction sor elements are operating

Effects a downsampling by selecting every tenth waveband

Ratio Sensor as flown on Landsat7 ETM satellite

Gaussian Crop circle sensor simulation with red – NIR illumination

ACS_210 Yellow_NIR

Downsampling_by_5

Gaussian

Ratio

Crop circle sensor simulation with yellow – NIR illumination

Effects a downsampling by selecting every fifth waveband

To change the current sensor select a sensor from the sensor drop down list box (Figure 24).

Figure 24: Selection of current sensor

A. Hueni \UserGuide_V02.doc Version 0.2 / 16.06.2006

SpectraProc

User Guide

Page 20 of 43

It may be advisable to apply the downsampling only to smoothed data in order to avoid aliasing. The combination of smoothing and downsampling is called decimation (Fliege, 1994).

4.6.4 Derivative Calculation

The derivative order is set to zero by default, i.e. no derivative is calculated. To change the order use the up and down arrows next to the box showing the derivative order (Figure 25). If the order is changed to a non-zero value, the calculation method can be selected.

Iterative calculation is based on the gradient of the linear curve segment between two wavebands:

ρ

' (

b i b i

+ 1

)

=

ρ

(

b i

+ 1

λ

(

b i

+ 1

)

) −

ρ

(

b i

)

λ

(

b i

) where

ρ

(

b i

)

= reflectance of band i

λ

(

b i

)

= wavelength of band i

ρ

' (

b i b i

+

1

)

= first derivative of linear curve segment between reflectances of band i and band i+1

The n-th derivative is thus calculated by applying the above formula n times.

The SavGol method uses Savitzky-Golay coefficients to calculate the derivatives. In this case the derivative is actually the derivative of the approximating function at the wavelength of interest. Thus the SavGol method automatically performs a smoothing of the data. This can be useful when dealing with noisy data where a single smoothing operation is not removing enough noise.

Both methods loose N data points per valid segment where N is the derivative order.

Figure 25: Derivative calculation settings: zero derivative (left), first derivative iterative (middle) and first derivative SavGol method (right)

For the SavGol method the filter size and polynomial order can be modified by clicking on the ‘Edit’ button. This brings up the Derivative Filter Details dialog (Figure 26). The filter size and order are automatically restricted to meet the following minimal conditions for the calculation of the derivative:

polynomial

_

order

derivative

_

order filter

_

size

≥ max(

polynomial

_

order

+

1 ,

derivative

_

order

+

1 )

Thus the minimal filter size depends on both the polynomial and the derivative order. A minimal filter size of (derivative_order+1) will result in the removal of n = derivative_order number of points.

Figure 26: Savitzky-Golay settings for derivative calculation

A. Hueni \UserGuide_V02.doc Version 0.2 / 16.06.2006

SpectraProc

User Guide

Page 21 of 43

Note that too big filter sizes remove relatively lot of data because the derivative calculation takes place after the data reduction step (sensor synthesizing/downsampling). E.g. a filter size of 5 applied to data downsampled by factor ten will remove 4 data points, i.e. 40nm, per segment.

4.6.5 Feature Space Transformation

Three types of feature spaces are implemented:

□ Derivative Indices (DI)

□ Normalized Two Band Indices (NTBI)

□ Principal Component Transformation (PCT)

4.6.5.1 DI (DGVI)

DGVIs (Derivative Greenness Vegetation Indices) are an example of a DI. These indices are trying to describe the shape of the reflectance curve. The DGVI calculation is based on the equations used by Thenkabail et al. (Thenkabail et al., 2004) and Elvidge and Chen (Elvidge and Chen, 1995):

DGVI

=

n

i

=

m

ρ

'

(

b i

− 1

b i

)

ρ

'

(

b i b i

+ 1

)

b i

where

ρ

’(b i-1 b i

) = first derivative of reflectance curve between b i-1

and b i m..n = start and end band number of DGVI area b i i

= centre wavelength of band i

= band number

∆ b i

= step width: b i+1

– b i-1

4.6.5.2 NTBI

Normalized two band indices are calculated by:

NTBI

=

ρ

(

b x

)

ρ

(

b y

)

ρ

(

b x

)

+

ρ

(

b y

)

4.6.5.3 PCT

Principal components transformation requires as input the eigenvectors of a given dataset. A principal component analysis (PCA) must be carried out before PCT can be applied to the data. PCA results in eigenvalues and eigenvectors. The eigenvectors are used in the transformation matrix G.

If only the first n components are to be used, the transformation matrix is a sub-matrix of G, consisting of the first n column vectors. The dimension of the resulting feature space is therefore n:

y n

× 1

=

n

G

×

m

x m

×

1 where m = original size of data space n = new size of data space, equal to the number of selected components

4.6.5.4 Creation, Modification and Deletion of Feature Space Definitions

Based on the three basic feature space types new feature spaces can be created. Existing feature spaces can be edited or deleted. For any of these functions navigate to the ‘Chain Settings’-

>‘Feature Space’ submenu (Figure 27).

A. Hueni \UserGuide_V02.doc Version 0.2 / 16.06.2006

SpectraProc

User Guide

Page 22 of 43

Figure 27: Feature Space Submenu

To create a new feature space select ‘Copy and Edit’ or ‘New’ in the ‘Feature Space’ submenu.

‘New’ opens up an empty dialog for the definition of a new feature space. ‘Copy and Edit’ copies the settings of the currently selected feature space into the dialog. The settings can subsequently be changed before creating the new feature space in the database.

To modify an existing feature space select ‘Edit’.

All the above three commands invoke the same dialog, shown here on the example of a NTBI feature space consisting of three individual NTBIs (Figure 28). The name of the feature space can be any alphanumeric string of maximal 45 characters, the description can be any alphanumeric string of maximal 250 characters. The bands listbox lists the band combinations used to calculate the indices. E.g. index 1 uses bands 550nm and 468nm. New band combinations are entered by clicking on ‘New’, existing band combinations are modified by clicking modify. Both commands bring up a dialog box to define the two bands (Figure 29). To delete an existing band combination select the combination in the list and click ‘Delete’.

The dimension field in the Feature Space dialog is non editable for DIs and NTBIs where it reflects the number of band combinations. For PCT feature spaces, the dimension field specifies how many components are to be used for the transformation. In the given example, the first 10 components will be used (Figure 30).

Figure 28: Feature Space edit dialog on the example of a NTBI feature space

Figure 29: Waveband Definition Dialog

A. Hueni \UserGuide_V02.doc Version 0.2 / 16.06.2006

SpectraProc

User Guide

Page 23 of 43

Figure 30: PCT Feature Space dialog

To delete an existing feature space make sure the currently selected feature space is the one to be deleted. Then select ‘Delete’ from the ‘Feature Space’ submenu. The current feature space is removed from the database.

4.6.5.5 Feature Space Selection

The feature space selection is comprised of two panels: (a) a selection of the feature space type offering DI, NTBI and PCT and (b) a drop down list box that contains all feature spaces of the current type (see Figure 31).

Figure 31: Feature space settings

4.7 File Export

SpectraProc features a powerful file export that facilitates the analysis and visualisation of the data in third party products. All output files are written to the directory

The output is controlled by a single dialog (see Figure 32). The file export dialog is displayed by selecting ‘File’->’Export’.

Data can be exported at any processing stage:

□ Raw

□ Filtered (removal of noisy band regions)

□ Smoothed

□ Synthesized

□ Derived

□ Feature Space transformed

A. Hueni \UserGuide_V02.doc Version 0.2 / 16.06.2006

SpectraProc

User Guide

Page 24 of 43

The processing for these stages depends upon the parameters that are defined at that point in time for the processing cascade. If e.g. ‘Synthesized’ is selected, every spectrum is put through the cascade of filtering, smoothing and synthesizing and is then written to the file. I.e. if a certain stage is selected all the preceding stages are automatically performed.

The data can be averaged either on site level or on species level. If the site mean is chosen a column containing the site number will be added to the output file. Selecting the ‘All spectra’ option performs no averaging but writes every spectrum to the file. The averaging is the last step of the processing, e.g. if the ‘Synthesized’ and ‘Mean per species’ is selected, then all spectra of each species are processed till the synthesizing stage and then averaged and written to the file.

Three file formats are currently supported:

□ CSV (Comma Separated Values)

□ ENVI spectral library file

□ ARFF (WEKA file format)

The species that are written to the file can be controlled by selecting on of four options:

□ Include all species: all species are written

□ Include only library relevant species: only species that have equal or more than the minimum number of spectra are written. The minimum number of spectra is set when defining a new study (see 4.2)

□ Only endmembers: only species that have been designated as endmembers are written

□ Only mixtures: only species that are not designated endmembers are written

The name that is used as species name can be one out of five choices:

□ Common: common name is defined in database

□ Maori: Maori name as defined in database

□ Latin: Latin name as defined in database

□ Folder: name of the folder that contains the site directories of the species

□ Auto number: incrementing numbers are assigned to the species according to their order in the database

The current version of the software supports only one file level setting: study. I.e. all species of the current study are written to one single file.

Four checkboxes allow for the following operations:

□ Transpose: the observations are written as columns instead of rows. This option can make the import of data into other programs easier.

□ Include filtered bands: this is for visualisation purposes only. The regions that were removed in the waveband filtering stage are written as empty cells, i.e. when plotted the filtered regions will appear as gaps.

□ Offset: each species will be set off by a freely choose able percentage of 100% reflectance. This is to be used for visualisation purposes only.

□ Interpolate: this option is only available when the ‘Synthesized’ stage is selected as output step. The synthesized data is interpolated to ASD data using straight line segments. This is useful for the assessment of the smoothing effect of the synthesizing operation e.g. by plotting a noise spectrum (smoothed minus interpolated synthesized data).

□ Sort species numerically: Sorts the species by interpreting the species names as numbers

A. Hueni \UserGuide_V02.doc Version 0.2 / 16.06.2006

SpectraProc

User Guide

Page 25 of 43

Species restriction selection

Name type selection

File level selection

Processing stage for file export

Data averaging options

File format selection

Sort the species names numerically

Transpose observations

Include the filtered band regions

Offset each species by a given percentage

Interpolate output by linear segments

Figure 32: File Export dialog

4.8 Spectral Mixtures

Spectral mixtures occur when more than one endmember is present in the field of view. If the endmembers are known the mixtures can be unmixed to give estimates of the abundance of each endmember in the mixture. For the process of unmixing refer to 4.10.4.

The unmixing process needs to have a set of endmembers and a set of mixtures to be unmixed using the endmember information. For this purpose species can be designated as endmembers.

If the spectral mixtures are the result of a controlled mixing experiment the abundances of the endmembers are usually know. These known endmember fractions can be entered into the database for each mixture, thus allowing the automatic calculation of the error involved in the estimation by the unmixing procedure.

4.8.1 Endmember Designation

To display the endmember designation dialog choose ‘Spectral Mixing’->‘Endmember Selection’

(Figure 33).

Figure 33: Endmember Selection menu entry

Initially, all species will be listed in the ‘Classes’ list while the ‘Endmembers’ list is empty. To designate a species as an endmember select the class by clicking on it, then click on the arrow pointing

A. Hueni \UserGuide_V02.doc Version 0.2 / 16.06.2006

SpectraProc

User Guide

Page 26 of 43 towards the right to shift this class into the Endmember list. Similarly, to move an endmember back to the Classes list, select the endmember and click the arrow pointing towards the left.

Figure 34: Endmember Selection dialog

4.8.2 Abundance Settings

Before invoking the abundance settings the endmembers should be selected as described in 4.8.1.

To display the abundance settings dialog select Spectral Mixing’->‘Abundance Settings’.

All mixtures are listed in the ‘Classes (excl endmembers)’ list box. The abundances of the endmembers are set for each of these mixtures by:

1. Selecting the mixture

2. Select the endmembers that are contained in the mixture one after another from the

‘Available endmembers’ list box and shift them down into the ‘Selected endmembers’ list box.

3. For every selected endmember do: select the endmember by clicking on it, then enter the abundance in the provided box. Click ‘Save’. The total abundance is updated.

When defining the abundance of the last endmember, click on ‘Complement’. This automatically assigns the abundance to the last endmember by summing the total abundance to 1.

Figure 35: Abundance Settings dialog

A. Hueni \UserGuide_V02.doc Version 0.2 / 16.06.2006

SpectraProc

User Guide

Page 27 of 43

4.9 Library Building

A library is a collection of statistics data built for a certain configuration of the processing chain. The

‘Library status’ field in the main window indicates if a library has been built for the current processing settings. To build a library select ‘Library’->’Build’.

4.10 Analysis

4.10.1 Separability

Before running the separability analysis the library for the current pre-processing settings must be built. See 4.9 on how to build libraries. The JM and B distance analyses are carried out in the currently selected feature space. I.e. the discriminating power of feature spaces can be assessed by the separability analysis.

To run the separability analysis select ‘Statistics’->’Separability Report’. In the displayed JM/B Distance dialog select the name and distance type and press OK (Figure 36).

A short message will be displayed in the message window, looking similar to:

Separability Analysis

Using JM and B distance

JM and B output written to file:

C:\Data\MPhil Remote Sensing\SpectraProc_output\JM and B output.csv

The output filename is shown in this message. Note that only library relevant species are included in this report.

Figure 36: JM/B Distance dialog

The JM distance is listed in the upper triangular matrix, the B distance in the lower triangular matrix

(see Table 1). The JM distance is asymptotic to 2.0. A JM distance of 2.0 indicates a complete separability and a distance of 1.9 a good separability (Richards, 1993). The B distance is not asymptotic and can get infinite (Inf.) due to limits in numerical precision.

Table 1: Example of a JM and B output using a 10 dimensional DI feature space

Blackfern

Cabbage_tree

Lemonwood

Blackfern Cabbage_tree Lemonwood

- 1.93 1.90

3.41

2.99

-

2.11 -

1.76

A. Hueni \UserGuide_V02.doc Version 0.2 / 16.06.2006

SpectraProc

User Guide

Page 28 of 43

4.10.2 Discriminant Analysis

Discriminant analysis (DA) uses a discriminant function to assign a class to samples. In the case of the calibration set being subjected to DA the result is an assessment of the separability of the classes. The DA is carried out in the currently selected feature space. The DA result is thus also an assessment of the discriminating power of the feature space.

Note that the discriminant function used is the one currently selected in the main window.

Before running DA the library for the current pre-processing settings must exist. See 4.9 on how to build libraries.

To run the DA select ‘Statistics’->’Discriminant Analysis’. A dialog will be displayed to select the names to be used in the output (Figure 37).

Figure 37: Discriminant Analysis dialog

A message stating the file name of the error matrix and the overall accuracy is reported in the message window, similar to the example shown below.

Discriminant Analysis

---------------------------------------------------

Error matrix written to file:

C:\Data\MPhil Remote Sensing\SpectraProc_output\Error Matrix.csv

---------------------------------------------------

Overall Accuracy: 78.46154

---------------------------------------------------

The error matrix written to a file contains the errors of omission and commission, the total accuracy, the producer and user accuracy and the minimum, maximum and mean for both producer and user accuracy.

Table 2: Error matrix of DA using a 3 dimensional NTBI feature space

Blackfern

Cabbage_tree

Lemonwood

Column Total

Producer Accuracy

Prod. Acc. Min

Prod. Acc. Max

Prod. Acc. Mean

User Acc. Min

User Acc. Max

User Acc. Mean

Blackfern

11.00

7.00

0.00

18.00

61.11

61.11

95.00

77.96

59.38

100.00

84.94

Cabbage_tree

0.00

19.00

1.00

20.00

95.00

Lemonwood

0.00

6.00

21.00

27.00

77.78

Row Total

11.00

32.00

22.00

65.00

User

Accuracy

100.00

59.38

95.45

78.46

A. Hueni \UserGuide_V02.doc Version 0.2 / 16.06.2006

SpectraProc

User Guide

Page 29 of 43

To run the DA with an independent data set choose ‘Statistics’->’Discr. Analysis using indep. set’. A dialog is displayed to select the study that contains the independent set (Figure 38).

Note that in order to work correctly the folder names of corresponding species in the training and in the independent study must be identical.

Figure 38: Selection dialog for independent data sets

4.10.3 Principal Component Analysis

Principal components analysis (PCA) is applied after the derivative calculation step. I.e. the PCA is carried out for the current settings of waveband filtering, sensor synthesizing and derivative calculation. To run PCA select ‘Statistics’->’PCA’. A message stating the output file similar to the one below is displayed in the message window:

Principal Component Analysis

---------------------------------------------------

PCA output written to file:

C:\Data\MPhil Remote Sensing\SpectraProc_output\pca_output.csv

---------------------------------------------------

The two first components will be plotted as a scatterplot in a graphics window, this plot is also known as scoreplot.

The pca_output file contains the eigenvalues, proportions and cumulative proportions (see Table 3 for an example). The eigenvalues and eigenvectors are stored in the database.

Table 3: First ten eigenvalues and proportions of a PCA

PC#

5

6

7

8

1

2

3

4

9

10

Eigenvalue Proportion Cumulative

2.211 0.944 0.944

0.097

0.019

0.042

0.008

0.986

0.994

0.007

0.005

0.001

0.000

0.003

0.002

0.001

0.000

0.997

0.999

0.999

0.999

0.000

0.000

0.000

0.000

0.000

0.000

1.000

1.000

1.000

A. Hueni \UserGuide_V02.doc Version 0.2 / 16.06.2006

SpectraProc

User Guide

Page 30 of 43

4.10.4 Unmixing

The unmixing procedure implemented is experimental at this stage and works on Landsat7 data only. Before starting the unmixing, make sure that the current sensor is Landsat7, the study has the mixtures and endmembers defined and the abundances are entered for each mixture.

The model used is unconstrained (Keshava and Mustard, 2002):

a

ˆ

U

=

(

S

T

S

)

− 1

S

T x

where:

x

= spectrum vector to be unmixed (L x 1)

S

= endmember matrix (L x M) consisting of M endmembers with the columns being the endmember spectra vectors

aˆ

= the unconstrained least squares solution for the abundances of the endmembers in the spec-

U

trum vector

x

To start the unmixing select ‘Spectral Mixing’->’Unmix’. For each mixture the unmixing results are displayed in the message window; first stating the calculated abundances for each endmember, then the abundances stored in the database and finally the absolute errors for each endmember abundance (see example of such an output below).

Spectral Unmixing

---------------------------------------------------

120V_240P

---------------------------------------------------

0.69 0V_360P

0.35 360V_0P simulated mixture abundances from database:

0.67 0V_360P

0.33 360V_0P absolute errors:

0.03 0V_360P

0.02 360V_0P

4.11 Adding new Sensors

New sensors can be added to the database if their properties are known. The sensors details must be entered in a tabulator formatted file as described below. Two classes of sensors are supported:

Gaussian and Ratio (see also 4.6.3).

The header structure is identical for both sensor classes:

<Sensor name>

<Sensor description>

<Column headers>

The sensor name should not include any of the following characters: \ / : * ? < > “ |.

The column header line is not interpreted by the program and exists only for the convenience of the user preparing the sensor description file.

For Gaussian sensors the file body contains one line per sensor band:

<Band> TAB <Average Wavelength (nm)> TAB <FWHM (nm)> TAB <Calibrated (Y/X)>

For Ratio sensors the file body contains 1 to many lines per band to model the response (see also

Figure 22):

<Band> TAB <Wavelength(nm)> TAB <Ratio>

A. Hueni \UserGuide_V02.doc Version 0.2 / 16.06.2006

SpectraProc

User Guide

Examples of Gaussian and Ratio sensor description files are given in tables 4 and 5:

Page 31 of 43

A. Hueni \UserGuide_V02.doc Version 0.2 / 16.06.2006

SpectraProc

User Guide

Page 32 of 43

Table 4: Example of a Gaussian sensor (Hyperion)

Hyperion

Hyperion Sensor as flown on EO1

Hyperion Band Average Wavelength (nm) FWHM (nm)

1 355.59

Calibrated

11.3871 X

2

3

4

5

6

365.76

375.94

386.11

396.29

406.46

11.3871 X

11.3871 X

11.3871 X

11.3871 X

11.3871 X

7

8

9

10

11

12

13

14

416.64

426.82

436.99

447.17

457.34

467.52

477.69

487.87

11.3871 X

11.3871 Y

11.3871 Y

11.3871 Y

11.3871 Y

11.3871 Y

11.3871 Y

11.3784 Y

Table 5: Example of a Ratio sensor (Landsat7)

Landsat7

Landsat 7 Thematic Mapper Sensor

Band Wavelength(nm)

1 435

1 436

1 437

1 438

1 439

1 440

1 441

1 442

1 443

Ratio

0.016

0.027

0.048

0.094

0.167

0.287

0.459

0.605

0.728

1 444 0.769

To import a new sensor select ‘Database’->’Import Sensor’. In the Import Sensor dialog select the sensor response type and the input file by clicking on ‘Browse’. Then click OK and the sensor data will be imported. An according message will be displayed in the message window when the sensor is loaded.

Figure 39: Import Sensor dialog

A. Hueni \UserGuide_V02.doc Version 0.2 / 16.06.2006

SpectraProc

User Guide

Page 33 of 43

4.12 Linking to a GIS System

If a GPS unit is used to record the spatial position of the sampling sites during the sampling process every site in the database contains the latitude, longitude and altitude in WGS84 format.

The database tables can be accessed by a GIS system if ODBC connections are supported.

The connection is shown here on the example of ArcGIS.

4.12.1 Setting up the Spectral DB as a Data Source

Note that before the Spectral DB can be configured as a data source the MySQL ODBC driver

(available from the MySQL website) must be installed on the system.

Start ‘Data Sources (ODBC)’ from the Windows administrative tools. In the New Data Source dialog

(Figure 40) select the MySQL ODBC Driver and click on ‘Finish’.

Figure 40: New Data Source dialog

This brings up the ODBC configuration dialog (Figure 41). Type in a name for the Data Source

Name. Set the Server to the address of the database server. Enter the User name and Password

(SpectraProc). Select the database schema containing the spectral data from the Database dropdown list.

Figure 41: MySQL ODBC Configuration dialog

4.12.2 Establishing a Database Connection in ArcCatalog

Start ArcCatalog and select ‘Add OLE DB Connection’ (Figure 42).

A. Hueni \UserGuide_V02.doc Version 0.2 / 16.06.2006

SpectraProc

User Guide

Page 34 of 43

Figure 42: Adding an OLE DB Connection in ArcCatalog

In the Data Link Properties select ‘Microsoft OLE DB Provider for ODBC Drivers’ (Figure 43) and click ‘Next>>’.

Figure 43: Data Link Properties; Provider tab

In the Connection tab select the data source from the dropdown list and enter again the username and password (SpectraProc) (Figure 44). Click ‘OK’.

Figure 44: Data Link Properties; Connection tab

A. Hueni \UserGuide_V02.doc Version 0.2 / 16.06.2006

SpectraProc

User Guide

Page 35 of 43

The database is now listed in ArcCatalog and the tables are displayed when selecting the database

(Figure 45).

Figure 45: SpectralDB and tables in ArcCatalog

4.12.3 Adding Table Data to a Map

To add table data to a map select ‘Add Data’ in ArcMap and choose the SpectralDB under Database Connections (Figure 46). Then select the site table as the data source (Figure 47).

The sampling sites can then be displayed on the map by selecting ‘Display XY Data’ and using the longitude and latitude fields of the site table as X and Y coordinates.

Figure 46: Selecting the SpectralDB as data source

A. Hueni \UserGuide_V02.doc Version 0.2 / 16.06.2006

SpectraProc

User Guide

Figure 47: Selecting the site table as data source

Page 36 of 43

A. Hueni \UserGuide_V02.doc Version 0.2 / 16.06.2006

SpectraProc

User Guide

5 Tutorial

Page 37 of 43

5.1 Overview

The tutorial is based on an example dataset comprised of three plant species. The folder containing the species folders is called ‘Vegetation_example’.

5.2 Examine the Folder and File Structure

Open the folder ‘Vegetation_example’. It contains three species folders: Blackfern, Cabbage tree and Lemonwood. Open each of these species folders and examine the contents of the site directories contained in them. Blackfern has only one sample site while Lemonwood and Cabbage tree have two. Also browse inside the site directories to find the ASD binary files.

5.3 Creating a new Study and Loading the Spectra

Create a new study as detailed in 4.2. As study name type ‘your-name_example’, e.g.

‘Mikes_example’. The ‘Species directory path’ is in this case the path pointing to the folder named

‘Vegetation_example’. Set the number of samples per species for library inclusion to 10.

Once the new study is created it will be automatically selected as the current study (otherwise select it (see 4.4)) and import the spectra (see 4.5).

5.4 Get to Know Your Data

Check the settings of the processing chain:

□ Smoothing filter: none

□ Current Sensor: Hyperion

□ Derivative: 0

Use the ‘Export’ function (see 4.7) to export the Hyperion synthesized spectra to a CSV file:

□ Processing Stage: Synthesized

□ Data Details: All Spectra

□ Species Restriction: Include all species

□ Names: Folder

□ Transpose: Ticked

□ Format: csv

This will create a file in the SpectraProc output directory. The output file name is shown in the message window. Open this file by double-clicking which should load it directly into Excel. The columns displayed should be similar to the example shown in Figure 48. Column A contains the average wavelengths of the Hyperion bands. The following columns contain the spectral data for each spectrum. The column headers are the species folder names.

A. Hueni \UserGuide_V02.doc Version 0.2 / 16.06.2006

SpectraProc

User Guide

Page 38 of 43

Figure 48: Example of a SpectraProc CSV output

Create an X-Y plot. The result should be similar to the one shown in Figure 49. Maybe surprisingly the plot shows not the expected spectra but some noise that reaches values of up to 140. To see the vegetation spectra rescale the Y axis to a maximum of 1.0 (see Figure 50).

The noise is caused by atmospheric absorption (Thenkabail et al., 2004). As first step this waterband noise should be removed from the data before any further processing or analysis is carried out.

To remove the noise affected regions from the spectra open the waveband filter dialog (see 4.6.1) and click on ‘Add default ranges’. This adds three wavelength regions that are known to contain waterband noise.

1.60E+02

1.40E+02

1.20E+02

1.00E+02

8.00E+01

6.00E+01

4.00E+01

2.00E+01

0.00E+00

0.00E+00 5.00E+02 1.00E+03 1.50E+03 2.00E+03 2.50E+03 3.00E+03

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Cabbage_tree

Cabbage_tree

Cabbage_tree

Figure 49: A plot showing mainly noise

A. Hueni \UserGuide_V02.doc Version 0.2 / 16.06.2006

SpectraProc

User Guide

Page 39 of 43

1.00E+00

9.00E-01

8.00E-01

7.00E-01

6.00E-01

5.00E-01

4.00E-01

3.00E-01

2.00E-01

1.00E-01

0.00E+00

0.00E+00 5.00E+02 1.00E+03 1.50E+03 2.00E+03 2.50E+03 3.00E+03

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Cabbage_tree

Cabbage_tree

Cabbage_tree

Figure 50: The same plot as above but rescaled to a reflectance of 1.0

Once the waveband filter is setup, repeat the export function (close the previously written file first in

Excel) and load it again into Excel and plot it. The plot will look similar to the one shown in Figure

51. Note that now the noise regions are plotted as straight line segments. For visualization purposes this should be avoided as it suggests that data exists in the filtered regions.

1.20E+00

1.00E+00

8.00E-01

6.00E-01

4.00E-01

2.00E-01

0.00E+00

0.00E+00 5.00E+02 1.00E+03 1.50E+03 2.00E+03 2.50E+03

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Cabbage_tree

Cabbage_tree

Cabbage_tree

Figure 51: Spectra after setting up the waveband filter regions

To plot the filtered regions as gaps, select the option ‘Include filtered bands’ in the file export dialog.

If the data is plotted again the result will be similar to the one shown in Figure 52.

Two other points are noteworthy: (a) some reflectance values are higher than one and (b) one Cabbage tree spectrum seems to be an outlier.

The reflectances higher than one could be the effect of changing illumination conditions during the sampling or a result of the BRDF.

The outlier problem can currently not be solved automatically. Two options exist: (a) remove the concerned spectrum from the file system and then delete, recreate and reload the study or (b) re-

A. Hueni \UserGuide_V02.doc Version 0.2 / 16.06.2006

SpectraProc

User Guide

Page 40 of 43 move the concerned spectrum from the database using SQL commands. Future versions of the software may feature automatic outlier detection and elimination.

1.20E+00

1.00E+00

8.00E-01

6.00E-01

4.00E-01

2.00E-01

0.00E+00

0.00E+00 5.00E+02 1.00E+03 1.50E+03 2.00E+03 2.50E+03 3.00E+03

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Blackfern

Cabbage_tree

Cabbage_tree

Cabbage_tree

Figure 52: Waveband filtered data export with the 'include filtered bands' option

To generate stacked plots select the option ‘Offset by [%]’ in the file export dialog and set the percentage to 100. Then plot the data again. The spectra of each species are offset by the same amount (see Figure 53). If only the species mean is selected, the resulting plot is similar to the ones shown in diverse journal articles (see Figure 54). Note that when creating graphs in these cases the

Y axis scale should not displayed as it makes physically not much sense (reflectances bigger than

1).

350 550 750 950 1150 1350 1550

Wavelength[nm]

1750 1950 2150

Figure 53: Stacked plots offset by 100% showing all spectra per species

2350

A. Hueni \UserGuide_V02.doc Version 0.2 / 16.06.2006

SpectraProc

User Guide

Page 41 of 43

Blackfern

Cabbage_tree

Lemonwood

250 450 650 850 1050 1250 1450 1650 1850 2050 2250 2450

Wavelength[nm]

Figure 54: Stacked plot showing the mean per species

5.5 Discriminant Analysis in PCT Feature Space

Discriminant Analysis (DA) is carried out in feature spaces. As an example the data will be transformed by a principal component transformation (PCT). The number of components used in the transformation has a direct impact on the classification accuracy achieved by DA as will be demonstrated hereafter.

Select ‘PCT’ as the current feature space type. Note that the ‘Feature Spaces’ dropdown listbox states that there exist no feature spaces of this type (see Figure 55). A PCT is only possible if a

Principal Components Analysis has been carried out. Perform PCA (see 4.10.3 for details).

Figure 55: PCT feature spaces

Open the PCA output file in Excel and plot the first 10 eigenvalues against the component number.

The result should be similar than the one showed in Figure 56. This graph is called ‘screeplot’. The sharper the drop off in eigenvalues the better are the results that can be expected from the following PCT.

Note that once the PCA has been performed the ‘Feature Spaces’ dropdown listbox contains a selection of feature spaces. The numbers at the end of the names are equal to the number of components used in the feature space. E.g. PCT_10 will effect a PCT using the first 10 components of the transformation matrix, i.e. the data will be reduced to 10 variables (dimensions).

Select ‘PCT_5’ as feature space.

The discriminating power of feature spaces can be assessed by carrying out a discriminant analysis. Before the DA can be executed the library for the current processing chain settings must be built (see 4.9 how to build libraries). Then subject the data to DA (see 4.10.2 for details). The resulting overall accuracy should be around 90.7%. The DA results are detailed in the DA output file ‘Error_Matrix.csv’ (see Table 6). The columns are the species that were classified. The number of correctly classified spectra is contained in the diagonal elements. The errors of omission are written in the off-diagonal elements. E.g. 6 Blackfern spectra were wrongly classified as Cabbage tree.

Close the DA output file, select PCT_10 as feature space and repeat the process of library building and DA and inspect the DA output file once more. The classification accuracy should have risen to

100%.

A. Hueni \UserGuide_V02.doc Version 0.2 / 16.06.2006

SpectraProc

User Guide

Scree Plot

2.5

2

1.5

1

Eigenvalue

0.5

0

0 2 4 6 8 10

Figure 56: Scree plot of the eigenvalues of the first 10 components

Table 6: Error Matrix for DA on PCT_5 feature space

Blackfern

Cabbage_tree

Lemonwood

Column Total

Producer Accuracy

Blackfern Cabbage_tree Lemonwood Row Total

12 0 0 12

User

Accuracy

1.00E+02

6

0

20

0

0

27

26

27

7.69E+01

1.00E+02

18

6.67E+01

20

1.00E+02

27

1.00E+02

65

9.08E+01

Page 42 of 43

A. Hueni \UserGuide_V02.doc Version 0.2 / 16.06.2006

SpectraProc

User Guide

Page 43 of 43

6 References

Bojinski, S., Schaepman, M., Schlaepfer, D. & Itten, K. (2003). "SPECCHIO: a spectrum database for remote sensing applications." Computers & Geosciences 29: 27-38.

Elvidge, C. D. & Chen, Z. (1995). "Comparison of broadband and narrowband red and near-infrared vegetation indices." Remote Sensing of Environment 54: 38-48.

Fliege, N. J. (1994). Multirate Digital Signal Processing. Chichester, John Wiley & Sons.

Keshava, N. & Mustard, J. F. (2002). "Spectral Unmixing." IEEE Signal Processing Magazine 19(1):

44-57.

Landgrebe, D. (1997). On Information Extraction Principles for Hyperspectral Data, Purdue University.

Papula, L. (1994). Mathematik fuer Ingenieure und Naturwissenschaftler, Viewegs.

Richards, J. A. (1993). Remote Sensing Digital Image Analysis. Berlin, Springer Verlag.

Savitzky, A. & Golay, M. J. E. (1964). "Smoothing and Differentiation of Data by Simplified Least

Squares Procedures." Analytical Chemistry 36(8): 1627-1639.

Shaw, G. & Manolakis, D. (2002). "Signal Processing for Hyperspectral Image Exploitation." IEEE

Signal Processing Magazine 19(1): 12-16.

Thenkabail, P. S., Enclona, E. A. & Ashton, M. S. (2004). "Accuracy assessment of hyperspectral waveband performance for vegetation analysis applications." Remote Sensing of Environment 91:

354-376.

University of Waikato (2005). WEKA.

http://www.cs.waikato.ac.nz/~ml/weka/ .

Vane, G. & Goetz, A. F. H. (1988). "Terrestrial Imaging Spectroscopy." Remote Sensing of Environment 24: 1-29.

A. Hueni \UserGuide_V02.doc Version 0.2 / 16.06.2006

Was this manual useful for you? yes no
Thank you for your participation!

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project