Neural networks for time series analysis Submitted in fulfillment of part of the Requirements for the degree of Department of Statistics Faculty of Science University of Pretoria Acknowledgements 2. Dr Johan Holm, my co-promoter. for his advice and encouragement especially in the field of neural networks. Summary Promoter: Dr H Boraine Co-promoter: Dr JEW Holm The analysis of a time series is a problem well known to statisticians. Neural networks form the basis of an entirely non-linear approach to the analysis of time series. It has been widely used in pattern recognition, classification and prediction. Recently, reviews from a statistical perspective were done by Cheng and Titterington (1994) and Ripley (1993). One of the most important properties of a neural network is its ability to learn. In neural network methodology, the data set is divided in three different sets, namely a training set, a cross-validation set, and a test set. The training set is used for training the network with the various available learning (optimisation) algorithms. Different algorithms will perform best on different problems. The advantages and limitations of different algorithms in respect of all training problems are discussed. In this dissertation the method of neural networks and that of ARlMA. models are discussed. The procedures of identification, estimation and evaluation of both models are investigated. Many of the standard techniques in statistics can be compared with neural network methodology, especially in applications with large data sets. To illustrate the adaptability of neural networks the problem of forecasting is considered. A few alternative theoretical approaches to obtain forecasts for non-linear time series models are discussed. It is shown that bootstrap methods can be used to calculate predictions, s~dard errors and prediction limits for the forecasts. Neural network methodology is applied to predict an electricity consumption time senes. Studieleier: Dr H Boraine Medestudieleier: Dr J.E.W Holm Voorgele ter vervulling van 'n deel van die vereistes vir die graad Magister in Wisklmdige Statistiek Die ontleding van tydreekse is 'n bekende probleem vir statistici. Neurale netwerke vorm die basis van 'n geheel en al nieliniere benadering tot die analise van 'n tydreeks. Dit word in 'n wye spektrum van probleemgebiede gebruik, byvoorbeeld patroonerkenning asook klassifikasie en vooruitskatting. Slegs onlangs is neurale netwerke deur navorsers soos Cheng en Titterington (1994) en Ripley (1993) vanuit 'n statistiese oogpunt beskou. Een van die belangrikste eienskappe van neurale netwerke is die vermoe om te leer. Neurale netwerkmetodologie behels dat die datastel verdeel word in drie dele, naamlik: 'n leerstel, 'n kruisvalidasiestel en 'n toetsstel. Die leerstel word gebruik om die netwerk af te rig deur gebruik te maak van verskeie leeralgoritmes. Sekere algoritmes presteer beter as ander, na gelang van die tipe probleemsituasie. Die vooren nadele van leeralgoritmes word bespreek. In hierdie verhandeling word die metodiek van neurale netwerke en die van ARMA modelle hespreek. Die prosedures van identifikasie, hemming en evaluasie van heide modelle word bespreek. Baie van die standaardtegnieke in statistiek word vergelyk met neurale netwerkmetodologie en spesifiek met die toepassing van groot datastelle. Om die aanpasbaarheid van neurale netwerke te illustreer, word die probleem van vooruitskatting in oenskou geneem. 'n Paar a1tematiewe teoretiese beskouings om vooruitskattinings te maak vir nieliniere tydreekse, word bespreek. Daar word aangetoon dat bootstrap-metodes gebruik kan word om standaardfoute en intervalle vir vooruitgeskatte waardes te bereken. vooruitskattings, Symbol Commentary A set of input variables for example: Yl-], Yt-2. temperature. days ... EUf) I E6i) ="2 f,; {yj N f(:!j;~r (H). = o2E(~ Y Ow.Ow. I J b= OE(~I - ow!!o Error function used for maximum likelihood estimation Contents Chapter 2: Neural Networks 3 2.1 Introduction 3 2.2 Historical Development 3 2.3 Artificial Neural Networks 4 2.4 Relevance of Neural Networks in Research 5 2.5 The Model of a Neural Network 7 2.6 Architecture of a Neural Network 11 2.6.1 Single-Layer Feedforward Networks 12 2.6.2 Multi-Layer Feedforward Networks 12 2.6.3 Recurrent Networks 14 Appendix:Terminology 17 Chapter 3: Learning in Neural Networks 19 3.1 Introduction 19 3.2 The Error Function 19 3.3 Taylor Series Approximations 22 3.4 Training 24 3.4.1 The Method of Gradient Descent 24 3.4.2 3.4.3 Superlinear methods 25 The Newton Methods 28 3.5 Cross- Validation as part of the Learning Process 29 3.6 Conclusions 30 Chapter 4: ARMA Processes and Neural Network Models 31 4.1 Introduction 31 4.2 The ARMA(p,q)-Process 31 4.3 Identification 4.4 Estimation 4.4.1 Maximum Likelihood Estimation (MLE) 4.4.2 4.5 Estimation when using neural networks Evaluation Chapter 5: Forecasting a Time Series Using Neural Networks 51 5.1 Introduction 49 5.2 The Forecasting Model 49 5.3 Non-linear Forecasting 51 5.3.1 The Naive Method 51 5.3.2 Closed Form 51 5.3.3 Monte Carlo 52 5.3.4 Bootstrap 52 5.3.5 Direct Method 53 The Bootstrap Technique 54 5.4.1 Prediction Limits for Autoregressive Models 54 5.4.2 Multi-step Forecasts for Neural Network Models 55 5.4.3 Prediction Limits for Yn+h 57 5.4 Chapter 6: Forecasting Electricity Time Series by using Neural Networks 6.1 Introduction 6.2 Model Identification 6.3 Estimation 6.4 Evaluation of the Model 62 6.5 Forecasting by using the Neural Network Model 6.5.1 The Naive Method 6.5.2 The Direct Method 6.5.3 The Bootstrap Method Appendix: Fortran Computer Programme Chapter 7: Conclusions and Recommendations for Further Research References Detecting trends and patterns in a time series traditionally involves statistical methods such as clustering and regression analysis. However, these methods are linear and may fail to forecast a non-linear data set. Neural networks form the basis of a different approach to the non-linear analysis of time series. This study investigates the use of a linear time series ARIMA model in comparison with a non-linear neural network model for application in forecasting. Chapter 2 deals with the history and development of neural networks. This is done in the framework of a variety of problems, such as classification, pattern recognition and forecasting. The model of a neural network is illustrated along with different types of applied neural networks. Of all the possible networks, multi-layered feedforward neural networks are used in this dissertation. A very important property of a neural network is its ability to learn, which is the phase during which parameter estimation takes place. The data set is divided into three different sets, namely a training set, cross-validation set, and a test set. Neural networks are data driven in the sense that they use data to train and build statistical models of a recorded process. The training set is used for training the network with various available learning (optimisation) algorithms. These various available learning algorithms are treated in Chapter 3. Optimisation is the process in which an error function, E, is minimised. The cross-validation set is used to prevent the network from over-training. Over-training occurs when the network is able to represent the data very well but without the ability to generalise. In well-behaved optimisation problems, the value of the error function will have a downward slope in regard to the parameters or weights. The algorithm will terminate when a specified criterion is met, such as a required degree of accuracy. A stopping criterion that is often used in neural network methodology, to prevent over-training, is the value of the error function calculated for the cross-validation set (a data set not used for the estimation of the parameters of the model). If this value shows an upward trend, while the corresponding value for the training set decreases, over-training is indicated. The test set does not form part of the learning process and is not considered in the estimation of the model. Further in Chapter 3. different optimisation algorithms are discussed. It is emphasised that it is not possible to highlight one specific algorithm because all algorithms have advantages and limitations. Chapter 4 reviews the ARMA(p.q)-process.The different stages of modelling. namely identification. estimation by means of maximum likelihood and evaluation are discussed. This is illustrated by examples where both an AR(2) model and a nemal network model are fitted to two time series. a computer generated AR(2) time series and an electricity consumption time series. In Chapter 6. the model used to describe the electricity consumption is extended in order to describe complex seasonal patterns. Time series models are often used for forecasting. Since neural network models are nonlinear. minimum mean squared error forecasts are not of a simple form. In Chapter 5. a procedure is proposed to calculate one- and multi-step forecasts as well as prediction limits for the forecasts based on bootstrap methodology. In Chapter 6. more complex models are considered to fit electricity consumption. The aim of Chapter 6 is to demonstrate the use of neural networks as a statistical tool for forecasting. Forecasting is done for one to twelve and twentyfour hours ahead. Three of the forecasting procedures. discussed in Chapter 5. are also implemented Chapter 7 concludes this work and suggests three extensions to this research. namely the inclusion of bootstrap methods in new software tools. comparative studies and applications. An introduction to neural networks will be given in this chapter. In Section 2.2 the history of Artificial Neural Networks (ANN), the development thereot: the different fields of applications as well as the contributions that have been made between Neural Networks and Statistics are discussed. To conclude the section, the connection between a biological neuron and the neural network is illustrated. The capabilities and properties of neural networks are given in Section 2.3 and 2.4. The construction of a neural network as well as network architectures will be discussed in Section 2.5 and Section 2.6. The first time interest was taken in the term Artificial Neural Networks was when Mc Culloch and Pitts (1943) introduced simplified neurons which represented biological neurons, and which could perform computational tasks. In 1969 Minsky and Papert (1969) published their book Perceptrons, which focussed on the deficiencies of perceptron models. Unfortunately this caused many researchers to leave the field Interest in neural networks in the early eighties re-emerged after the publication of several important theoretical results. This renewed interest is clearly visible in the number of societies and journals associated with neural networks. The INNS (International Neural Network Society) is widely known, and ranges from Europe (ENNS) to Japan (JNNS). The field of Electrical and Electronic Engineering is served by the journal series IEEE, and the field of Agriculture by the journal Neural Network Application Agriculture (NNAA). In the field of Economics and Finance there are the Journal of Economics and Complexity and the Journal of Computational Intelligence in Finance etc. Other well known journals include Neural Networks and Neural Computation. Several conferences are held regularly, and for example the World Congress of Neural Networks (WCNN'95) took place in Washington in 1995. A.I.arge 3 variety of software packages are available in the market today. Neural networks are popular because of the many application fields in which they can be used Many articles were written in recent years in several interesting fields, for example on classification (Ripley, 1994) and pattern recognition, where applications include the automatic reading of handwriting and SPeeChrecognition (park et oJ, 1991). Another field of application for neural networks is prediction. In the world of finance, Tam and Kiang (1992) showed that the neural network is a promising method of predicting bankruptcy. In the medical world neural networks are used for instance to predict mortality following cardiac surgery (Orr, 1995). Several articles have been published in which statistical principles are used with neural networks. Cheng and Titterington (1994) pointed out some links between statistics and neural networks and encouraged cross-disciplinary research to improve results. De Jongh and De Wet (1991) introduced neural networks as a powerful tool that is close to statistical methods in solving various problems like pattern recognition, classification and forecasting. The neural network and statistical literature contain many of the same concepts but usually with different terminology (Sarle, 1996). A list of statistical and neural network terminology is given in the Appendix of this chapter. In the following paragraphs a complete discussion will be given of what a neural network is and how it comPares with a biological neural network. Artificial Neural Networks (ANN) are computer algorithms or computer programs develoPed to model the activity of the brain (Stem, 1996:205). The term "neural network" originates from the attempt by scientists to imitate the ability of the brain to recognise patterns and to classify. This is the reason why it is called "Artificial NeuroJ Networb". Scientists prefer to refer to ANN only as NN (neural networks). In this dissertation they will also be referred to only as neural networks. The brain is a highly complex, non-linear and massively parallel computer. It can perform computations like pattern recognition, perception and motor control many times faster than any digital computer. This is possible because of the existence of neurons. Neurons are simple elements in the brain connected to each other to form a network of neurons. In Figure 2.1 the biological neuron is illustrated A neuron is activated by another activated neuron(s) and, in ~ activates another neuron(s). Synapses are units that mediate the interaction between the neurons. It can either excite or inhibit a receptive neuron. If a neuron receives a signal from a neighbouring neuron, which excites, the neuron's soma potential rises above some threshold value and the neuron "fires". This in turn sends a signal to the next neuron, and so on. New synaptic connections between neurons and the modification of existing synapses constantly take place as the brain learns. The axon.~ are the transmission lines, and the dendrites are the receptive zones. The neural network resembles the human brain in two ways, namely that the network has to learn to acquire knowledge and then the knowledge is stored in synaptic weights, which represent the strength betw"eenconnected neurons.(Haykin, 1994:2). For classification problems, discriminant analysis is used and linear regressIOn analysis is commonly used for prediction. Why would one use neural networks when there are so many other methods available? In this section only a few reasons mentioned by Haykin (1994) are given. Firstly a neural network develops out of interconnections between neurons which defines a non-linear relationship between the neurons whereas the procedure for the above-mentioned techniques are linear. Secondly neural network methodology is a non-parametric because the network learns from examples by constructing input-output mapping for the specific problem. Although there are usually many Parameters in a neural network model, these Parameters are essentially artefacts in the fitting process. The emphasis is therefore not on interpreting their values but on finding the optimal set for the classification or prediction problem. One of the training methods is supervised learning. This involves the changing of the synaptic weights by Presenting a training set of examples to the network. Every time the weights are changed, an error signal is computed This error signal is defined as the difference between the actual response of the network and the desired response. The moment the error signal reaches a minimum value, the desired weights for the specific problem have been obtained. Neural networks are therefore adaptable. A more detailed discussion on supervised learning as a training method is given in Chapter 3. In classification problems, neural networks'are not only designed to give information on which specific pattern to select, but also on the confidence in the decision made. Evidential response is thus an important property of neural networks in classification problems. Neural networks are fault tolerant in the sense that, for example if a neuron or its connecting links should be damaged, the neural network will gradually degrade in performance, rather than suddenly. Another important issue regarding neural networks is that the broad approach is a very computer- intensive method, but with the constant improvement in computer technology it becomes cheaper and faster to do calculations. Lastly, neural networks are perhaps more user-friendly than other methods because the user only has to differentiate between the independent (input) and dependent (output) variables before selecting a model. There are other broad classes of neural networks such as Hopfield associative networks, Linear networks, Probabilistic neural networks and Radial Basis Function neural networks. A feed forward neural network, as used in this dissertation, is a name used for a wide variety of mathematical models used to define the connection between a number of explanatory variables and one or more dependent variables. In Section 2.5 it is shown that the architecture of the model depicts the mathematical outlay of the neural network. As mentioned in Section 2.2, the neurons are the elements that are connected to each other to form the network. In Figure 2.2 the non-linear model of a neuron is illustrated. The different components of a non-linear model are given. Let x I, X2, ... Xp be a set of independent or explanatory variables and let Yk be the non- linear function Activation function Input Output signals y}; Synaoeic weights Figure 2.2: The non-linear model ofa neuron (Haykin, 1994:8) The Xj'S are the input signals. They are each connected to a weight or strength of its own. A signal Xj at the input of synapse j connected to neuron k is multiplied by the synaptic weight wkj. The first subscript refers to the neuron in question and the second subscript refers to the input end of the synapse to which the weight refers. If the weight is positive, the associated synapse is excitatory. If negative, the synapse is inhibitory. The threshold function, Bt, is a bias or constant parameter added as an extra input with value in most cases chosen as one. The non-linear regression model can be described as where Yk is the outp~ wkj, j = 1,2, ...p are the parameters or weights associated with each input valuexj,j= 1,2,...p and &.t is the residual term. The linear combiner described as I sums the input signal Xj, weighed by the respective synapses of the neuron. The activation Junction, ffJ, transforms the amplitude of the output of a neuron. Normally the amplitude will lie in the interval (0.0;1.0) or (-1.0;1.0), depending on the type of activation function. Three types of activation functions, namely the threshold function, the piecewise linear function, and the sigmoidal function are discussed. P Vk = LW.qxj j=l -8k I if V> 0 qJ( V) = { 0 if v < 0 1 0.8 0.6 0.4 0.2 o -1.5 -2 , -1 -0.5 I if v ~ 0.5 v if -0.5<v<0.5 qJ(V)= { o if v:::;-o.5 2 1.8 1.6 1.4 1.2 1 0.8 0.6 0.4 0.2 o-2 -1.5 -1 -0.5 0 0.5 The sigmoidal function is the most popular activation function. The fact that it is a differentiable function is especially important in the training of neural networks. The reason for this will be discussed later. An example of a sigmoidal function is the logistic !W2Ction. qJ(v) = 1 . l+exp(-av) Note that the activation functions range from 0 to 1. In Figure 2:5 the sigmoidal function is illustrated for different values of a. 2 1.8 1.6 :p(v) 1.4 1.2 1 0.8 0.6 0.4 0.2 0 -10 -8 -6 -4 -2 0 2 4 6 8 10 v The threshold has the effect of lowering the net input to the activation function. If the opposite effect is needed a hia') term can be implemented. The bias and threshold function are, therefore nUIl?ericallythe same but with different signs. where Yk is the output of the signal threshold The use of the threshold transformation to the output rp(.) is the activation function and 8k is the 8k has the effect of applying an affine of the Jinear combiner. See Figure 2.6. Uk. Total internal activity level, Vk Threshold 8k < 0 8k=0 8k>0 Linear combiner's output, Uk Figure 2.6: Affine transformation produced by the presence of a threshold (Haykin, 1994:9) Whether the threshold 8k is positive or not, the relationship between the activation potential Vk of neuron k and the linear combiner output Uk stays constant. A neural network is an interconnection of neurons. Neurons can be connected in many different ways to define different neural network architectures. In Section 2.5 the architecture of the neural network is illustrated along with the different classes of neural network architectures. Neural networks are mostly illustrated as block diagrams in which the vanous elements of the model are described. There are four different classes of network architectures, namely single-layer feedforward networks, multi-layer feedforward networks, recurrent networks and lattices structures. Only the first three will be discussed. When a network is organised in a form of layers, it is referred to as a layered network. A linear regression model is equivalent to a feedforward neural network in its simplest form with an input and an output layer and a linear activation fimction. In Figure 2.7, a feedforward network with a single layer of neurons is illustrated The most simple feedforward neural network has an input and output layer. Figure 2.7:Feedforward network with a single layer of neurons (Schalkhoff, 1992:250) This network consists of an input layer, one or more hidden layers and an output layer. The computation nodes of the hidden layer are called the hidden neurons or units. This type of network defines a more complicated model. The relationship between the inputs and outputs can be non-linear functions of non-linear functions, so that the degree of non-linearity increases with each additional layer. In Figure 2.8 a fully connected feedforward network with one hidden layer is illustrated. In a fully connected feedforward network, every input neuron in the input layer is connected to every neuron in the follqwing hidden layer, as well as every neuron in the last hidden layer connected with every neuron in the output layer (Schalkhoff, 1992:236-258). 0·· Figure 2.8: Fully connected feedforward network with one hidden layer and output layer(De Jongh and De Wet, 1994:5) There are many fields of application for the multi-layered feedforward network, like classificatio~ pattern recognition and forecasting. The following example illustrates a time series prediction problem. Letyt. Yt-l, Yt-2 ... be a stationary time series and let Then a set of d such values Yt-d+], ... Yt Yt+l be the value to be predicted. can be selected from the time series to be the inputs to a feedforward network, and the next value Yt+ 1, used as the target for the output. In Figure 2.9 a feedforward neural network with one hidden layer is illustrated (Bishop, 1996:303). y t ~---., t t I I I I I I I • • •Yt Yt+l• Yt-? Yt-l ~ A recurrent network differs from a feedforward network in the sense that it has at least one feedback loop. If feedback exists in a neural network, it can be shown that its output depends on all previous input values. Let Yl• Yl •... YN be a time series. Suppose that a recurrent network architecture is used to define a model for the time series and that the observation at time t can be written as Yt-1 and the previous output value, Y t-I, which is characteristic of a recurrent neural network model. By recursive substitution it can be shown that Yt can be written as a function of Yt-1.Yt-l•... : y, = f(Y,_I,Yt-l,~+&, = f(Yt-l,f(Y,-2'Y 1-2,w),~ + &, If Yt is generated by a moving average process (see par 4.2), it can be expressed as a linear function of an infinite nwnber of previous time series observations: Yt-J• Yt-2•... (see 4.2.7). A recurrent network will therefore be suitable in the case of any process with moving average terms. Basic ModelGenlM 1.0 of Crusader systems, the software package that was used in this study, offers both feed forward and Elman networks, which are recurrent In Figure 2.10 The Elman-net is illustrated Figure 2.10: An Elman Recurrent network (Basic ModelGenlM1.0, 1997:53-54) In an Elman network one or more of the hidden units from the previous time step are used as an input at the next time step. This is useful when the output depends on the history of the inputs, which is the case in this study. O'Brien feedforward (1997) uses a neural network and an Elman recurrent neural network to extract knowledge of a physical system (O'Brien, 1997). A neural network can thus be compared with a non-linear model which defines a connection between certain input or explanatory or independent variables and output or dependent variables. Neural networks are used for different applications such as classification and prediction. In order to predict or classify, the parameters in the case of the non-linear model and known in neural network literature as weights, first have to be estimated The estimation of the weights is referred to as '~ining" Chapter 3 investigates the various methods of training a network. the network. Below is a list of neural network terms and the corresponding statistical terms.(Sarle, 1996: 1-5) .---- Neural network term Statistical term Architecture Model Training, Learning, Adaptation Estimation, Model fitting, Optimisation Classification Discriminant analysis Mapping, Function approximation Regression Supennsedleanting Regression, Discriminant analysis Unsupennsed learning, Principal components, Cluster analysis, Self-organization Data reduction Training set Sample, Construction sample Test set, Validation set Hold-out sample Input Independent variables, Predictors, Regressors, Explanatory variables, Carriers Output Predicted values Forward propagation Prediction Training values, Target values Dependent variables, Responses, Observed values Training pair Observation containing both inputs and target values Neural network term Statistical term Errors Residuals Noise Error term Generalisation Interpolation, Extrapolation, Prediction Prediction Forecasting Squashing function Bounded function with infinite domain Error bars Confidence intervals Weights, Synaptic weights (Regression) coefficients, Parameter estimates Bias Intercept Them~rencereM~nilie~pected Bias value of a statistic and the corresponding true value Backpropagation Computation of derivatives for a multilayer perceptron and various algorithms based thereon Least mean squares (LMS) Ordinary least squares (OLS) One of the most important properties of a neural network: is its ability to leam In neural network methodology, the data set is divided into three different sets, namely a training set, a cross-validation set, and a fest set. The training set is used for training the network with the various available leaming (optimisation) algorithms. These are discussed in this chapter. Section 3.2 defines the error function associated with neural network learning. In Section 3.3 and Section 3.4 methods based on first and second order Taylor expansions are reviewed, including the gradient descent method, Newton methods, conjugate gradient and quasi-Newton methodv. The powerful superlinear methods namely Levenberg-Marquardt which involve combinations discussed. process, The procedure is described algorithm and Snyman's leap-frog method, of first and second order Taylor expansions, are also of cross-validation, which forms part of the learning in Section 3.5. Different algorithms will perform best on different problems. It is therefore not possible to highlight one specific algorithm. The advantages and limitations of different algorithms in respect of all training problems are discussed. Optimisation is the process in which an error function E is minimised There are many optimal error functions, of which the sum of squares error function is the most widely used. The error is a function of the weights in the network as well as the training data. For a multi-layered perceptron (MLP) (discussed in Chapter 2), the derivatives of an error function with respect to the weights can be obtained efficiently by using backpropagation. The gradient information is of central importance in the use of algorithms for network training. There are four main stationary points at which the local gradient of an error function can possibly vanish. Figure 3.1 illustrates this point by showing the value of an error function E against a single weight w. Point A is called a local minimum because it is not the global minimum value in the error space. Point B is a local maximum, while point C is called a "saddle point". Point 19 9. is a region where the error function can be flat and some algorithms tend to get stuck on this flat surface for long periods. This behaviour is also found with local minima and can lead to premature termination of the optimisation algorithm. The desired error minimum is point D, the global minimum, because the value of the error function is minimal at this point. To illustrate basic terminology, the simple case where the error function depends on only two weights, Wl WI and Wl, is considered The problem is to find values for WI and where the error function reaches a global minimum. Figure 3.2 is a geometrical representation of an error function E(!YJ as a surface lying above the weight space. Point A represents a local minimum while point B represents the global minimum of the error function. If C is any point in the error surface, then the local gradient of the error function is VE, where VE is the gradient of the error function with respect to the weights. The gradient in Figure 3.2 will thus be the two-dimensional vector of partial derivatives with respect to the weights: Neural networks usually have many weight parameters, which may lead to extended training times. As a result, effective algorithms are necessary to find a suitable local minimmn of the error function in the shortest time possible. The parameters or weights are not identifIable because more than one set of parameters can give the same error function value. For instance, a two layered neural network: with M hidden neurons exhibits a symmetry factor of MJrJ (Bishop, 1996:256) equivalent points which generate the same network mapping and therefore give the same value for the error function. It is clear that the error function therefore does not have a unique global minimum. As a result, the specifIc values of the weights are not important. When the error surface is locally convex, the error function can be written as a Taylor series expansion in the vicinity of a local minimum. Optimisation algorithms can be classified according to their relation with the terms in the Taylor expansion. Four categories can be distinguished. Firstly, methods based on zero order Taylor expansions are simplex and random walk. These methods are well treated in different texts, and will not be of interest in this work (Fletcher, 1987). A second category includes methods based on first order Taylor expansions such as gradient descent. A third category which can involve combinations of either first and of second order Taylor expansions are superlinear methods such as the Levenberg-Marquardt, leap- frog, quasi- Newton and conjugate gradient methods. Finally, the fourth category \ '~lolt?-z.(."l 'b \ 4 :,C;<t, 0 b '2.. includes methods based purely on second order Taylor expansions such as the Newton methods. Because of the non-linearity of the error function, it is impossible to find global solutions to the error. To overcome this problem, algorithms that search through the weight space are used; formulated in mathematical terms as follows: where w is a weight vector (chosen at random) in a specific update direction, n the iteration step and Aw(n) the adjustment of the weight vector. Different algorithms involve different choices for Aw(n). Algorithms such as conjugate gradient and quaviNewton are formulated not to increase the error function as the weights change. Non- linear optimisation algorithms cannot guarantee global minimum and therefor the disadvantage of these methods is that the error function can get stuck in a local minimum, which, in turn, leads to premature termination. The process of learning or estimation involves the process of optimising the error function by finding the vector of weights which minimises the error function. By obtaining a local quadratic approximation by using a Taylor series expansion, the error function can be minimised in a convex region around a local minimwn. Let E be a function of more than one argument, EU!) = [(WI. W2. ...• w,J (Hamilton, 1994:735-738). Furthermore, let E: fit' -+91 with continuous second order derivatives. A first order Taylor series expansion of EU!) around any point WI is given by T oE ow = VET is a (lxn) vector (the transpose of the gradient) and RI(.) is a remainder If E(.) has continuous second order derivatives, a second order Taylor senes expansion of E(!!!J around WI is given by Consider a second order Taylor series approximation of the multi-variable error function E(!!!J: It' ~ RI around some point minimised. When E(!!!J is WI in the weight space, which is to be twice differentiable, the second order Taylor series approximation of E(w) around WI is b = oE(!i) - ow I ~l The first derivative of the approximated error function (3.3-3) with respect to given by W is The gradient is set equal'to zero at a stationary point, say !!2 = w * and solved for ~ then This forms the basis of many of the learning algorithms because, for points w that are close to w·, these above expressions will give acceptable approximations to the error function and its gradient. The method of gradient descent, also known as steepest descent, makes use of the first order Taylor expansion of the error function, E<!f) as given by (3.3-1). As mentioned in Section 3.2, different algorithms involve different choices for L1w(n), the weight vector update. An initial guess for the weight vector, denoted by wo, is required. The weight vector is updated in such a way that with each iteration step n the direction of the movement is towards the negative gradient of the error function oE~), A!!" evaluated at !fn. This is a discretisation of a first-order differential equation in w so that the trajectory in weight space leads to a stationary point w*. Thus for the error function oE~ --"- ow" )T . =!!, the weIght vector update is A ( aE(w,,) ) l..lwn =-TJ--- - aw" 2 O<TJ<--, Amax where A.max is the largest eigenvalue of H(J!) (Haykin, 1994:49). This leads to guaranteed convergence, although in practice 1] is determined empirically or by rule of thumb. The algorithm that implements the gradient descent learning strategy for feed forward neural networks is known as back propagation. It can be described in two steps. The first step is a forward propagation of the input pattern from the input to the output layer of the network. The second step is a back propagation of the error vector from the output layer to the input layer of the network (Berthold et ai, 1999: 228-229). Superlinear methods involve either first or second order Taylor expansions, such as Levenberg-Marquardt, leap-frog, and quasi-Newton where the Levenberg-Marquardt methods. Consider (3.3-8) methods replace B-1 with rt,.l) where 1] is a step size parameter and I the unity matrix. When TJ is equal to 1, rt,.l)/Ldominatesand the method resembles to gradient descent and when TJ becomes small the method becomes second order. It is important to mention that if residuals, which is the difference between the model output and the desired output, are not small. it should be better to use general quasi- methods or hybrids thereof (see later in this section). The Levenberg-Marquardt algorithm (Bishop, 1996:290-292) minimises the error function while at the same time it tries to keep the step size small to ensure that the linear approximation remains valid The leap-frog optimisation method of Snyman (1982:449-462) is a reliable and robust algorithm which will generally compute acceptable minima without being overly sensitive with respect to difficult environmental conditions. The method is based on the motion of a particle of unit mass in an N-dimensional conservative force field where the total energy of the particle is conserved In the case of neural networks the force field is the weight space. The energy of the particle consists of two components, namely potential energy and kinetic energy from the fact that a force acts on a unity mass particle. The force accelerates the particle to a point where it has maximum kinetic energy and minimum potential energy. When this happens the particle stops because the force is zero. The force acting on the particle is the negative of the gradient of the potential energy (Snyman, 1983). The leap-frog method is based on Euler relations for dynamic systems. The updating equation (cf 3.2-2) for this method where Aw(n) g(n) = rln)At(n) is the step size in the weight space, At(n) the time step, and the acceleration of the particle at step n. (3.4.2-1) relates the trajectory of the particle in the weight space to the particle velocity! and acceleration Q. Quasi-Newton methods use the local gradient of the error function, but instead of calculating the Hessian directly and then evaluating its inverse, an approximation A of the inverse Hessian is built up over a number of steps. The minimum of the quadratic error function is typically reached after W steps (where W is the number of weights and biases in a network) as is the same with the method of conjugate gradient A sequence of matrices is generated to represent increasingly accurate approximations to the inverse Hessian Ifl. This is accomplished by the use of only first order derivatives of the error function (by using backpropagation). By starting from a positive definite matrix such as the unity matrix, the problem of non-positive definite Hessian matrices is eliminated. The update procedures are such that the approximation to the inverse Hessian Ifl is guaranteed to remain positive definite although the condition number of Ifl may become inconveniently large. Two commonly used update procedures are the Broyden-Fletcher-Goldfarb-Shanno (BFGS) and the Davidson-Fletcher-Powell (DFP) (Bishop, 1996:288) with BFGS the method of choice. For the method of gradient descent, the direction vector is the negative of the gradient vector. Small steps are taken to reach the minimum of the error function, which can be a time consuming effort. The method of conjugate gradients guarantees the error function is minimised within W search steps, without calculating the Hessian matrix. This property is an improvement on the gradient descent method because the gradient descent needs many steps to minimise a simple quadratic error function. For conjugate gradient the choice for L1w(n) (the weight update in (3.2-2» is where pIn) denotes the search direction vector at iteration n of the algorithm and TJ is the learning rate parameter. The initial direction vector, ]l(O), is set equal to the negative gradient vector, b = oE(~)1 • _0 ~ w=w uw -- p(O) = -~o Each successive direction vector is then calculated as a linear combination of the current gradient vector and the previous direction vector: p(n + 1) = -Q"+l + p(n)p(n) where JX...n) is a time varying parameter that is determined in terms of the gradient vectOTSQ"and Q,,+l' The Fletcher-Reeves fonnula and the Polak-Ribiere fonnula can be used to detennine JX...n) (Bishop, 1996:280-281). This method was applied to small Boolean learning problems and it was shown that backpropagation learning based on conjugate gradient required fewer iterations than the standard backpropagation algorithm, but is more complex to compute (Kramer and Sangiovanni-Vincentelli, 1989 and Johansson et ai, 1992). There are known problems with conjugate gradient when the dimensionality of the network becomes large. This is due to curnmuIative mis-adjustment on the search direction update formula given in (3.4.2-5). Newton method,; may take only a few steps to converge, but are relatively slow because the Newton methods make explicit use of the full Hessian matrix, H, which has to be calculated. These can lead to computational expense. However, these methods are elegant and applicable and must be treated in this text. Consider the second order Taylor expansion given in (3.3-3) and assume that EUf) is twice differentiable. Then from (3.3-8), where the weight vector w corresponding to the minimum of the error function satisfies w= w· -H-1b. - - - The vector _H-1Q is known as the Newton direction or step (Bishop, 1996:287). Since the quadratic approximation used to determine (3.3-3) is not exact, the weight update equation has to be applied iteratively and H re-evaluated at each new search point. The Newton method has significant disadvantages. For non-linear networks, the Hessian matrix take NW 2 steps to compute, where W is the number of weights in the network and N the number of patterns in the training set. It also has to be inverted, 1..NW 3 requiring 2 steps. The Hessian has to be positive definite to ensure descent in the Newton direction. Cross validation is the process whereby a trained network's performance is measured on unseen data records. The data is divided into three different subsets, namely the training set, cross validation set and the test set. The cross validation set is used during the training process in order to stop training when the network starts to overtrain. With each iteration, the root mean square error (RMSE) of both the training and the cross validation sets are calculated and compared. The RMSE of the training set will typically decrease, and after a number of iterations, stabilise. The RMSE as a measure of performance can be determined by calculating the sum of squares of the error function whereYj is the target output, f(~j;~ a function of the input vector ~and the weight vector wand N is the number of training patterns. The MSE (mean squared error) is then M~E=E~ N Overtraining occurs when the network model fits the training set well but fails to generalise on samples of the same population. The problem is more accute in situations where the number of observations is small in comparison with the number of weights. To overcome the problem of overtraining, the neural network model fitted to the training set is also (in a sense) fitted to the cross validation set. A measure of fit such as the root of the mean squared error (RMSE) is calculated for both data sets. If the RMSE of the cross validation set increases after a number of iterations while the corresponding value decreases for the training set this is a sign of overtraining and unacceptable generalisation results. The remaining data is used to test the network's performance after training. This will be discussed in Chapter 4 by means of an example, when the test set is used to test the performance of the model after training. In this chapter different training or learning algorithms are given. It is emphasised that there is no ideal learning algorithm and that discretion must be used in selecting the correct method for a specific problem. In the next chapter time series analysis with neural network models and Box-Jenkins autoregressive moving average (ARMA) models are discussed. The aim of this chapter is to illustrate the identification, estimation and evaluation of neural network models and autoregressive moving average (ARMA) models for a stationary time series. Neural network models have been used in time series prediction (Le. Baron and Wiegen~ 1996). Autoregressive moving average models (ARMA) are well-known for purposes of time series analysis (Ansuj et aI. 1996). These models define linear relationships between a time series observation at time t. the dependent variable. and a set of time series observations that occurred prior to time t. Any linear model can be expressed as a simple feed forward neural network model with only linear activation functions. This is referred to as a regressor. The versatility of a neural network lies in the fact that it is used to model non-linear relationships between input and output variables. This can result in very complicated models with a large number of parameters. where the parameters have no physical meaning. In model building there is always a trade-off between models with a large number of parameters that fit very well, but with poor generalisation abilities. and models with fewer parameters that do not fit all that well but produce better forecasts because of better generalisation abilities. In Section 4.2 to Section 4.5 the ARMA{p,q}-process is defined. maximum likelihood estimation and the evaluation of the results are discussed. It is shown that these procedures can also be applied as part of the neural network methodology. Suppose that Y,. t = ... -I, 0, I ... is an equally space~ weakly stationary or covariance stationary. time series. A well-known class of linear models for the analysis of time series in the time domain belongs to an autoregressive moving average (ARMA) class of the form (4.2-1) (Hamilton, 1994:59-61), where {&t} is a sequence of uncorrelated variables, also referred to as a white noise process, with conditions = o and E{et) E{e e ) = t' and ~ r {CT: for t = r 0 th . (PI. ... o efWlse ¢p, 01, ... Oq are unknown constants or parameters. The model (4.2-1) is an ARMA(p,q) model or Box-Jenkins model. The process is stationary if the roots of the equation i(BJ = 0 all lie within the unit circle. TheARMA(p,q) model can be expressed as a moving average (AM) model, TI(B)Y, = p. +&, (J(B) with n(B) = =-~(B) Let n(B) = 1-JrtB-Jr2B2 - •••• The equation (4.2-6) can be written as Any ARMA model can therefore be written as an AR or MA model (Cryer, 1986:7374). Box and Jenkins (Box, Jenkins and Reinsel, 1994) developed a general framework:for time series modelling. They suggested a model-building strategy where model identification, estimation and diagnostic checking are done iteratively to select the best possible model for a given series. The ARMA model is based on linear relationships between successive observations as measured by the autocorrelation function. Plots of the sample autocorrelation function (ACF) and the sample partial autocorrelation function (PACF) are compared to the corresponding population functions to identify the ARMA model by selecting the values of p and q, the autoregressive and moving average order, of the model. (Cryer, 1986:111). . r r = COV(Y'_l , &, ) = 0 :;t: if k > 0 0 if k ~ 0 _r1 ro P 1- t/ti is the correlation coefficient in the bivariate distribution of Yt. Yt-k conditional on Yt-I, Yt-l, ... , Yt-k+I (Cryer, 1986:106). The ACF of an AR(P) process shows an exponential decay or a damped sine wave, while its PACF is zero for lags greater than p. A pure moving average model of order q has nonzero autocorrelations wdY on the first q lags and its partial autocorrelations are not zero after q lags. Two time series are used to illustrate the concepts of identification, estimation and evaluation of a model. Time Series 1 consists of 150 tenns generated from an AR(2) model with t/JJ. =1.3, th = -0.5, and zero mean, and St - N(O,a;). Time Series 2 consists of 336 data points of electricity consmnption, measured hourly, for two weeks. As mentioned in Section 4.3, the sample autocorrelation function and sample partial autocorrelation function can be used to identify a possible model that can be used to describe a time series. A time plot can be used to identify a trend, seasonal patterns, outliers, discontinuities, etc. A time plot of hundred observations of Series I is given in Figure 4.1. 10 5 o -5 I-Vtl -10 -15 Figure 4.2 is the sample autocorrelation function. It has the appearance of a damped sine wave which is typical of an AR model while the sample partial autocorrelation illustrated in Figure 4.3 gives a clear indication of an AR(2) model. The first two partial autocorrelations differ significantly from zero while the rest do not differ from zero on the 5% significance level. Lao Covariance Correlation ·1 o 7.670817 1.00000 1 6.013996 0.78401 2 3.712388 0.48396 3 2.166189 0.28239 4 1.322617 0.17242 5 0.765067 0.09974 6 0.505147 0.06585 7 0.830336 0.10825 8 1. 042072 O.13585 9 0.744163 0.09701 10 0 •323893 0 .04222 11 -0.124980 ·0.01829 12 -0.368546 ·0.04805 13 ·0.454332 -0.05923 14 -0.431367 ·0.05823 15 ·0.391593 ·0.05105 16 -0.338712 ·0.04418 17 -0.359287 ·0.04884 18 -0.789711 ·0.10034 19 -1.156028 ·0.15070 20 ·1.868117 ·0.21748 21 -2.225459 ·0.29012 22 -2.784961 ·0.36045 23 ·2.763432 ·0.36025 24 -1.983370 ·0.25856 Std ,I················ . I······. I'·' I'• I' I" I'·' I" ,. .,., .,., .,., .., I "'1 ····1 ....... ,, ........ . ....., marks two standard o 0.100000 O.'49310 0.164249 0.169035 0.170784 0.171368 0.171619 0.172300 0.173368 0.173910 0.174012 0.174028 0.174160 0.174362 0.174543 0.174692 0.174804 0.174929 0.175504 0.176793 0.179448 0.184079 0.191007 0.197685 errors Figure 4.2: Sample autocorrelation function for an AR(2) with ;1=1.3 and ria. = -0.5 (SAS System for Windows v6.12) Lag Correlation ·1 9 8 7 6 5 432 I 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 0.88347 ·0.42102 0.00224 0.00211 ·0.00117 0.02936 0.02602 0.08493 0.06300 0.11536 -0.06218 ·0.00949 ·0.08332 0.19943 -0.13970 -0.04186 -0.00209 -0.14519 ·0.07380 0.02051 0.04710 -0.06309 -0.15938 ·0.00652 I 0 I 2 3 4 5 6 7 891 ........ I·················· , I I I I" I' I'• I' I" '1 1 "1 I···· ····1 '1 1 .* •• , '1 ,. ., .... , 1 I Figure 4.3: Sample partial autocorrelation function for an AR(2) with ;1 =1.3 and -0.5 (SAS System for Windows v6.12) th = ~ ~ ~= .~ :::(,)-,..., 3 2.5 2 .- c + b 0 CD 1.5 (,)._ 0 CD .•••~ _ Cox 1 W E ~ U) I I 1-.ELEK:yt: 0.5 C o o CJ There are explicit daily, 12-hourly and weekly seasonal patterns in the electricity consumption, due to consumer behaviour. The autocorrelation function depicted in Figure 4.5 shows evidence of a sine wave, which may suggest an AR(2) component. The sample autocorrelations from lags 20 to 24 are quite high due to the 24-hour seasonal pattern. Lag Covariance Correlation o 5842606 1.00000 1 5379454 0.92073 2 4337118 0.74233 3 3086162 0.52822 4 1901803 0.32551 5 917772 0.15708 6 194390 0.03327 7 -238615 -0.04084 8 -361234 -0.06183 9 -197386 -0.03378 10 146418 0.02506 11 456639 0.07816 12 511863 0.08761 13 243946 0.04175 14 -219729 -0.03761 15 -631504 -0.10809 16 -783852 -0.13416 17 -635189 -0.10872 18 -208851 -0.03575 19 465597 0.07969 20 1324200 0.22665 21 2276192 0.38959 22 3199092 0.54755 23 3893269 0.66636 24 4086444 0.69942 't:****'/r+**** ********+.*** •••• ****+ ••••• The partial.a.utocorrelation function's first two lags are significant and the remaining lags are significantly reduced which indicates an AR(2)- process. This is illustrated in Figure 4.5. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 0.89902' ·0.40365 ·0.08511 ·0.12721 ·0.22907 ·0.17068 0.15599' 0.13347' 0.04207 0.02022 -0.04473 -0.12335 -0.09437 0.08936' ·0.19785 -0.09166 ·0.00793 ·0.13995 0.00804' -0.10087 -0.02190 -0.06457 -0.00557 0.03435' I·················· I1 • •••••• , 1 ·1 · "I , 1 •• •• 1 ••••• 1 I , 1 · •• ·1 1 I •• I •• 1 I·... ,. I I 1 , 1 I I , '1 1 .., .., , I , I I 1 I • •• , 1 I 1 '1 I 1 I ,. 1 • , · I I •••• •• , 1 ,.. . · I I , 1 · ••• , , , , I I Figure 4.6: The partial autocorrelation function for the sample of electricity consumption. A spectral analysis of the data can be used to identitYthe most prominent cycles in the data. A periodogram of the data is given in Figure 4.7 .. _ 6 co + - 8 4 X - 2 ~ N ~ M ~ ~ ~ M ID ~ ~ ro ~ ~ ~ 0 ~ ~ N ~ The graph s~~ws peaks at 12-hour. 24-hour (the most) and 48-hour periods. A large percentage of the total variation in the data can therefore be ascribed to the identified cycles. and the information can be used to define input variables for a neural network model. Through the method of spectral analysis. the series of observations Yt can be described as a weighed sum of periodic functions of the form cos(~). where {j} denotes a particular frequency (Hamilton, 1994:152). For Series 2 with dominant periods of 24 hours and 12 hours. as seen in Figure 4.7. the frequencies can be calculated as follows: Period =24 = 2Jr therefore lU I = 0.2618 lUl Period =12 = 2Jr therefore lU2 = 0.5236. lU2 Different numbers of nodes in the hidden layer were tried and it was decided to use no more than two nodes in the hidden layer. The added complexity introduced by adding nodes in the hidden layer did not result in a significant improvement of the model fit. Let fl, f2, ... , f N denote a stationary time series, and suppose that the observation at time t can be described by the model where w is the parameter vector or weight vector and the error terms ... ~-I,~,~+I •.. are assumed to be white noise. Iff is a linear function, the above model is an AR(P) model. The parameters of an ARMA model can be estimated by the method of maximum likelihood. There are various other methods, for example, the conditional least .~quare method (CLS) and the unconditionallea.,>;tsquares method (ULS). In the case of an AR model, if the error tenns are assumed to be independent nonnal, it can be shown that the maximum likelihood method is equivalent to the least squares method. For purposes of this dissertation only the method of maximum likelihood will be discussed. be the vector of parameters for anAR(p) model and let (Y ••f2. ... fM>be a stationary time series. The value of l'that maxirnises the joint probability or likelihood function is the maximum likelihood estimate. The likelihood function for an AR(p )-process, conditions on both Y' sand i's. lfthe initial values of ~o = (Yo'Y-J'···'Y-P+)' the sequence {e) ,e2, ••• ,eN } and fo =(Eo,E_),...~E_q+)'are given, can be calculated from {Y)'Y2' ...'YN }by iterating on with the option of setting the initial y's and cs equal to their exPected values (Hamilton, 1994:132). The error function used for maximum likelihood estimation can be written as In neural network terminology, estimation of the parameters of the model takes place during the training period, during which the network learns with generalisation through certain learning algorithms. Weights connected to each input value (parameters) are constantly updated until the error function, E(!!J, which is often the sum of squared errors (see Chapter 2), reaches its global minimum. An AR(2) model was fitted, by using the SAS System for Windows v6.12, to Series 1 by the method of maximum likelihood. The estimates, approximate standard errors and corresponding t-ratios are given in Table 4.1. Parameter Estimate Approx. Std Error T-Ratio Lag MU -o.672n 0.89122 -0.75 0 AR1,1 1.26298 0.09195 13.74 1 AR1,2 -0.42242 0.0937 -4.51 2 For large sample sizes the t-ratio may be used to test the significance of the estimates with respect to the null hypothesis that the corresponding parameter is zero. The tvalue for the mean is small, not rejecting the null hypothesis. This is expected, since the time series was generated with a zero mean. The parameter is redundant and should be excluded from the model. The t-values for the two estimates are large, indicating that ;J and th should not be excluded from the model. In Figure 4.8 the time plots of the actual and the fitted series is given. 10 5 o l--Target - - .... Model -5 -10 A neural network is fitted to the electricity consumption time series. The program package used for this purpose is Basic ModelGen™l.O from Crusader Systems. Estimation ~.f a neural network model takes place during training by means of the training algorithm. For the training set 70% of the data is used, 20% for the crossvalidation set and the rest is used for testing the model. The model (4.3-6) has six input variables, as mentioned in Example 4.1, as well as an intercept term included in both the input layer and the hidden layer. The neural network model is non-linear with two neurons in the hidden layer. The sigmoidal function (2.5-7) is used as activation function. The sum of squared error function (3.5-1), which is a function of the weights in the network, is minimised. This is done efficiently by using backpropagation. As mentioned in Chapter 3, the cross-validation set is also used during training. The root mean square error (RMSE) of both the training set and the cross-validation set decrease with each iteration. The RMSE of the training set will usually decrease more than the RMSE of the cross-validation set because the network is trained on the data of the training set. If too many parameters are included in the model relative to the number of data points in the training set, over-training can occur. An increase in the RMSE of the cross-validation set gives an indication that the network is starting to over-train and training stops immediately. In Figure 4.9 a plot of the RMSE of the training set and the cross-validation set is illustrated. ,.-.,', , .. ; •. I O.Q4.~ : ·5 w ,,~OD36 .' . : . : :. '; : .. . .;" ., .. .:'. '.:. .,' -, :' . " i\11 , c' " " :', " ':': ' ..' t: oms .' 'lk:. "-" < . ;. . :,: :".', .' i \ . 7,- .... ',,' .. ..• c' , o.03i : I . ,', .. :'" ' ;"/. . " :. ". . , " . (. " < '.< ,, ; • '.', J, ~ • .. '-:: -~ .' - ,-~. .. ~ .' , :": ".:,' : ... " -'1'.:,',_'- ,"" The output of the model is a function of the weights but the weights t~emselves do not have any significance value. When the vector of weights, which minimises the gradient of the error function, has been detennined through a learning algorithm, training stops. The output of the model containing the six input variables (and one hidden laye~~th two neurons) as indicated in Section 4.3 fitted to Series 2 is given in Figure 4.10. a- 1.2 1 ~ .1) 0.8 ~ § 0.6 i ~0.4 U c .- 0 a. ~ 8 II) --Series1 ----...Series2 0.2 0 ~ m ~ ~ ~ N M M ~ ~ m ~ ~ ~ ~ M <0 ~ Time Although the data set is quite small, the model fits the data reasonably well, considering Figure 4.10. A rule of thumb that is often used and accepted by practitioners is that the number of observations should be at least a factor of ten times the number of weights (N=JOW). In this section, the estimation of linear and neural network models is discussed The evaluation of the estimated model is discussed in the next section. After a model has been identified and the parameters estimated, the model must be evaluated. Evaluation procedures are the same for both the ARIMA and neural network models. If the model fits the data well, then the residuals should almost have the properties of uncorrelated, identically distributed, random vat:iables with zero mean and fixed standard deviation (Cryer, 1986:147). If the estimated value of given by r. is The residuals of a fitted model are useful indicators of any inadequacies in the specification of the model or violations of underlying assumptions. Examination of various plots of the residuals, is an indispensable step in the evaluation process of any model (Box and Jenkins, 1994:289). If a plot of the residuals exhibits a trend over time, it is an indication of a trend in the data that is not adequately modelled. A histogram of standardised residuals should correspond with a symmetrical normal curve if the model fits the data well. Any outliers will imply significant differences between the Y, and the corresponding observed value, Yt that should be investigated since the model fits the data poorly at those points. The sample autocorrelation function of the residuals, r k , can be observed to check for the independence of the residuals in the model. Usually the sample autocorrelations are approximately uncorrelated and normally distributed with mean zero and variance l/n. A '"i (Chi-squared) test is used to test whether residuals are correlated (Cryer, 1986:153). Different programme packages evaluate neural network models differently. A comprehensive residual analysis is offered by the BasicModelGen™l.O Crusader systems. The mean square error (MSE) in (3.5-2) and the root of the mean square error (RMSE) are determined for every model and the model with the smallest of these values should be selected. As mentioned, the residuals play an important role in the evaluation process. A fast Fourier transform of the model residuals is calculated If the spectrum of the residuals is constant, white noise is implied and it can be concluded that the model fits the data relatively well. Other meth~ of evaluation are a sensitivity analysis performed to establish the most influential input variable of the model, and output analysis where the relationship between an output and specific input is shown. The t-test statistic is used to determine whether a model's output differs from the desired output. Together with this test the mean and the standard deviation of the model's output are calculated. The quality of the model can be observed by viewing a graphical representation of the model's output versus the desired output. Another useful measure of evaluation is the correlation (R) between the target output and the actual model output. In the following example the evaluation of the models fitted to the AR(2) and the electricity demand time series is illustrated. The residuals for the estimated AR(2) model in Example 4.2 were analysed A time plot and other residual plots revealed no deviations from the model and assumptions. A X2 (Chi-squared) test for independence of the residuals was performed and the null hypothesis was not rejected. AR(I), ARMA(2,I) and AR(3) models were also fitted, but the AR(2) model fit was superior as measured by Akaike's information criterion (Cryer, 1986:122) and the significance of the estimated pammeters. This is an expected result, since the time series is generated by an AR(2)-process. The evaluation procedure points out some inconsistencies with respect of the neuml network model fitted to the electricity data, which can be expected because the data is seasonal and the model selected to fit the series perhaps does not include all the variables necessary to fit the data. This problem is examined in Chapter 6. The observed spectrum of the residuals of the neural network model has some peaks which indicate that there is some seasonal information present in the residuals. The sensitivity analysis, illustmted in Figure 4.11, shows that the variable ft-l has the greatest influence on the model output which is quite reasonable because it represents the electricity consumption during the previous hour. 700600500400 300 .~ 1ii 200 - 8100 "2. 0 g -100, &-200 ..,. !! -300 -< ~. . -400 -500·· .. -600 -700-800 The t-test is used to determine how significantly a model's output differs from the desired or target output. The time plot of the model output and the desired output for the test set are given in Figure 4.12. Model Output versus Target Output The correlation between the target output and model output is 0.897. This value differs significantly from O. Table 4.2 provides this information together with the mean and standard deviation of the model and the desired outputs for the test set, as well as a 95% confidence interval for R. Table 4.2: The correlation coefficient for the test set as determined by the neural network model Standard 95% Confidence Mean Deviation R Backprop net 19861.1 2065.443 0.89722 Desired 19461.3 2144.093 Ii 0.80501 UmitsforR [0.82EH:>.940] Although the evaluation of the model for the electricity consumption may indicate some inadequacies, one can always improve the model by trial and error. In Chapter 6 an electricity load forecasting will be done on one year's data. In this chapter the process of identification, estimation and evaluation was discussed by means of examples. When the process of evaluation has been completed, and the model gives a satisfactory description of the data, the model can be used for forecasting or classification. In Chapter 5 forecasts and prediction intervals are derived and calculated for neural network models. In this chapter a neural network model is used to forecast a time series. It is shown that bootstrap methods can be used to calculate standard errors and prediction limits for the forecasts. In Section 5.2 one- and two-step minimum mean square error (MSE) forecasts are given for both a linear autoregressive model and a non-linear neural network model. In case of the non-linear model, a different approach has to be considered. A few alternative theoretical approaches to obtain forecasts for non-linear time series models are discussed in Section 5.3 and the bootstrap technique is selected for the purpose of obtaining prediction intervals. Section 5.4 shows how bootstrap methodology can be used to construct prediction intervals. One- and multi-step predictions, together with their standard errors and 95% prediction intervals are calculated for the simulated AR(2) series. Consider the ARMA model (4.2-1) for a stationary time series fit} with t =1,2, ..., N. The linear AR( 1) model is given by where {Et} is a white noise process that satisfies the conditions (4.2-2). Suppose that the white noise variables are identically distributed. The minimum mean square error (MSE) linear predictor for one step is the conditional expectation of YN+l, given Yl• Y1•... YN: YN(l) = E(YN+.IY.,Y2,··.,YN) = E(f;'YN + E,v+l!Y.,Y2,···,YN) = tAYN By using (5.2-1) for the second step prediction, Y,v+1, the U';;E linear predictor can be written in terms of the observed data Y;v: YN(2) = E(YN+2IY.'Y2'···'YN) = ;IE(Y,v+IIY1'Y2, ...'YN )+0 = ;IYN(l) =;.2YN = E[f(YN+I,a)+EN+21~'Y2' = E{f[f(Y N ,a) + EN+.],a YN(2) = E{flYN(I)+&N+d,al~'Y2' ...'YN] lY 1, Y2,..., YN } ...'YN} Alternative approaches have been proposed to calculate the forecasts; such methods are discussed in the following paragraphs. Five techniques for forecasting have been discussed by Brown and Mariano (1989) and summarised by Lin and Granger (1994). Some of the methods are naive in the requirement of assumptions regarding the distribution of the white noise terms. The methods vary in respect of ease of implementation and computing sensitivity. This technique is widely used and easy to implement but due to the fact that the forecast is b~ it is not satisfactory. If one considers the non-linear model for time series forecasting given in (5.2-5), the naive technique states that the two-step forecast of YN+2 for YN+l is given by in the model equation. The estimator a is obtained by minimising the sum of squared error terms or by maximising the likelihood function of the observations. Usually the expected value of the function is not equal to the function of the expected value and therefore the forecast will be biased Even if the functional form f( ) is known, the bias will not go to zero jf the sample size N is large. J YN(2) = f{(YN(I)+&),~)dF(&), = JfTf(Y N,~)+&),~]dF(&) where F() is the distribution function of & necessary to know the distribution of which is in practice generally unknown. &, andj(.) a non-linear function of &. It is Brown and Mariano (1989) assume a N(O,I) distribution, but this has to be verified Numerical integration can be used to approximate (5.3.2-1). Apart from the fact that this forecast can be difficult to calculate, it may be incorrect, because of the wrong assumption on the distribution of &. The dimensionality of the error distribution will increase with the lead time of the forecast, and will consequently result in an increase in computer time. For large values of N this technique has the same disadvantage as the closed form technique (in that the distribution the error terms is needed) but it is easier to implement. The two-step forecast is defined as where the sequence ej's are independent and identically distributed and chosen from the error terms. This is a particular form of numerical integration based on simulations. The mathematical expectation (5.2-7) is approximated by the arithmetic average from a sample of possible realisations of YN+2 given Yl, Y2, ... , YN. The bootstrap technique is based on the estimated residuals. The two-step forecast is given by are the realised one-step forecast errors arising up to time N. No assumption on the distribution of the error terms is therefore necessary. The forecast improves as time advances and is easily formed (Lin and Granger, 1994:2). This technique is used for purposes of this dissertation and discussed further in Section 5.4. This method involves the direct modelling of the relationship between YN+2 and YN by using the same model as before but estimating a new set of coefficients directly. The previous techniques involved the use of the same function with the same set of coefficients, f(f( )), for the two-step forecasts, which decreases its quality. The f( ) is rarely known in practice and has to be approximated from a specific search. If could be considered For a non-parametric procedure such as neural networks this would be a sensible technique. Lin and Granger (1994) investigated all these techniques with a simulation study for the two-step case. The results lead to a mild recommendation of the bootstrap predictor because of its small bias and not more than 5% inefficiency in mean squared error, when compared with the parametric model, using the corr~ Further study on broader classes of time series models is recommended specifications. The bootstrap technique can be applied to obtain interval forecasts for an autoregressive time series. Masarotto (1990) finds the bootstrap technique useful for three reasons, namely: it is distribution-free, it takes into account that the Parameters and order of the model are unknown, and improved computer technology makes the difficult calculations involved with the bootstrap technique easier. Boraine (2000) showed that the bootstrap results for linear models can be extended to non-linear time series models. A discussion is given in this section. In Section 5.4.1 the prediction limits for linear autoregressive models are given. These are the standard results given in any time series text book. The multi-step forecasts for the neural network models are introduced in Section 5.4.2 and the prediction limits for the multi-step forecasts are given in Section 5.4.3. where YN(J) = YN-i if j < o. The forecasts are calculated recursively and converge to the mean of the time series for large values of h. YN +h IYl, Y2 , ••. , YN is normal with mean YN(h) and variance h-l· a; (1+ Ltp~) where the j 's j:l The approximation of I-a probability limits for fN +hll; , f2 ,""", fN is given by where Z a 1- 2 is the 100(1 - ~)Ih 2 percentile of the standard normal distribution (Box, Let fl, f1, ... ,fN be a stationary time series described by a non-linear neural network model where j{ ) is a non-linear function defined by the neural network architecture, p is the number of input variables in the network, and w the weight vector. The c,'s are uncorrelated, identically distributed random variables with mean zero and variance u/. The observation at time N+ 1 can be written as For the estimation of the minimum MSE forecast, the bootstrap methodology can be used and therefore the bootstrap forecast for YA(2) is proposed as AIm A YN (2) -- - "f *(j) • LJ (YN+l'YN""'YN+2-p,!:!:0 m j=1 ~{l is an observation, and & -. drawn with replacement, from &p+1,..., &N with AIm A Y (h) "f(y*(j) y*(j) y*(j). ",\ N - - LJ N+h-1' N+h-2"'" N+h-p'~ m j=1 Y;}!2 = f(Y;~L1,Y;}!L2, " ... ,Y;}!Lp;!:!:0+&~{1 *(j) - Y YN+h-p - N+h-p 1'fh -p -< and 0 The bootstrap procedure can be summarised in a few steps: First fit the model (5.4.2-1) to the time series Yl, Yl, ...,YN. Then calculate estimates of the residual term using (5.4.2-8). Thirdly calculateY;+1, ...,r:~+h conditional on Yl, Yl, ... ,YN: Y;+Ir-p = YN+h-p if h-p:::; Oand&~+h is an observation drawn randomly with replacement from &p+l, ...,&N .Repeat this for m times where m =100, usually. Now calculate the one- to h-step forecasts for the time series generated in the previous step using (5.4.2-3), (5.4.2-6) and (5.4.2-9). In the next .s~on it is shown that bootstrap methodology can also be used for the construction of prediction intervals for YN+h. The ideas used in this section are based on the method proposed by Masarotto (1990) for linear time series models. The prediction error is By using the bootstrap methodology the distribution of r};{h) can be approximated using the Monte Carlo algorithm. rL andru are the B(Y )-th andB(l- Y )- th order statistic of rN(h); •...• rN(h)~ where 2 2 B is the number of bootstrap replications. eN(h) are required Note that rN(h)is rN (h). a function of u. A uis required for each For good approximation of the distribution of rN (h). at least 1000 bootstrap replications are required First calculate the bootstrap time series ~. ,..., Y;+h. To do this a neural network model of the form (5.4.2-1) is fitted to the observed time series, Yl, Y1,... YH. If w is the estimated weight vector, the estimated residuals as in (5.4.2-8) are &, = Y, - fO~-t,···,}'~-p;~ with A Y,· = f(Y'~J'Y'~2""'Y'~P;~+&' t = p+ l,p+2, ...,N. ",* A* where &, is drawn randomly with replacement from It is assumed that the bootstrap time series values Y:3P-t = Y:3P-2 = ...= y,,:,p = Y. u· , the standard error of YN (h), is calculated by using a bootstrap procedure within the bootstrap procedure, that estimates the distribution of the standardised residual The neural network model of the form (5.4.2-1) is fitted to the time series, ~. ,..., Y;. The residuals of the model are determined by using (5.4.2-8). By using this model residuals, a bootstrap series is generated namely, The forecasts, YN (h) are calculated by taking into account the first N observations, Yl- ,Y;...,Y;. The prediction error defined in (5.4.3-1) is e;(h) = Y;+h - Y N(h). This is repeated m times. The standard deviation of e;(h)(l), ...,e;(h)(m) is used as an ....• time series J;* ' ...'Y;+h' as well as the calculation of YN(h) andu A* and 7N(h)*, B times where B must be at least 1000, the interval (5.4.3-4) can be constructed. A practical application of forecasting a time series from a linear AR(2) model is given in the following example. A Fortran program (see Appendix: Chapter 6) was developed to train the network and to implement the bootstrap for forecasting and the calculation of the prediction limits. The program uses a Gauss-Newton algorithm to estimate the parameters of the neural network. As an exam:l~lea generated AR(2) time series (see Example 4.1 to 4.3) consisting of 200 values is used to illustrate the calculation of the forecasts and the corresponding 95% prediction limits. The generated process is linear, and therefore it can not be expected that the neural network model would produce better prediction results than a linear model. A feed forward neural network with two input nodes for two past values of the time series and one hidden layer with a sigmoidal activation function was trained. Bootstrap methodology was used to calculated the prediction limits. Figure 5.1 and Figure 5.2 illustrate the prediction results. The results for the neural network where bootstrap methods were used and the linear model correspond more or less. Autoregressive time series: predictions and 95% prediction limits (neural network) 0.8 0.6 0.4 0.2 o -0.2 -0.4 -0.6 -0.8 -1 -1.2 •.•. 2 •.•. •. •.- 4 --- 6 8 - - ~--- ..•.... 10 12 - - .. .... .. •. Lead Pred ictio n ---- - ·L95 - - - - - 'U95 I "Autoregressive time series: predictions and 950/0 prediction limits (linear regression model) 0.8 0.6 0.4 0.2 o -0.2 -0.4 -0.6 -0.8 -1 ----Prediction - - - - - ·L95 - - - - - 'U95 I It is shown that bootstrap methodology can be used to calculate one- and multi-step predictions of neural networks with their standard errors and prediction limits. A minimum of 1000 time series have been resampled from the observed times series and for each resampled time series a network is trained to do one step predictions. These highly computer intensive calculations were no problem considering the everincreasing speed of computers. In can be seen in the two examples given that neural networks as well as linear regression models are used to model the time series. Predictions and prediction limits are calculated. The results compare reasonably well. Further research is required to establish whether the bootstrap results can be improved. For an example, the increase in the number of bootstrap replications may lead to an improvement in the results. The objective of this chapter is to forecast a time series by using neural networks. Some scientists devote careers to the building of models for electricity load forecasting. The purpose of this chapter is not to improve on the work done on model estimation but only to introduce the neural network as a forecasting tool. Several studies have been done on electricity load forecasting. For example: the modelling of one-hour-ahead hourly electricity demand prediction has been done by Connor (1996) while Hwang and Ding (1997) focused on the construction of prediction intervals for electricity load forecasting. Lee et of (1992) used neural networks for short term load forecasting by dividing the data into different classes of daily and weekly load variations while Peng et 01 (1993) proposed a new strategy in selecting training cases for a neural network model. The data used here is ESKOM data taken hourly. measuring the average electricity consumption for the Republic of South Africa. Section 6.2 deals with the analysis of the data. The different strategies and techniques used to prepare the data for estimation of the models are proposed in Section 6.3. The model is investigated in Section 6.4 by means of evaluation. Forecasting one to twelve hours ahead by using the selected model is done in Section 6.5. Unfortunately the programme package. Basic ModelGen does not include the bootstrap method as a forecasting technique. Two methods. namely the naive technique discussed in paragraph 5.3.1 and the direct method discussed in paragraph 5.3.5 are used and compared in Section 6.5 by the use of the programme package. Basic ModelGen. The bootstrap method is then used in Section 6.5.3 on the above mentioned electricity data. As mentioned earlier. ESKOM data. taken hourly. measuring electricity consumption was analysed. A one-year period was considered starting from the 25th of October 62 1997 to the ~.5thof October 1998, a total of N = 8760 observations. The data therefore contains seasonal components, as seen in Figure 6.1, where a four-week period is illustrated. 3 e -------------------------------------1 0 +i:;r Q.+ 2.5 E CD ::so (1)- ex o .. CJ= ~tO ..2 -~ ~ ...= 0 CJ_ .!! w 2 1.5 1 0.5 Seasonal components cause difficulties in the estimation of the model parameters. The sample autocorrelation function (ACF) and sample partial autocorrelation function (PACF) point out a definite AR(2) component. Therefore the consumption in periods I-I and 1-2will be considered as two of the inputs in the neural network model. From the electricity data it is clear that there is more than one seasonal component. Several seasonal components can influence the behaviour of a time series. A spectral analysis can be performed to attribute the total variation in the electricity consumption to cycles of different frequencies. The results of the spectral analysis indicated the 24-hour and the 12-hour periods respectively as the more important seasonal components. Several other factors were considered, for example the effect of the day of the week and of public holidays, and the average temperature for each day. Galpin (1997) investigated the inclusion of rainfall, temperature and humidity as input variables in a regression model. Alt~~ugh she found that these climate data decrease the forecast error, it was found that public holidays have an overriding impact on the forecasting process. These variables were all considered as explanatory variables. Networks with different combinations of input variables and a different number of hidden inputs were trained The model that was finally selected to describe the electricity data has as explanatory variables periodic components that were determined by a spectral analysis (see Example 4.1), and the electricity consumption of the two previous hours. If Yt denotes the electricity consumption at time I, the proposed model is: Y, = I(Yt-), Y'_2,COS[liJ)(I -1)],sin[liJ)(I -1)],COS[liJ2(1 -1)],sin[liJil -I)];.!!) +&, (6.2-1) as well as two nodes in the hidden layer. The &,'S are assumed to be generated by a white noise process and liJ} = 0.2618 and liJ2= 0.5236. The ideal network is one that is not redundant, in other words, the specific network with the fewest parameters and which represents the data the best, is preferred (Hwang and Ding, 1997:748-757). The data is divided into a training set (first 70% of data), a cross-validation set (10% of data excluding the training set) and the rest is used as a test set (last 20%). By calculating the MSE for each model, the one with the smallest MSE value was selected. This was done for both the feed forward neural network, which uses backpropagation, and the Elman recurrent neural network The feedforward neural network is a function of a fixed number of previous values of a time series, while the Elman recurrent neural network, implicitly depends on all previous observations of a time series. The feedforward models considered in this dissertation, performed better as can be seen from the MSE values in Figure 6.2. - -------------------1. 50 30 I I 'ii 20 I N • -CD 0 )( 40 1 OMSE:FF CD :s .MSE:ELMAN I ~ w 10 UJ :E 0 ~ty "\ty Model The labels on the horizontal axis denote the number of input nodes and the number of hidden nodes of the different models considered. For example: 3.2 indicates a neural network with three inputs and two hidden nodes. The number of parameters, W, in the fully connected model is calculated as follows: where nj is the number of input nodes and nh is the number of nodes in the hidden layer. (The term, one, that is added. provides for a bias or intercept in both the input and hidden layer.) Therefore the neural network model (6.3-1) with six input nodes and two hidden nodes, has 17 parameters or weights. This is acceptable because the total number of patterns in the training set should ideally not be less than 32W(Bishop, 1996:410), but 10Wis also an accepted norm in some cases. The software package of the Basic ModelGen™ 1.0 from Crusader Systems was used. In the training phase the input signals are passed through an activation function (see Chapter 3). The activation function used here is a symmetric sigmoidal function (see 2.5-7). The backpropagation method was used to calculate the gradient of the error function. The training stopped after 250 iterations. After the network has been trained, the model has to be evaluated. Procedures for evaluation differ from programme package to programme package. A few of the procedures available in the Basic ModelGen™ 1.0 from Crusader Systems, are discussed in Section 6.4. Figure 6.3 is a representation of the root mean squared error (RM()E), and therefore also the mean square error (MSE), of the training and cross validation sets during training. This gives the user an indication of how well the training progresses with each iteration or epoch and, should there be some default, training can be stopped immediately. 0.049 0.046 0.044 0.041' (5 l::0.038, w ~0.036- ~ \ \ 0.033- ; 0.031 ) t I 0.028' !I " ';' ; 1 1 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 _______ ,.,_. . "_,, __.•"._, Epochs .__ ,,.. ,,_,...__ ..__ .. ,_. . , . ! .1 The RMSE for both the training and cross-validation sets fall sharply at the beginning of the iterative process. The RMSE of the training set is lower and decreases steadily, while the RMSE of the cross-validation set shows a slight upward trend before it stabilises after about 130 iterations. This is an indication that the model should fit reasonably well to an independent data set from the same population. The investigation of the model residuals by means of the Fourier transform gives a spectrum that is not zero. This means the residuals may still contain some information of the underlying time series. The sensitivity analysis points out that the model is the most sensitive for the input variables Yt-1 and Yt-2• To inspect the sensitivity of the model output to the other input variables, Figure 6.4 can be observed. A graph of the output of the model and the desired outputs also gives an indication of the quality of the fit of the model. Figure 6.5 shows these results for a randomly selected section of the time series. The graph appears satisfactory because the model output is relatively similar to the target or desired output values. The last peak is overestimated by the model. The performance investigated further. of the model on weekends should be ----, j II I-To_... - - ••••. I Model: New Vt The following table provides the correlation coefficient, R, between the outputs predicted by the model and the actual desired output on the test set. The test set is used for model evaluation. This is data that was not used during the training phase. Standard Mean Deviation Feed Forw 0.096 0.341 Desired 0.035 0.341 Desi"ed 0.035 0.341 MODELS 95% Confidence R R2 UmitsforR 0.9821 0.9645 [0.980 - 0.984] The R of 0.9821 is quite high and lies in the 95% confidence interval. This indicates that the model fits the data well. The narrow interval is due to the large number of observations in the test set. The evaluation of the model has been done by means of v~ous methods of measurements. If this model is satisfactory, the forecasting process can take place. In Section 6.5 the forecasting of twelve hours ahead is done. The model (6.2-1) was estimated and evaluated in the previous sections. This model is now used to forecast Basic ModelGen™ 1.0 from Crusader Systems unfortunately is not a specialised time series analysis package. It only provides one-step predictions. For a given set of input variables, the predicted model value (6.2-1) can be calculated. It was decided to use the nalve-, the direct method and the bootstrap method (see Section 5.3 and 5.4) to forecast twelve electricity consumption values. A Fortran programme (see Appendix: Chapter 6), was used to produce forecasts and prediction intervals using bootstrap methodology. In the next section these three methods are implemented. = fey N(I),YN,cos[0,2618(N + 1)],sin[0.2618(N + 1)], cos[O.5236(N + 1)],sin[O.5236(N + 1)];w) = fCY N(2),Y N Cl),cos[O,26H{N + 2)],sin[O.261a:N + 2)], cos[0.523((N + 2)],sin[0.523((N + 2)];~ The following graph shows how this method performed. The forecast output values are compared with the target outputs for twelve hours. - e o -'¢ 0.+ .- E CD ::10 2.5 -------------------------.--i 2 ex o .. 1.5 i, II .- 3= .2 0 •• II (I),,"" u= ~as ~= .4 10.5 - o- -.JI:: u_ .!!! w 4.~ i • Target fTIDirect I I n, I I ~ I I ~ I xCb I I ,,~ I ,,'1, 4.~ 4.~ 4.~ 4.~ 4.~ 4.~ By inspection of the graph one can see that after two forecasts the forecast values predicted by the model are less than the target values. This can be a problem because it can cause an underproduction of electricity. The next method implemented is the direct method. For the direct method, a different model has to be estimated for each lead time. To forecast two steps, the neural network is trained to estimate Y 1+2 = g(Y I, Y,-l' cos[O,2618(t + 1)],sin[0.2618(t + 1)], cos[O.5236(t + 1)],sin[O.5236(t + 1)]; w) + &1+2 Yt+2 from .Yi: and .Yi:-J: Y Hi = h(Y, ,Yt-I ,cos[0,2618(t + l-l)],sin[0.2618(t + 1-1)], cos[0.5236(t + 1-1)],sin[0.5236(t + I-I)]; w) + CHi Figure 6.7 illustrates the performance of the direct method with once again the forecast output values compared with the target outputs for twelve hours. c o ;; .•...•... D.~ 2.5 --·--------·-.--------------- .. -----1 i E + ~ c! 2 H en c >< o .. u t:: 1.5 ~ .a-as~ ..~ -~ ~= 0 u_ .!! w I I DDirect 1 0.5 o .Target J , -\." Here the model- forecast values differ a bit less from the target values. To compare the naive method with the direct method Figure 6.8 is shown. .. e o ...-.-. + Q,"l:t E ::s8 (1)- ex o .. u= ~«S .- ~ ..u_= .!:! 0 -.lllI: .!! w 3 -..----.--------------------.----.-----~---------, 2.5 2 1.5 1 ~ I 'n r-.-T-a-r-g-et-' EJNa"ive mJDirect '--------' 0.5 - o- Figure 6.8 illustrates that the direct method performs better. The predicted values of the model for this method are for most of the time periods the nearest to the target. In this chapter a neural network was fitted to hourly electricity consumption. Two methods for determining the forecasts were discussed. The direct method performed better for most of the forecast values. Chatfield (1996) investigated alternative approaches to forecasting. The investigation of neural networks as a forecasting model showed that models with less parameters generally give better out-of-sample predictions than those neural network models with more parameters, even though this smaller models fit worse than less parsimonious models. Chatfield (1996) also found that wider prediction intervals may reflect model uncertainty better. Neural network software programs may be improved by including methods for determining prediction intervals. The same neural network model used in Chapter 5 is used to predict the electricity consumption for 1 to 24 hours ahead. A time plot of the data is given in Figure 6.9. Hourly electricity load d e man d tim e s e rie s 0.8 0.6 0.4 0.2 o -0.2 -0.4 -0.6 -0.8 Bootstrap methodology is used for prediction together with their standard errors and a 95% prediction interval. The results are illustrated in Figure 6.10. Predicted (Bootstrap ~ o .5 () "t: t) GI 'ii '0 GI ii hourly electricity load. approach applied to neural network.) '0 CIS 0 .2 -0 .5 () en - 1 -----Predicted -----·L95 -----·U95 I A linear regression model was also used to predict the electricity lo~d. The same input variables were used and it was assumed that the prediction errors are normal distributed when determining the prediction limits. In Figure 6.11 the prediction limits as well as the predictions are illustrated. Predicted (linear hourly electricity load regression model) 1 ~ (J °C (J •• ~ (I) 0.5 "C fCS .2 0 "C .!!! fCS (J U) -0.5 .•. -. -_ ... -1 I ----Predicted - - - - - 'L95 - - - - - 'U95 II It can be seen that the forecasts obtained by the neural network (Figure 6.10) and the linear model (Figure 6.11) compares well. In the case of the linear model the predictions and the prediction intervals are smoother than that obtained by the bootstrap method for the neural network model. This is because the bootstrap method is data driven and does not rely on any assumptions on the probability distribution of the data. PREDICTNN is an artificial feed forward neural network that does one-step and multistep time series prediction, as well as the calculation of standard errors and prediction intervals based on bootstrap methodology. (Reference). Minimum hardware specifications: The program runs under DOS Version 4. Memory: 32 MB CPU: 486 Disk space: 1 MB Disclaimer: The authors do not take responsibility in any terms for any damages that may result from using PREDICTNN. Create a new directory, say TSPREDICT. Copy PREDICTNN.EXE to the TSPREDICT directory. All input and output files will be written to or read from TSPREDICT directory. Go to DOS Window. cd\path to TSPREDICT Type PREDICTNN. PREDICTNN requires parameter input (from screen or ASCII file) and the training data set (from ASCII file). The user can either specify the parameters by typing it in as prompted by the program, or the parameters can be read from an input file: I Enter 0 if input from screen or 1 if input from file:--] If the user choose to specify the parameters by typing it in from the screen, the following values must be entered as prompted by th~ program: Enter Enter Enter Enter Enter Enter Enter Enter Enter Enter the the the the the the number number number number number number of patterns in the training set: of bootstrap samples to·be drawn: of input variables: of hidden neurons: of lags of the dependent variable: of forecasts to be calculated: lOO*(l-?)% confidence interval: the maximum number of iterations: the number of nn models to fi t: the input file name: It is assumed that the training set is the first pattern in the data file up to the number of patterns in the training set as specified. The number of bootstrap samples to be drawn refers to the number of samples to be used to calculate the prediction interval. A minimum of 1000 bootstrap samples is recommended in the literature. Note that an A1~ is trained on every bootstrap sample. If you only want predictions and standard errors, you may specify a value of3 for this parameter. The program will draw 100 bootstrap samples from the original time series to calculate predictions and standard errors. The number of input variables is the number of independent! explanatory variables in the model including the number oflagged values of the dependent! output variable. The dependent variable is a time series. The number of lags of the dependent variable is the autoregressive order of the model. Number of hidden neurons - number of neurons in hidden layer. Number offorecasts to be calculated - size of forecasting window. 100*(1-?)% confidence interval- a value of 0.05 will, for instance, specify a 95% confidence coefficient. Maximum number of iterations - iterative optimization algorithm will stop after the maximum number of iterations has been reached, even if convergence to the global minimum has not taken place. More complex problems require more iterations. In relatively small problems, (20 parameters), 60 iterations should suffice. The number of nn models tofit ~efers to the number of ANN's to be trained from random starting weights. The network with the lowest mean square error is then selected. Values between 3 and 10 should suffice. The input file name is the name of the data file. (Not longer than 20 characters). The values entered are automatically writt.en to the file PARAMB.DAT. \ If the parameters are read from a file, the program will read the parameter values from the ASCII file PARAMB.DA T one value per line in the following order: NTRAIN: Number of patterns in the training set. NBOOT: Number of bootstrap samples to be drawn. NINP: Number of input 'variables. NHID: Number of hidden neurons. NLAGS: Number of lags of the dependent variable. LEAD: Number of forecasts to be calculated. PALPHA:IOO*(l-?)% confidence interval. MAXITE: Maximum number of iterations. NFIT: Number of neural network models to fit. FNAME: Input file name. The data file is an ASCII file. The data are real numbers separated by one or more blanks. The name of the data file is specified by the user (see input parameters). The first pattern in the training set is given in the first row of the data file, starting with the output variable, the independent variables (if any) the first lag of the output variable, the second lag of the output variable, and so on. More than one line of data may be used for each pattern in the training set. The computer will go to a new line after a pattern has been read. FORECAST.OUT Format of output: Forecast lead (e.g. 1 for one step), forecast, standard error of forecast, lower and upper prediction limits, length of prediction interval. BOOT.OUT Format of output: Bootstrap sample number, value of root mean squared error for trained network (on training set). OPTIM.OUT Work file used by optimization algorithm~ This file may have useful information on optimization problems. Iteration results are given for the last network trained. Neural network architecture: Pre processing: Input and output variables are scaled to (-I: I). Initialization of weights: A set of weights are generated using a random number generator. In order to prevent local minimum problems, a number (as specified by user) of ANN's are trained from random points, and the best ANN is selected. One hidden layer with sigmoidal activation function. Error function: Root mean squared error. One output. Number of input variables (NINP) and number of nodes in hidden layer (NHID): NHID*(NINP+2)<100 Number of patterns in training set (NTRAIN): NTRAIN<IOOO (NINP+ 1)*NTRAIN < 6000 Number of forecast lead times (LEAD): LEAD<100 Number of bootstrap samples (NBOOT): NBOOT < 2000 LEAD*NBOOT < 10000 The data file ELECS.TXT contains 716 patterns of an electricity load demand time series. Each pattern consists of the output variable and 8 input variables including two lagged values of the output variable. Suppose we want to train a network with two nodes in the hidden layer on the first 400 patterns and produce 5 forecasts with 95% prediction limits, using 1000 bootstrap samples. 400 1000 8 2 2 5 0.05 40 3 elecs.txt To run the program, type PREDICTNN. The pr~m will estimate the required running time. If you want to terminate the program, hit 19!21 ~. . The output file are FORECAST. OUT and BOOT.OUT. The temptation exists to use neural networks as "black boxes", and scientists often succumb. This is very unfortunate because a neural network is a powerful statistical device, used for many different problems. Statisticians should provide the knowledge to implement statistical methods together with neural networks to bring out its best performances. In this dissertation, neural networks were used to analyse a time series and especially in request of the problem of forecasting. Statistical methods, used for time series analysis, are usually linear but may not succeed in forecasting a non-linear data set. This study specifically investigated the use of ARMA model in comparison with the neural network model for the problem of forecasting. Two time series, namely a generated AR(2) and an electricity consumption time series, were used to illustrate the process of time series analysis. Box and Jenkins proposed three phases for the analysis of a time series: identification, estimation and evaluation. These phases were given for ARMA models and also applied to neural network models in Chapter 4. To illustrate the use of neural networks for the problem of forecasting, electricity consumption data as a time series was used. A neural network model was estimated to predict twelve hours ahead. The model for electricity consumption can be improved. Different forecasting methods suitable for non-linear models were discussed A program was developed to detennine prediction intervals for the examples of the linear model and the neural network model by using the bootstrap method. The neural network program package that was used in this study can only calculate one-step predictions and does not permit the implementation of the bootstrap method. Calculation of multi-step forecasts and prediction intervals should be implemented in neural network software. Neural. ~etworks as a versatile, adjustable tool are highly recommended. This study focussed on the identification, estimation and evaluation of neural network models for time series, and forecasting. Many of the standard techniques in statistics can be compared with neural network methodology, especially in application with large data sets. Further research includes the use of bootstrap methods to calculate predictions, standard errors and prediction intervals for the forecasts. Bootstrap techniques can be implemented in neural network computer software packages. Research in the application and comparison by use of bootstrap techniques with neural networks are recommended. ANSUJ, A.P., CAMARGO M.E., RADHARAMANAN (1996). Sales forecasting R., and PETRY D.G. using time series models and neural networks. Computers and Indu~trial Engineering, 31(1,2), 421-424. BISHOP, C.M. (1996). Neural networks for pattem recognition. Oxford University press, New York. BERTHOLD, M., HAND, D.J., (1999). Intelligent data analysi/t: An introduction. Springer- Verlag. BROWN, B. and MARIANO, R (1989). Residual based stochastic predictors and estimation in non-linear models. Econometrica, 52, 321-343. BORAINE, H (2000). Prediction intervals for forecasts of neural network models for time series. Proceedings of the second ICSC Smymposium on Neural Computation, Berlin, Germany (23-26 May, 2000). BO~ G.E.P., JENKINS, G.M and REINSEL, G.C. (1994). Time Serie.~ Analysis: Forecasting and Control. Third Edition, Prentice Hall, Englewood Cliffs, New Jersey. CHATFIELD, C (1996). Model Uncertainty and Forecast Accuracy. Journal ofForeca~ting, 15,495-508. CHENG, B. and TITTERINGTON, D.M (1994). Neural networks: A view from a statistical perspective. Statistical Science, 9(1),2-54. CONNOR, J.T. (1996). A robust neural network filter for electricity demand prediction. Journal ofForeca~ting, 15(6),458(22). CRUSADER guide, 1-142. SYSTEMS (Pty)Ltd. (1998). Basic ModeIGen1MJ.O user's DE JONGH, P.J. and DE WET, T. (1994). An Introduction to neural network:;;, 1-19. FLETCHER, R (1987). Practical Methods of Optimization (Second ed.). New York: John Wiley. GALPIN, J.S. (1997). Investigation of improvement of load forecasting by using rainfall, temperature and humidity data. ESKOM Technology research report, 2-93. GEMAN, S., BIENENSTOCK E., DOURSAT, R (1992). Neural Networks and the Bias/Variance Dilemma. Neural Computation, 4, 1-58. HA YKIN, S. (1994). Neural network<; - A comprehen.vive foundation. Macmillan College Publishing Company, Inc., printed in USA. HWANG, J.T.G. and DING, AA (1997). Prediction intervals for neural networks. Journal of the American Statistical A.<;sociation, 92(438), Theory and Methods, 748-757. HAMILTON, J.D. (1994). Time serie.<;ana/y,<;i,<;. Princeton University Press, Princeton, New Jersey. JOHANNSON, E.M, DOWLA, F.U. and GOODMAN, D.M (1992). Backpropagation learning for multi-layer feedforward neural networks using conjugate gradient method. International Journal of Neural Systems, 2(4), 188-197. KRAMER, AH and SANGIOVANNl-VINCENTELLI, A (1989). Efficient parallel learning algorithms for neural networks. In Touretzky (Ed), Advance.v in Neural Information Processing Systems, I , 40-48. San Mateo, CA: Morgan Kaugmann. LE BARON, B. and WEIGEND, AS. (1996). Evaluating neural network predictors by bootstrapping, http://www.cs.colorado.edulhomeslandreas/public htmllHome.html.(January 1998). LEE, K Y., CRA, Y.T., PARK, J.R, (1992). Short term load forecasting using an artificial neural network. IEEE: Transactions on Power systems,7, 124-132. LIN, J.L. and GANGER, C.W.J. (1994). Forecasting from Non-linear Models in Practice. Journal offoreca.<;ting,13,1-9. MASAROTIO, G. (1990). Bootstrap prediction intervals for autoregressions. International Journalfor Forecayting, 6, 229-239. MC CULLOCH, W.S. and PITIS, W. (1943). A logical calculus of ideas immanent in nervous activity. Bulletin of Mathematical Biophysics, 5, 115133. MC LEOD, A.I. (1978). On the distribution ofresiduaI autocorrelation in BoxJenkins models. Journal of Royal Statistical Society, A40, 296-302. MINSKY, M. and PAPERT, S. (1969). Perceptrons. Cambridge, MA: MIT Press. ORR, KD. (1995). Use of an artificial neural network to predict lCO length of stay in cardiac surgical patients. Society of Critical Care Medicine. San Francisco, CA, January 1995. .l~ARK, D.C., EL-SHARKAWI, M.A., MARKS II, R.I., ATLAS, L.E., DAMBORG, MJ., (1991). Electric load forecasting using neural networks. IEEE Transactions on Power System~, 6(2), 442-449. PENG, T.M, HUBELE, N.F. and KARADY. G.G. (1993). An adaptive neural networks approach to one week ahead load forecasting. IEEE transactions on Power Systems, 8(3), 1195-1203. RIPLEY. B.D. (1993). Statistical aspects of neural networb. Invited lectures for SemStat, Sandbjerg, Denmark, 25-30 April 1992. To appear in the proceedings to be published by Chapman & Hall (January 1993). RIPLEY. B.D. (1994). Neural networks and related methods of classification. Journal of the Royal Statistical Society, 56(3), 409-456. SARLE, W.S. (1996). Neural Network and Stati.~tical Jargon. [email protected], (April 29, 1996). SCHALKHOFF, R (1992). Pattern Recognition: Statistical, Structural and Neural approaches. John Wiley and Sons, Inc., Canada. SNYMAN. J.A. (1982). A New Dynamic Method for Unconstrained Minimization. Applied Mathematical Modelling, 6, 449-462. SNYMAN, J.A. (1983). An updated version of the Original Leap-frog, Dynamic Method for Unconstrained Minimisation (LFOP1(b)). Applied Mathematical Modelling, 7, 216-218. STERN. as. (1996). Neural networks in applied statistics. American Statistical Association and the American Society for Quality Control: Technometrics, 38(3),205-213. TAM, K.Y. and KlANG, MY. (1992). Managerial applications of neural networks in the case of bank failure predictions. Journal of Management Science, 38(7), 926-947.

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Download PDF

advertisement