Black-Box Modelling of Integrated Circuits for Electromagnetic Compatibility Simulations
KVALIFIKACIJSKI DOKTORSKI ISPIT 1 Black-Box Modelling of Integrated Circuits for Electromagnetic Compatibility Simulations Marko Magerl University of Zagreb Faculty of Electrical Engineering and Computing Unska 3, Zagreb, Croatia Email: [email protected] Index Terms—black-box modelling, artificial neural networks, echo state networks, electromagnetic compatibility, direct power injection, circuit simulations. output weights (trained) input weights (trained) u(n) y(n) ... Abstract—This paper presents an overview of the state-of-the-art in black-box modelling with focus on modelling integrated circuits using artificial neural networks. The trade-offs of different model architectures are discussed, including the model building time, training algorithm complexity, and model execution time in circuit simulators. An interchangeable black-box model of a bandgap reference block for direct power injection simulations, based on the echo state network architecture, is presented and discussed. Guidelines for the future work in the research of the topic are given. output neurons input neurons internal neurons Fig. 1: Generic diagram of a single-layer feedforward neural network (SLFN). I. I NTRODUCTION T HE growing complexity of integrated circuits (IC) has made the black-box modelling approach very appealing to the IC industry for reducing the complexity of the circuit simulation models. The electromagnetic compatibility (EMC) simulations are becoming increasingly important for IC manufacturers as they can reduce the number of prototypes in the development cycle of a product. The idea behind the black-box, or behavioural, approach is to model the behaviour of the modelled system without using the information about its inner structure . The advantages of this approach are the model order reduction (MOR), which can enable the reduction of the circuit simulation time, and the intellectual property (IP) protection, which makes the models free for the distribution to third parties. In this context, a research project is established that investigates the applicability of the black-box approach to modelling integrated circuits for EMC applications. The primary focus of this research project is to use artificial neural networks (ANN) to model electronic circuits in the time-domain. The goal of this paper is to present the state-of-the-art in black-box modelling, as well as to present initial results of the research project. This paper is organised as follows. An overview of the black-box modelling theory is given in Section II. Section III presents the state-of-the-art in black-box modelling of circuits and for other applications. The initial results of this research are presented in Section IV, and the guidelines for future work are given in Section V. The paper is concluded in Section VI. II. B LACK -B OX M ODELLING T HEORY Artificial neural networks (ANN) are networks of interconnected units – neurons. Each neuron is associated with a time-domain signal, the neuron activation which is defined by the interaction between the neuron and the neighbouring connected neurons. Different neural network architectures are defined by the configuration of the neurons in the network and by the weights of the connections between them. Two basic types of ANN exist: feedforward and recurrent neural networks. The feedforward neural networks, shown schematically in Fig. 1, do not contain any circular paths between the neurons. It is proven that the single-layer feedforward neural network (SLFN), shown in Fig. 1, has the potential to approximate any continuous function , . This is referred to as the universal approximation property. The feedforward architectures do not retain any information about previous input values, therefore, they cannot be used to model systems with memory, such as most electronic circuits. In order to incorporate the temporal component in the model, feedback connections between neurons are added, resulting in recurrent neural networks (RNN). In  it is shown that the internal dynamics of an RNN with random fixed weights resemble a Markovian process if the the input connection weights have small values. This observation forms the basis for the echo state network (ESN) approach, as shown in . The echo state network is discussed in Section II-D. KVALIFIKACIJSKI DOKTORSKI ISPIT 2 B. ANN training algorithms Fig. 2: Flow chart for building a generic artificial neural network (ANN) model. A. Model building The flow chart for building a generic ANN model is shown in Fig. 2. The hyper-parameters are parameters specific to a particular ANN architecture, e.g. the number of neurons, the weight distributions, the activation functions, the leaking rate, the number and value of delays of the input signal, the sparsity of the neuron connections. Adjusting the hyper-parameters is usually done in a trial-and-error process. Generating the ANN corresponds to initializing the data structures that define the architecture of the particular ANN. Within this research project, the ANN models are coded in R Python and MATLAB . In order to model a particular function or system, the ANN is trained using data from the training dataset – a collection of input-output values of the modelled system, obtained either from simulations, e.g. SPICE simulations of an electronic circuit, or from measurements. An overview of the ANN training algorithms is presented in Section II-B. In order to test its generalisation properties, the trained model is verified using a dataset that does not include the data points used in training. If the model performance is not satisfactory, the hyper-parameters are re-adjusted and the flow is reiterated. Once a satisfactory model is obtained, the ANN is implemented in a hardware description language (HDL) in order to be usable in a circuit simulator. This topic is discussed in Section II-C. Once a particular architecture of the ANN is chosen, the ANN is trained by adjusting the weights of the connections according to the learning data (training dataset) based on a training algorithm. The “classical” training algorithms are iterative methods based on gradient descent on an error function defined between the desired and the actual model output. Commonly used examples include the Bayesian regularisation backpropagation algorithm , and the Levenberg-Marquardt algorithm . These algorithms are readily available in software packages for neural networks, e.g. the Neural Network ToolboxTM in R MATLAB . An overview of the gradient descent algorithms is available in . These algorithms have inherent limitations for recurrent neural networks. E.g. such a problem is bifurcations, a phenomenon where a small change in a parameter of a nonlinear system results in the fundamental change of the nature of its state trajectories . Bifurcations that occur during an iterative learning process can significantly slow down the learning process, or completely prevent the algorithm from reaching the optimal solution . Further problems include the information latching problem, caused by the error function vanishing in time, which limits the memory ability of the RNN ; the convergence can be very slow for large networks ; the algorithms may get stuck at local optimum points that are not optimal if the optimization problem is not convex . The above training methods are offline methods, because the complete training dataset is available during learning. Online learning methods adjust the connection weights as the training data is being applied to the network, and also during the execution of the model. The advantage of this approach is the ability of the model to adaptively respond to disturbances during the model execution. Online algorithms include the least mean square (LMS) and the recursive least squares (RLS) algorithms. In  the RLS algorithm with the adaptive forgetting factor is used to adjust the weights of an echo state network. In  the training algorithm is taken a step further, by using an evolutionary algorithm that adjusts not only the connection weights of an echo state network (ESN), but it also chooses the optimal hyper-parameters. Similarly, in  the particle swarm optimisation (PSO) evolutionary algorithm is used to adjust the output layer weights of an ESN in an online fashion. C. Model implementation in circuit simulators A number of hardware description languages (HDL) supports analog and/or mixed signals: Verilog A, VHDL-AMS, MAST . In the context of this project, the R R Cadence Spectre circuit simulator is used to simulate black-box models coded in Verilog A. An example of implementing a 3-layer feedforward neural network in Verilog A is given in . The presented methodology can be extended to different types of neural networks, KVALIFIKACIJSKI DOKTORSKI ISPIT D. Echo state networks This section presents an overview of the echo state network (ESN), and Section IV presents the initial results of this research project based on using the ESN. The echo state network, first proposed in , is a recurrent neural network that has fixed input connections and fixed internal connections, both with randomly generated weights. A generic diagram of the echo state network is shown in Fig. 3. The activations of the hidden neurons represent “echoes” of the input sequence history due to the echo state property (ESP) of the ESN. The ESP is conceptually based on the architectural bias of the RNN towards a Markovian process . The existence of the ESP can be mathematically guaranteed by imposing certain conditions on the internal weight matrix of the ESN, as it is discussed in . The rich dynamics in the hidden layer of the ESN, available due to the ESP, enable constructing the output of the ESN using linear regression only on the output connection weights. This property makes the ESN training algorithm extremely fast and robust compared to the RNN training algorithms presented in Section II-B. An additional benefit of having a fast training algorithm is a significantly reduced time required for tuning the hyper-parameters according to the flow chart in Fig. 2. Due to these favourable properties, the echo state network is chosen as the starting point for this research project. The numerical properties of the trained ESN are determined by the linear regression method used in the ESN training process. Most researchers use ridge regression, also known as regression with Tikhonov regularisation, e.g. , , or the simple pseudo-inverse solution, e.g. . These methods are based on minimising the L2-norm of the error vector. An alternative method is the LASSO regression , which minimises the sum of the L2-norm of the error vector and the L1-norm of the output weights vector. It is shown in  that the LASSO method forces most output weights to zero, internal weights input weights output feedback weights output weights (trained) Ub y(n) u(n) ... e.g. the ESN. Furthermore, the HDL code generation can be automated using a scripting language, e.g. Python. The two basic figures of merit for black-box models are the building time and the execution time. In the context of using the black-box models in circuit simulators, the model execution time is critical because the primary goal of using the black-box approach is the reduction of the simulation time. It is empirically observed that the number of neurons has the main influence on the execution time, because the neurons are implemented in a HDL as interconnected electrical nodes. Therefore the number of neurons determines the number of simultaneous nonlinear equations the simulator has to solve. Another consideration for the execution time is the smoothness of the signals produced by the model because discontinuities or sudden jumps in the signals cause the simulator to reduce the timestep, increasing the simulation time. The execution time can also be significantly improved by pre-compiling the HDL model in a low-level language such as C. This conversion can be done automatically, e.g. using a feature of R R the Cadence Spectre circuit simulator, called the compiled model interface (CMI). 3 output neurons input neurons x(n) internal neurons Fig. 3: Generic diagram of the echo state network (ESN). generating a sparse solution. Finally, the elastic-net regression is a method that combines the above two approaches and their favourable properties . Adding a small noise term to the training data obtained from simulations is shown to improve the stability and generalisation properties of the trained ESN according to . Although without a formal proof, the reasoning behind this idea is that adding noise to the numerical values of the training dataset reduces the risk of the ESN to overfit the training data, degrading its generalisation properties. The echo state network quickly became popular in the research community due to its conceptual simplicity, as well as its fast training algorithm and good generalisation properties. However, it is shown that many theoretical results are not well understood by the users of the ESN. For example, the original work by H. Jaeger  gives an “almost sufficient” condition for the echo state property that involves globally rescaling the internal weight matrix of the ESN such that its spectral radius is less than one. Many research papers, e.g. , , , readily use this condition citing it as the sufficient condition for the existence of the ESP. In  Jaeger shows that examples can be found where the ESP does not hold even if the spectral radius of the “almost” sufficient condition is satisfied. In that work, as well as in , alternative methods for constructing the internal weight matrix are proposed and evaluated. In  a better intuitive understanding of the ESN is attempted by constructing several deterministic ESN configurations and testing them using the standard benchmark tests. In  the average state entropy (ASE) is introduced as a quantification of the ESN reservoir “richness”, and the ESN architecture is interpreted from the standpoint of signal processing of basis functions. In  the theoretical properties of the ESN are further investigated. The reason why the “almost sufficient” condition for the existence of the ESP often holds is elaborated mathematically. The ESN has two types of feedback connections: the internal feedback connections in the hidden layer, and the feedback connections from the output layer to the hidden layer. While the internal feedback connections are a fundamental part of the ESN, the output feedback connections are optional . The mathematical conditions that guarantee the echo state property given in  apply only in the case the output feedback connections are not used. In  the stability of KVALIFIKACIJSKI DOKTORSKI ISPIT the ESN with output feedback connections is investigated. The authors propose regularising the output layer and the hidden layer using Tikhonov regularisation. In  an ESN with output feedback connections is replaced by an ESN without feedback connections, however the training process is modified to include incremental changes in the internal weight matrix that reflect the influence of the output feedback connections of the original ESN. In this way, the stability of the ESN is improved by removing the output feedback connections, while retaining the favourable properties of using the output feedback connections. In  the experimental results of  are re-evaluated, and a critical assessment is made. The stochastic nature of the ESN generation process is shown to influence the performance of certain ESN instances that have the same hyper-parameters. In  the influence of the spectral radius on the performance of an ESN with 50 neurons is evaluated experimentally, showing that using the spectral radius close to 0.8 gives optimal results. In  the short term memory of the ESN and its continuous-time counterpart, the liquid state machine (LSM), is assessed by introducing a simplified ESN (SESN) and comparing its memory performance to the ESN and the LSM. III. S TATE OF THE A RT Coming from the area of computer science, artificial neural networks are traditionally evaluated using standard benchmark test cases, such as predicting discrete time-series: e.g. a 2nd order dynamical system, such as in , the NARMA series , or the Mackey-Glass chaotic series , , . The reservoir sizes used for these benchmarks are in the order of several hundred neurons (compare ). This paper will focus on examples from literature in which neural networks are applied to practical engineering problems. As it is discussed in Section II-C, keeping a low number of neurons is key to achieving the simulation speed-up in the circuit simulator. In  neural networks are used for the model predictive control (MPC) of an unknown nonlinear system. The ESN architecture is used for system identification, and the control problem is formulated as a quadratic programming (QP) problem that is solved using a single-layer neural network, referred to as the simplified dual neural network (SDN). In  and  the ESN is used in a harmonic extraction block of a shunt active power filter (SAPF). The SAPF eliminates the higher harmonics from the output current caused by nonlinear loads in power systems. The ESN is used to extract the fundamental component of the load current. Although 100 neurons are used in  to obtain good performance, it is shown in  that an ESN with only 30 neurons is sufficient. The achieved performance is shown to be superior to a feedforward neural network, referred to as the multilayer perceptron (MLP). In the electronic circuit modelling applications,  presents the model of an output buffer of a digital IC for signal integrity simulations. A combination of feedforward and recurrent neural networks is used. Single-layer feedforward neural networks (SLFN) with 4 neurons are used to model the 4 the I-V characteristics of the buffer transistors, and the total buffer output current is modelled by weighting the currents of the output transistors, where the weights are implemented as recurrent neural networks with 10 neurons each. This model overcomes the limitation of using the I/O buffer information specification (IBIS)  that cannot properly model analog nonlinear behaviour, such as ringing and attenuation. In  a similar test case is presented using an ESN with 30 neurons. The stability of the model is shown by plotting the locus of the linearised model eigenvalues. In  a 3-layer feedforward neural network is used to model the response of a digital IC to a radiofrequency (RF) disturbance on its supply pin. The input of the model is the RF disturbance signal and its derivative and, in order to have a constant number of input samples per period for varying frequency, the sampling rate of the input data is adjusted to the input frequency of the signal. The model is implemented in the circuit simulator as a mathematical expression. In  a 2-layer feedforward neural network is used to model a voltage regulator for direct power injection (DPI) simulations. In order to enable the model to interact with the previous stage, the input impedance of the circuit is modelled as well. The training dataset is obtained by sampling the RF frequency–power design space using the LOLA-Voronoi adaptive sampling algorithm , which enables putting more focus on the more nonlinear regions of the design space. IV. R ESULTS A. EMC modelling test case In this section, the initial results of this research project are presented. A circuit model for direct power injection (DPI) simulations is built and evaluated. DPI is an electromagnetic compatibility precompliance test for measuring the immunity of integrated circuits to electromagnetic interference (EMI). It is defined in the IEC-62132-4 standard . A radiofrequency (RF) disturbance signal of increasing forward power is injected into the IC pin under test until the functionality of the circuit fails according to a predefined criterion. The frequency of the RF disturbance is swept between 150 kHz and 1 GHz. A testbench for DPI simulations is shown in Fig. 4a. A bias-tee circuit is used to superpose an RF disturbance signal vRF on a DC voltage VDC , in this case on the power supply of the device under test (DUT). Being able to simulate the performance of an IC on the DPI test before chip fabrication is extremely useful for IC manufacturers, as it reduces the development cost of the product and shortens the time-to-market. In  an electrical model of the measurement setup for DPI is presented for use in SPICE simulations. Included in the model are the IC package, the injection probe, the printed circuit board (PCB), the capacitors for decoupling and the IC itself. However the time-domain simulations in  are shown to be very slow due to the complex netlist of the test IC. A similar pattern is observed throughout the IC industry: the DPI simulations are inherently slow, especially if the extracted parasitics are included in the netlist, which is often required in order to observe realistic EMC problems in the IC. KVALIFIKACIJSKI DOKTORSKI ISPIT 5 iin 430 µH VDC iout v(vin ) + 6.8 nF vin VDD DUT VBG GN D − VDD GN D VDIGREG BLKS VOU T VDD OB GN D VOU T VIN IIN,BG VIN,BG VBG − Fig. 5: Black-box model architecture. (a) The top-level testbench. VOUT,BG i(vin ) vout 1 MΩ 0.001 Ω VDD BG GN D − + vRF + VBG (b) The schematic of the DUT. Fig. 4: The schematics used for DPI simulations. In this research project, an interchangeable black-box model of a bandgap reference block for direct power injection (DPI) simulations is built using the echo state network (ESN) approach. The goal is to improve the simulation-speed of the DPI simulations in the circuit simulator, as well as to hide the structure of the test IC. Also, since the block is used in a larger schematic, the goal is to properly load the previous stage by modelling the input impedance. The schematic of the complete circuit is given in Fig. 4b, where the modelled circuit block is labelled as BG. The remaining blocks, the bulk switch (BLKS) and the output buffer (OB), are netlist blocks. The architecture used for this test case is presented in Fig. 5. In order to make the BG block model interchangeable within the schematic, an approach similar to  is used to model the input impedance. However, instead of modelling the value of an impedance block as a function of the RF frequency and amplitude, the modelled quantity is the input current waveform in the time-domain as a function of the input voltage. Therefore, the input impedance is modelled using a voltage controlled current source (VCCS). The output of the circuit is modelled using a voltage controlled voltage source (VCVS). It was shown experimentally that the presented architecture can properly load the netlist block BLKS. Furthermore, another black-box model that has the same architecture would be properly loaded as well. The operating conditions for the model are chosen to include the frequency range between 30 MHz and 300 MHz, and the forward power range from −20 dBm to 0 dBm. This design space can be represented in a 2D plane. The input impedance is modelled using an ESN with 20 neurons, while the output voltage is modelled using an ESN with 35 neurons. The training dataset consists of a series of uniformly sampled time-domain waveforms with the sampling time of 200 ps, containing approximately 2 · 106 data points. The time required to train the ESN is approximately 40 seconds using a machine with an TM R Intel Core i5-4460 CPU @ 3.2 GHz and 8 GB of RAM. The achieved accuracy of the model is presented in Fig. 6. Figs. 6a, 6b show the MSE of the input impedance model, i.e. the modelled input current waveform IIN,BG and the input voltage waveform VIN,BG in Fig. 4b. The MSE for IIN,BG varies between 5.772·10−13 and 2.713·10−9 , and between 1.658·10−7 and 1.348·10−3 for VIN,BG . It should be noted that the input voltage VIN,BG is modelled implicitly, through the interaction between the netlist of the BLKS block and the VCCS of the BG block model. Figs. 6c, 6d show the MSE of the modelled output voltage of the BG block, and the relative error of the mean value of the output voltage of the entire circuit. The mean value of the bandgap reference output voltage is often used as a failure criterion in the DPI test. The MSE for VOU T,BG varies between 6.187·10−6 and 1.501·10−3 , and the relative error of the mean output voltage varies between 0.009% and 4.644%. In order to give proper interpretation of the MSE values, the time-domain waveforms for the worst-case MSE points of the modelled input impedance and the modelled output voltage are shown in Fig. 7. The simulation speed-up of the BG block model is presented in Table I. The simulation times for the BG block without the BLKS and OB blocks, Tmod,BG and Tmod,DU T for the netlist and the model respectively, is evaluated by driving the BG block using the input waveforms from file. All simulations are run for 100 periods of the RF disturbance. B. Discussion The presented test case shows the achieved model accuracy is comparable to the results reported in . Compared to the time required to build the model in , which is in the order of several hours, the presented ESN model is trained extremely fast. The comparison between the simulation time for the BG block netlist and the model (shown in columns Tref,BG and Tmod,BG in Table I) shows an increase in the netlist simulation time in the order of 5 times with increasing power KVALIFIKACIJSKI DOKTORSKI ISPIT MSE >10−9 ] 6 MSE −3 >10 @ fR F 0 +] PRF>G%[email protected] (a) The mean square error of the VCCS model: IIN,BG . MSE −3 >10 @ PRF>G%[email protected] (b) The mean square error of the VCCS model: VIN,BG . err >@ fR F 0 +] fR F 0 +] PRF>G%[email protected] (c) The mean square error of the VCVS model: VOU T,BG . fR F 0 +] PRF>G%[email protected] (d) The error of the mean value of the output voltage: VBG . Fig. 6: The MSE performance of the black-box model. levels. This can be interpreted as the consequence of the transistors-level models entering the nonlinear regime for higher power levels, which increases the complexity of the netlist. The ESN model, on the other hand, shows a minimal increase in simulation time with increasing power levels. This is an expected result because the model has the same complexity regardless of the forward power level. It should be noted that the model is built using the training dataset obtained by uniformly sampling the frequency–power design space. This simplistic approach is used to obtain an initial estimation of the applicability of the ESN for building circuit models for DPI simulations. As it can be seen in , using the LOLA-Voronoi sampling algorithm  enables obtaining a training dataset that puts more focus on the nonlinear regions of the design space, while avoiding undersampling the linear regions. V. F UTURE WORK This section presents an overview of the modelling methods that are related to the ESN, or a direct modification of the ESN. The promises and trade-offs of these methods are discussed. Their application for this research project is to be investigated in future work. A. Extreme learning machines As it can be seen in the examples presented in Section III, the feedforward neural network is frequently used for functional approximation. Having the universal approximation property and a simple structure, circuit models based on the feedforward neural network promise significant simulation speed-up in cases where they are applicable. The main drawback of the feedforward neural network are the training algorithms that suffer from the problems described in Section II-B. In  a training algorithm that promises to solve this problem is proposed, called the extreme learning machine (ELM). It is mathematically shown that single-layer feedforward neural networks (SLFN) with random fixed input weights and bias values retain the universal approximation property of the SLFN with all trained weights, if the neuron activation function is infinitely differentiable. Fixing the input weights and bias values reduces the training algorithm to a linear regression task on the output weights, making it extremely efficient. This approach is similar to the algorithm used in the ESN. A comprehensive feasibility study of the ELM can be found in , , and an overview of the ELM applications in different problem domains is available in . B. Support vector machines The support vector regression machine (SVR) is a neural network that uses kernel functions to transform the regression problem to a higher dimensional space. The motivation is to obtain a linear problem at the cost of increased dimensionality. The main drawback of SVR is that the number of neurons KVALIFIKACIJSKI DOKTORSKI ISPIT IIN,BGP$ 02'(/ '(6,5(' (5525 0+] G%P error$ 7 tQV TABLE I: The simulation times required for simulating 100 periods. Columns Tref,BG and Tref,DU T are the simulation times obtained using the transistor-level models of the BG block and of the entire DUT, respectively. Columns Tmod,BG and Tmod,DU T are the simulation times obtained using the black-box model of the BG block. fRF [MHz] PRF [dBm] Tref,BG [s] Tmod,BG [s] Tref,DU T [s] Tmod,DU T [s] 30 -20 -10 0 5.9 20.4 47.8 4.6 4.9 6.5 13.3 23.0 30.6 14.2 18.9 26.8 100 -20 -10 0 7.9 17.3 40.3 4.0 5.2 7.1 16.6 26.0 34.0 15.8 20.8 29.5 300 -20 -10 0 6.6 10.7 18.9 6.7 4.5 8.8 14.2 21.7 30.1 14.3 18.1 28.1 (a) Input current. fRF = 300 MHz, PRF = 0 dBm. 0+] G%P VOUT,BG9 02'(/ '(6,5(' (5525 errorP9 tQV (b) Output voltage. fRF = 30 MHz, PRF = 0 dBm. Fig. 7: The time-domain responses of the input current and output voltage signals of the modelled IC block taken at their worst-case MSE points, respectively. The instantaneous error is shown. (support vectors) depends on the size of the training dataset due to the architecture of the SVR. In the case of the echo state network the number of neurons is not inherently dependent on the size of the training dataset. In  the SVR approach is modified to produce recurrent models in the time-domain with sparse solutions. The proposed approach is evaluated on standard benchmark tests for neural networks, presented in Section III. In future work the applicability of the SVR to the modelling that are not successfully handled by the ESN will be investigated, e.g. the startup sequence of a bandgap reference circuit. C. Time-warping invariant neural networks It was shown during this research project that it is extremely difficult to model signals that have fast transients followed by long periods of “flat” signal values, e.g. the startup sequence of a bandgap reference. The discrete-time implementation of any ANN implicitly assumes the timestep between every two samples is equal. The above mentioned signals have to be sampled with very short timesteps due to the fast transients, at the price of having extremely large datasets due to the long flat signals. Using non-uniform sampling rates is referred as time-warping. Examples of implementing neural networks that are able to model such datasets can be found in the literature. E.g.  introduces the time-warping invariant neural network (TWINN). This concept is based on replacing the implicit information about the timestep by the norm of the change in the signal between two consecutive non-uniform timesteps. In  the time-warping invariant echo state network (TWIESN) is introduced. Following the principle shown in , the ESN is modified for pattern recognition of highly time-warped patterns. So far the TWI concept has not been reported for modelling non-uniformly sampled time-series, which will be investigated. D. X-parameters Artificial neural networks are used as time-domain black-box models. In the frequency domain, the most commonly used black-box models are the S-parameters. As an already mature modelling method, many books cover the theory behind S-parameters, e.g. . The most appealing property of S-parameters is that they can be readily obtained by measurements using the vector network analyser (VNA), and subsequently used as a model in circuit simulators, e.g. Keysight Advanced Design System (ADS). The main drawback of S-parameters is that they can be used to describe only linear devices. Many methods have been proposed to extend the S-parameters to nonlinear devices, e.g. “hot” S-parameters , however without much success. In , the X-parameters are introduced as a true extension of the S-parameters for nonlinear devices. The X-parameters represent a mathematical framework for including the influence of each harmonic component on every other harmonic component, a phenomenon that occurs in nonlinear systems that is not included in S-parameters. Being a relatively new concept, and being proprietary to Keysight, the X-parameters are not yet accepted as an industry standard. Furthermore, the application of X-parameters in  KVALIFIKACIJSKI DOKTORSKI ISPIT is mainly for power amplifier design. However, the mathematical background behind X-parameters shows promise to be used as the theoretical basis for modelling the input impedance of circuits for EMC simulations in the frequency domain, analogously to the time-domain approach presented in Section IV. 8 reference block for computer simulations of the direct power injection EMC test, are encouraging. Finally, the guidelines for the future work are given. ACKNOWLEDGMENT This research is funded by ams AG. E. Echo state network modifications In  the decoupled echo state network (DESN) is introduced to tackle the stochastic nature of generating the ESN. The main idea is to use multiple ESN reservoirs in parallel with a lateral inhibition mechanism that reduces the correlation between the different reservoirs in order to enable the generation of more rich dynamics. The reservoir sizes used are in the order of 100, which is plausible for use in circuit simulators, and the experimental results show very good performance. In  an augmented echo state network for use in adaptive filtering is introduced, that can process complex-valued signals. The augmented ESN shows better overall performance than the standard ESN also for the real-valued signals in the one-step-ahead prediction tests. F. Concluding remarks The next steps of this research project should include investigating two paths. The first one is to explore how to build the EMC models directly from DPI measurements. This requirement follows from the fact that the circuit simulations for high forward power levels of the RF disturbance invariably break, or produce non-verified results. The challenge in this approach is to incorporate the DPI measurements, which are the measurements of power levels and do not include the phase information, in the ESN model building method. The second path is to explore the applicability of alternative neural network architectures overviewed in this paper for the EMC simulations. In particular, the circuit models should not only model the RF disturbance well, but also the normal functionality of the circuit as well. The example of modelling the startup sequence of a bandgap reference circuit, mentioned in Section V-C, showed to be a very difficult modelling task during this research project. VI. C ONCLUSION In this paper an overview of the state-of-the-art in black-box modelling is given. The theory of using artificial neural networks for building black-box models is presented, including generic model building methods and an overview of the training algorithms. Implementing the obtained models in commercially available circuit simulators is discussed. A more detailed overview of the echo state network is presented, due to its favourable properties that include an extremely fast and robust training method, excellent generalisation properties, and conceptual simplicity. Examples from the literature of using echo state networks and artificial neural networks in general and in engineering applications are presented. Initial results of this research project, specifically using the echo state network to produce an interchangeable circuit model of a bandgap R EFERENCES  J. Wood and D. E. Root, Fundamentals of Nonlinear Behavioral Modeling for RF and Microwave Design, ser. Artech House microwave library. Artech House, 2005.  K. Hornik, “Approximation capabilities of multilayer feedforward networks,” Neural Networks, vol. 4, no. 2, pp. 251 – 257, 1991.  P. Baldi and K. Hornik, “Universal Approximation and Learning of Trajectories Using Oscillators,” in 1995 Advances in Neural Information Processing Systems (NIPS), Nov 1995, pp. 451–457.  M. Makula, M. Cernansky, and L. Benuskova, “Approaches Based on Markovian Architectural Bias in Recurrent Neural Networks,” in SOFSEM 2004: Theory and Practice of Computer Science, ser. Lecture Notes in Computer Science. Springer Berlin Heidelberg, 2004, vol. 2932, pp. 257–264.  M. Cernansky and M. Makula, “Feed-forward Echo State Networks,” in 2005 IEEE Int. Joint Conf. on Neural Networks (IJCNN ), vol. 3, July 2005, pp. 1479–1482.  F. Dan Foresee and M. T. Hagan, “Gauss-Newton approximation to Bayesian learning,” in 1997 Int. Conf. on Neural Networks, vol. 3, Jun 1997, pp. 1930–1935.  M. T. Hagan and M. B. Menhaj, “Training feedforward networks with the Marquardt algorithm,” IEEE Trans. Neural Netw., vol. 5, no. 6, pp. 989–993, Nov 1994.  A. Atiya and A. Parlos, “New Results on Recurrent Network Training: Unifying the Algorithms and Accelerating Convergence,” IEEE Trans. Neural Netw., vol. 11, no. 3, pp. 697–709, May 2000.  H. Khalil, Nonlinear Systems, ser. Pearson Education. Prentice Hall, 2002.  H. Jaeger, “The “echo state” approach to analysing and training recurrent neural networks - with an Erratum note,” Fraunhofer Institute for Autonomous Intelligent Systems, Tech. Rep., Jan 2001. (rev. 2010.).  S. Boyd and L. Vandenberghe, Convex Optimization. Cambridge University Press, 2004.  Q. Song, X. Zhao, Z. Feng, and B. Song, “Recursive Least Squares Algorithm with Adaptive Forgetting Factor Based on Echo State Network,” in 2011 9th World Congr. on Intelligent Control and Automation (WCICA), June 2011, pp. 295–298.  A. A. Ferreira, T. B. Ludermir, and R. R. de Aquino, “An approach to reservoir computing design and training,” Expert Systems with Applications, vol. 40, no. 10, pp. 4172–4182, 2013.  J. Fan and M. Han, “Online designed of Echo State Network based on Particle Swarm Optimization for system identification,” in 2011 2nd Int. Conf. on Intelligent Control and Information Processing (ICICIP), vol. 1, July 2011, pp. 559–563.  P. A. Duran, A Practical Guide to Analog Behavioral Modeling for IC System Design. Norwell, MA, USA: Kluwer Academic Publishers, 1998.  K. Suzuki, A. Nishio, A. Kamo, T. Watanabe, and H. Asai, “An Application of Verilog-A to Modeling of Back Propagation Algorithm in Neural Networks,” in 2000 43rd IEEE Midwest Symp. on Circuits and Systems, vol. 3, 2000, pp. 1336–1339 vol.3.  I. B. Yildiz, H. Jaeger, and S. J. Kiebel, “Re-Visiting the Echo State Property,” Neural Networks, vol. 35, pp. 1–9, 2012.  A. Rodan and P. Tino, “Minimum Complexity Echo State Network,” IEEE Trans. Neural Netw., vol. 22, no. 1, pp. 131–144, Jan 2011.  Y. Pan and J. Wang, “Model Predictive Control of Unknown Nonlinear Dynamical Systems Based on Recurrent Neural Networks,” IEEE Trans. Ind. Electron., vol. 59, no. 8, pp. 3089–3101, Aug 2012.  V. Ceperic and A. Baric, “Reducing Complexity of Echo State Networks with Sparse Linear Regression Algorithms,” in 2014 16th Int. Conf. on Computer Modelling and Simulation (UKSim), Mar 2014, pp. 26–31.  J. Xu, J. Yang, F. Liu, Z. Zhang, and A. Shen, “Echo state networks based method for harmonic extraction in shunt active power filters,” in 2011 6th Int. Conf. on Bio-Inspired Computing: Theories and Applications (BIC-TA), Sept 2011, pp. 135–139. KVALIFIKACIJSKI DOKTORSKI ISPIT  Y. Xia, B. Jelfs, M. Van Hulle, J. Principe, and D. Mandic, “An Augmented Echo State Network for Nonlinear Adaptive Filtering of Complex Noncircular Signals,” IEEE Trans. Neural Netw., vol. 22, no. 1, pp. 74–83, Jan 2011.  T. Strauss, W. Wustlich, and R. Labahn, “Design Strategies for Weight Matrices of Echo State Networks,” Neural Comput., vol. 24, no. 12, pp. 3246–3276, Dec. 2012.  M. C. Ozturk, D. Xu, and J. C. Principe, “Analysis and Design of Echo State Networks,” Neural Comput., vol. 19, no. 1, pp. 111–138, Jan. 2007.  B. Zhang, D. Miller, and Y. Wang, “Nonlinear System Modeling With Random Matrices: Echo State Networks Revisited,” IEEE Trans. Neural Netw. Learn. Syst., vol. 23, no. 1, pp. 175–182, Jan 2012.  R. F. Reinhart and J. J. Steil, “Regularization and stability in reservoir networks with output feedback,” Neurocomputing, vol. 90, pp. 96–105, 2012.  D. Sussillo and L. Abbott, “Transferring Learning from External to Internal Weights in Echo-State Networks with Sparse Connectivity,” PLoS ONE, vol. 7, no. 5, p. e37372, 05 2012.  D. Prokhorov, “Echo state networks: appeal and challenges,” in 2005 IEEE Int. Joint Conf. on Neural Networks (IJCNN), vol. 3, July 2005, pp. 1463–1466.  G. K. Venayagamoorthy and B. Shishir, “Effects of spectral radius and settling time in the performance of echo state networks,” Neural Networks, vol. 22, no. 7, pp. 861–863, 2009.  G. Fette and J. Eggert, “Short Term Memory and Pattern Matching with Simple Echo State Networks,” in Artificial Neural Networks: Biological Inspirations ICANN 2005, ser. Lecture Notes in Computer Science. Springer Berlin Heidelberg, 2005, vol. 3696, pp. 13–18.  J. Dai, G. Venayagamoorthy, R. Harley, and K. Corzine, “Indirect adaptive control of an active filter using Echo State Networks,” in 2010 IEEE Energy Conversion Congr. and Expo. (ECCE), Sept 2010, pp. 4068–4074.  Y. Cao, I. Erdin, and Q.-J. Zhang, “Transient Behavioral Modeling of Nonlinear I/O Drivers Combining Neural Networks and Equivalent Circuits,” IEEE Microw. Compon. Lett., vol. 20, no. 12, pp. 645–647, Dec 2010.  I/O Buffer Information Specification (IBIS), Electronic Industries Alliance Std. IBIS Version 6.0, Sept 2013.  I. Stievano, C. Siviero, I. Maio, and F. Canavero, “Guaranteed Locally-Stable Macromodels of Digital Devices via Echo State Networks,” in 2006 IEEE Electrical Performance of Electronic Packaging, Oct 2006, p. 65-68.  I. Chahine, M. Kadi, E. Gaboriaud, A. Louis, and B. Mazari, “Characterization and Modeling of the Susceptibility of Integrated Circuits to Conducted Electromagnetic Disturbances Up to 1 GHz,” IEEE Trans. Electromagn. Compat., vol. 50, no. 2, pp. 285–293, May 2008.  C. Gazda, D. Ginste, H. Rogier, I. Couckuyt, T. Dhaene, K. Stijnen, and H. Pues, “Harmonic balance surrogate-based immunity modeling of a nonlinear analog circuit,” IEEE Trans. Electromagn. Compat., vol. 55, no. 6, pp. 1115–1124, Dec 2013.  K. Crombecq, L. De Tommasi, D. Gorissen, and T. Dhaene, “A Novel Sequential Design Strategy for Global Surrogate Modeling,” in 2009 Winter Simulation Conf. (WSC), Dec 2009, pp. 731–742.  Integrated circuits - Measurement of electromagnetic immunity 150 kHz to 1 GHz - Part 4: Direct RF power injection method, International Electrotechnical Commission Std. IEC 62 132-4, 2006.  A. Alaeldine, R. Perdriau, M. Ramdani, J. Levant, and M. Drissi, “A Direct Power Injection Model for Immunity Prediction in Integrated Circuits,” IEEE Trans. Electromagn. Compat., vol. 50, no. 1, pp. 52–62, Feb 2008.  G.-B. Huang, Q.-Y. Zhu, and C.-K. Siew, “Extreme learning machine: Theory and applications,” Neurocomputing, vol. 70, no. 13, pp. 489–501, 2006.  X. Liu, S. Lin, J. Fang, and Z. Xu, “Is Extreme Learning Machine Feasible? A Theoretical Assessment (Part I),” IEEE Trans. Neural Netw. Learn. Syst., vol. 26, no. 1, pp. 7–20, Jan 2015.  S. Lin, X. Liu, J. Fang, and Z. Xu, “Is Extreme Learning Machine Feasible? A Theoretical Assessment (Part II),” IEEE Trans. Neural Netw. Learn. Syst., vol. 26, no. 1, pp. 21–34, Jan 2015.  E. Cambria, G.-B. Huang, L. L. C. Kasun, H. Zhou, C. M. Vong, J. Lin, J. Yin, Z. Cai, Q. Liu, K. Li, V. C. Leung, L. Feng, Y.-S. Ong, M.-H. Lim, A. Akusok, A. Lendasse, F. Corona, R. Nian, Y. Miche, P. Gastaldo, R. Zunino, S. Decherchi, X. Yang, K. Mao, B.-S. Oh, J. Jeon, K.-A. Toh, A. B. J. Teoh, J. Kim, H. Yu, Y. Chen, and J. Liu, “Extreme Learning Machines [Trends & Controversies],” IEEE Intell. Syst., vol. 28, no. 6, pp. 30–59, Nov 2013. 9  V. Ceperic, G. Gielen, and A. Baric, “Recurrent sparse support vector regression machines trained by active learning in the time-domain,” Expert Systems with Applications, vol. 39, no. 12, pp. 10 933–10 942, 2012.  G.-Z. Sun, H.-H. Chen, and Y.-C. Lee, “Time Warping Invariant Neural Networks,” in Advances in Neural Information Processing Systems 5. Morgan-Kaufmann, 1993, pp. 180–187.  M. Lukosevicius, D. Popovici, H. Jaeger, and U. Siewert, “Time Warping Invariant Echo State Networks,” Jacobs University Bremen, Technical Report 2, May 2006.  D. Pozar, Microwave Engineering. Wiley, 2004.  D. E. Root, J. Verspecht, J. Horn, and M. Marcu, X-Parameters: Characterization, Modeling, and Design of Nonlinear RF and Microwave Components. Cambridge University Press, 2013.  Y. Xue, L. Yang, and S. Haykin, “Decoupled Echo State Networks with Lateral Inhibition,” Neural Networks, vol. 20, no. 3, pp. 365–376, 2007.
* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project
Identification and Predictive Control Using Recurrent Neural Networks International Master’s Thesis Technology
Exploiting Temporal Difference for Energy Disaggregation via Discriminative Sparse Coding ERIC LEIJONMARCK