We investigate Solutions provided by the finite-context predictive model called neural prediction machine (NPM) built on the recurrent layer of two types of recurrent neural networks (RNNs). One type is the first-order Elman’s simple recurrent network (SRN) trained for the next symbol prediction by the technique of extended Kalman filter (EKF). The other type of RNN is an interesting unsupervised counterpart to the “claissical” SRN, that is a recurrent version of the Bienenstock, Cooper, Munro (BCM) network that performs a kind of time-conditional projection pursuit. As experimental data we chose a complex symbolic sequence with both long and short memory structures. We compared the Solutions achieved by both types of the RNNs with Markov models to find out whether training can improve initial Solutions reached by random network dynamics that can be interpreted as an iterated function system (IFS). The results of our simulations indicate that SRN trained by EKF achieves better next symbol prediction than its unsupervised counterpart. Recurrent BCM network can provide only the Markovian solution that is not able to cover long memory structures in sequence and thus beat SRN.
The predictive performance of Echo State neural networks were optimized for electrical load forecasting and compared to the results achieved by competitors in the worldwide Eunite Competition #1. The test data used were the actual results of the competition, attached to a specific region. A regular adaptation of an \textit{Echo State }neural network was optimized by adapting the weights of the dynamic reservoir through Anti-Hebbian learning, and the weights from input and output neurons to the hidden neurons were optimized using the Metropolis algorithm. The results achieved with such an optimized Echo State neural network would gain a strong second place within the Eunite competition.
Recurrent neural networks (RNNs) have much larger potential than
classical feed-forward neural networks. Their output responses depend also on the time position of a given input and they can be successfully used in spatio-temporal task Processing. RNNs are often used in the cognitive science community to process symbol sequences that represent various natural language structures. Usually they are trained by common gradient-based algorithms such as real time recurrent learning (RTRL) or backpropagation through time (BPTT). This work compares the RTRL algorithm that represents gradient based approaches with extended Kalman filter (EKF) methodology adopted for training the Elman’s simple recurrent network (SRN). We used data sets containing recursive structures inspired by studies of cognitive science community and trained SRN for the next symbol prediction task. The EKF approach, although computationally more expensive, shows higher robustness and the resulting next symbol prediction performance is higher.