Computational Finance
   Hybrid Systems
   Recurrent Networks
   Support Vector Machines
   Input Variable Selection
   Classification Methods

   Book Chapters
   Journal Papers
   Conference Papers

   Windale Technologies


   Contact Me
View Andrew Back's profile on LinkedIn

   NIPS 95 Workshop

Neural Networks for Signal Processing

Friday Dec 1, 1995
Marriott Vail Mountain Resort, Colorado

Workshop Organizers:
Andrew D. Back
Department of Electrical and Computer Engineering
University of Queensland
Brisbane, Qld 4072. Australia

C. Lee Giles, Bill G. Horne
NEC Research Institute
4 Independence Way
Princeton, NJ 08540. USA
Ph: 609 951 2642, 2676
Fax: 609 951 2482
Email: {giles, horne}

A Postscript summary of workshop is available for downloading.

Intended Audience:

Researchers interested in nonlinear signal processing based on neural networks.




Workshop Aims

Nonlinear signal processing methods using neural network models form a topic of some recent interest. A common goal is for neural network models to outperform traditional linear and nonlinear models. Many researchers are interested in understanding, analysing and improving the performance of these nonlinear models by drawing from the well established base of linear systems theory and existing knowledge in other areas. How can this be best achieved ?

In the context of neural network models, a variety of methods have been proposed for capturing the time-dependence of signals. A common approach is to use recurrent connections or time-delays within the network structure. On the other hand, many signal processing techniques have been well developed over the last few decades. Recently, a strong interest has developed in understanding how better signal processing techniques can be developed by considering these different approaches.

A major aim of this workshop is to obtain a better understanding of how well this development is proceeding. For example, the different model structures raise the question, "how suitable are the various neural networks for signal processing problems ?". The success of some neural network models in signal processing problems indicate that they form a class of potentially powerful modeling methods, yet relatively little is understood about these architectures in the context of signal processing.

As an outcome of the workshop it is intended that there should be a summary of current progress and goals for future work in this research area.

Back to top

Workshop Schedule

Session 1 - Speech Focus

7:30-7:40: Introduction
7:40-8:10: Herve Bourlard, ICSI and Faculte Polytechnique de Mons
Hybrid use of hidden Markov models and neural networks for improving state-of-the-art speech recognition systems
8:10-8:40: John Hogden, Los Alamos National Laboratory
A maximum likelihood approach to estimating speech articulator positions from speech acoustics
8:40-9:10: Shun-ichi Amari, A.Cichocki, H. Yang, RIKEN
Blind separation of signals - Information geometric point of view
9:10-9:30: Discussion

Session 2 - Recurrent Network Focus

4:30-4:40: Introduction

4:40-5:10: Andrew Back, University of Queensland
Issues in signal processing relevant to dynamic neural networks

5:10-5:40: John Steele, Aaron Gordon, Colorado School of Mines
Hierarchies of recurrent neural networks for signal interpretation with applications
5:40-6:10: Stephen Piche, Pavilion Technologies
Discrete Event Recurrent Neural Networks
6:10-6:30: Open Forum, discussion time.

Back to top


Hybrid use of hidden Markov models and neural networks for improving state-of-the-art speech recognition systems

Herve Bourlard
Faculte Polytechnique de Mons, Mons, Belgium, and International Computer Science Institute Berkeley, CA 94704. USA

Recently it has been shown that Artificial Neural Networks (ANNs) can be used to augment speech recognizers whose underlying structure is essentially that of Hidden Markov Models (HMMs). In particular, we have shown that fairly simple layered structures, which we lately have termed "Big Dumb Neural Networks" (BDNNs), can be discriminatively trained to estimate emission probabilities for HMMs. Many (relatively simple) speech recognition systems based on this approach, and generally referred to as hybrid HMM/ANN systems, have been proved, on controlled tests, to be both effective in terms of accuracy (recent results show this hybrid approach slightly ahead of more traditional HMM systems when evaluated on both British and American English tasks, using a 20,000 word vocabulary and a trigram language model) and efficient in terms of CPU and memory run-time requirements.

In this talk, we will first give a short description of the basic HMM/ANN approach as successfully used today for hard speech recognition problems, as well as some recent results. We will then discuss some current research topics on extending these results to somewhat more complex systems, including new theoretical and experimental developments on transition-based recognition systems and training of HMM/ANN hybrids to diretcly maximize the global posterior probabilities.

This talk will assume some background in both hidden Markov models and artificial neural networks.

Back to workshop schedule

A maximum likelihood approach to estimating speech articulator positions from speech acoustics

John Hogden
Los Alamos National Laboratory
Los Alamos, New Mexico 87545. USA

An algorithm called maximum likelihood continuity mapping (MALCOM) wil be presented. MALCOM recovers the positions of the tongue, jaw, lips, and other speech articulators from measurements of the sound-pressure waveform of speech. MALCOM differs from other techniques for recovering articulator positions from speech in three critical respects: it does not require training on measured or modeled articulator positions, it does not rely on any particular model of sound propagation through the vocal tract, and it recovers a mapping from acoustics to articulator positions that is linearly, not topographically, related to the actual mapping from acoustics to articulation. The algorithm categorizes short-time windows of speech into a finite number of sound types, and assumes the probability of using any articulator position to produce a given sound type can be described by a Gaussian density function. MALCOM then uses maximum likelihood estimation techniques to: 1) find the most likely smooth articulator path given a speech sample and a set of Gaussian distributions (one distribution for each sound type), and 2) change the parameters of the Gaussian distributions to better account for the data. Using this technique improves the accuracy of articulator position estimates compared to continuity mapping -- the only other technique that learns the relationship between acoustics and articulation solely from acoustics.

Back to workshop schedule

Blind separation of signals - information geometric point of view

S.Amari, A.Cichocki and H.Yang
Laboratory For Information Representation,
Frontier Research Program
The Institute of Physical and Chemical Research (RIKEN)
Hirosawa, 2-1, Wako-shi, Saitama, 351-01, Japan

We propose a new efficient on-line learning algorithm for blind separation of mixtured signals. The algorithm is motivated from the Gram-Charlie expansion of density functions and information-geometric dynamics of the manifold of matrices. We give a more informal and informative presentation of underlying statistical and information-geometric ideas than that presented in the poster session of NIPS'95 for heated discussions.

Back to workshop schedule

Issues in signal processing relevant to dynamic neural networks

Andrew Back
Department of Electrical and Computer Engineering
University of Queensland
Brisbane, Qld 4072. Australia 

Signal processing and neural networks are often seen as largely disparate topics. There is however an increasing awareness that the well grounded theory of linear systems developed over the past 50 years, should be taken into account when developing nonlinear models derived from recent connectionist modelling approaches. This talks examines some topical issues in signal processing relevant to dynamic neural network models.

Back to workshop schedule

Hierarchies of recurrent neural networks for signal interpretation with applications

John P.H. Steele, Aaron Gordon
Colorado School of Mines
Golden, CO 80401. USA

We have done a number of projects using recurrent neural network architectures. In particular we have had good success with Elman's approach, i.e., using feedback of the internal state of the net. Applications include radar signal identification, trajectory estimation, robot control, and online monitoring of epoxy cure processes. In addition, we have developed our own version of constructive nets, (i.e., recurrent cascade correlation, after Fahlman) and we have developed a variety of sequentially fired nets, called progressive neural nets. This architecture has lead to higher precision solutions.

As our experience has evolved, we found that the use of hierarchies was beneficial to developing better solutions for some types of problems. This approach has allowed us to "divide and conquer" more difficult problems by developing individual recurrent nets to solve pieces of the problem and then combining the system of nets into a hierarchy, which can respond to the total problem. We believe that hierarchies can be very useful in solving more complicated and expansive problems, and that they provide an avenue for evolving more sophisticated neural network systems.

Back to workshop schedule

Discrete Event Recurrent Neural Networks

Stephen Piche
Pavilion Technologies
Austin, TX. USA

A recurrent neural network architecture for processing discrete event data will be presented. The network is composed of an input filter referred to as a discrete event infinite impulse response filter which is used to drive the input of a feedford neural network. A backpropation-through-time type algorithm is used to adjust the parameters (time constants) of the discrete event filters. An application of the algorithm to a real-world high-bandwidth signal processing problem will be reviewed.

Back to workshop schedule

Back to top

Last modified on 16 Nov 1995




Home   Research  Contact Me  Feedback  Publications

Copyright 1989-2008 Andrew Back. All Rights Reserved.