NIPS 95 Workshop
Neural Networks for Signal Processing
Friday Dec 1, 1995
Marriott Vail Mountain Resort, Colorado
Workshop Organizers:
Andrew D. Back
Department of Electrical and Computer Engineering
University of Queensland
Brisbane, Qld 4072. Australia
C. Lee Giles, Bill G. Horne
NEC Research Institute
4 Independence Way
Princeton, NJ 08540. USA
Ph: 609 951 2642, 2676
Fax: 609 951 2482
Email: {giles, horne} @research.nj.nec.com
A Postscript summary of workshop is available for
downloading.
Intended Audience:
Researchers interested in nonlinear signal processing based on neural networks.
Aims
Schedule
Abstracts
Workshop Aims
Nonlinear signal processing methods using neural network models form a topic of some
recent interest. A common goal is for neural network models to outperform traditional
linear and nonlinear models. Many researchers are interested in understanding, analysing
and improving the performance of these nonlinear models by drawing from the well
established base of linear systems theory and existing knowledge in other areas. How can
this be best achieved ?
In the context of neural network models, a variety of methods have been proposed for
capturing the time-dependence of signals. A common approach is to use recurrent
connections or time-delays within the network structure. On the other hand, many signal
processing techniques have been well developed over the last few decades. Recently, a
strong interest has developed in understanding how better signal processing techniques can
be developed by considering these different approaches.
A major aim of this workshop is to obtain a better understanding of how well this
development is proceeding. For example, the different model structures raise the question,
"how suitable are the various neural networks for signal processing problems ?".
The success of some neural network models in signal processing problems indicate that they
form a class of potentially powerful modeling methods, yet relatively little is understood
about these architectures in the context of signal processing.
As an outcome of the workshop it is intended that there should be a summary of current
progress and goals for future work in this research area.
Back to top
Workshop Schedule
Session 1 - Speech Focus
-
- 7:30-7:40: Introduction
- 7:40-8:10: Herve Bourlard, ICSI and Faculte Polytechnique de Mons
Hybrid use of hidden Markov models and neural networks for
improving state-of-the-art speech recognition systems
- 8:10-8:40: John
Hogden, Los Alamos National Laboratory
A maximum likelihood approach to estimating speech articulator
positions from speech acoustics
- 8:40-9:10: Shun-ichi Amari,
A.Cichocki, H. Yang, RIKEN
Blind separation of signals - Information geometric point of view
- 9:10-9:30: Discussion
Session 2 - Recurrent Network Focus
-
- 4:30-4:40: Introduction
4:40-5:10: Andrew Back,
University of Queensland
Issues in signal processing relevant to dynamic neural networks
- 5:10-5:40: John Steele,
Aaron Gordon, Colorado School of Mines
Hierarchies of recurrent neural networks for signal interpretation
with applications
- 5:40-6:10: Stephen Piche, Pavilion Technologies
Discrete Event Recurrent Neural Networks
- 6:10-6:30: Open Forum, discussion time.
Back to top
Abstracts
Hybrid use of hidden Markov models and neural
networks for improving state-of-the-art speech recognition systems
Herve Bourlard
Faculte Polytechnique de Mons, Mons, Belgium, and International Computer Science
Institute Berkeley, CA 94704. USA
Email: bourlard@tcts.fpms.ac.be
Recently it has been shown that Artificial Neural Networks (ANNs) can be used to
augment speech recognizers whose underlying structure is essentially that of Hidden Markov
Models (HMMs). In particular, we have shown that fairly simple layered structures, which
we lately have termed "Big Dumb Neural Networks" (BDNNs), can be
discriminatively trained to estimate emission probabilities for HMMs. Many (relatively
simple) speech recognition systems based on this approach, and generally referred to as
hybrid HMM/ANN systems, have been proved, on controlled tests, to be both effective in
terms of accuracy (recent results show this hybrid approach slightly ahead of more
traditional HMM systems when evaluated on both British and American English tasks, using a
20,000 word vocabulary and a trigram language model) and efficient in terms of CPU and
memory run-time requirements.
In this talk, we will first give a short description of the basic HMM/ANN approach as
successfully used today for hard speech recognition problems, as well as some recent
results. We will then discuss some current research topics on extending these results to
somewhat more complex systems, including new theoretical and experimental developments on
transition-based recognition systems and training of HMM/ANN hybrids to diretcly maximize
the global posterior probabilities.
This talk will assume some background in both hidden Markov models and artificial
neural networks.
A maximum likelihood approach to estimating speech articulator
positions from speech acoustics
John Hogden
Los Alamos National Laboratory
Los Alamos, New Mexico 87545. USA
Email: hogden@lanl.gov
An algorithm called maximum likelihood continuity mapping (MALCOM) wil be presented.
MALCOM recovers the positions of the tongue, jaw, lips, and other speech articulators from
measurements of the sound-pressure waveform of speech. MALCOM differs from other
techniques for recovering articulator positions from speech in three critical respects: it
does not require training on measured or modeled articulator positions, it does not rely
on any particular model of sound propagation through the vocal tract, and it recovers a
mapping from acoustics to articulator positions that is linearly, not topographically,
related to the actual mapping from acoustics to articulation. The algorithm categorizes
short-time windows of speech into a finite number of sound types, and assumes the
probability of using any articulator position to produce a given sound type can be
described by a Gaussian density function. MALCOM then uses maximum likelihood estimation
techniques to: 1) find the most likely smooth articulator path given a speech sample and a
set of Gaussian distributions (one distribution for each sound type), and 2) change the
parameters of the Gaussian distributions to better account for the data. Using this
technique improves the accuracy of articulator position estimates compared to continuity
mapping -- the only other technique that learns the relationship between acoustics and
articulation solely from acoustics.
Blind separation of signals - information geometric point of view
S.Amari, A.Cichocki and H.Yang
Laboratory For Information Representation,
Frontier Research Program
The Institute of Physical and Chemical Research (RIKEN)
Hirosawa, 2-1, Wako-shi, Saitama, 351-01, Japan
Email: amari@sat.t.u-tokyo.ac.jp
We propose a new efficient on-line learning algorithm for blind separation of mixtured
signals. The algorithm is motivated from the Gram-Charlie expansion of density functions
and information-geometric dynamics of the manifold of matrices. We give a more informal
and informative presentation of underlying statistical and information-geometric ideas
than that presented in the poster session of NIPS'95 for heated discussions.
Issues in signal processing relevant to dynamic neural networks
Andrew Back
Department of Electrical and Computer Engineering
University of Queensland
Brisbane, Qld 4072. Australia
Signal processing and neural networks are often seen as largely disparate topics. There
is however an increasing awareness that the well grounded theory of linear systems
developed over the past 50 years, should be taken into account when developing nonlinear
models derived from recent connectionist modelling approaches. This talks examines some
topical issues in signal processing relevant to dynamic neural network models.
Hierarchies of recurrent neural networks for signal interpretation with
applications
John P.H. Steele, Aaron Gordon
Colorado School of Mines
Golden, CO 80401. USA
Email: steele@matchless.mines.edu
We have done a number of projects using recurrent neural network architectures. In
particular we have had good success with Elman's approach, i.e., using feedback of the
internal state of the net. Applications include radar signal identification, trajectory
estimation, robot control, and online monitoring of epoxy cure processes. In addition, we
have developed our own version of constructive nets, (i.e., recurrent cascade correlation,
after Fahlman) and we have developed a variety of sequentially fired nets, called
progressive neural nets. This architecture has lead to higher precision solutions.
As our experience has evolved, we found that the use of hierarchies was beneficial to
developing better solutions for some types of problems. This approach has allowed us to
"divide and conquer" more difficult problems by developing individual recurrent
nets to solve pieces of the problem and then combining the system of nets into a
hierarchy, which can respond to the total problem. We believe that hierarchies can be very
useful in solving more complicated and expansive problems, and that they provide an avenue
for evolving more sophisticated neural network systems.
Discrete Event Recurrent Neural Networks
Stephen Piche
Pavilion Technologies
Austin, TX. USA
Email: piche@pav.com
A recurrent neural network architecture for processing discrete event data will be
presented. The network is composed of an input filter referred to as a discrete event
infinite impulse response filter which is used to drive the input of a feedford neural
network. A backpropation-through-time type algorithm is used to adjust the parameters
(time constants) of the discrete event filters. An application of the algorithm to a
real-world high-bandwidth signal processing problem will be reviewed.
Back to top
Last modified on 16 Nov 1995
|
Research