EEL6825 Course Materials and Schedule

Many of the lecture examples in this class are contained in Mathematica notebooks. If you have access to Mathematica, you can run all the experiments contained therein. If you do not have access to Mathematica, you can still view the notebooks through MathReader available for free from Wolfram for the Linux, Windows and Macintosh operating systems.

Topic Subtopics & materials
Course introduction & background
zip file (5.3 Mb)
8/26, 8/28		 Syllabus
Course syllabus, Fall 2003, EEL6825 (2 pages, 6 kb).
Introduction to pattern recognition
Richard O. Duda, Peter E. Hart and David G. Stork, Pattern Classification, 2nd ed., Chapter 1, pp. 1-19, John Wiley & Sons, New York, 2001.
Chapter 1 figures, Fall 2003, EEL6825 (8 pages, 2 Mb).
Probability and statistics
Richard O. Duda, Peter E. Hart and David G. Stork, Pattern Classification, 2nd ed., Appendix A.4, pp. 611-23, John Wiley & Sons, New York, 2001.
Probability and statistics refresher, Fall 2003, EEL6825 (10 pages, 176 kb).
Sample feature extraction: color
Visualizing the RGB cube, Fall 2003, EEL6825 (1 slide, 288 kb).
Selected classification problems
Object detection (faces & cars)
Road following
Horizon detection
Horizon tracking for Micro Air Vehicles
Classifying monkey neural activity (1 page, 20 kb)
Bayesian decision theory
zip file (9.6 Mb)
9/2, 9/4		 Bayesian decision theory
Richard O. Duda, Peter E. Hart and David G. Stork, Pattern Classification, 2nd ed., Chapter 2, pp. 20-31, John Wiley & Sons, New York, 2001.
Chapter 2 figures, Fall 2003, EEL6825 (25 pages, 1.7 Mb).
Normal density and related discriminant functions
Richard O. Duda, Peter E. Hart and David G. Stork, Pattern Classification, 2nd ed., Chapter 2, pp. 31-45, John Wiley & Sons, New York, 2001.
Lecture slides, Fall 2003, EEL6825 (41 slides, 11 pages, 6.3 Mb).
Discriminant examples for the Normal density, Fall 2003, EEL6825 (2 pages, 72 kb).
Statistical modeling and classification definitions
(needed for many of the Mathematica notebooks below this entry), Fall 2003, EEL6825 (52 kB).
Statistical modeling and classification definitions, Fall 2003, EEL6825 (web version of above Mathematica notebook).
Normal distribution: discriminant functions
(requires Mathematica definitions notebook), Fall 2003, EEL6825 (8.7 Mb).
Normal distribution: discriminant functions, Fall 2003, EEL6825 (web version of above Mathematica notebook).
Maximum-likelihood estimation
zip file (4.5 Mb)
9/9 - 9/16		 Introduction to maximum-likelihood estimation
Richard O. Duda, Peter E. Hart and David G. Stork, Pattern Classification, 2nd ed., Chapter 3, pp. 84-90, John Wiley & Sons, New York, 2001.
Maximum-likelihood slides, Fall 2003, EEL6825 (4 slides, 1 pages, 12 kb).
Introduction to maximum-likelihood estimation, Fall 2003, EEL6825 (4 pages, 36 kb).
Maximum-likelihood estimation for Normal density
Maximum-likelihood examples: Normal density
(requires Mathematica definitions notebook), Fall 2003, EEL6825 (1.7 Mb).
Maximum-likelihood examples: Normal density, Fall 2003, EEL6825 (web version of above Mathematica notebook).
Application example: sky/ground segmentation
Feature test for sky/ground classification
(requires Mathematica definitions notebook, and image files from below), Fall 2003, EEL6825 (6.9 Mb).
Sample image files (required for Mathematica notebook above)
Feature test for sky/ground classification, Fall 2003, EEL6825 (web version of above Mathematica notebook).
Application example: object segmentation
Color-based object detection and image segmentation: slides, Fall 2003, EEL6825 (9 slides, 1.6 Mb)
C source code and sample files for above slides, Fall 2003, EEL6825 (500 kb zip file).
Expectation-Maximization (EM) & mixture modeling
zip file (454 kb)
9/23 - 10/7		 Theoretical foundations of EM
Richard O. Duda, Peter E. Hart and David G. Stork, Pattern Classification, 2nd ed., Chapter 3, pp. 124-128, John Wiley & Sons, New York, 2001.
Maximum-likelihood estimation for mixture models: the EM algorithm, Fall 2003, EEL6825 (21 pages, 388 kb).
Mixture modeling and the EM algorithm: slides, Fall 2003, EEL6825 (38 slides, 10 pages, 120 kb).
EM/mixture modeling examples
zip file (28.8 Mb)
9/23 - 10/7		 Motivating examples for mixture modeling
Difficult to classify examples: Normal density
(requires Mathematica definitions notebook), Fall 2003, EEL6825 (1.3 Mb).
Difficult to classify examples: Normal density, Fall 2003, EEL6825 (web version of above Mathematica notebook).
Synthetic mixture modeling examples
Mixture-of-Gaussians example #1
(requires Mathematica definitions notebook), Fall 2003, EEL6825 (4.2 Mb).
Mixture-of-Gaussians example #1, Fall 2003, EEL6825 (web version of above Mathematica notebook).
Mixture-of-Gaussians example #2
(requires Mathematica definitions notebook), Fall 2003, EEL6825 (4.1 Mb).
Mixture-of-Gaussians example #2, Fall 2003, EEL6825 (web version of above Mathematica notebook).
Animated EM examples
Animated EM algorithm examples for above two synthetic examples, Fall 2003, EEL6825.
Application example: two-object classification
Two objects classification examples
(requires Mathematica definitions notebook, and image files from below), Fall 2003, EEL6825 (5.9 Mb).
Sample image files (required for Mathematica notebook above)
Two objects classification examples, Fall 2003, EEL6825 (web version of above Mathematica notebook).
Two-parameter mixture-modeling example
Two-parameter mixture-modeling example
(requires Mathematica definitions notebook), Fall 2003, EEL6825 (332 kb).
Two-parameter mixture-modeling example, Fall 2003, EEL6825 (web version of above Mathematica notebook).
Detailed mixture-modeling example
Detailed mixture-modeling example
(requires Mathematica definitions notebook), Fall 2003, EEL6825 (5.9 Mb).
Detailed mixture-modeling example, Fall 2003, EEL6825 (web version of above Mathematica notebook).
Detailed mixture-modeling animations, Fall2003, EEL6825.
Importance of EM initialization: a case study
Bad initialization example
(requires Mathematica definitions notebook), Fall 2003, EEL6825 (2.6 Mb).
Bad initialization example, Fall 2003, EEL6825 (web version of above Mathematica notebook).
Animations of EM algorithm with different initializations, Fall 2003, EEL6825.
Number of component densities
Varying the number of component densities
(requires Mathematica definitions notebook), Fall 2003, EEL6825 (748 kB).
Varying the number of component densities, Fall 2003, EEL6825 (web version of above Mathematica notebook).
Examples of EM convergence with varying number of components, Fall 2003, EEL6825.
EM/mixture modeling examples 2
zip file (10.4 Mb)
10/7		 Application example: soda-can classification I
Soda cans classificantion examples I
(requires Mathematica definitions notebook, and image files from below), Fall 2003, EEL6825 (16.4 Mb).
Sample image files (required for Mathematica notebook above)
Soda cans classificantion examples I, Fall 2003, EEL6825 (web version of above Mathematica notebook).
Application example: soda-can classification II
Soda cans classificantion examples I
(requires Mathematica definitions notebook, and image files from below), Fall 2003, EEL6825 (16.2 Mb).
Sample image files (required for Mathematica notebook above)
Soda cans classificantion examples I, Fall 2003, EEL6825 (web version of above Mathematica notebook).
Odds & ends
zip file (4.8 Mb)
10/9		 Generating non-uniform random numbers
W. H. Press, et. al., Numerical Recipes in C: The Art of Scientific Computing, 2nd. ed., Section 7.2, pp. 287-290, Cambridge University Press, Cambridge, 1992 (4 pages, 56 kB).
(This book section contains a discussion on how to generate a random number from a Gaussian distribution.)
Application example: texture (non-color) based classification
Texture-based classification experiments
(requires Mathematica definitions notebook, and image files from below), Fall 2003, EEL6825 (4.6 Mb).
Sample image files (required for Mathematica notebook above)
Texture-based classification experiments, Fall 2003, EEL6825 (web version of above Mathematica notebook).
Threshold-based classification of single models
Threshold-based classification of single model: synthetic-data example
(requires Mathematica definitions notebook), Fall 2003, EEL6825 (3.5 Mb).
Threshold-based classification of single model: synthetic-data example, Fall 2003, EEL6825 (web version of above Mathematica notebook).
Threshold-based classification of single model: real-data example
(requires Mathematica definitions notebook, and image files from below), Fall 2003, EEL6825 (7.2 Mb).
Sample image files (required for Mathematica notebook above)
Threshold-based classification of single model: real-data example, Fall 2003, EEL6825 (web version of above Mathematica notebook).
Vector quantization & histogram modeling
zip file (3.3 Mb)
10/9-10/16		 Vector quantization
Vector quantization: a lmiting case of EM, Fall 2003, EEL6825 (11 pages, 396 kb).
Iterative vector quantization animations, Fall 2003, EEL6825.
Histogram modeling
Vector quantization & histogram modeling: slides, Fall 2003, EEL6825 (14 slides, 4 pages, 196 kb).
Histogram modeling & classification examples
Simple histogram modeling examples
(requires Mathematica definitions notebook), Fall 2003, EEL6825 (3 Mb).
Simple histogram modeling examples, Fall 2003, EEL6825 (web version of above Mathematica notebook).
Another histogram modeling example
(requires Mathematica definitions notebook), Fall 2003, EEL6825 (3.3 Mb).
Another histogram modeling example, Fall 2003, EEL6825 (web version of above Mathematica notebook).
Markov systems
zip file (3.3 Mb)
10/16-		 Introduction to Markov systems
Richard O. Duda, Peter E. Hart and David G. Stork, Pattern Classification, 2nd ed., Chapter 3, pp. 128-138, John Wiley & Sons, New York, 2001.
Introduction to Markov systems, Fall 2003, EEL6825 (41 pages, 528 kb).
Introduction to Markov systems: slides, Fall 2003, EEL6825 (52 slides, 13 pages, 332 kb).
L. R. Rabiner, "A tutorial on hidden Markov models and selected applications in speech recognition,", Proc. of the IEEE, vol. 77, no. 2, pp. 257-86, 1989 (30 pages, 2.2 MB).
HMM Mathematica examples
Hidden Markov model definitions
(needed for the HMM Mathematica notebooks below this entry), Fall 2003, EEL6825 (60 kb).
Hidden Markov model definitions, Fall 2003, EEL6825 (web version of above Mathematica notebook).
Different HMMs, same observation probability
(requires HMM Mathematica definitions notebook), Fall 2003, EEL6825 (204 kb).
Different HMMs, same observation probability, Fall 2003, EEL6825 (web version of above Mathematica notebook).
Comprehensive set of HMM examples
(requires HMM Mathematica definitions notebook, and sample HMM and sample observation sequence files from below), Fall 2003, EEL6825 (1.4 Mb).
Data files used in above Mathematica notebook:
Comprehensive set of HMM examples, Fall 2003, EEL6825 (web version of above Mathematica notebook).
HMM applications
zip file (21.9 Mb)
 Application example: speech recognition case study
An isolated-word, speaker-dependent speech recognition system, Fall 2003, EEL6825 (12 pages, 1.0 Mb).
An isolated-word, speaker-dependent speech recognition system: slides, Fall 2003, EEL6825 (17 slides, 1.1Mb).
Sound (wav) files used for speech recognition case study:
  • Repeated instances of word "one" (1.8 Mb)
  • Repeated instances of word "two" (1.8 Mb)
  • Repeated instances of word "three" (1.8 Mb)
  • Repeated instances of word "four" (1.8 Mb)
  • Repeated instances of word "five" (1.7 Mb)
  • Repeated instances of word "dog" (1.8 Mb)
  • Repeated instances of word "god" (1.7 Mb)
HMM convergence from different initial random models, Fall2003, EEL6825.
Application example: gesture recognition
Simple gesture recogntion example: slides, Fall 2003, EEL6825 (4 slides, 288 kb).
Application example: HMM-based stochastic similarity measure
HMM-based similarity between stochastic time series: slides, Fall 2003, EEL6825 (10 slides, 272 kb).
M. C. Nechyba and Y. Xu, "Stochastic Similarity for Validating Human Control Strategy Models,", IEEE Trans. on Robotics and Automation, vol. 14, no. 3, pp. 437-51, 1998 (15 pages, 816 kb).
Application example: discontinuous driving control
Discontinuous driving control: slides, Fall 2003, EEL6825 (16 slides, 1.1 Mb).
M. C. Nechyba and Y. Xu, "On Learning Discontinuous Human Control Strategies,", Int. Journal Of Intelligent Systems, vol. 16, no. 4, pp. 547-70, 2001 (24 pages, 500 kb).
M. C. Nechyba, Learning and Validation of Human Control Strategies, Ph.D. Thesis, CMU-RI-TR-98-06, Robotics Institute, Carnegie Mellon University, 1998 (208 pages, 4.7 Mb).
Short human driving sequence in driving simulator used for the above two applications.
Application example: predicting neural-spike activity
Predicting neural-spike activity: slides, Fall 2003, EEL6825 (12 slides, 1.7 Mb).
S. Darmanjian, S. P. Kim, M. C. Nechyba, S. Morrison, J. Principe, J. Wessberg, M. A. L. Nicolelis, "Bimodal Brain-Machine Interface for Motor Control of Robotic Prosthetic," to be presented at IEEE Int. Conf. on Intelligent Robots and Systems, Las Vegas, October, 2003 (7 pages, 616 kb).
Feature-dimensionality reduction
zip file (6.6 Mb)
 Feature dimensionality reduction: PCA and the Fisher linear discriminant
Richard O. Duda, Peter E. Hart and David G. Stork, Pattern Classification, 2nd ed., Chapter 3, pp. 114-121, John Wiley & Sons, New York, 2001.
Synthetic-data examples
Feature reduction examples on synthetic data sets
(requires Mathematica definitions notebook), Fall 2003, EEL6825 (1.0 Mb).
Feature reduction examples on synthetic data sets, Fall 2003, EEL6825 (web version of above Mathematica notebook).
Choosing the best discriminant line: an animation, Fall 2003, EEL6825.
Real-data examples
Feature reduction in sky/ground segmentation
(requires Mathematica definitions notebook, and image files from below), Fall 2003, EEL6825 (12.7 Mb).
Sample image files (required for Mathematica notebook above)
Feature reduction in sky/ground segmentation, Fall 2003, EEL6825 (web version of above Mathematica notebook).
Neural networks
zip file (6.6 Mb)
 Introduction to neural networks
Introduction to neural networks, Fall 2003, EEL6825 (20 pages, 360 kB).
Introduction to neural networks: slides, Fall 2003, EEL6825 (60 slides, 15 pages, 476 kB).
Advanced parameter optimization
Introduction to advanced parameter optimization, Fall 2003, EEL6825 (13 pages, 240 kB).
Introduction to advanced parameter optimization: slides, Fall 2003, EEL6825 (54 slides, 14 pages, 464 kB).
W. H. Press, et. al., Numerical Recipes in C: The Art of Scientific Computing, 2nd. ed., Sections 10.1-10.2, pp. 397-405, Cambridge University Press, Cambridge, 1992 (9 pages, 208 kB).

Conjugate gradient algorithm
Conjugate gradient algorithm for training neural networks, Fall 2003, EEL6825 (19 pages, 364 kB).
Conjugate gradient algorithm for training neural networks: slides, Fall 2003, EEL6825 (80 slides, 20 pages, 480 kB).

Advanced neural network techniques
Advanced neural network techniques: slides (includes scaled conjugate gradient slides), Fall 2003, EEL6825 (72 slides, 18 pages, 652 kB).
M. C. Nechyba and Y. Xu, "Cascade neural networks with node-decoupled extended Kalman filtering," Proc. IEEE Int. Symp. on Computational Intelligence in Robotics and Automation, vol. 1, pp. 214-9, 1997 (6 pages, 80 kB).
Neural network applications
Neural network applications: slides, Fall 2003, EEL6825 (57 slides, 15 pages, 1.9 MB).
ALVINN
D. A. Pomerleau, "Efficient Training of Artificial Neural Networks for Autonomous Navigation," Neural Computation, vol. 3, no. 1, pp. 88-97, 1991 (10 pages, 148 kB).
[provided courtesy of Dean Pomerleau; some figures are missing.]
Gaussian fitting of ALVINN ouput, Fall 2003, EEL6825 (324 kb).
Gaussian fitting of ALVINN ouput, Fall 2003, EEL6825 (web version of above Mathematica notebook).
RALPH
D. A. Pomerleau and T. Jochem, "Rapidly Adapting Machine Vision for Automated Vehicle Steering," IEEE Expert, vol. 11, no. 2, pp. 19-27, 1996 (9 pages, 1.5 MB).
RALPH examples , Fall 2003, EEL6825 (1.7 MB).
[The above notebook loads in a straight road image (40 kB) and a curved road image (40 kB).]
RALPH examples Fall 2003, EEL6825 (web version of above Mathematica notebook).
Face detection
S. Baluja, H. A. Rowley and T. Kanade, "Neural Network-Based Face Detection," IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 20, no. 1, pp. 23-38, 1998 (28 pages, 392 kB).
References
T. M. Mitchell, "Chapter 4: Artificial Neural Networks," Machine Learning, McGraw-Hill, Boston, 1997.
C. M. Bishop, "Chapter 7: Parameter Optimization Algorithms," Neural Networks for Pattern Recognition, Oxford University Press, Oxford, 1995.
Nonparametric methods

 Nonparametric methods
Richard O. Duda, Peter E. Hart and David G. Stork, Pattern Classification, 2nd ed., Chapter 4, pp. 161-214, John Wiley & Sons, New York, 2001.
Last updated October 21, 2003 by Michael C. Nechyba