Theory of Neural Information Processing Systems

Theory Of Neural Information Processing Systems Provides An Explicit, Coherent, And Up-to-date Account Of The Modern Theory Of Neural Information Processing Systems. It Has Been Carefully Developed For Graduate Students From Any Quantitative Discipline, Including Mathematics, Computer Science, Physics, Engineering, Biology, And Has Been Thoroughly Class-tested By The Authors Over A Period Of Some 8 Years. Exercises Are Presented Throughout The Text And Notes On Historical Background And Further Reading Guide The Students Into The Literature. All Mathematical Details Are Included And Appendices Provide Further Background Material, Including Probability Theory, Linear Algebra And Stochastic Processes, Making This Textbook Accessible To A Wide Audience.--jacket. Pt. 1. Introduction To Neural Networks -- 1. General Introduction -- 1.1. Principles Of Neural Information Processing -- 1.2. Biological Neurons And Model Neurons -- 1.3. University Of Mcculloch-pitts Neurons -- 1.4. Exercises -- 2. Layered Networks -- 2.1. Linear Separability -- 2.2. Multilayer Networks -- 2.3. The Perceptron -- 2.4. Learning In Layered Networks : Error Back Propagation -- 2.5. Learning Dynamics In Small Learning Rate Perceptrons -- 2.6. Numerical Simulations -- 2.7. Exercises -- 3. Recurrent Networks With Binary Neurons -- 3.1. Noiseless Recurrent Networks -- 3.2. Synaptic Symmetry And Lyapunov Functions -- 3.3. Information Processing In Recurrent Networks -- 3.4. Exercises -- 4. Notes And Suggestions For Further Reading-- Pt. 2. Advanced Neural Networks -- 5. Competitive Unsupervised Learning Processes -- 5.1. Vector Quantization -- 5.2. Soft Vector Quantization -- 5.3. Time-dependent Learning Rates -- 5.4. Self-organizing Maps -- 5.5. Exercises. 6. Bayesian Techniques In Supervised Learning -- 6.1. Preliminaries And Introduction -- 6.2. Bayesian Learning Of Network Weights -- 6.3. Predictions With Error Bars : Real-valued Functions -- 6.4. Predictions With Error Bars : Binary Classification -- 6.5. Bayesian Model Selection -- 6.6. Practicalities : Measuring Curvature -- 6.7. Exercises -- 7. Gaussian Processes -- 7.1. The Underlying Idea -- 7.2. Examples Of Networks Reducing To Gaussian Processes -- 7.3. The 'priors Over Functions' Point Of View -- 7.4. Stationary Covariance Functions -- 7.5. Learning And Prediction With Gaussian Processes -- 7.6. Exercises -- 8. Support Vector Machines For Binary Classification -- 8.1. Optimal Separating Plane For Linearly Separable Tasks -- 8.2. Representation In Terms Of Support Vectors -- 8.3. Preprocessing And Svm Kernels -- 8.4. Exercises -- 9. Notes And Suggestions For Further Reading. Pt. 3. Information Theory And Neural Networks -- 10. Measuring Information -- 10.1. Brute Force : Counting Messages -- 10.2. Exploiting Message Likelihood Differences Via Coding -- 10.3. Proposal For A Measure Of Information -- 11. Identification Of Entropy As An Information Measure -- 11.1. Coding Theory And The Kraft Inequality -- 11.2. Entropy And Optimal Coding -- 11.3. Shannon's Original Proof -- 12. Building Blocks Of Shannon's Information Theory -- 12.1. Entropy -- 12.2. Joint And Conditional Entropy-- 12.3. Relative Entropy And Mutual Information -- 12.4. Information Measures For Continuous Random Variables -- 12.5. Exercises -- 13. Information Theory And Statistical Inference -- 13.1 Maximum Likelihood Estimation -- 13.2. The Maximum Entropy Principle -- 13.3. Exercises -- 14. Applications To Neural Networks -- 14.1. Supervised Learning : Boltzmann Machines -- 14.2. Maximum Information Preservation -- 14.3. Neuronal Specialization -- 14.4. Detection Of Coherent Features -- 14.5. The Effect Of Non-linearities -- 14.6. Introduction To Amari's Information Geometry -- 14.7. Simple Applications Of Information Geometry -- 14.8. Exercises -- 15. Notes And Suggestions For Further Reading -- Pt. 4. Macroscopic Analysis Of Dynamics -- 16. Network Operation : Macroscopic Dynamics -- 16.1. Microscopic Dynamics In Probabilistic Form -- 16.2. Sequential Dynamics -- 16.3. Parallel Dynamics -- 16.4. Exercises. 17. Dynamics Of Online Learning In Binary Perceptrons -- 17.1. Probabilistic Definitions, Performance Measures -- 17.2. Explicit Learning Rules -- 17.3. Optimized Learning Rules -- 17.4. Exercises -- 18. Dynamics Of Online Gradient Descent Learning -- 18.1. Online Gradient Descent -- 18.2. Learning From Noisy Examples -- 18.3. Exercises -- 19. Notes And Suggestions For Further Reading -- Pt. 5. Equilibrium Statistical Mechanics Of Neural Networks -- 20. Basics Of Equilibrium Statistical Mechanics -- 20.1. Stationary Distributions And Ergodicity -- 20.2. Detailed Balance And Interaction Symmetry-- 20.3. Equilibrium Statistical Mechanics : Concepts, Definitions -- 20.4. A Simple Example : Storing A Single Pattern -- 20.5. Phase Transitions And Ergodicity Breaking -- 20.6. Exercises -- 21. Network Operation : Equilibrium Analysis -- 21.1. Hopfield Model With Finite Number Of Patterns -- 21.2. Introduction To Replica Theory : The Sk Model -- 21.3. Hopfield Model With An Extensive Number Of Patterns -- 21.4. Exercises -- 22. Gardner Theory Of Task Realizability -- 22.1. The Space Of Interactions -- 22.2. Capacity Of Perceptrons -- Definition And Toy Example -- 22.3. Capacity Of Perceptrons -- Random Inputs -- 23. Notes And Suggestions For Further Reading. Appendix A : Probability Theory In A Nutshell -- A.1. Discrete Event Sets -- A.2. Continuous Event Sets -- A.3. Averages Of Specific Random Variables -- Appendix B : Conditions For The Central Limit Theorem To Apply -- B.1. Moment Condition -- B.2. Lindeberg's Theorem -- Appendix C : Some Simple Summation Identities -- Appendix D : Gaussian Integrals And Probability Distributions -- D.1. General Properties Of Gaussian Integrals -- D.2. Gaussian Probability Distributions -- D.3. A List Of Specific Gaussian Integrals -- Appendix E : Matrix Identities -- E.1. Block Inverses -- E.2. The Woodbury Formula -- Appendix F : The [delta]-distribution -- F.1. Definition -- F.2. [delta](x) As Solution Of The Liouville Equation -- F.3. Representations, Relations, And Generalizations -- Appendix H : Metrics For Parametrized Probability Distributions -- H.1. Local Distance Definitions -- H.2. The Triangular Inequality -- H.3. Global Distance Definitions -- Appendix I : Saddle-point Integration -- References -- Index. A.c.c. Coolen, R. Kühn., P. Sollich. Includes Bibliographical References And Index.