Hopfield Models and Neural Networks

One of my main research interests has been centered around the so called Hopfield model. The Hopfield model has become popular as a model for an autoassociative memory and is something of a paradigmatic model for neural networks. Personally, I am mostly concerned with it as a rather interesting example of a strongly disordered spin system. You can find detailed reviews on what has been done in the book  (see here for a summary) "Mathematical Aspects of Spin Glasses and Neural Networks", edited by myself and Pierre Picco, and that was published by Birkhauser Verlag in 98. In particular, there is a long review paper by myself and Veronique Gayrard in it. Recently Michel Talagrand has done some very nice work on the subject, and if you are interested in the field you should keep a look at his  homepage . An introduction can also be found, together with a lot more on disordered systems, in my MaPhySto lecture notes.

What has been achieved, what are the main questions today? When asking such questions, we must look into the wider context of these problems. Let me place them in the restricted setting of disordered mean field models of Ising type. What is that? Most generally speaking, consider the hypercube in dimension N and assume that you are given a random function H on the vertices of the hypercube. You may alternatively think of this as a stochastic process on the hypercube. Conceptually, the simplest examples pould be Gaussian processes that then would be characterized by the covariance function. Indeed, these correspond to models studied in physics and that are known as the random energy model (if the process is white noise), the Sherrington-Kirkpatric model (if the correlation is a quadratic function of the Hamming distance), etc. But of course you can think of many other processes, and the Hopfield Hamiltonian corresponds to just some class of such processes (with some very special properties). In principle, one would like to study the properties of such processes: e.g. what are its maxima or minima, what is the distribution of the size of the level sets, what is the numbre of connected components of a level set (as a function of the level), etc.. The approach of statistical mechanics is to look at this processes via the probability measures they produce by deforming the uniform measure on the hypercube by an exponential factor exp(-beta H). These things are called Gibbs measures and they encode most of the interesting information about the stochastic process H. Now the study of such processes and the associated measures appears to be a rather horrendous problem. The theory of random process on high dimensional spaces is a vast topic (see e.g. the book "Probability in Banach Spaces", by M. Ledoux and M. Talagrand (Spinger, 1994). The amazing thing is that theoretical physicist have proposed "exact solutions" for such problems in many cases which predict much sharper results than mathematicians have been able to proof (and even conjecture) previously. In particular, this leads to amazingly good solutions of a number of problems in combinatorical optimization (I suggest to look at the homepages of Remi Monasson and of Stephan Mertens). The methods for coming up with these heuristic solutions are highly non-trivial in themselves and invoke most strikingly innovative concepts (like 0-by-0 matrices , functions of them and optimization problems in the space of such matrices (the good reference to start learning about this is the book "Spin glass theory and beyond", by M. Mezard, G. Parisi and M.-A. Virasoro (World Scientific, 1987). It is clear that to understand these methods in rigorous mathematical terms and to see precisely when and where these results can be justified is of great interest, if one is a) interested in such problems at all and b) mathematically minded.