Pattern Classification and Scene Analysis

The 1973 book by Duda and Hart was a classic. It surveyed the literature on pattern classification and scene analysis and provided the practitioner with wonderful insight and exposition of the subject. In the intervening 28 years the field has exploded and there has been an enormous increase in technical approaches and applications. With this in mind the authors and their new coauthor David Stork go about the task of providing a revision. True to the goals of the original the authors undertake to describe pattern recognition under a variety of topics and with several available methods to cover each topic. Important new areas are covered and old but now deemed less significant are dropped. Advances in statistical computing and computing in general also dictate the topics. So although the authors are the same and the title is almost the same (note that scene analysis is dropped from the title) it is more like an entirely new book on the subject rthan a revision of the old. For a revision, I would expect to see mostly the same chapters with the same titles and only a few new chapters along with expansion of old chapters. Although I view this as a new book, that is not necessarily bad. In fact it may be viewed as a strength of the book. It maintains the style and clarity of the original that we all loved but represents the state-of-the-art in pattern recognition at the beginning of the 21st Century. The original had some very nice pictures. I liked some of them so much that I used them with permission in the section on classification error rate estimation in my bootstrap book. This edition goes much further with beautiful graphics including many nice three-dimensional color pictures like the one on the cover page. The standard classical material is covered in the first five chapters with new material included (e.g. the EM algorithm and hidden markov models in Chapter 3). Chapter 6 covers multilayer neural networks (a totally new area). Nonmetric methods including decision trees and the CART methodology are covered in Chapter 8. Each chapter has a large number of relevant references and many homework exercises and computer exercises. Chapter 9 is "Algorithm-Independent Machine Learning" and it includes the wonderful "No Free Lunch" theorem (Theorem 9.1), a discussion of the minimum desciption length principle, overfitting issues and Occam's razor, bias - variance tradeoffs,resampling method for estimation and classifier evaluation, and ideas about combining classifiers. Chapter 10 is on unsurpervised learning and clustering. In addition to the traditional techniques covered in the first edition the authors include the many advances in mixture models. I was particularly interested in that part of Chapter 9. There is good coverage of the topics and they provide a number of good references. However, I was a bit disappointed with the cursory treatment of bootstrap estimation of classification accuracy (section 9.6.3 on pages 485 - 486). I particul

I liked this book because it does a great job explaining the concepts and the reasoning behind the mathematical formulae. Other books such as "The Elements of Statistical Learning" toss the Math formulas at you and expect you to figure out the significance or the importance of 'em. The book does not shy away from Math - but does a great job presenting it.

This book is not for the novice, and it assumes some mathematical skills on the reader's side. Having read the book a few times now, I must conclude that this one covers a lot of ground regarding pattern classification, and is probably more complete than any other book currently on the market. If you're really interested in pattern recognition, you will get through this book with success, and will feel very thankful about the many useful algorithms, all perfectly clarified with pictures and even pseudo code. For those having mathematical problems, you might have to read more than once or twice to get a good grip on it. Yes, there are a few bugs in there, but this is the same with anyother book (there is an errata available on the web). I've also been implementing many of the algorithms discussed, and I think anybody seriously involved with pattern classification should have at least a copy of this book nearby. For those who complain that the book doesn't cover enough topics, related to distributed processing, machine learning, statistical inference, etc, I think these topics don't belong in here (they deal less with pattern classification), and others have dedicated separate books for all of those topics. For those readers/students complaining about "too complicated", or "too many errors", or "hard-to-understand concepts", I recommand a better science teacher.I haven't read the first edition, but having been in the commercial field of data mining / data fore casting / data clustering for many years now, I think this book is very up-to-date. I have come to understand what works and what doesn't, and yes, maybe not everything is covered, but the things that are covered are definitely current and leading-edge technology.

Before this book was published, I considered "Pattern Recognition", by Theordoridis to be the best text for learning pattern recognition and classification. Although Theordoridis' book has some difficulties (not enough concrete exercises, ommission of structural methods, and not enough material on Bayesian Networks and HMMs), it seemed significantly better than previous texts. However, not only does Duda, Hart, and Stork's book succeed in those areas where the former fails, but it also has other strengths that the former book does not have: better illustrations, boxed formulas and algorithms, and highlighted defintions. Although somewhat superficial, these improvements mark the fact that pattern recognition is now considered a mainstream subject, and thus requires a mainstream text that keeps the integrity and rigor of the subject matter, while simultaneously making it more accessible to the average engineer. The new champ, however, does not come without it's own shortcomings. For example, I believe the last 3 chapters of Theodoridis' book should be read by anyone who wants a deeper understanding of clustering techniques for unsupervised learning. Moreover, this book fails to acknowledge the brilliant work done in computational learning by Vapnik and Chervonenkis, which reveals the authors' bias towards practice over theory. I believe it deserves more than passing mention in the historical notes section of unsupervised learning.

The 1973 edition of Pattern Classification by Richard Duda and Peter Hart is one of the most cited books in the fields of image processing, machine vision, and classification. It contains perhaps the clearest, most comprehensible descriptions of statistical inference ever written. Though intended for the image processing audience, it is general in its approach, and is broader in coverage than other contemporary books like the redoubtable Van Trees (1969). The section on Bayesian Learning anticipates the EM algorithm which appeared a few years later (Dempster, et al. 1977) and their description of Parzen windows for density estimation is more often cited than Parzen's own papers. The appearance of the 2000 2nd edition led this writer to wonder if D & H could repeat with an offering as good as their first. In particular, would D & H have kept up with the considerable growth in methodology in the 1990s? Well, they have! With the addition of David Stork as third author, the second addition re-presents the basic theory, illustrated with some beautiful and complex figures, and knits it neatly with an exposition of neural networks, stochastic methods for posterior determination, nonmetric classification (tree search and string parsing), and clustering. Chapter 9 is a particularly interesting review of the recent machine learning research making the point that, absent knowledge of a problem's specific domain, no one classifier is better that any other. This chapter also reviews solutions to the problem of training on too-small samples including the Jackknife and bootstrap methods, and newer bagging and boosting algorithms popular in data mining applications. Each chapter is well-designed, with a summary, many exercises (including computer exercises), and references to the literature (typically 50-100) including many recent references. This book is designed for an upper-level undergraduate/graduate audience. It doesn't assume a knowledge of statistics, but requires some familiarity with methods from calculus, real analysis, and linear algebra. The first edition was a particularly important element in this writer's education; the second edition is certain to find a similar place in the working and intellectual lives of many new readers.