Statistical Language Learning

A few weeks ago I was browsing in a bookstore and saw this book on the shelf. As I reread the preface for the first time in 13 years, memories of a time gone by rushed over me. Memories of 1993, back when I was young, good-looking, and had a full head of hair...ahhhh those happy grad school years. I can still remember reading this book's preface so many years ago, and feeling the excitement! A new age is upon us--the old paradigms are crumbling, but not to worry! Statistical parsing will save the day! And Dr. Charniak gave us such an incredibly easy on-ramp: this book is fat-free, crystal-clear, and mercifully short. Ideal for busy professors and grad students, you too could change your paradigm in only 10 days of assiduous reading. But in 2006, this book doesn't sound anymore like a manafesto for a new era. Indeed, it sounds more like a survey and summary of the last 10 years worth of research in statistical language processing. After so many years, statistical parsers are still only around 90% accurate in picking the best parse for a sentence. Rereading this book makes it PAINFULLY obvious that the field of NLP has been treading water for over a decade. What could have gone so wrong??? I went back to the preface to find out, and my conclusion is that although Dr. Charniak accurately diagnosed the disease, he prescribed the wrong medicine. What was the disease? Listen to how cogently he characterises what was wrong with NLP in 1993: "...language understanding depends on a lot of "real-world knowledge" ...But....the study of knowledge representation....is not going anywhere fast....AI has become notorious for the production of countless non-monotonic logics and almost as many logics of knowledge and belief, and none of the work shows any obvious application to actual knowledge representation problems". The coup-de-grace is delivered by Dr. Charniak as follows: "Thus many of us in AI-NLP have found ourselves in the position of basing our research on the successful completion of other's reasearch--a completion that is looking more and more problematic." One would expect Dr. Charniak at this point to say something like "so lets all pitch-in and give the knowledge-representation folks a hand for a while." But no. Instead, he suggests that we all start inducing statistical parsers. Now let's step back for a moment and marvel at what a great strategy this is. If your current search space is exhausted, you have no choice--you must create a new search space for yourself. I'm reminded of the marketing plan for Altoids. The market for breath-mints is incredibly crowded, so what's a newcommer to do? Create a _new_ space (strong mints) and give the consumer a reason to buy Altoids (they're curiously strong!) Dr. Charniak made a similar move when he wrote this book--create a new space (statistical parsing) where he could be number one (I wrote the book). Historically, this strategy worked flawlessly--statistical parsing

Note that this is a small book (roughly 160 pages of text) so don't expect to find a broad coverage of the field in it. The (only) 3 topics covered in a satisfactory detail are Hidden Markov Models, Probabilistic Grammars and Word Sense Disambiguation. The first chapter ("The Standard Model") is probably included just for comparison to the statistical model, so it's a bit surprising to find the good coverage of the Chart Parsing there. If you are interested in some of the 3 topics mentioned above, consider buying this book. Don't forget that it's been written in 1993 (so it's pretty old). For this reason (or maybe others) it is less known than other similar writings in the field, so you may also surprise some of your colleagues with it :-)

This book is one of the most concise presentations of the subject of natural language processing using statistical theory. It gives a very thorough introduction in the areas of hidden Markov models (HMMs), PCFG (probabilistic context-free grammar) and other related areas, with good coverage of the mathematics behind the theory. There is quite an extensive use of relevant samples, and the bibliography is extensive. It is a very good introduction to this area for professionals and students alike.Juntung Wu University College (University of Oxford) Oxford, England