Probability Theory: The Logic of Science [Hardcover]

To frequentist statisticians, probability theory is the study of relative frequencies or of proportions of a population; those are "probabilities".

To Bayesian statisticians, probability theory is the study of degrees of belief. Bayesians may assign probability 1/2 to the proposition that there was life on Mars a billion years ago; frequentists will not do that because they cannot say that there was life on Mars a billion years ago in precisely half of all cases -- there are no such "cases".

To _subjective_ Bayesians, probability theory is about subjective degrees of belief. A subjective degree of belief is merely how sure you happen to be.

"Noninformative" _objective_ Bayesians assign "noninformative" probability distributions when they deal with uncertain propositions or uncertain quantities, and replace them with "informative" distributions only when they update them because of "data". "Data", in this sense, consists of the outcomes of random experiments.

"Informative" _objective_ Bayesians -- a rare species -- ask what degree of belief in an uncertain proposition is logically necessitated by whatever information one has, and they don't necessarily require that information to consist of outcomes of random experiments.

Jaynes is an "informative" objective Bayesian. This book is his defense of that position and his account of how it is to be used.

"Pure" mathematicians will not find that this book resembles that branch of "pure" mathematics that they call probability theory.

Jaynes rails against those he disagrees with at great length. Often he is right. But often he simply misunderstands them. For example, writing in the 1990s, he said that pure mathematicians reject the use of Dirac's delta function and its derivatives, and related topics. That is nonsense; the delta function has long been considered highly respectable, and required material in the graduate curriculum. Unfortunately Jaynes's misunderstandings may cause some others to misunderstand him when he is right. Statisticians are more informed than "pure" mathematicians and will disagree with Jaynes for better reasons. _Some_ statisticians will agree with him.

Jaynes has many flaws, made all the more annoying by the fact that we need to overlook them in order to understand him. His message is important.

This book develops probability theory from first principles as an extension of deductive logic. In deductive logic, propositions can have only three possible truth values: true, false, and irremediable uncertainty. Therefore, the goal of the book is to describe a consistent extended logic that assigns real numbers to the plausibility of propositions. The requirements for such a system are derived from five simple desiderata, which serve as the postulates of this theory - and it turns out that *any* such system is equivalent to probability theory, to within a monotonic transformation.

Probability theory is then developed through applications to problems which grow more and more complex. The author demonstrates its use in direct sampling problems and so-called inverse problems, aka Bayesian probability. He derives procedures for multiple hypothesis testing, parameter estimation, and significance testing, and shows that although there are close connections between probability and frequency of occurrence in a large number of trials, no probability is *simply* a frequency.

Following this, the author presents solutions to the problem of assigning prior probabilities, and develops decision theory as an adjunct to probability theory. The author then compares and contrasts mainstream or "orthodox" statistical theory with probability theory as extended logic, and (perhaps unsurprisingly) finds severe deficiencies in the orthodox methods. The final chapters concern even more advanced applications.

Math Requirements

Readers should be well versed in simple calculus and multivariate calculus; some familiarity with convolution integrals and finite combinatorics is also an asset, but not essential. In isolated places, the author uses or refers to the calculus of variations and the theory of function spaces (in this case Hilbert spaces); but lack of familiarity with these branches of mathematics will not seriously hamper the reader.

Critical Review

This book represents a major step forward in the understanding of what probability theory is and how to use it. In particular, a lack of solutions to the problem of prior probabilities is the main reason that for the past 100 years, mainstream probability theory was taught as a theory of frequencies instead of as an extenstion of logic; therefore, having solutions to the problem of assigning priors in a textbook is a great step forward in the development of probability theory.

The book is a pleasure to read, with a text-to-equation ratio that is uncharacteristically high for a textbook of probability theory. That is not to say that the equations are simplistic; on the contrary, solutions to quite challenging problems are presented. In addition, the author's polemics against orthodox theory are quite entertaining (and convincing); he wields an acerbic pen when describing the efforts of those who actively reject probability theory as extended logic.

One negative feature of the book is its incompleteness: the author passed away before finishing the book, so occasionally large chunks of planned text are missing. The editor has cleverly mitigated this flaw by inserting "Editor's Problem Boxes", which challenge the reader to fill in the missing text. Still, as one reads the book, one gets the vaguely disquieting feeling that the author wanted to include much, much more information, but didn't have the chance.