Waltzing With Bears: Managing Risk on Software Projects [Paperback]

I believe it behoves renowned experts to do their homework and get the basics right before they pollute the education system with their half-baked 'wisdom'. Do yourself a favour, ignore what the book says about risk analysis and go and by a good book on Bayesian Methods and Decision Theory. You don't have to take my word for this, just type in 'epistemic uncertainty and Monte Carlo' into your Internet search engine and take it from there. In the meantime, here are some background notes to help explain my remarks:

•There are two types of uncertainty: epistemic and aleatory.
•As the name suggests, epistemic uncertainty results from gaps in knowledge. For example, one may be uncertain of an outcome because one has never used a particular technology before. Such uncertainty is essentially a state of mind and hence subjective.
•Aleatory uncertainty results from variability that is intrinsic to the behaviour of some systems (alea is the Latin for die). For example, I can be confident regarding the long term frequency of throwing sixes but I remain uncertain of the outcome of any given throw of a dice. This uncertainty can be objectively determined.
•There are two branches of probability theory: Frequentist and Bayesian.
•Frequentist probability theory is used to analyse systems that are subject to aleatory uncertainty.
•Bayesian probability theory is used to analyse epistemic uncertainty.
•For most risk assessments a Project Manager has to undertake, there is both epistemic and aleatory uncertainty but epistemic uncertainty is always significant due to the novelty of the situation under assessment.
•Standard Monte Carlo Simulation uses frequentist probability theory to analyse risk.
•When Monte Carlo is used to model schedule risk, the schedule uncertainties are being treated as if they are aleatory, even though they are predominantly epistemic. This is now considered to be unrealistic and is known to give incorrect results. The main problem is that the second order uncertainties are often too large to be ignored, i.e. the required shape for the chosen probability distribution curves is more important than the tool vendors would have you believe and yet they are usually imprecisely known.
•Using standard Monte Carlo to analyse schedule risk also requires unrealistic assumptions to be made regarding the correlations between the probabilities for the individual outcomes, e.g. that there are no correlations or that they are all of the same nature. In practice, there are correlations to be considered when analysising schedule risk and they are of both a positive and negative nature.
•As a result of the above drawbacks many expert authorities are warning against the use of Monte Carlo Simulation where the historical data upon which the analysis is premised is incomplete. For example, its use in ecological and economic models is now controversial (see the World Congress on Risk website).
•In software development there is a growing appreciation of the importance of Bayesian Methods in analysing problems in software quality assurance.
•Bayesian methods are appropriate in situations where there are gaps in information (i.e. where there is epistemic uncertainty). They involve the creation of Bayesian Belief Networks (BBNs) to model causal relationships. Data is fed into the model to enable the probability of specified outcomes to be calculated given the current body of knowledge.
•Even more interesting, Bayes Theorem can be used to assess the likelihood that pre-conditions exist in the light of outcomes becoming known.
•BBNs can be used in any situation where one is trying to calculate the likelihood of an outcome, or an unknown situation, when there is only limited information. It is useful in Decision Theory when a risk-based decision is required in the face of epistemic uncertainty.

(1) Ignore it.
(2) Pretend it does not exist.
(3) Look upon the messanger as "Not a Team Player" and finally ...
(4) Pressure

DeMarco and Lister blow the lid off this approach by making a compelling argument that 1-4 are irresponsible and unethical, then pointing to a better way.

The "toolset" that DeMarco and Lister provide is very specific and will help, but I think they are getting to something deeper. Ever since "PeopleWare", the impression I have of these authors is that they are trying to get IS folks to think for themselves - to have a very large toolbox and to pick the right tool for that particular job, instead of "Standardized Procedure where you do everything by the book."

The fact is, IS people can add value and clarity by knowing that we can't do everything and picking the highest ROI items to work on. (Or helping Sr. Management to decide what to work on ... however you want to word it.)

The book can help your organization deal with reality, instead of hoping for fantasyland. If you struggle with projects with unrealistic deadlines and artificially compressed schedules (because if the project was scheduled in a sane way, the ROI wouldn't make it worth doing) - this book can help.

The explanation of the relationship between risk and benefit analysis was both insightful and seemed like it would be useful. It provides a pragmatic framework for a lot of what are considered 'good engineering practices', such as incremental deliverables that can be measured and verified in meaningful ways.

The only downside is that it's very difficult to understand how to take advantage of some of the frameworks he provides without being in a management position already. Individual contributors won't get a lot out of this.