Modern Decision Analysis Methods

The tools and methods of decision analysis have undergone highly practical improvements in the last few decades, with key contributions coming from a number of practitioners in the corporate and consulting worlds.  These improvements have largely to do with the need to engage senior decision makers directly rather than have DA be seen as “back room” operation.  Only in this way, experience has shown, does DA reliably lead to high-stakes major decisions that executives “buy in” to and actually commit to executing.

This paper describes one key method and two key tools that have arisen from this endeavor.  It uses a space mission to Europa, an extraordinary and unusual moon of Jupiter of intense scientific interest, to illustrate these.

The Dialog Process

The key to successful decision making in organizations is to engage the decision makers in a highly structured dialog.  In this dialog, decision makers run the show.  Decision analysts facilitate the dialog and guide the decision analysis process.  The figure below illustrates:

Dialog process illustrated

At the outset, the decision makers have identified a decision that needs to be made.  The decision team proposes a frame and reports back to the decision makers.  The decision makers spend what is usually 3-4 hours adjusting the frame until all the decision makers have agreed on the scope and decision space they wish the decision team to explore. 

Note that not one speck of analysis has been done to this point—the world’s best analysis is worthless if its frame is poor.  The greatest source of decision failure is a poor frame, not a poor analysis.  Further, unless decision makers have agreed among themselves on the frame, trouble will brew down the road.  Often, decision makers do not realize that there are disagreements and frequently have not clearly considered, as individuals, the full scope of alternatives contemplated by various other of their colleagues.  Without clarification of these issues by decision makers, a decision team’s hard work is in peril, and subject to be thwarted down the road by dysfunctional behaviors of decision makers.

But with an agreed frame in hand, the decision team can be confident that their work program has been endorsed and will not be subject to capricious behavior by decision makers at some late stage of the effort.

In the second phase, the decision team analyzes the decision alternatives agreed to by the decision makers and reports back the results. While this is conceptually shown in the figure as one encounter between the decision team and the decision makers, it frequently involves several meetings. The decision makers learn from the results, debate their meaning, and provide guidance on the directions in which the analysis should be deepened. Together with the decision team, they may discover from this process a hybrid alternative that promises better financial and risk performance than those evaluated at the outset and direct the team to analyze it.

Finally, the decision team returns to the decision makers with a set of recommendations that decision makers debate and evaluate.  Importantly, at this final meeting the decision makers are obligated to make the final decision and direct the decision team to formulate an implementation plan.

The importance of this structured dialog is that it prevents the most common cause of decision failure (poor frame) and prevents costly “do-overs” due to poor communication.

Strategy Tables

Frequently the decision structure of the problem is complex, especially when the decision being considered relates to a large-scale initiative involving many sub-decisions.  The Strategy Table helps both decision makers and the decision team to simply and clearly define the alternatives.

The example below illustrates how this works for the case of a decision involving a space mission to Europa, a moon of Jupiter.  As background, Europa is a moon shrouded with ice that is believed to harbor a liquid water ocean beneath the ice covering.  Some scientists suspect this ocean could contain living microorganisms, perhaps similar to the “extremophiles” that inhabit the thermal vents in the deep oceans of Earth.  The heat source permitting liquid water to exist, and perhaps thermal vents, arises from the action of tidal forces generated by Jupiter that continually stress Europa’s core.

Mission planners must consider several important decisions, including significantly different design options and how aggressive to make the schedule.  The Strategy Table is shown below:

Table of alternative strategies

The decisions are the column headers of the table—target launch date, scientific scope, probe vehicle, and degree of reliability to build in. In each column are the options associated with each decision. Different options have different cost implications, timing implications, and different implications for the success of the mission.

The team then defines the alternatives to be analyzed. In this case the alternatives are called “strategies,” sensibly-connected combinations of decision options. A strategy alternative is a pathway through the table, illustrated below:

Pathway through alternative strategies

For example, the “Maximum Discovery” (green) alternative is a strategy that seeks to maximize the likelihood of scientific success by including a science package (A++) that contemplates several different possibilities for the circumstances in which microbial life may be found, and a design for the probe vehicle that will have flexibility and controllability of operation once it has penetrated the ice sheet. Designing and fabricating such a package will require a longer development time, but this longer time allows more thorough testing so redundancies can be reduced while maintaining substantial reliability.

The “Assured Arrival” alternative builds in aggressive redundancies to maximize the likelihood that the probe will arrive on target and function properly. Development and fabrication can be accomplished more quickly than with Maximum Discovery if a less sophisticated science package (A) is chosen. The probe vehicle design can still be of the more flexible kind, but the less sophisticated science package it carries reduces the likelihood of finding life.

The “Cheap and Fast” alternative addresses the beliefs of some decision makers that more sophisticated science packages and probe design are overkill for the science mission; that the most likely microbial forms will be readily apparent, if they exist; that the mission can be accomplished much more cheaply than some propose; and that there is high scientific value to proving or excluding simpler life forms as quickly as feasible for benefit of the scientific community.  It also offers the potential for a quick and badly-needed political “win” for the organization.  So this alternative uses a simpler scientific package (B) and a simple probe (Design #1) that has limited mobility and controllability once it reaches liquid water.

These are the alternatives to be analyzed, now clearly laid out on a Strategy Table and crystal clear to all involved.

Influence Diagrams

The above sets the decision components of a decision tree.  Uncertainties can be structured in a team setting using influence diagrams that explicitly show dependencies, both between uncertainties and between decisions and uncertainties.  The influence diagram for the Europa mission is shown below:

Diagram of influences

The yellow rectangles represent the decisions; the green ellipses represent the uncertainties; and the blue rounded rectangle represents the value measure used. The arrows indicate dependencies: for instance, the ellipse labeled “Microbial Life Exists on Europa” represents the uncertainty as to whether life actually exists on Europa; the ellipse “Mission Goal Achieved,” while uncertain for other reasons as well, obviously depends on whether life actually exists; this dependency is indicated by the arrow between them.

Some uncertainties depend on decisions: for instance, the likelihood of System(s) failures depends on the redundancy built into the payload system.

Not only do influence diagrams enable teams to more easily come to consensus on the uncertainties they need to assess and what dependencies they need to consider, they provide a blueprint for the uncertainties component of the decision tree and a blueprint for the quantitative model to be attached to the tree for calculating endpoint values.  Further, there is a formal mathematical equivalence between influence diagrams and decision trees, and some software packages, such as DPL™, automatically generate the decision tree from the influence diagram created (sometimes requiring some adjustments to be made to accommodate asymmetries).

Influence diagrams also provide a useful, easy to absorb, communication device that decision teams can present to decision makers.

Cumulative Probability Distributions
and Sensitivity Analysis

In this example, the team determines (using methods not described here) that the scientific value (i.e., excluding costs) of the mission given discovery of microbial life is $20 Billion. Cost uncertainties, schedule uncertainties and the other uncertainties shown in the influence diagram are assessed from subject matter experts. A spreadsheet model for calculating the mission value delivered along each branch of the decision/probability tree is constructed and attached to the tree.

From the tree are created cumulative probability distributions, along with an expected value calculation, shown below:

Comparison of decision alternatives

The way to read these cumulative probability distributions is to pick some probability on the left vertical axis, move horizontally from that point to one of the curves, drop a vertical line from this intersection point to the horizontal axis, and note the value of this point on the horizontal axis. The probability initially chosen on the vertical axis will be the likelihood that choosing this alternative will deliver a value of less than or equal the value you have found on the horizontal axis. For instance, if I pick a probability of 95% on the left axis, a line drawn horizontally from this point to the green line (Cheap and Fast alternative) intersects it at approximately a value of $8 Billion when a line is dropped to the horizontal axis. This says that if I were to choose the Cheap and Fast alternative, there is a 95% probability that it will deliver a value of less than or equal $8 Billion. Said another way, there is a 5% probability that it will deliver a value of greater than $8 Billion.

The monetary expected value (MEV) of the Cheap and Fast alternative is showing as negative $1.4 Billion. The expected value calculation of course is the sum of all the endpoint values of the tree weighted by their probability of occurrence, and is a summary measure that appropriately weighs the good outcomes against the bad outcomes. In the Cheap and Fast alternative, the bad outcomes dominate (magnitude- and probability-wise), so the MEV is negative.

But such curves also provide decision makers further information on the relative risks of the strategy alternatives. One measure of risk is found by looking at the point where the strategy lines intersect the $0 value line. For example, the Assured Arrival strategy alternative intersects the $0 value line at a cumulative probability of 88% and the Maximum Discovery strategy intersects it at a cumulative probability of 69%. In the first case, the interpretation is that there is an 88% probability that the Assured Arrival strategy will deliver a value less than or equal zero. In other words, there is an 88% chance that this strategy will deliver negative value—will destroy value rather than creating it. Similarly, there is a 69% probability that the Maximum Discovery strategy will destroy value.

Another measure of the risk of a strategy alternative is the worst case value destruction it can deliver. This is the intersection point where each strategy line touches the horizontal axis. While in this case the strategies are comparable in their worst case scenarios, you can see that the Maximum Discovery could lead in the worst case to greater value destruction (more negative value) than the others, somewhere around negative $3.5 Billion.

(A qualifying note on these particular cumulative probability curves: In most cases, you will find that such curves are much smoother and better behaved and typically will take on an “S” shape—often they are called “S-curves” for shorthand. The curves here represent an unusual case. The “dogleg” is due to large changes in value given the binary outcomes operational success and existence of microbial life.)

The Maximum Discovery alternative has the highest expected value and the lowest likelihood of a negative value, but it also has the highest downside risk (by a slight margin). Most decision-makers will view it as the least risky, especially in light of its high MEV and much lower likelihood of destroying value.

Further sensitivity analysis can generate added insights. In this case, it turns out mission value is highly sensitive to one’s assessment of the likelihood microbial life exists on Europa to begin with:

Sensitivity analysis of microbial life

The Maximum Discovery alternative clearly dominates the other two alternatives along this dimension unless one believes the likelihood of life is miniscule.  It is also apparent that proceeding with the mission has positive value provided one believes the probability of microbial life on Europa exceeds 15% (must be higher than this if the other alternatives were to be chosen).  The mission should be undertaken if there is belief that the likelihood of life on Europa exceeds 15%.

Hybrid Strategies

Often the decision team and decision makers can generate significant additional value by a deep examination of the underlying causes creating differences in risk and value for the different strategy alternatives.  Additionally, sometimes guidance arises from some challenge put forward by the decision makers to the decision team, such as, “If we could only find a way to gather more information with the simpler probe, we could deliver high mission value with high reliability at low cost. Can we do this somehow?”

Decision team members can return to the Strategy Table to explore the decision space and, now armed with knowledge of the key drivers, generate other plausible decision options.  With sufficient effort, new ideas and insights will be generated by bearing in mind what would be required, in this case along the following lines:

  • Achieving the Mission Goal
    • What if we exploited the fact that Europa’s ocean circulates?  We could design a probe vehicle that drifts long distances without having fancy control mechanisms, if inserted at the right location.  “Go with the flow,” as it were.
  • Cost
    • This idea would require a probe design that is much less expensive than the autonomous vehicle of Design #2.  Let’s call it Design #3.
  • Schedule
    • This probe could be designed and fabricated more quickly than with “Maximum Discovery.”
  • Reliability
    • We have only considered “all or nothing” redundancy, where we essentially duplicate every system.  We have not considered an option where we build redundancy only in selected critical systems.  Our reliability experts last week showed us that we could achieve more than adequate reliability by following this route.  This is another decision option for the Strategy Table.
  • Evidence Gathered
    • The new, simpler probe, since it does away with complex control and propulsion systems, leaves us with enough “real estate” to include our best science package, package A++.

This hybrid strategy alternative looks as follows:

Hybrid strategy alternatives

The analysis of the “Go With the Flow” strategy alternative looks like the following:

Analysis of the “Go With the Flow” strategy alternative

Not only does this hybrid strategy alternative have the highest expected value, it further delivers the lowest downside risk.  On top of this, it requires the lowest probability assessment of microbial life existing to deliver a positive mission value.  A clear winner:

Hybrid probability assessment

It is not always possible to create such hybrid strategies.  But, for sizeable projects, when one compares the potential for value increase to the cost of added time invested by the team, the return on that investment is potentially so enormous that it is almost always worthwhile attempting it.

Conclusion

Modern methods and tools, developed by numerous practitioners to address DA challenges in real-world settings, have much to commend them.

Methods-wise, the structured Dialog Process has proven to be, time and again, a highly efficient way to ensure quality decisions that actually get implemented.

Tools-wise, the Strategy Table and Influence Diagram have also proven themselves to be extremely effective additions to the toolkit of the decision analyst.

Publications

  • “It is to be hoped that Hulett's approach will be adopted... to produce more realistic... projections of project outcomes - to set more achievable targets and to point towards more effective risk mitigation actions.”
    -- Philip Rawlings, Director, Euro Log Ltd

  • “This is an extremely important book, which presents a depth of understanding of estimating and scheduling rarely seen in industry.”
    -- John Greenwood

David T. Hulett, Ph.D. FAACE