In SDM, evaluation criteria are used to characterize the degree to which different alternatives are expected to meet objectives. They are used to:

• compare alternatives accurately and consistently;
• expose trade-offs including trade-offs among different degrees of uncertainty;
• generate productive discussion about better alternatives;
• prioritize information needs;
• communicate the rationale for and improve the transparency of decisions.

It isn’t easy to define good evaluation criteria that are widely agreed upon by stakeholders, experts and decision makers. However, the up-front investment pays off in streamlined decision making, for two principal reasons:

• because data, modeling and expert judgment processes are focused on producing decision-relevant information;
• because large numbers of very complex options can be consistently and efficiently evaluated by multiple decision makers.

Brainstorming evaluation criteria

Each lowest level objective in the hierarchy will need an evaluation criterion. There are a number of things to consider in selecting or designing good criteria which we cover in the next section. However, the first step is simply brainstorming an initial list of candidate criteria. It’s easiest to know what criteria will be useful if you have a clear idea of the alternatives under consideration. (In fact most often you can’t really identify useful criteria at all without this knowledge!) Therefore, a good approach involves:

• Brainstorm a list of alternatives. Ask, what are all the possible ways we could achieve these objectives? You don’t need the details of the alternatives at this point, but you do need to identify the full range of them;
• Sketch out a consequence table, with the objectives listed in the rows and a few sample alternatives shown in the columns.
• Ask, “What specific metric could we use to report the impact of these alternatives on this objective?” Or “What specific information would you like to see to be able to evaluate the impact of these alternatives on this objective?”
• At this point, write down all the possible responses. It is possible that the responses will reveal that some people have different interpretations of the objectives. This is good. One of the key reasons for structuring objectives is to build common understanding and improve communication. Refine the objectives if you need to.

For complex systems (biological, economic or social), it will be helpful to have or to develop influence diagrams to show the major factors affecting the objective. Useful evaluation criteria are often found in the influence diagram.

Key Ideas

• Evaluation criteria are required for comparing and making trade-offs between alternatives
• Each of the lowest-level objectives needs an evaluation criterion
• Sketching out a list of alternatives and a consequence table helps identify evaluation criteria