Types of Evaluation Criteria: A Summary

Generally, proxies tend to be overused, so be on guard. But they can also be extremely useful. A practical approach involves:

  • Use proxies at the scoping stage (they are generally much easier to estimate than the corresponding natural criteria).
  • Based on preliminary analysis, eliminate, if possible, some proxies from further consideration (because they are insensitive to the alternatives for example, or closely correlated with another criterion).
  • Be explicit about the range of possible and/or presumed relationships between the proxy and the endpoint/objective.
  • When using one proxy to represent multiple objectives, make sure there are no hidden trade-offs.
  • If trade-offs involving a proxy are pivotal to the decision, use a structured expert judgment process to better understand the relationship of the proxy to the endpoint/objective.

Conclusion

  • Use conceptual models such as influence diagrams to help identify appropriate criteria and focus information gathering and model building efforts.
  • Use natural criteria if you can. Consider expert judgment to estimate them if modeling is not available.
  • Use constructed scales as a second choice. If they are carefully constructed, they are surprisingly effective.
  • Use proxies freely at the scoping stage. Use them cautiously at later stages if they are key to final choices about difficult trade-offs. They are surprisingly problematic.
  • There are no right or wrong criteria, but there are good ones and bad ones. Know the limitations of the criteria you use.

Key Ideas

  • Be cautious when using proxies: there are a number of pitfalls associated with them
  • Natural attributes make the best evaluation criteria, followed by constructed attributes, and then proxies.