Exploring & Identifying Indicators

Why Indicators and Metrics Are Needed

As we developed this website and toolkit, we took to heart a number of common questions from adaptation practitioners:

  • How can we know that our adaptation actions are moving our community in the right direction?
  • How can we know that our adaptation efforts are enough?
  • How can we know what’s working and what’s not?
  • How can we show we’re making progress toward our vision?
  • How can we show we’re better off now than we were before the disaster that prompted our investment in adaptation?
  • How can we justify our adaptation expenses to our elected officials?
  • How can we explain our options to our investors?
  • If we can’t know whether our adaptation actions are effective until another storm hits, how can we show accountability to our community?

All of these common questions point to the five fundamental ways that indicators and metrics can help decision-makers in their day-to-day work:

Indicators and metrics reflect the concrete interests of decision-makers and stakeholders. They also provide the data needed to support the responsibilities outlined above. For a more detailed discussion, visit Why Think About Adaptation Success?

Clarifying Terminology

Metrics

Many terms are used to describe the process of measuring progress and success. This is due in part to varying usage across disciplinary and professional traditions, in part to carelessness and imprecision. The figure to the right shows some of these commonly used terms.

In a multidisciplinary field like adaptation, inconsistency in the use of terminology can cause a lot of confusion. For clarity, across this website, we use two fundamental terms: indicators and metrics.

An indicator is a quality, trait, or state of a system that suggests ("indicates") or hints at something one is interested in. More specifically, an indicator is a sign that a particular set of adaptation actions are yielding the desired result and/or making progress in the right direction.

Examples of indicators might include reduced damage to homes from flooding or uninterrupted food supply for all residents during storms. Visit Terminology for a more detailed discussion of indicators.

A metric is a variable that can be measured (if quantitative) or otherwise tracked (if qualitative) that represents the indicator.

The indicator, reduced flood damage, may be measured in several ways. If flood reduction activities were implemented after a devastating storm, and another comparable storm occurred later with less damage, then the metric might be the actual difference in damages, measured in dollars. If we can’t compare what happened in one event to what happened in another—or if decision-makers don’t want to wait until nature runs that potentially damaging experiment—then analysts can use models to compare the likely damages under current conditions to the likely damages if the same area were flood-proofed. The difference, again measured in dollars, is another metric to quantify the benefits of action.

Alternative metrics for this indicator might be the number of homes elevated, or the number of generators and other critical equipment moved above a given flood elevation. These metrics measure actions taken, as opposed to actual outcomes, assuming the actions indeed reduce flooding. Visit Terminology for a more detailed discussion of metrics.

We do not use the term “measures” as a noun interchangeably with either indicators or metrics, because we have found that many professionals use “adaptation measures” to denote actions they are taking. Taking action—while critically important for progress—should not be equated with adaptation success. See Adaptation and Resilience for a detailed discussion of adaptation success.

We also do not equate indicators with “targets.” Targets are not always set or formalized in an adaptation process, but when they are, they hint at a quantifiable goal achieved by a specific time.

  • In business operations, targets may focus on things like protected production facilities or the number of secured supply chains within the next five years.
  • In a local government’s work plan, targets may focus on things like the number of flood-proofed homes in a particular area of town or the number of staff trained in the basics of adaptation over the course of the next budget year.
  • In a natural resource management context, targets may focus on things like the number of restored acres of a particular habitat or forests treated for fuel management before the winter or the next wildfire season.

Targets, when specified, often involve metrics that matter to decision-makers. They do not spell out the actions needed to achieve the targets, nor do they “indicate” what achievement would signify. However, targets can be directional and highly motivational to staff in an organization.

Indicators and metrics can serve as “signs,” “markers,” “guides,” or ways to “gauge” progress, but these all are less precise phrases, so we avoid them. And finally, indicators can represent “goals” and “objectives” or interim “milestones,” but these terms should not be conflated. Sometimes “target” is used interchangeably with these latter terms.

Monitoring is the process of tracking the evolution of an indicator over time. For both qualitative and quantitative metrics, this may involve data collection through a variety of methods or review of data someone else has collected.

Evaluation is the step that follows monitoring. It is the interpretive assessment of whether progress is sufficient, adequate, and beneficial. Evaluation may involve comparison of the current state against a baseline, a target, a set of quality criteria, or an emerging need. Visit Terminology for a more detailed discussion of evaluation.

Monitoring and evaluation (often abbreviated as “M&E”) are key steps in the ongoing process of adaptation. They are a finger on the pulse of adaptation efforts. They show what is working and point to what’s needed next. They propel the adaptation process forward, building on past efforts, encouraging additional actions because climate change continues, and accelerating and intensifying the positive outcomes experienced by a given community.

Start with a Focus on Indicators

When people are keen to measure progress and success, they may be tempted to jump right to concrete metrics. Often these metrics are based upon data that is convenient to collect or obtain. Sometimes the proposed metrics are the ones everyone else tracks, or they seem simple and intuitive. But beware of choosing metrics that are not related to your vision of success. You must be able to explain why a chosen metric is important.

Meaningful indicators and metrics go beyond familiarity, convenience, and off-the-cuff ideas. Meaningful indicators come from thoughtful answers to questions like these:

  • How would we know if we reached a particular aspect of our vision?
  • How would it look to achieve x/y/z?
  • How will we know if we’ve moved in the right direction?
  • How will we know if we’ve gotten off track? 
  • What qualities would this community/system exhibit if it were well-adapted?

Answers to these questions yield meaningful indicators. From there you can develop specific metrics based on what those involved in the adaptation process actually want to know.

Brainstorming Indicators

Another common (and related) observation from our work is this: When people start to brainstorm possible indicators, they tend immediately to limit the range of possibilities. We frequently hear comments like, “How would you measure that?” or, “We don’t have data for that.”

More subtly, people will quietly drop an idea because they lack confidence in their ability to follow through on measuring it. For example, a desire to know whether adaptation will result in greater social equity or human well-being—both important indicators—may be abandoned simply because the “how” of measurement is not readily apparent.

Our experience suggests that initial brainstorms should not be constrained by concerns about data, cost, or familiarity. With an approach that is open, creative, and welcoming of all ideas, the brainstorming process is more likely to help make the vision of success more refined, specific, and vividly imaginable.

One particularly promising practice is to explore indicators as you are developing specific strategies to address climate risks and other current problems. See the work with Kachemak Bay NERR in Alaska and the City and County of Honolulu’s Resilience Strategy, which attaches a performance metric to each strategy.

The Kachemak Bay National Estuarine Research Reserve brought together stakeholders from the City of Homer, Alaska, and the surrounding borough, for a workshop on adaptation success and measuring progress. Early on, participants developed a vision of success and explored the implications of different climate scenarios for the region. (See work the with Kachemak Bay NERR.)

When it was time to explore indicators, participants returned to the current problems facing the city and borough and developed possible response strategies. They tested their feasibility under different climate scenarios, along with their alignment to the vision they had developed. They used all of this information to inform the indicator brainstorm. They chose this guiding question: How can we tell if the city and borough are making progress toward their goals in a way that is consistent with their vision of success?

The indicators that resulted from this exercise were directly related and relevant to the action strategies stakeholders had identified. Planners and stakeholders could track whether current problems would be solved in ways that accounted for climate change and moved the city and borough closer to their vision.

This is why we—and other experts in indicator development—recommend tying the exploration of indicators to the development of strategy options or the design of concrete actions.

Refining the List of Indicators

After the initial brainstorm, your list of possible indicators may be very long—and it may be incomplete in certain areas. It should be refined and augmented through careful deliberation with colleagues, stakeholders, and subject matter experts before you reach your final set of indicators. (See Job Aid on Choosing and Prioritizing Indicators.)

There are a number of ways to ensure that your list is complete, balanced, and focused on the things that matter most to those involved. These four lines of inquiry will support your efforts: ​​​​​

  • Importance of indicators: What do stakeholders and decision-makers really want to know about the state of adaptation, the progress made, and the results achieved? This line of inquiry gets to the core interests of those involved. Questions to ask about whether the core interests are actually being addressed may include:
    • If a particular (set of) indicators was tracked, would it answer the key questions a decision-maker or interested stakeholder might actually have?
    • Is a particular indicator actually meaningful?
    • Are there ways to get even more directly at the issues of greatest interest?
    • What matters most?
  • Completeness and balance of indicators: Have key sectors, areas of concern, or interests of decision-makers and stakeholders been addressed sufficiently and evenly? This line of inquiry gets to the completeness and balance of your indicator set. Questions to ask about whether the list of indicators is complete and balanced may include:
    • Are there some indicators for each system of concern that was identified in a risk and vulnerability assessment?
    • Are there some indicators for the topical areas, sectors, systems or dimensions identified in the vision? (See Visioning Success)
    • Are there at least some indicators for each of the six dimensions of adaptation success? (see Six Dimensions of Adaptation Success)
    • If current baseline conditions can be considered the outcome of past actions and inactions, how can better understanding how these current conditions arose help identify meaningful indicators for future conditions? 
  • Audience for indicators: Who is most invested in these indicators? Who wants to know? Who needs to know? This line of inquiry helps define your audience. Questions to ask about the immediate and ultimate audience(s) of the indicators may include: 
    • Does an indicator or set of indicators actually resonate with or is it meaningful to those who will hear about them? 
    • Who would really want to know about this? Who would be interested?
    • Who has a need-to-know about the state of adaptation or the progress being made? Walking in the shoes of [fill in] (e.g., funders, superiors, journalists, all or certain local residents, regulators, other stakeholders), what matters most?
  • Use of Indicators: How and for what purpose will your indicators and metrics ultimately be used? This line of inquiry clarifies your purpose—the reason for developing indicators. Questions to ask about the intended use of indicators is closely related to the audience and may include:
    • Who would use indicator information for which purpose?
    • Or turning it around, if information on a set of indicators were available, what would happen to it? How, where, when and with whom would it be used?
    • Is certain information absolutely required in certain situation? When? Why?
    • If there was information to be collected, what would make the case stronger for further work on and investment in adaptation?
    • Is there information about the state of adaptation or progress on adaptation that would support decision-makers’ communication with certain audiences?
    • If the goal was to learn what works and doesn’t work, what information would give decision-makers that kind of insight?

Clearly, there is more to choosing, refining, and augmenting your indicators than simply making a long list. Leverage your group’s collective understanding and institutional knowledge to identify the indicators that best illuminate whether—or not—your system’s behavior is fundamentally improving, and whether—or not—your system is prepared for the consequences of climate change.

For example, shuttered homes may be a more powerful indicator of the economic status of a particular neighborhood than residents’ average household income, as the former reveals to residents and local governments that things are not going well for their community. Moreover, while income and shuttered homes are clearly related, one may be more meaningful and significant to stakeholders.

Another example: Habitat protected may be an important indicator for adaptive natural resource management. However, increased outdoor recreation on that very same protected land may be more meaningful to stakeholders because they know that kids and adults don’t play outside unless there is healthy, attractive, safe habitat for them to use. This indicator shows that their community is doing well.

The lines of inquiry described here also suggest that the refinement process is iterative. Asking about audiences and uses may reveal that key indicators were omitted from the initial list, or that indicators initially included are not particularly important to stakeholders. In the same vein, making an extra effort to ensure a meaningful list of indicators may well open doors to new audiences and uses not previously considered.

Supporting Resources

This toolkit offers a number of useful resources for exploring and identifying indicators and metrics.