PLANNING AND EVALUATING THE SOCIETAL IMPACTS OF

PLANNING AND

EVALUATING

THE SOCIETAL

IMPACTS OF

CLIMATE CHANGE

RESEARCH

PROJECTS:

A guidebook for

natural and physical

scientists looking to

make a difference

TABLE OF CONTENTS

Foreword 3

Authors 3

Acknowledgements 3

Introduction 4

Part 1. Introducing Societal Impacts 4

Why consider the societal impacts of research? 4

What are Societal Impacts? 6

Long-term vs. short-term change 6

Five categories of societal impacts 6

How do researchers generate social impacts? 8

Who are societal partners? 10

Contribution or Attribution? 11

Part 2. Using Logic Models to Document Research Impacts 13

Societal Impacts Framework for Project Planning and Design 13

Societal Impacts Framework for Impact

Assessment and Project Reflection 20

Documenting Impacts – Worksheets and Case Studies 26

Part 3. Additional Context and Information 34

Appendix A: Exploring Societal Impacts Categories 34

Process as an Indicator of Impacts 35

Impacts in Context 35

Summary 36

Appendix B: Socially Engaged Research 37

Appendix C: Challenges of Impacts Assessment 40

Knowledge Generation and Information Use 40

Methodological Barriers 40

Appendix D: Concerns about Societal Impacts Evaluation 42

References 43

FOREWORD

As scientists, we aim to generate new knowledge and insights about the world around

us. We often measure the impacts of our research by how many times our colleagues

reference our work, an indicator that our research has contributed something new

and important to our field of study. But how does our research contribute to solving

the complex societal and environmental challenges facing our communities and our

planet? The goal of this guidebook is to illuminate the path toward greater societal

impact, with a particular focus on this work within the natural and physical sciences.

We were inspired to create this guidebook after spending a collective 20+ years

working in programs dedicated to moving climate science into action. We have seen

firsthand how challenging and rewarding the work is. We’ve also seen that this

applied, engaged work often goes unrecognized and unrewarded in academia. Projects

and programs struggle with the expectation of connecting science with decision

making because the skills necessary for this work aren’t taught as part of standard

academic training.

While this guidebook cannot close all of the gaps between climate science and

decision making, we hope it provides our community of impact-driven climate

scientists with new perspectives and tools. The guidebook offers tested and proven

approaches for planning projects that optimize engagement with societal partners, for

identifying new ways of impacting the world beyond academia, and for developing the

skills to assess and communicate these impacts to multiple audiences including the

general public, colleagues, and elected leaders.

We would like to thank our colleagues in the Climate Assessment for the Southwest

(CLIMAS), the Southwest Climate Adaptation Science Center (SW CASC), the Northwest

Climate Adaptation Science Center (NW CASC), and the Great Lakes Integrated Sciences

and Assessments (GLISA) for inspiring this work and for their contributions to making

our communities and environments healthier and stronger, even as we face the

challenge of climate change.

Sincerely,

Alison Meadow

Gigi Owen

Suggested citation: Meadow, Alison M. and Gigi Owen (2021) Planning and

Evaluating the Societal Impacts of Climate Change Research Project: A guidebook for

natural and physical scientists looking to make a difference.

AUTHORS

Alison M. Meadow, Ph.D.

Associate Research Professor – University of Arizona; Arizona Institutes

for Resilience

[email protected]

Alison Meadow has a background in environmental anthropology and urban

planning. Her research focuses on the process of linking scientists with

decision makers to improve the usability of climate science, with a particular

emphasis on evaluating the outcomes of such research partnerships.

Gigi Owen, Ph.D.

Research Scientist, University of Arizona, Arizona Institutes for Resilience;

Climate Assessment for the Southwest (CLIMAS)

[email protected]

As a qualitative social scientist with a background in geography and political

ecology, Gigi’s research interests center on the interactions between humans

and their environments. Her current research portfolio includes building

equitable and resilient local food systems, assessing climate adaptation

strategies, and evaluating how interdisciplinary and transdisciplinary research

supports a thriving Southwest region.

ACKNOWLEDGEMENTS

We would like to thank The Center for Advancing Research Impacts in Society

(ARIS) for the opportunity to participate in the 2020 Fellows Program; our

colleagues in the 2020 ARIS fellows cohort; the Arizona Institutes for Resilience

at the University of Arizona; and Molly Hunter, Michelle Higgins, Connie

Woodhouse, Sarah Olsen, Natasha Wingerter, Julie Risien, and Peter Ruggiero

who reviewed and helped us refine, strengthen, and improve this guidebook.

This document was developed as part of the Advancing Research Impacts in

Society (ARIS) Center Fellows Program supported through a grant from the NSF

(#1810732). The findings and recommendations are those of the author(s) and do

not necessarily reflect the views of the National Science Foundation.

photography throughout courtesy of Unsplash

As scientists, we aim to generate new knowledge and insights about the

world around us. We often measure the impacts of our research by how many

times our colleagues reference our work – an indicator that our research has

contributed something new and important to our field of study. Considering

the challenges facing our society and environment, this type of impact is no

longer enough. Many researchers are seeking ways to have greater and more

direct impacts on communities and our environment by helping solve complex

challenges such as climate change.

But how can we structure our research to contribute directly to solving the

complex social and environmental challenges facing our communities and our

planet? This guidebook offers techniques to help researchers plan to increase

their impacts beyond academia as well as document and evaluate those impacts.

In order to identify and describe the impacts that may be generated by research,

we use a set of five impact types that range from creation of new connections

between researchers and partners in broader society to tangible, beneficial

changes in social and/or ecological systems (i.e. socio-ecological systems).

Collectively, we refer to this range of impacts as societal impacts.

The framework we present is applicable to all types of academic research.

However, we specifically tailored this guidebook for physical and natural

scientists working in the field of climate change. Global society is in a moment

where swift and intentional action is needed, but it seems difficult to know

which actions will be most effective and for whom. The environmental and

societal challenges associated with climate change are diverse and complex.

It is necessary to understand how climate research can most effectively guide

climate action, where it has been most successful doing so, and reveal the types

of action that are currently missing.

This guidebook is formatted into three main parts. In the first part we provide

background information about the societal impacts of research. In the second

part, we present a framework for planning for and assessing the societal

impacts of research, including worksheets and case studies. The third part

contains a series of appendices that provide additional context for the

information in this guidebook.

INTRODUCTION

Why consider the societal impacts of research?

1 See for example NSF Dear Colleague Letter 21-059, A Broader Impacts Framework for Proposals Submitted to NSF's Social, Behavioral, and Economic Sciences

Directorate.

The harmful social and environmental impacts associated

with global warming and climate change present urgent

and complex challenges for humankind. These challenges

have encouraged natural and physical scientists to

make their research more responsive to communities,

resource managers, and policy makers. Many researchers

choose the profession because they see possibilities for

generating new knowledge to benefit the environment and

society. However, academic researchers are not always

supported by their institutions or departments to engage

with societal partners, produce research outputs for public

audiences, or conduct applied research.

Funders and universities are starting to pay more

attention to the ways that academic research generates

positive impacts for society – in economic terms and

far beyond. In much of Europe, the United Kingdom, and

Australia, universities are required to report on their

societal impacts as part of the requirements to receive

federal research funding. In the U.S., the National Science

Foundation (NSF) includes broader impacts criteria in its

funding calls, which require applicants to describe their

plans to achieve impacts beyond the production of high-

quality research. For example, applicants may develop

activities that connect their fundamental research efforts

to empowering people and improving their quality of life.

Other federal funding programs also expect researchers to

engage with decision makers in the course of producing

“actionable” or “usable” knowledge. The Department of

the Interior’s Climate Adaptation Science Center network

is one example; the National Oceanic and Atmospheric

Administration’s long-running Regional Integrated Science

and Assessment program is another. Both programs

award 5-year grants to academic institutions that can

demonstrate skill and experience in producing science

about climate variability and change that can be directly

linked with resource management and policy making.

Crafting a societal impacts plan and an evidence-based

assessment of project impacts can improve your chances

of receiving funds from agencies that prioritize societal

and broader impacts.

PART 1.

Introducing Societal Impacts

Larger issues of trust and accountability are also in play

in considering societal impacts. Academic researchers can

help build relationships of trust between universities and

their surrounding communities through public engagement

and problem-solving at a local scale. The Association

of Public and Land Grant Universities’ recent report,

Public Impact Research: Engaged Universities Making

the Difference, makes the case that using public impact

research along with fundamental research, “communicates

powerfully to the public the value of university research

and could help restore public trust in our institutions.”

Researchers contribute to their university’s trust-building

work through effective engagement and by documenting

impacts in local, regional, or international communities.

2 Abel, Steve, and Rod Williams. 2019. The Guide: documenting, evaluating and recognizing engaged scholarship.: Purdue University, Office of Engagement.

3 See for example:

Alvesson, Mats, Yiannis Gabriel, and Roland Paulsen. 2017. Return to meaning: A social science with something to say: Oxford University Press.

Foster, Kevin Michael. 2010. Taking a stand: Community-engaged scholarship on the tenure track. Journal of Community Engagement and Scholarship 3 (2):20.

Hicks, Diana, Paul Wouters, Ludo Waltman, Sarah De Rijcke, and Ismael Rafols. 2015. Bibliometrics: the Leiden Manifesto for research metrics. Nature News 520

(7548):429.

4 See Appendix D for more information regarding concerns about the role of societal impacts in academic performance reviews.

At the scale of individual researchers, societal impacts

have not often been included in academic assessments for

promotion and tenure. This trend, however, is changing.

For example, Purdue University now includes promotion

and tenure criteria that is specific to engaged scholarship.

Other efforts have been made to broaden academic

performance metrics beyond the restrictive focus on

publications, citations, and grants and include measures of

societal relevance and impact.

A well-crafted and well-

supported statement on the societal impacts of research

can be integrated into performance review and promotion

packets as an example of outreach and service.

What are Societal Impacts?

Societal impacts are the ways that research, and the process of conducting research, influences the world

beyond the academic realm.

While often in the climate and environmental sciences, we aspire to long-term and large-scale impacts like

measurable risk reduction or changes in ecosystem health, these kinds of impacts take substantial time to

develop and are dependent on a wide range of variables, such as the ability of societal partners to enact

new policies or practices. For example, a forest may not be demonstrably more resilient to fire in the span

of 2-3 years—the length of a typical research grant. Likewise, a city may not know if it is more resilient to

storm surges until the next time a large storm hits. However, there are a range of nearer-term impacts that

occur along a pathway to long-term and large-scale impact that can tell us about the ways in which our

research is helping to make beneficial changes.

In this guidebook, we present five categories of societal impacts. However, we focus on four that are more

likely to manifest on short- to medium-term timescales. We can observe and document these impacts and

use them as indicators of the likelihood of future socio-environmental impacts. Throughout this guidebook,

you will see most of our discussion involves the first four categories, but we will occasionally reference the

fifth (socio-environmental impacts) as part of the larger context of impacts evaluation.

FIVE CATEGORIES OF SOCIETAL IMPACTS

Instrumental applications

your research led to tangible changes to plans, decisions,

practices, or policies

Conceptual impacts

your research contributed to changes in people’s knowledge

about or awareness of an issue

Capacity building impacts

your research contributed to enhancing the skills, expertise, or

resources of an organization or group of people

Connectivity impacts

your research led to new or strengthened relationships,

partnerships, or networks that endure after the project ends

Socio-environmental impacts

changes to social and/or ecological systems, such as

improvements in health and well-being or in ecosystem

structure and function, that result from actions taken because

of your research.

Societal impact

categories

Example in practice #1 Example in practice #2

Instrumental

impacts

Changes to plans, decisions,

practices, or policies

“We worked with the staff of the local wildlife

management agency throughout this research.

When it came time for them to update their

species management plan, they cited our report

and journal article in the plan. They also asked

both myself (PI) and my Co-I to review their

plan to ensure that the research findings were

explained accurately.”

“The state Department of Transportation used

our research findings regarding drought and

the sources and patterns of dust storms,

which have caused numerous highway

casualties, to apply for federal funds to improve

highway signs, warnings, and road markings.

Funding was approved; new infrastructure was

installed in 2017.”

Conceptual impacts

Changes in people’s knowledge

about or awareness of an issue

“Health professionals who participated in this

project reported that they understood the

science behind regional temperature projections

much better than when the project started.

Two of these professionals took the initiative

to present the findings of our research to an

internal group in their health center, in order to

explain the research methods and findings to

their colleagues.”

“Results from our paleoclimate project,

conducted with a federal agency, provided

new insight into how temperatures impact

streamflow and drought in the Colorado River

basin. During subsequent meetings, water

managers explicitly discussed how they could

apply this insight into new techniques for

water management.”

Capacity building

impacts

Enhancing the skills, expertise,

or resources of an organization

or group of people

“Three graduate students, from backgrounds

often under-represented in STEM fields,

participated in this project. They gained data

collection, analysis, and project management

skills, participated in the writing of four

academic papers, and all three were accepted

into post-doctoral climate research programs.”

“We provided municipal planners with data

about flooding impacts on their city. We worked

with them in several workshops and phone

calls to ensure the data were clearly presented

and addressed the planning needs of the

city. Our partners have said they already feel

more confident and capable when discussing

flooding with city residents.”

Connectivity impacts

New or strengthened

relationships

“A group of city planners met for the first time

at a workshop we held as part of our project.

After the workshop, this group began meeting

regularly to collaborate on a funding proposal

for their city. They continue to invite me [the

researcher] to attend their meetings, and I do on

a regular basis.”

“In 2015, utility employees identified several

climate and environmental risks that could

impact their operations. Our research

team provided them with tailored data and

analyses to address their concerns. A year

later, utility employees contacted us again to

propose collaborating to develop scenarios for

carbon reduction. has turned into an ongoing

new project.”

Socio-environmental

impacts

Changes to social or ecological

systems, such as improved

health and well-being

or ecosystem structure

and function

“Our research team modeled the likelihood of

future heatwaves in our region. After working

with us on the research project, the City revised

its extreme heat response protocols. We now

have 10 years of data on heat-related deaths

in the city which show a significant decline,

despite the fact that we’ve had more and hotter

heatwaves during this same 10-year period.”

“We worked with local residents to develop a

reforestation plan, using findings from previous

research. The residents got funding from the

City for the project and they planted over 200

trees. Over the last three years, residents

noticed that several bird and mammal species

had repopulated the area. The reforestation

plan is bringing back biodiversity to the region.”

These categories do not represent all possible

types of societal impacts.

However, they provide

a foundation for exploring the societal impacts

of climate-related research. If your research

generates impacts that do not fit in one of these

five categories, you can still use the framework

presented in this guidebook to identify and

document your research impact and insert your

own impact categories as appropriate.

Impact categories are not hierarchical – one is not

more important than another. They all represent

meaningful results from the process of conducting

research. Impacts may interact within and across

categories and will often amplify each other.

For example, a network established to connect

researchers and policymakers may improve

relationships and trust among network members.

Within the network, policymakers feel free to ask

key questions, consider new data, and ultimately

become sufficiently confident in the research to

include it in a new environmental policy. All the

impact categories can contribute to beneficial

socio-environmental changes. They are worth

tracking to understand the changes that have

already occurred; they also indicate the likelihood

that more changes are coming.

5 Other frameworks can be found in:

Pedersen, David Budtz, Jonas Følsgaard Grønvad, and Rolf

Hvidtfeldt. 2020. Methods for mapping the impact of social sciences

and humanities—A literature review. Research Evaluation 29 (1):4-21.

Reed, Mark S. 2018. The Research Impact Handbook. 2nd ed: Fast

Track Impact.

Edwards, David M, and Laura R Meagher. 2020. A framework to

evaluate the impacts of research on policy and practice: A forestry

pilot study. Forest Policy and Economics. 114: 101975.

6 See Appendix A for further discussion about categories of

societal impacts.

How do researchers generate societal impacts?

There are several pathways that connect research to societal impacts. Often, they

flow through the following steps:

#1. You do robust, credible, and relevant research.

#2. That research connects to society.

#3. Societal partners use that research.

#4. That research changes something in the world.

It is in Step #2 — how research connects to society – that a number of different

pathways toward generating societal impacts open up. We introduce a few of these

pathways below.

Indirect Connections between Research and Research Users

One pathway involves indirect connections between researchers and research users. Perhaps someone comes across

your research findings in a book, journal article, or report and uses it to address their needs. They might cite these

research findings in a management or policy document. Along the indirect pathway, the researcher and the person

using research findings never interact in person. Indirect pathways are sometimes referred to as a “classical pipeline” or

“loading dock” model.

7 Read more about the “classical pipeline” pathway in: Muhonen, Reetta, Paul Benneworth, and Julia Olmos-Peñuela. 2020. From productive interactions to impact

pathways: Understanding the key dimensions in developing SSH research societal impact. Research Evaluation 29 (1):34-47.

Read more about the “loading-dock” model in: Cash, D. W., J. C. Borck, and A. G. Patt. 2006. Countering the loading-dock approach to linking science and decision

making - Comparative analysis of El Nino/Southern Oscillation (ENSO) forecasting systems. Science, Technology and Human Values 31 (4):465-494.

RESEARCH

OUTPUT

POLICY MAKER OR

PRACTITIONER

SOCIETAL

IMPACT

Adapted from Muhonen et al. 2020

Connecting through Public Engagement

A second pathway involves public engagement activities like giving talks to the general public or having your research

appear in news media, social media, or popular media like podcasts. In this case, members of the public may choose to

use your findings in their lives and the information might be conveyed from the general public into a policy or practice

use, such as if citizens advocate for a particular governmental action.

Direct Engagement Between Researchers and Societal Partners

A third pathway involves direct engagement between researchers and societal partners over the course of a project.

These partners might include natural resource managers, policy makers, or those responsible for managing the impacts

of climate change within their community, city, region, or country.

PUBLIC

AUDIENCE

SOCIETAL

IMPACT

RESEARCH

POLICY MAKER OR

PRACTITIONER

Adapted from Muhonen et al. 2020

SOCIETAL

IMPACT

RESEARCH

SOCIETAL

PARTNERS

RESEARCHERS

RESEARCH

QUESTION

SOCIETAL

NEED

Adapted from Muhonen et al. 2020

This guidebook focuses mostly on this third pathway – direct interaction between researchers and societal partners

throughout the research process. This pathway stems from decades of evolution in socially-engaged research practices,

which are more consistently effective in generating research that is useful to and used by society.

This approach

might involve interactive activities such as: researchers and societal partners jointly developing research questions and

project design, conducting fieldwork together, collectively analyzing data, and producing co-authored outputs, such as

papers or reports.

Who are societal partners?

In this guidebook, we use the term “societal partners,” or

“partners” for short, to refer to people who are engaged

in a research project but live or work outside of the

formal, academic research enterprise. Other common

terms for this group of people include practitioners,

decision makers, stakeholders, and community partners.

Our use of the term societal partners indicates that the

people engaged in the research process, and those who

are affected by the research, are active participants

in the research effort—not passive recipients of the

research outcomes.

Although engaged research approaches are the most effective

in moving research into use, these approaches come with some

challenges. They tend to be more time and resource intensive, which

imposes some “costs” on participants in terms of their own time,

energy, and resources. This is true for both researchers – who also

need to devote time and energy to their academic impacts – and

for societal partners – who also need to devote time and energy to

their own work and life responsibilities. Often, societal partners are

asked to participate in multiple research efforts due to their unique

perspectives or experiences. It is important to be aware of partner

burn-out

and researcher stress.

Some of these additional “costs”

can be mitigated through careful project planning and ensuring

adequate research funding and time.

8 See Appendix B for more information on socially-engaged research practices.

9 Young, Nathan, Steven J Cooke, Scott G Hinch, Celeste DiGiovanni, Marianne Corriveau, Samuel Fortin, Vivian M Nguyen, and Ann-Magnhild Solås. 2020. “Consulted

to death”: Personal stress as a major barrier to environmental co-management. Journal of environmental management 254:109820.

10 Cvitanovic, Christopher, Mark Howden, RM Colvin, Albert Norström, Alison M Meadow, and PFE Addison. 2019. Maximising the benefits of participatory climate

adaptation research by understanding and managing the associated challenges and risks. Environmental Science & Policy 94:20-31.

Contribution or Attribution?

Although it can be tempting to try to connect a research effort to a new policy or to an environmental change,

generating societal impacts from research is rarely a straight line. It is not always (quite rarely, really) possible to

attribute a change in policy, practice, or in the socio-ecological system directly to one particular research finding or

project. Multiple factors influence whether research is used and what impact it has for the intended users. Perhaps your

partners realize they need to build more internal capacity before being able to move forward with a new plan. Or, as is

often the case, the organization’s decision-making framework may require evidence from multiple sources before feeling

confident in taking action. Sometimes, a perfectly planned and executed project does not result in anticipated changes

due to factors outside of your control, such as budgetary constraints or changes in leadership.

Although it is not always possible to attribute specific impacts to individual projects, it does not mean that a project

has not been successful. More often, research has been a contributing factor – or one of several factors – in creating

change. While a singular research project or finding is not typically the sole factor in generating an impact, we can still

illustrate how and in what ways it contributed to the impact. Contributing to change is a positive outcome and should be

reported as such.

In a contribution model, like the one below, you might see that your research was one part of the process of making

changes in practice that ultimately contributed to improved environmental conditions.

Practitioners

work to make

changes in their

organizations

Sufficient

amount of

evidence

accumulated

Necessary new

skills acquired

Other

research

efforts

Other

research

efforts

Your

research

Improved

societal and

environmental

conditions

New

Management

practice in use

Professional

development

opportunity

How Research Contributes to Societal Impacts

First, your research might be one of several projects examining the phenomenon in question. A management agency is

likely to require several sources of evidence before it is willing to consider a change in practice. Therefore, having your

research as part of that evidence is crucial to the process. Second, a change in practice may require agency staff to

develop new skills. The agency may seek that training from another source and that source becomes another contributor

to the impact. When the new evidence and training is combined, the agency may be more able and willing to make the

change in practice. However, work still must take place within the agency to craft new policy language, move it through

internal decision channels, and secure appropriate funding – all of which require significant effort and know-how on the

part of agency personnel. In this example, that change in practice eventually leads to improved environmental outcomes.

The conceptual model below illustrates how one research project is part of a larger system that supports a change in

practice and, ultimately, a change in environmental conditions. While your research project may not be solely responsible

for changes in practice, policy, or socio-environmental conditions, using a contribution model like this one helps

demonstrate how your work has contributed to those greater impacts.

11 See Appendix C for more information about the challenges of documenting societal impacts.

The process for identifying and documenting research

impacts presented in this guidebook is built on an

outcome-oriented logic model framework.

Logic models

have similar basic components, but do not always look

the same. They can be adjusted to apply to projects and

programs in multiple ways. Logic models are not rigid;

they are flexible tools that can be used in a variety of

ways. A general way to understand logic models is that

they offer a comprehensive explanation of how and why

a desired change is expected to happen in a particular

context. This framework invites researchers to make

explicit connections between project resources, activities,

outputs, and how these lead to specific

impacts or changes.

12 See W.K. Kellogg Foundation http://www.wkkf.org/knowledgecenter/resources/2006/02/WK-Kellogg-Foundation-Logic-Model-Development-Guide.aspx

In this section, we present two applications of the logic

model framework. The first variation is meant to guide

researchers who are in the planning and design stages

of a project. The second variation is geared toward

researchers who are in the middle or final stages of a

project and want to document impacts that have already

occurred. To illustrate how this framework functions, we

will work through both variations separately and apply

them to a case study example about coastal flooding and

sea level rise.

PART 2.

Societal Impacts Planning

and Assessment Frameworks

2.1 Framework for Project Planning and Design

Generating societal impacts requires re-thinking the way research projects are designed. For example, the degree

to which societal partners are involved (or not) at various steps of the research process will determine the types of

impact.

Used during research planning, design, and proposal writing stages, the following logic model framework

encourages researchers to think through and explicitly connect what they anticipate doing and producing during their

project, to the societal impacts they anticipate achieving at the end of their project.

13 Shirk, Jennifer L, Heidi L Ballard, Candie C Wilderman, Tina Phillips, Andrea Wiggins, Rebecca Jordan, Ellen McCallie, Matthew Minarchek, Bruce V Lewenstein, and

Marianne E Krasny. 2012. Public participation in scientific research: a framework for deliberate design. Ecology and society 17 (2).

ACTIONS/ACTIVITIES

OUTPUTS

How will you

address the issue?

• Methods for

engagement

with partners

• Project activities

INPUTS

What are the

interests of both the

researchers and

societal partners?

What resources

and expertise are

necessary for

this project?

What tangible

products do you

expect to produce?

• Reports

• Papers

• Circula

•Databases

SOCIO-ENVIRONMENTAL IMPACTS

Changes to social or ecological

systems over time

For whom,

will things change

CONCEPTUAL

IMPACTS

INSTRUMENTAL

IMPACTS

CONNECTIVITY

IMPACTS

CAPACITY BUILDING

IMPACTS

Gain new knowledge

or awareness

Direct applications

of research

Establish or

strengthen

partnerships

Gain or improve

skills or expertise

What societal problem do you aim

to address in your research?

PROBLEM STATEMENT

What societal problem do you aim to address in your research?

Write out the specific societal or

environmental issue that your research

addresses. It should speak to the broader

issue that your research will inform. Most

likely, it will be different from your actual

research questions or objectives. In your

statement, include who is affected by this

issue.

Floodsville Project Problem Statement:

A tropical storm caused major flooding in a neighborhood of

the City of Floodsville. Ocean water breached the existing

coastal wall, leading the city’s planners and emergency

managers to question whether they had the most accurate

projections for sea level rise. They need to up-to-date sea

level rise information so they can make better informed

policies and decisions for the future of the city.

INPUTS

• What resources are available or necessary to conduct your research?

• What are the interests of both the researchers and the research partners?

Describe the inputs that you have available or

that you will need to conduct your research.

Your inputs will vary depending on the type

of project you are conducting. Fundamental

resources include time and funding. Consider

the human resources needed to carry out your

research, including skills, types of knowledge,

and expertise. Describe the members of your

research team (including yourself) and the

expertise, knowledge, and skills each person

offers to the project. If you do not already have a

research team established, write down the types

of members you will seek out. Once your team is

established, each member should think through

their individual motivations for participating

in the research. What kinds of societal impact

goals motivate each research team member?

How will they contribute toward those impacts?

The financial and human resources available

and the motivations of the research team will

drive the research design, as well as the project’s

activities, outputs, and impacts.

Floodsville Project Inputs:

Human resources needed:

• Climatologist with skill in climate modeling, experience

with sea level rise

• GIS specialist to map model projections

• Floodsville City official with knowledge about

city policies and the climate adaptation plan and

connections to city planners and emergency managers

• Neighborhood Association representative where

flooding occurred

• Social scientist to gather qualitative data about

residents’ experiences

Physical resources needed:

• Funding for climate modeler, GIS specialist, social

scientist, neighborhood association representative

• Time from city officials who participate in the project

• Time from community members who participate

in the project

Motivations, objectives, and interests of the

research team:

• Climatologist: to apply their modeling knowledge to a

specific societal issue

• GIS specialist: to fulfill their contract, but wants to help

solve the City’s issue

• City official: to ensure safety for the future of the City

• Community member: to ensure safety for

neighborhood residents

• Social scientist: to engage with local community

members and help inform a better adaptation plan

ACTIVITIES

• What activities will you do to address the broader societal issue?

• Do your proposed project activities incorporate the project inputs and the interests of your

research team?

In the activities section, focus on the

infrastructure of your project, which may

include things such as: a calendar timeline

of your proposed activities; the research

methods you will use; a description about

how you and your research team plan to

interact and communicate throughout the

project; or describing the responsibilities of

each member of the research team.

Floodsville Project Activities:

This two-year project will include two main

components of research:

1. Risk assessment: analyzing current climate models

for sea level rise for the region around Floodsville. The

climatologist will run the model analyses and produce

downscaled climate projections. She will coordinate with

the GIS specialist to construct risk maps for the region

based on these analyses.

2. Qualitative analysis of people’s experiences with the

flood and future plans to deal with flood risk. The

social scientist will coordinate with the community

representative and the Floodsville city official to develop

protocol for focus groups with the neighborhood

association and with city planners and emergency

managers. Three focus groups will occur with the

neighborhood association and three will occur with the

city planners and emergency managers in the first year.

The entire research team will meet monthly throughout the

project to discuss updates and coordinate next steps. Between

regular meetings, team members will communicate via email

and phone as necessary.

At the end of the project, the research team will present

findings and recommendations for the city’s adaptation plan to

city officials, planners, and emergency managers. We will also

present our findings at a neighborhood association meeting.

OUTPUTS

• What do you plan to produce through your research?

In this section, write out the tangible research

outputs that you and your team plan to produce

over the course of your project. These outputs

may include several different types of products,

such as: data analyses and models; public

outreach materials; white papers and peer-

reviewed articles; teaching curricula or training

materials; reports tailored to specific audiences;

websites; or fact sheets. As you outline this

section, think about when you want these

outputs to be completed, keeping in mind that

some of them may fall outside of your anticipated

timeline.

Floodsville Project Outputs:

1. Climate model analysis

(Completed in 6 months)

2. Risk map of Floodsville region

(Completed by end of Year 1)

3. Policy recommendations for City’s Adaptation Plan

regarding sea level rise

(Completed in 18 months)

4. Fact sheet report for neighborhood where flooding

occurred (printed and available on City website;

completed by end of Year 2)

5. Peer-reviewed paper submitted

(completed 1 year after end of project)

IMPACTS

• What do you expect to change as a result of your activities, your interactions with the public and your

research partners, and your outputs?

• For whom will things change or who will change because of your research?

In this section, think through the types of

impacts you anticipate having through your

research project. Your impacts section does

not need to be comprehensive, but it should

contain a list of attainable goals. As we know,

research projects often do not proceed as

intended. Not all research projects will yield

outcomes in all categories. Achieving one

impact may depend on accomplishments

in other impact categories. At the end of

a project, you may also discover several

unanticipated impacts.

We suggest starting with the first four of the

five categories of impact we introduced earlier

in this guidebook, which include:

• Conceptual impacts – new knowledge or

awareness gained

• Connectivity impacts – partnerships

established or strengthened

• Capacity building impacts – skills or

expertise strengthened or acquired

• Instrumental impacts – direct

applications of research

As you work through your anticipated impacts,

define who your societal partners are, such as:

resource management agency representatives,

policy makers or government officials,

community members, industry professionals,

students, or members of the general public. As

part of this section, you might also consider

the demographics of the people who will be

impacted by your research. Will your research

impact the audiences that you aim to reach?

Floodsville Project Impacts:

Conceptual impacts

• City officials, planners and emergency managers have a

better understanding of future flood risk and sea level rise;

they also understand the experiences of the residents

whose neighborhood was flooded

• Neighborhood residents have increased awareness about

why the recent flooding happened and the future impacts

they might face

• Researchers have a better understanding of the city

planning processes and experiences of flooding in the city

Capacity building impacts

• City residents feel better equipped to prepare for

future flood risk

• City officials feel equipped to discuss climate projections

and make decisions to address future flood risk

Connectivity impacts

• Connect city officials with neighborhood residents

• Increase connections between emergency planners and

city managers – currently there is little communication

between these agencies.

Instrumental impacts

• Inform Floodsville’s new multi-hazard mitigation plan

regarding flood

• Inform emergency management policies to deal with

future flood risk

Socio-ecological impacts (long-term aspirations)

• As sea-level rise occurs, Floodsville does not experience

any further flooding

• Floodsville saves significant amounts of money by

implementing actions from their hazard mitigation plan

PLANS TO COLLECT EVIDENCE OF CHANGE

• How will you know that your research changed things for people?

Your research design should incorporate plans

to systematically collect pieces of evidence that

corroborate the impacts of your research. Some

research projects will require external evaluation,

which can generate high quality data from formal

surveys, interviews, and even randomized control

trials. However, societal impacts evidence can

also be collected by research team members

and societal partners. These sources of evidence

might include: feedback from partners in the

form of emails, letters, or conversations you are

able to document; reference to your work in a

management/policy document; or feedback from

the general public.

Floodsville Project Evidence of Change:

• Citations in Floodsville’s Multi-Hazard Mitigation Plan

• Emails and discussions with City officials, planners, and

emergency managers and with neighborhood residents

• Solicit feedback via survey at final project meetings

with City officials, planners, and emergency managers

and with neighborhood residents

REVIEW YOUR PLAN

Once you have completed outlining each piece

of your project framework, review your plan to

see if each step logically flows into the next

step. Make sure your inputs relate to your project

activities; that activities relate to outputs; that

your outputs relate to your anticipated impacts;

and that impacts address the original interests

and motivations of the research team. Write a

quick summary about how your impacts address

the societal problem you are trying to inform with

your research.

Floodsville Project Design Review:

This project will provide up-to-date projections for sea

level rise in the region around the City of Floodsville.

By working with city officials, planners, and emergency

managers, as well as gathering evidence with

neighborhood residents, our project will provide insight

into how people cope with flood-related disasters

and will help them prepare for, and hopefully prevent,

future flood risk.

2.2

Societal Impacts Framework for Impact

Assessment and Project Reflection

What societal problem did you aim

to address in your research?

ACTIONS/ACTIVITIES

OUTPUTS

INPUTS CONTEXT

CONCEPTUAL

IMPACTS

INSTRUMENTAL

IMPACTS

CONNECTIVITY

IMPACTS

CAPACITY BUILDING

IMPACTS

SOCIO-ENVIRONMENTAL

IMPACTS

How did you try to

address it?

• Methods for

engagement

with partners

• Project activities

What resources, skills,

and existing relationships enabled

you to conduct your research?

What social, environmental, and

political factors facilitated or

inhibitied your research?

What tangible

products did

you produce?

• Reports

• Papers

• Circula

•Databases

New knowledge or

awareness gained

Direct applications

of research

Partnerships

established or

strengthened

Skills, expertise

strengthened or

acquired

Changes to social or

ecological systems

over time

For whom,

did things change

PROBLEM STATEMENT

• What societal problem did you aim to address in your research?

Summarize the specific societal or

environmental issue that you addressed in

your research in one or two sentences. It will

likely be different from your actual research

questions or objectives. In your statement,

include who was affected by this issue.

Floodsville Project Problem Statement:

A tropical storm caused major flooding in a neighborhood of

the City of Floodsville. Ocean water breached the existing

coastal wall, leading the city’s planners and emergency

managers to question whether they had the most accurate

projections for sea level rise. They wanted to up-to-date sea

level rise information so they could make better informed

policies and decisions for the future of the city.

ACTIVITIES

• What actions did you take to address the broader societal issue?

In this section, state your

research methods briefly.

Then provide a description

of how you interacted or

communicated with societal

partners through your research.

Floodsville Project Activities:

• Downscaled climate and sea level rise projections to the scale of

Floodsville and the surrounding region.

• Attended 4 meetings with city planners and emergency managers during

the winter and spring of 2017. We presented our research (using maps) at

each meeting and engaged in discussions with decision makers about the

research process as well as implications of our findings.

• City officials called us regularly throughout their planning process to

confirm their understanding of the research (March – November 2017).

• Attended two community association meetings focused on the flood event

and community plans for flood relief (summer 2017). Answered questions

about projections of future flood events linked to sea level rise.

• Conducted two focus groups with community members (one with 10

participants and one with 8), one focus group with city planners (5

participants), and one with emergency managers (8 participants). All

occurred in spring and summer 2017. These focus groups helped us

understand the scale of analysis that was most helpful to all three groups,

how to present the data, and made us more aware of questions and

concerns of the community members.

• Provided city officials and community members with detailed maps

and summaries of our analysis during a presentation with each group

(December 2017).

OUTPUTS

• What did you produce through your research?

Briefly describe your research

findings. Then write out the tangible

research outputs you produced.

If you anticipate producing more

outputs (such as a peer-reviewed

article) note that here as well.

Your outputs could take many

forms, such as: data analyses and

models; public outreach materials;

white papers and peer-reviewed

articles; teaching curricula or

training materials; reports tailored

to specific audiences; websites; or

fact sheets.

Floodsville Project Outputs:

Findings: We found that the City’s current sea wall was not high

enough for the current sea levels or future levels. We provided

estimates to inform the height of a new wall.

Products:

• Downscaled climate projections for the Greater Floodsville region.

• Maps of projected sea level rise in the region – maps comply with

FEMA data standards for easy use in planning documents.

• Presentations summarizing research findings provided to

city planners.

• Two-page factsheet, aimed at community audience, summarizing the

research and addressing questions and concerns raised by community

members at meetings.

• A peer-reviewed publication has been submitted.

IMPACTS

• What changed as a result of your activities, your interactions with the public and your research

partners, and your outputs?

• For whom did things change or who changed because of your research?

In this section, identify what your project activities and outputs did for your societal partners. Think through

how your activities led to certain outputs, and how your activities and outputs contributed to change. In

reviewing your original research design, did you have the type of impacts you anticipated? Did you have any

unanticipated impacts?

Begin your description of impacts with one of the impact categories we introduced above: conceptual impacts –

new knowledge or awareness, connectivity impacts – partnerships established or strengthened, capacity building

impacts – skills or expertise strengthened or acquired, and instrumental impacts – direct applications of research.

Identify for whom things changed because of your research. This list may include resource management agency

representatives, policy makers or government officials, community members, industry professionals, students, or

members of the general public. You might also consider the demographics of the people who were impacted by

your research. Did your research impact the audiences that you aimed to reach?

IMPACTS CONTINUED

Floodsville Project Impacts:

Conceptual impacts:

• Our research team received emails from city council about their increased understanding about the risks of sea

level rise.

• Community members asked increasingly in-depth questions about the impacts that they might face, and used

this to think about new ways they would prepare for floods

People impacted: Floodsville community members from a neighborhood that had historically been underfunded

for infrastructure improvements by the Floodsville city government; Floodsville city councilmembers.

Capacity building impacts:

• A city council member used our Powerpoint presentation files to give a presentation to another city that was

recently flooded.

• Floodsville City Government gave staff members time to learn GIS skills from our research team in our lab

facilities, which helped them to be able to map the projections. This was a new skill set for the staff.

People impacted: Floodsville city government staff members.

Connectivity impacts:

• We built new relationships with city planners and community members.

• Our existing relationships with emergency managers were strengthened.

• The city planners invited us to do a new project about implementing green infrastructure.

• Our research team continues to get requests from the city for presentations.

People impacted: Floodsville community members; Floodsville city planners and emergency managers; members

of the project research team.

Instrumental impacts

• The new projection maps were used in the city’s new multi-hazard mitigation plan.

• The city’s multi-hazard mitigation plan was accepted by FEMA.

People impacted: Floodsville city government officials.

Long-term socio-ecological impacts (assessed using long-term evaluation process)

• 25 years later, sea level rise of 1 meter occurred, but flooding of the town was avoided

• Floodsville saved $5 billion USD by implementing actions from their hazard mitigation plan

EVIDENCE OF CHANGE

• How do you know that your research changed things for people?

Here, you can cite evidence that shows how

your research was used or how things changed

because of your research. Some examples of

evidence include:

Feedback from your partners

• Formal (letters of recommendation or

partnership)

• Informal (email or phone calls)

Reference to your work in a management/

policy document

• Citation to support statements

• Research findings forming the basis of

policy document

• Citation in management reports or publications

Feedback from the general public

• Audience surveys

• Emails or other engagement from public

• Media interviews/reference to your work

Formal evaluation of your work

• Randomized control trials

• Pre-post tests

• Surveys/interviews of partners

Floodsville Project Evidence of Change:

• City of Floodsville Multi-hazard Mitigation Plan. 2018.

Floodsville Department of Emergency Management.

• City of Floodsville Long-Range Infrastructure Plan. 2018.

Floodsville Department of Planning.

• Documents available online at:

floodsvillestoppedflooding.gov

• Emails from community members

and government employees

• Qualitative data from a survey distributed to city

employees and community members

PROJECT REFLECTION AND REVIEW

• What resources and factors enabled this project and your research impacts?

• What components of the research were effective and which ones were not as effective?

Once you have completed outlining your

research impacts, you may want to reflect on

the parts of your project that were effective

and those that were not effective. Write down

some of the resources you had available

that enabled your research. Think about the

environmental, social, and political factors

that helped or hindered your research. Some

examples include:

• Process oriented factors – the way the

problem was framed; project management;

the way findings were disseminated; outputs

matched partner needs

• Inputs – key skill sets, funding, and

resources of the research team

• External factors – the political, economic, or

social context supported, or did not support,

action or change

Floodsville Project Review:

Overall, our project turned out to be effective. After the flood,

the City looked back at their hazard mitigation plan regarding

flooding and decided to update it. They want to keep in touch

about future updates to sea level rise projections. Residents

in the neighborhood that experienced flooding feel prepared

to deal with future risk and have more trust in the city’s

multi-hazard mitigation plan to reduce their future risk.

Some of the inputs that helped enable our project include:

• The knowledge and capacity on our research team to

provide projections

• Funding for research and engagement with partners

• Existing relationships between researchers an Floodsville

City Emergency Managers

Some of the context that enabled our project include:

• A flood event happened which galvanized

action in the city

• Community members were motivated to address the issue

• The city made it a priority to update their hazard plans

2.3

Documenting Impacts –

Worksheets and Case Studies

PROJECT TITLE:

Societal or Environmental Issue:

What societal problem do you aim to address in your research?

• Summarize your problem statement in 1 – 2 sentences

• Different from your research questions and research objective

• Address who is affected by this societal problem

Inputs

What resources are available or necessary to conduct your research?

What are the interests and motivations of each member of your research team?

• Describe the resources that you have available or will need to conduct your research

• Define your research team, and the skills, types of knowledge, and expertise each member offers to the project

• Identify the societal impact goals that motivate each research team member to be part of the research project

This section contains worksheets filled with guidance and additional case study examples to help you plan and

document your own research impacts. The first set of worksheets pertain to the Societal Impacts Framework for

Project Planning and Design. The second set of worksheets are based on the Societal Impacts Framework for Impact

Assessment and Project Reflection.

Societal Impacts Worksheet for Project Planning and

Design Guidance Document

Click here to download your own worksheet template for project

planning and design in Google Drive.

PROJECT CONTINUED:

Anticipated Activities

What actions will you take to address the societal issue?

• State your research methods

• Outline roles and responsibilities for each member of the research team

• Describe how the research team will engage and communicate with one another during the project

• Outline the activities that will take place throughout the project

Anticipated Outputs

What do you plan to produce through your research?

• Observations and data

• Tangible research outputs might include:

• Public Outreach Materials

• Models/Datasets

• Reports for Partners

• Fact Sheets

• Websites

• Peer-reviewed articles

• Curricula

• Others

Anticipated Impacts

What do you expect to change as a result of your activities, your interactions with partners, and your project

outputs?

• Conceptual outcomes - new knowledge or awareness gained

• Connectivity outcomes - partnerships established or strengthened

• Capacity Building outcomes - skills, expertise strengthened or acquired

• Instrumental outcomes - direct applications of research

For whom do you expect things will change?

Evidence of Impact

How will you know that things have changed?

• Describe your plan to collect, document evidence of change. Some examples include:

• Feedback from your partners

• Reference to your work in management or policy documents

• Feedback from the general public

• Formal evaluation of your work

PROJECT TITLE:

The Influence of Temperature and Soil Moisture on Colorado River Water Resources

Societal or Environmental Issue:

The U.S. Southwest is an arid region and is becoming hotter and drier due to climate change. Climate model-based

temperature projections indicate Colorado River streamflow reductions of 20 percent by 2050 and 35 percent by

2100. Approximately 40 million people in the western U.S. and northwestern Mexico rely on Colorado River water for

domestic, industrial, and agricultural uses.

This project aims to better understand how temperature has influenced Colorado River water resources. Water

resource managers who work in the Colorado River Basin will use this information to inform resource management

plans and operations, which has implications for people, cities, and ecosystems across the Southwest.

Inputs

To develop the proposal, we have engaged with a group of Colorado River basin water managers who have

assisted in directing research questions and ensuring the project is relevant to resource management. This group

of water managers will form an advisory board to guide the analysis and generation of research products through

interaction with the science team. The members of the advisory board have long-standing relationships with

members of the science team from previous research engagements. They are interested in using the research

findings for their own purposes, and are informing the project design, but are not interested in collecting or

analyzing data.

The science team will be collecting and analyzing the data and communicating findings to the advisory board. The

science team is an interdisciplinary group with expertise in climatology, paleoclimatology, hydrologic modeling, and

climate projection analysis, with extensive experience working with resource managers. The members of the team

want to apply their expertise to address water management issues in the Southwest. One member of the team

will be using part of this research for her dissertation.

This project has funding for 2 years.

Anticipated Activities

• Engagement activities include:

• quarterly meetings with the advisory board

• email and phone calls between science team and advisory board members as needed

• a final project workshop with water resource managers

The science team will meet monthly to discuss project updates and will communicate as needed

by phone and email.

Societal Impacts Worksheet for Project Planning and Design

Case Study Example

The Influence of Temperature and Soil Moisture on Colorado River Water Resources

Anticipated Outputs

• Analysis of instrumental data

• Analysis of tree-ring chronologies

• Soil moisture model

• Project website, kept up to date as research is completed

• Quarterly reports to advisory board

• Presentations to academic audiences

• Workshop with water resource managers

• Final report for water resource managers

• Academic publication

Anticipated Impacts

• Inform decision making and planning for water management and conservation measures, by providing

information about the roles seasonal precipitation, temperature, and antecedent moisture conditions

• Raise awareness of possible impacts of warming temperatures on drought and expected runoff

now and in the future

• Increase awareness about how to use tree-ring data for future planning, recognizing t that the past

is not an analog for the future

• Increase awareness about the climate connections between winter precipitation and fall and spring

temperatures in the Colorado River Basin

Evidence of Impact

• Annual interviews with members of the advisory board about the research process and how they have used

or plan to use project findings

• Interviews with members of the science team about the research process, research findings, and

implications of their findings

• Feedback from workshop participants via online survey

• Evidence from emails and feedback from advisory board about use of information

PROJECT TITLE:

Summary Statement of Impacts

Return to this section after you have completed the rest of the worksheet.

Summary of the Research

What societal problem did you aim to address in your research?

• Summarize this problem statement in 1 – 2 sentences

• This is likely to be different from your research questions and your research objective

• Be sure to address who is affected by this societal problem

Project Engagement Activities

What actions did you take to address this issue?

• State your methods briefly to give a sense of the kind of research you did

• Provide a description of your approach to societal engagement

Research Outputs

What did you produce through your research?

• BRIEFLY describe your research findings

• Tangible research outputs:

• Public Outreach Materials

• Models/Datasets

• Reports for Partners

• Fact Sheets

• Websites

• Peer-reviewed articles

• Others

Societal Impacts Worksheet for Project

Reflection and Impact Assessment

Guidance Document

Click here to download your own worksheet template

for project planning and design in Google Drive.

PROJECT CONTINUED:

Details of Impacts

What changed as a result of your activities, your interactions with partners, and your outputs?

• Conceptual Impacts - new knowledge or awareness gained

• Connectivity Impacts - partnerships established or strengthened

• Capacity Building Impacts - skills, expertise strengthened or acquired

• Instrumental Impacts - direct applications of research

For whom did things change?

Evidence of impact

Examples of evidence of change

• Feedback from your partners

• Formal (letters of recommendation or partnership)

• Informal (email or phone calls)

• Reference to your work in a management/policy document

• Citation to support statements

• Research findings forming the basis of policy document

• Citation in management reports or publications

• Feedback from the general public

• Audience surveys

• Emails or other engagement from public

• Media interviews/reference to your work

• Formal evaluation of your work

• Randomized control trials

• Pre-post tests

• Surveys/interviews of partners

PROJECT TITLE:

Climate Change and Regional Fish Management

Summary Statement of Impacts

We worked closely a coalition of natural resource managers to identify the cumulative impacts of climate change,

habitat degradation, and invasive species on an important fish species in this region. Based on this research,

the coalition was able to prioritize funding for management of this species. In addition, a federal resource

management agency used the findings to inform its decision to build a physical barrier to protect the key species

from an invasive competitor.

Summary of the Research

This project used a novel modeling approach to develop multi-species climate change vulnerability assessments,

which offer the ability to relatively rank and prioritize populations of multiple sympatric species. We concluded

that climatic drivers, which are damaging the species’ habitat, as well as introduction of an invasive species are

contributing to the loss of the focal species in the region.

Project Engagement Activities

The lead researcher worked closely with the coalition group. The coalition’s mission is to support the management

of ecologically significant species in this region through building awareness and sharing best practices between

and among federal and state natural resource management agencies in the region. The researcher attended

coalitions meetings approximately 3 times each year during the project (2016 – 2018) and co-convened a workshop

with the coalition (Summer 2018) where they elicited feedback from other regional resource managers about early

project findings.

The researcher and the coalition leaders communicated regularly by phone and email, at least every month. The

coalition team members remarked on how often the researcher attended their general meetings.

Research Ouputs

• Several scientific papers analyzing the extent to which the focal species is affected by invasive fish species in

the region and the rate of habitat degradation expected due to climate change.

• Maps of currently stressed and fragmented habitat areas.

• Maps of areas likely to become more stressed in the future due to climate change impacts.

• Maps of presence of focal species and invasive species.

• All maps were integrated into an existing decision support tool used regularly by

resource managers in the region.

Societal Impacts Worksheet

Case Study Example Document

Climate Change and Regional Fish Management continued

Details of the Impacts

• The management coalition used our research findings as part of its decision to prioritize funding for

management of the focal species. They reallocated available resources to their efforts to restore habitat

in the region.

• A federal agency used the findings to inform its decision to build a physical barrier at one key location (identified

through our work) to protect the focal species from its invasive competitor.

Sources that corroborate the Impacts

• Regional Aquatic Species Management Coalition Strategic Plan 2018 – 2023.

• Federal Agency Species Management Plan 2019.

• Environmental Impact Statement regarding placement of physical barrier at No Invasives Creek. 2018.

Appendix A: Exploring Societal

Impacts Categories

A number of different impact categories have been

developed for use in different contexts. Many people

use a common core of impact types but refine or add

new categories to better suit a particular use. There are

good reasons to rely on a common typology, such as the

ability to compare and contrast projects, programs, and

organizations, and to provide consistency over time to

measure progress in achieving impact goals. However,

because the societal benefits accruing from research

are highly context-dependent (i.e. different communities,

decision makers, or policy makers need different things),

retaining the ability to demonstrate impacts in ways that

reflect the true societal benefits is important. Below we

summarize some of the common impacts definitions in

order to provide options and ideas for researchers looking

for appropriate descriptions of their work.

A standard typology, that emerged from policy studies

(Weiss 1973; Pelz 1978), and has been applied to societal

impacts assessments (Walter, Davies, and Nutley 2003;

Meagher, Lyall, and Nutley 2008; Meagher and Martin 2017),

focuses on two core categories of impact:

• Instrumental impacts - direct impacts of research on

policy and practice decisions where a specific piece

of research is used in making a specific decision or in

defining the solution to a specific problem

• Conceptual impacts - complex and indirect ways in

which research can have an impact on the knowledge,

understanding, and attitudes of policy makers and

practitioners.

Kuruvilla et al (2006) designed the Research Impact

Framework as a tool to assist researchers in documenting

their own impacts and added two additional impacts that

has also previously been described by Weiss (1998):

Mobilization of support - research findings provide

persuasive evidence to back ongoing and proposed

policy activities or raise awareness and support for new

policy-making. Also termed symbolic or justification by

Pelz (1978).

Wider influence - research impacts that lead to a wide

change or transformation of accepted beliefs and

practices, beyond the use or impact to a specific, direct

societal partner.

PART 3.

Additional Context and Information

Part 3 contains four appendices that provide additional context and information about several of the topics introduced

in this guidebook. The appendices are intended to dive deeper into the academic literature on societal impacts, engaged

research approaches, and research evaluation for readers who wish to explore these topics in greater detail.

Meagher and Martin (2017) expanded impacts categories to include impacts with an indirect, but still traceable influence,

on use of research findings:

• Capacity building impacts - training and/or developing collaborative abilities

• Enduring connectivity - establishment of long-lived external relationships

• Attitude or cultural change - increased willingness to engage in knowledge exchange activities, on the part of

individuals, institutions, or organizations.

• The National Science Foundation (NSF) includes “broader impacts” as a key component of research it funds. NSF’s

categories tend to emphasize improvements to the broader science enterprise through:

• Full participation of women, persons with disabilities, and underrepresented minorities in STEM

• Improved STEM education and educator development at any level

• Increased public scientific literacy and public engagement with science and technology

• Development of a diverse, globally competitive STEM workforce

• Increased partnerships between academia, industry, and others

• Enhanced infrastructure for research and education

The list of NSF impact examples also includes:

• Improved well-being of individuals in society

• Improved national security

• Increased economic competitiveness of the United States

Process as an Indicator of Impacts

Researchers also recommend evaluating the process of engagement as a key indicator of the likelihood of generating

impacts (see Appendix B for a more detailed discussion of this process). The Social Impact Assessment Methods for

research and funding instruments through the study of Productive Interactions (SIAMPI) framework was developed

by Spaapan and van Drooge (2011) to evaluate the return on investment of European Union-funded research. They

recommended documenting productive interactions between researchers and societal partners in three categories:

• Direct interactions - ‘personal’ interactions involving direct contacts between humans, interactions that revolve

around face-to-face encounters, or through phone, email or videoconferencing (i.e. research findings and processes

are conveyed directly between researchers and societal partners).

• Indirect interactions - contacts that are established through some kind of material ‘carrier’, for example, texts, or

artefacts such as exhibitions, models or films (i.e. research findings are transferred through a report, article, or

other media).

• Financial interactions - when potential partners engage in an economic exchange with researchers, for example, a

research contract, a financial contribution, or a contribution ‘in kind’ to a research program.

Similarly, Ford et al. (2013) link the likelihood of impacts to the extent to which the research has been intentionally

co-designed with decision makers. They note that all types of research, including basic or fundamental science as well

as applied or action-oriented science, have the potential to influence policy, inform decisions, and alert the public to

potential problems; but only “usable science” – or research that is explicitly designed to contribute to decision making

processes and have the characteristics of pertinence, quality, and timeliness – does so directly.

Impacts in Context

All research impacts are context-dependent (Boaz, Fitzpatrick, and Shaw 2009; Mach et al. 2020); they will differ

depending on the project goals, sector, processes, and participants. Thus, while maintaining a common core of impacts

categories is valuable for consistency and comparison, allowing for flexibility in reporting structures is crucial to ensure

that the impacts most important to societal partners are prioritized. For example, Reed (2018) presents a more fine-

grained list of impact categories, intended to help researchers plan and document their own impacts. In addition to

his categories for capacity/preparedness, understanding and awareness, attitudinal, and policy (which align well with

capacity-building, conceptual, and instrumental impacts), he suggests:

• Economic – monetary benefits arising from research

• Environmental – benefits from research to genetic diversity, species or habitat conservation, ecosystems, and

ecosystem services

• Health and well-being – research that leads to better outcomes from the health of individuals, social groups, or

public health.

• Cultural – changing in the prevailing values, attitudes, beliefs, discourse and patterns of behavior

A caveat to using pre-defined categories: Fazey et al. (2014) and Wiek et al. (2014) point out that while it is possible to

anticipate some impacts, it is important to be open to unanticipated or unintended outcomes. Unexpected results can

provide additional insights into the possibilities for future impacts.

Summary

There is some coalescence around a common core of impact categories, particularly in frameworks developed for use by

European research institutions, who are more likely to have mandated reporting of societal impacts. This common core

– instrumental, conceptual, capacity-building, and connectivity – can be beneficial when seeking to compare research

projects, program, and organizations or when seeking to track progress in meeting impact goals over time. However,

in order to accurately capture the impacts from diverse research and investigators, maintaining flexibility within

frameworks is important. Space should be maintained for unanticipated or unexpected impacts or for describing impacts

that do not immediately fit a pre-defined category. Exploring alternative impacts categories can help researchers identify

the breadth of categories they may generate and provide ideas for framing and describing impacts.

Appendix B: Socially Engaged Research

As noted throughout this guidebook, one pathway to

generating societal impacts is through direct interactions

between researchers and societal partners throughout

the research process. This pathway stems from decades

of development in the theory and practice of socially-

engaged research.

In the U.S., the conceptualization of how science brings

about innovation and social change can be traced

back to the post World War II era when the first clear

articulation of non-war federal science policy was put

forward by Vannevar Bush (Guston 2001). The vision

that Bush formulated in a report to President Truman—

Science, the Endless Frontier (1945)—emphasized the

need for government support for science to continue

during post-war times. Research and technology, he

believed, would spur innovations in health, agriculture,

national security, and other forms of public welfare.

But Bush also emphasized that the “freedom of inquiry

must be preserved. As long as universities are vigorous

and healthy and their scientists are free to pursue the

truth wherever it may lead, there will be a flow of new

scientific knowledge to those who can apply it to practical

problems in Government, in industry, or elsewhere” (1945).

This vision of science policy was originally developed

to maintain scholarly independence from government

or corporate influence (Dennis 2015). It assumes that

discoveries made by academic experts will be delivered

to members of society who will use the results to inform

policy, practice, and technology (Cash, Borck, and Patt

2006). However, the model does not explain how scientific

results will be delivered, consumed, or used. Evidence has

accumulated to indicate that when research is separated

from its intended users (such as in the loading-dock

model described by Cash et al. 2006), it is not often used

in practice and policy. For example, Bornmann et al. (2016)

found that only 1.2% of climate science research produced

between 1980 and 2014 was cited in international climate

policy documents.

In response to this gulf between research and practice,

new modes of research have evolved. These new modes

can be broadly categorized as socially engaged research.

Various approaches and methods within this vein include

action research (Lewin 1946; Greenwood and Levin 2007);

post-normal science (Funtowicz and Ravetz 1993); Mode 2

knowledge production (Gibbons et al. 1994; Nowotny, Scott,

and Gibbons 2001); boundary work (Guston 2001; Cash et

al. 2003); knowledge coproduction (Jasanoff 2004; Lemos

and Morehouse 2005); transdisciplinarity (Pohl 2008; Jahn,

Bergmann, and Keil 2012); transformational sustainability

science (Kates et al. 2001; Wiek et al. 2012); social learning

(Wenger 2010; Pahl-Wostl et al. 2007); and useful science or

usable knowledge production (e.g. Dilling and Lemos 2011;

Clark et al. 2016; Mach et al. 2020). See the table below

for brief descriptions. These frameworks converge and

diverge in terms of agenda and approach. However, they

all make one thing clear: physical and social scientists

must collaborate with people outside of the

academic realm if they want their research to inform

policy or create societal and environmental change.

Socially engaged research concepts

Research concept Description Exemplary citations

Action research Research primarily defined and driven by stakeholders.

Researchers support, facilitate, and provide technical input.

Objectives aim to effect

change in a stakeholder community.

Lewin 1946;

Greenwood and Levin 2007

Post-normal science Motivated by societal issues and needs rather than

researcher curiosity. Relies on participation of a ‘peer

community’ that extends beyond scientists.

Funtowicz and Ravetz 1993

Mode 2 knowledge

production

Contextualized by the societal factors surrounding a

particular issue and characterized by the inclusion of multiple

types of knowledge and research partners.

Gibbons et al. 1994;

Nowotny et al. 2001

Boundary work Aims to solve real-world issues through dialog and

collaboration across the ‘boundaries’ between scientific

experts and decision makers.

Guston 2001;

Cash et al. 2003

Knowledge

coproduction

Iterative engagement between researchers and nonacademic

partners will produce innovative, scientifically-robust, and

useful knowledge for real-world applications.

Jasanoff 2004;

Lemos and Morehouse 2005

Transdisciplinary

research

The integration of multiple types of expertise and knowledge

is necessary to address a common issue.

Pohl 2008;

Jahn et al. 2012

Transformational

sustainability science

Incorporates knowledge from outside academia into research

processes to support the creation of sustainable social

systems.

Kates et al. 2001;

Wiek et al. 2012

Social learning Social processes help create mutual understandings between

two or more parties, through which individual and societal

transformations can occur.

Wenger 2000;

Pahl-Wostl et al. 2007

Useful science, usable

knowledge

Research conducted to produce knowledge that is useful,

usable, and actually used by societal members.

Dilling and Lemos 2011;

Clark 2016;

Mach et al. 2020

Socially engaged research is more effective than indirect

research pathways in transitioning research into policy

and practice and generating societal impacts. By virtue of

the direct engagement between researchers and societal

partners, and incorporating partners’ perspectives and

knowledge into the research process, the research better

fits the decision contexts in which it will be used (Mach

et al. 2020). The process of engagement helps to build

relationships and trust between societal partners and

researchers (Kothari et al. 2011), which makes it more likely

that partners will find the research credible and usable

(Cash et al. 2003; Lacey et al. 2018; Association of Public

and Land-Grant Universities 2019). The research findings

are more accessible to societal partners both conceptually,

because the research process does not occur in a black

box (Jasanoff and Wynne 1998), and physically, because

the findings are do not sit behind academic pay-walls

(Cvitanovic et al. 2019). A principle of socially engaged

research is that it should include multiple knowledges and

values (Norström et al. 2020; Reed and Meagher 2019). This

inclusive process helps to make the outputs and outcomes

more applicable and acceptable to societal partners,

particularly those who have historically been excluded

from both research and decision making processes (David-

Chavez and Gavin 2018; Association of Public and Land-

Grant Universities 2019).

Studies of the outcomes of socially engaged research

show that the approach is particularly strong in terms

of fostering learning (conceptual impacts), strengthening

community (connectivity impacts), and using research

findings in policy or practice documents (instrumental

impacts) (Cvitanovic et al. 2019; Jagannathan et al. 2020). As

we note throughout this document, although expectations

for long-term socio-environmental change are embedded

within many discussions of socially engaged research

(Jagannathan et al. 2020), those changes are harder to

generate in the short- and medium-term and are harder to

identify and trace in the longer term.

Appendix C: Challenges of Impacts Assessment

There are two main sets of challenges associated with evaluating research impacts. The first set has to do with

the nature of knowledge generation and the use of knowledge and information. The second set deals with the

methodological barriers to identifying and quantifying the full range of possible impacts emerging from research.

Knowledge Generation and

Information Use

The nature of knowledge generation and use poses

significant challenges for tracing clear lines from a

particular research project to societal impacts. There

is often a significant time lag between when research

results emerge to when practitioners adopt new practices

(Bell, Shaw, and Boaz 2011; Penfield et al. 2014; Spaapen

and van Drooge 2011). For example, in medical research,

this lag can be up to 17 years (Moore et al. 2016). The lag

can be attributed to several factors, such as whether

an organization is prepared to integrate new knowledge;

the level of uncertainty surrounding new knowledge;

and the pertinence of the new knowledge to the specific

needs of the practitioners (Ford, Knight, and Pearce 2013;

Green et al. 2009; Oh and Rich 1996). The need for an

organization to be ready to use new information – and

the need for new information to be ready to use – and

the resulting time lag makes it more difficult to trace the

influence of one particular piece of research on a decision

or action (Bell, Shaw, and Boaz 2011; Boaz, Fitzpatrick, and

Shaw 2009; Penfield et al. 2014; Spaapen and van Drooge

2011; Wiek et al. 2014). Researchers can use contribution

analysis to gauge the extent to which research has

influenced practitioner decisions. Using a contribution

framework allows researchers to acknowledge the

complexities involved in creating societal change, while

also pointing to the ways that individual research efforts

(and researchers themselves) help the process of change

(Morton 2015). In addition to contribution analysis, using

multiple categories of impact, including the categories

presented in this guidebook, is an effective approach to

identifying the ways a research project helps influence

new thinking and behaviors while acknowledging that

systemic change (whether social or ecological) may

require additional time and inputs.

Methodological Barriers

Collecting tangible pieces of evidence can sometimes be

straightforward — for example, if an organization cites

your research findings as justification for a new policy

change. In other cases, identifying less-tangible impacts

can pose some methodological challenges to those

unfamiliar with using qualitative data. Conceptual impacts,

such as changes in understanding or attitude, are most

likely to be identified through qualitative assessment

methods that allow project participants to reflect in an

open-ended fashion on their experiences. Unanticipated or

unexpected impacts are also more likely to be identified

through open-ended qualitative inquiry (Spaapen and van

Drooge 2011; Meagher and Martin 2017).

Addressing these methodological challenges will depend

on the resources and time available for evaluation

(Bell, Shaw, and Boaz 2011). Rigorous qualitative data

collection and analysis can help to identify impacts and

understand the context in which they were generated.

The most common societal impacts reporting frameworks,

the Research Excellent Framework (UK), the Standard

Evaluation Protocol (Netherlands), and the Australian

Engagement and Impact Assessment all use a case study

approach to collecting impact reports. Case studies

are reviewed by panels of experts, usually made up of

a mix of researchers and societal partners. While more

time consuming than quantitative assessments, these

qualitative approaches provide a more realistic and richer

understanding of the role of research in societal change.

Another methodological challenge is related to the time

lag issue discussed above. Because it can take several

years for research to move into use and impact, details

about how and why it was able to generate such impacts,

particularly questions related to engagement processes

and participation, may be lost along the way (Bell, Shaw,

and Boaz 2011; Wiek et al. 2014) . Over time, memories

fade or events are re-interpreted by participants and it

can be challenging to reconstruct the details of particular

projects and decisions. It can also be challenging to

secure the participation of researchers and practitioners

in evaluation processes, particularly after projects have

ended and the participants have moved on to other

matters (Wall, Meadow, and Horangic 2017; Wiek et al. 2014).

These challenges can be overcome through on-going and

embedded evaluation approaches that follow projects

throughout their lifetimes and maintain contact with

participants.

Finally, the lack of baseline data on how and under what

conditions and at what frequency research has specific

societal impacts hinders our ability to evaluate project

success (Penfield et al. 2014; Jagannathan et al. 2020). This

challenge may be self-correcting as interest in societal

impacts grows and more data is generated about the

current state of impacts

Appendix D: Concerns about

Societal Impacts Evaluation

As societal impacts reporting and evaluation has gained

traction, it has also raised some concerns within the

research community. A societal impacts approach to

academic evaluation represents a significant shift for

many institutions and researchers. Instead of being

evaluated solely on the scientific rigor and significance of

their work, it requires demonstrating the significance of

their research outside of the academic realm.

Some researchers have expressed concern that

emphasizing societal impacts will effectively devalue basic

research and divert resources away from research that

is focused on “blue skies” questions and grand scientific

challenges (Chubb and Reed 2018; Watermeyer and Chubb

2019; Stern 2016). In a similar vein, some are apprehensive

that evaluation of their research impacts will threaten

their academic freedom by allowing external forces to have

more control over research agendas. Some researchers,

for example, express concern that government-funded

evaluations of government-funded research may be a

tool to suppress research critical of government policies.

Their concern may be related to fears about breaking

down the boundaries between science and society in

ways that some researchers perceive will dilute the

integrity of scientific research (Cvitanovic et al. 2019).

However, this critique perhaps overlooks the fact that

funding agencies have always had significant influence on

research agendas, and that societal values and interests

have always influenced which scientific research is valued,

funded, and undertaken (Greenwood and Levin 2007).

These concerns are not raised solely by the introduction of

societal impacts measures.

A more practical concern is the societal impacts reporting

may be required on top of current evaluation practices.

Most academic researchers are evaluated based on the

academic impacts of their work – as measured by the

number of publications and citations. The accuracy and

appropriateness of these academic metrics have been

challenged on multiple fronts (Adler and Harzing 2009;

Alvesson, Gabriel, and Paulsen 2017; Hicks et al. 2015), but

they have proved quite durable thanks to their relative

ease of use. Given the importance of academic metrics,

the move to societal impacts reporting can feel like an

added burden – and one with little payoff if the effort is

not directly linked to formal performance evaluation or

promotion and tenure policies.

Societal impacts evaluation and reporting does not seek to

supplant research excellence. Quite the opposite – societal

impact practices recognize that the foundation of societal

impacts is high quality research. Engagement with societal

partners, often a key component in generating societal

impacts, does not negate the need for rigor in scientific

methods or processes (Cvitanovic et al. 2019; Greenwood

and Levin 2007). Societal impacts reporting provides

opportunities for researchers who do socially engaged

research to be recognized and evaluated based on a more

accurate portrayal of the work they do – research that

involves scientific rigor plus additional time and resources

for direct engagement with societal partners. Societal

impacts reporting can become less of an additional burden

on researchers once the practice is fully integrated into

academic performance measures. It will help ensure that

researchers receive credit for the work of generating

societal impacts and the work of documenting, evaluating,

and reporting them.

References

Adler, Nancy J, and Anne-Wil Harzing. 2009. When knowledge wins: Transcending the sense and nonsense of academic

rankings. Academy of Management Learning & Education 8 (1):72-95.

Alvesson, Mats, Yiannis Gabriel, and Roland Paulsen. 2017. Return to meaning: A social science with something to say:

Oxford University Press.

Association of Public and Land-Grant Universities. 2019. Public Impact Research: Engaged Universities Making The

Difference. Washington, DC: Association of Public and Land-Grant Universities.

Bell, Sarah, Ben Shaw, and Annette Boaz. 2011. Real-world approaches to assessing the impact of environmental

research on policy. Research Evaluation 20 (3):227-237.

Boaz, Annette, Siobhan Fitzpatrick, and Ben Shaw. 2009. Assessing the impact of research on policy: a literature review.

Science & Public Policy (SPP) 36 (4):255-270.

Bornmann, Lutz, Robin Haunschild, and Werner Marx. 2016. Policy documents as sources for measuring societal impact:

how often is climate change research mentioned in policy-related documents? Scientometrics 109 (3):1477-1495.

Cash, D. W., J. C. Borck, and A. G. Patt. 2006. Countering the loading-dock approach to linking science and decision

making - Comparative analysis of El Nino/Southern Oscillation (ENSO) forecasting systems.

Science, Technology and Human Values 31 (4):465-494.

Cash, David W., William C. Clark, Frank Alcock, Nancy M. Dickson, Noelle Eckley, David H. Guston, Jill Jäger,

and Ronald B. Mitchell. 2003. Knowledge Systems for Sustainable Development.

Proceedings of the National Academy of Sciences 100 (14):8086-8091.

Chubb, Jennifer, and Mark S Reed. 2018. The politics of research impact: academic perceptions of the

implications for research funding, motivation and quality. British Politics:1-17.

Clark, William C., Lorrae van Kerkhoff, Louis Lebel, and Gilberto C. Gallopin. 2016. Crafting usable knowledge for

sustainable development. Proceedings of the National Academy of Sciences 113 (17):4570-4578.

Cvitanovic, Christopher, Mark Howden, RM Colvin, Albert Norström, Alison M Meadow, and PFE Addison. 2019.

Maximising the benefits of participatory climate adaptation research by understanding and managing the

associated challenges and risks. Environmental Science & Policy 94:20-31.

David-Chavez, Dominique M, and Michael C Gavin. 2018. A global assessment of Indigenous community engagement

in climate research. Environmental Research Letters 13 (12):123005.

Dennis, Michael A. 2015. Our monsters, ourselves: Reimagining the problem of knowledge in Cold War America.

Dreamscapes of modernity: Sociotechnical imaginaries and the fabrication of power:56-78.

Dilling, Lisa, and Maria Carmen Lemos. 2011. Creating usable science: Opportunities and constraints for

climate knowledge use and their implications for science policy. Global Environmental Change 21 (2):680-689.

Fazey, Ioan, Lukas Bunse, Joshua Msika, Maria Pinke, Katherine Preedy, Anna C. Evely, Emily Lambert, Emily Hastings,

Sue Morris, and Mark S. Reed. 2014. Evaluating knowledge exchange in interdisciplinary and multi-stakeholder

research. Global Environmental Change 25 (0):204-220.

Ford, James D., Maggie Knight, and Tristan Pearce. 2013. Assessing the ‘usability’ of climate change research for

decision-making: A case study of the Canadian International Polar Year. Global Environmental Change 23 (5):1317-1326.

Foster, Kevin Michael. 2010. Taking a stand: Community-engaged scholarship on the tenure track.

Journal of Community Engagement and Scholarship 3 (2):20.

Funtowicz, Silvio O., and Jerome R. Ravetz. 1993. Science for the post-normal age. Futures 25 (7):739-755.

Gibbons, M., C. Limoges, H. Nowotny, S. Schwartzmann, P. Scott, and M. Trow. 1994. The New Production of Knowledge:

The Dynamics of Science and Research in Contemporary Societies. London, UK: Sage.

Green, Lawrence W., Russell E. Glasgow, David Atkins, and Kurt Stange. 2009. Making Evidence from Research

More Relevant, Useful, and Actionable in Policy, Program Planning, and Practice: Slips “Twixt Cup and Lip”.

American Journal of Preventive Medicine 37 (6, Supplement 1):S187-S191.

Greenwood, Davydd J., and Morten Levin. 2007. Introduction to Action Research: Social Research for Social Change.

2nd ed. Thousand Oaks: Sage Publications.

Greenwood, Davydd, and Morten Levin. 2007. Scientific Method and Action Research In Introduction to Action Research.

Thousand Oaks, California: SAGE Publications, Inc.

Guston, David H. 2001. Boundary Organizations in Environmental Policy and Science:

An Introduction. Science, Technology, & Human Values 26 (4):399-408.

Hicks, Diana, Paul Wouters, Ludo Waltman, Sarah De Rijcke, and Ismael Rafols. 2015.

Bibliometrics: the Leiden Manifesto for research metrics. Nature News 520 (7548):429.

Jagannathan, Kripa, James C. Arnott, Carina A. Wyborn, Nicole Klenk, Katharine J. Mach, Richard H. Moss, and

K. Dana Sjostrom. 2020. Great Expections? Reconciling the aspiration, outcome, and possibility of co-production.

Current Opinion in Environmental Sustainability 42:22 - 29.

Jahn, Thomas, Matthias Bergmann, and Florian Keil. 2012. Transdisciplinarity:

Between mainstreaming and marginalization. Ecological Economics 79:1-10.

Jasanoff, Sheila, ed. 2004. States of Knowledge: The Co-production of Science and the Social Order,

International Library of Sociology: Routledge.

Jasanoff, Sheila, and Brian Wynne. 1998. Science and decisionmaking. In Human choice and climate change,

edited by S. Rayner and E. L. Malone. Columbus: Battelle Press.

Kates, Robert W, William C Clark, Robert Corell, J Michael Hall, Carlo C Jaeger, Ian Lowe, James J McCarthy,

Hans Joachim Schellnhuber, Bert Bolin, and Nancy M Dickson. 2001. Sustainability science. Science 292 (5517):641-642.

Kothari, Anita, Lynne MacLean, Nancy Edwards, and Allison Hobbs. 2011. Indicators at the interface:

managing policymaker-researcher collaboration. Knowledge Management Research & Practice 9 (3):203-214.

Kuruvilla, Shyama, Nicholas Mays, Andrew Pleasant, and Gill Walt. 2006. Describing the impact of health research:

a Research Impact Framework. BMC health services research 6 (1):134.

Lacey, Justine, Mark Howden, Christopher Cvitanovic, and R. M. Colvin. 2018. Understanding and managing trust

at the climatescience–policy interface. Nature Climate Change 8 (1):22-28.

Lemos, Maria Carmen, and Barbara J. Morehouse. 2005. The co-production of science and policy in integrated

climate assessments. Global Environmental Change 15:57-68.

Lewin, Kurt. 1946. Action research and minority problems. Journal of Social Issues 46 (2):34-46.

Mach, Katharine J., Maria Carmen Lemos, Alison M. Meadow, Carina Wyborn, Nicole Klenk, James C. Arnott,

Nicole M. Ardoin, Clare Fieseler, Richard H. Moss, Leah Nichols, Missy Stults, Catherine Vaughan, and

Gabrielle Wong-Parodi. 2020. Actionable knowledge and the art of engagement.

Current Opinion in Environmental Sustainability 42:30-37.

Meagher, Laura, Catherine Lyall, and Sandra Nutley. 2008. Flows of knowledge, expertise and influence: a method

for assessing policy and practice impacts from social science research. Research Evaluation 17 (3):163-173.

Meagher, Laura R., and Ursula Martin. 2017. Slightly dirty maths: The richly textured mechanisms of impact.

Research Evaluation 26 (1):15-27.

Moore, Julia E., Michelle Grouchy, Ian D. Graham, Maureen Shandling, Winnie Doyle, and Sharon E. Straus. 2016.

The Council of Academic Hospitals of Ontario (CAHO) Adopting Research to Improve Care (ARTIC) Program: Reach,

Sustainability, Spread and Lessons Learned from an Implementation Funding Model. Healthcare Policy 11 (4):27-39.

Morton, Sarah. 2015. Progressing research impact assessment: A ‘contributions’ approach.

Research Evaluation 24 (4):405-419.

Norström, Albert V., Christopher Cvitanovic, Marie F. Löf, Simon West, Carina Wyborn, Patricia Balvanera,

Angela T. Bednarek, Elena M. Bennett, Reinette Biggs, Ariane de Bremond, Bruce M. Campbell, Josep G. Canadell,

Stephen R. Carpenter, Carl Folke, Elizabeth A. Fulton, Owen Gaffney, Stefan Gelcich, Jean-Baptiste Jouffray,

Melissa Leach, Martin Le Tissier, Berta Martín-López, Elena Louder, Marie-France Loutre, Alison M. Meadow,

Harini Nagendra, Davnah Payne, Garry D. Peterson, Belinda Reyers, Robert Scholes, Chinwe Ifejika Speranza,

Marja Spierenburg, Mark Stafford-Smith, Maria Tengö, Sandra van der Hel, Ingrid van Putten, and Henrik Österblom.

2020. Principles for knowledge co-production in sustainability research. Nature Sustainability January:1-9.

Nowotny, Helga, Peter Scott, and Michael Gibbons. 2001. Re-Thinking Science:

Knowledge and the Public in an Age of Uncertainty.

Oh, Cheol H, and Robert F Rich. 1996. Explaining use of information in public policymaking. Knowledge and Policy 9 (1):3-35.

Pahl-Wostl, Claudia, Marc Craps, Art Dewulf, Erik Mostert, David Tabara, and Tharsi Taillieu. 2007.

Social learning and water resources management. Ecology and society 12 (2).

Pelz, D.C. 1978. Some expanded perspectives on use of social science in public policy. In Major Social Issues:

A Multidisciplinary View, edited by J. Yinger and S. SCutler. New York: The Free Press.

Penfield, Teresa, Matthew J Baker, Rosa Scoble, and Michael C Wykes. 2014. Assessment, evaluations, and definitions of

research impact: A review. Research Evaluation 23 (1):21-32.

Pohl, Christian. 2008. From science to policy through transdisciplinary research.

Environmental Science and Policy 11:46-53.

Reed, Mark S. 2018. The Research Impact Handbook. 2nd ed: Fast Track Impact.

Reed, Mark S, and Laura Meagher. 2019. Using evidence in environmental and sustainability issues. In What Works Now?

Evidence-informed policy and practice, edited by A. Boaz, H. Davies, A. Fraser and S. Nutley. Chicago: Policy Press.

Spaapen, Jack, and Leonie van Drooge. 2011. Introducing ‘productive interactions’ in social impact assessment.

Research Evaluation 20 (3):211-218.

Spaapen, Jack, Leonie van Drooge, Tilo Propp, Barend van der Meulen, Terry Shinn, Anne Marcovich,

Peter van den Besselaar, Stefan de Jong, Kate Barker, Deborah Cox, Kathryn Morrison, Thordis Sveinsdottir,

Diana Pearson, Beatrice D’Ippolito, Ad Prins, Jordi Molas-Gallart, Puay Tang, and Elena Castro-Martinrz. 2011.

SIAMPI Final Report.

Stern, Nicholas. 2016. Building on Success and Learning from Experience: An Independent Review of the

Research Excellence Framework.

Wall, Tamara, Alison Meadow, and Alexandra Horangic. 2017. Developing Evaluation Indicators to Improve the Process of

Co-producing Usable Climate Science. Weather, Climate, and Society 9 (1):95-107.

Walter, I., H. Davies, and S. Nutley. 2003. Increasing research impact through partnerships: evidence from

outside health care. J Health Serv Res Policy 8.

Watermeyer, Richard, and Jennifer Chubb. 2019. Evaluating ‘impact’in the UK’s Research Excellence Framework (REF):

liminality, looseness and new modalities of scholarly distinction. Studies in Higher Education 44 (9):1554-1566.

Weiss, Carol. 1973. The politics of impact measurement. Policy Studies Journal 1 (3):179.

Weiss, Carol H. 1998. Have we learned anything new about the use of evaluation?

American journal of evaluation 19 (1):21-33.

Wenger, Etienne. 2010. Communities of practice and social learning systems: the career of a concept.

In Social learning systems and communities of practice: Springer.

Wiek, Arnim, Barry Ness, Petra Schweizer-Ries, Fridolin S Brand, and Francesca Farioli. 2012. From complex systems

analysis to transformational change: a comparative appraisal of sustainability science projects.

Sustainability science 7 (1):5-24.

Wiek, Arnim, Sonia Talwar, Meg O’Shea, and John Robinson. 2014. Toward a methodological scheme for capturing

societal effects of participatory sustainability research. Research Evaluation 23 (2):117-132.