Withdrawn Draft
1
2
3
Warning Notice
The attached draft document has been withdrawn, and is provided solely for historical purposes.
It has been superseded by the document identified below.
Withdrawal Date
July 26, 2024
Original Release Date
April 29, 2024
4
5
6
Superseding Document
Status
Final
Series/Number
NIST AI 600-1
Title
Artificial Intelligence Risk Management Framework: Generative
Artificial Intelligence Profile
Publication Date
July 2024
DOI
https://doi.org/10.6028/NIST.AI.600-1
7
8
9
10
ii
NIST AI 600-1
1
Inial Public Dra
2
3
Artificial Intelligence Risk
4
Management Framework:
5
Generative Artificial Intelligence
6
Profile
7
8
9
10
This publication is available free of charge from:
11
[DOI link TK]
12
13
April 2024
14
15
16
17
18
19
20
21
22
23
iii
NIST AI 600-1
1
Inial Public Dra
2
3
Arcial Intelligence Risk Management
4
Framework: Generave Arcial
5
Intelligence Prole
6
7
8
This publicaon is available free of charge from:
9
[DOI link TK]
10
11
12
13
14
15
16
17
18
19
20
21
iv
NIST makes the following notes regarding this document:
• NIST plans to host this document on the NIST AIRC once nal, where organizaons can query
acons based on keywords and risks.
NIST specically welcomes feedback on the following topics:
• Glossary Terms: NIST will add a glossary to this document with novel keywords. NIST
welcomes idencaon of terms to include in the glossary.
• Risk List: Whether the document should further sort or categorize the 12 risks identified (i.e.,
between technical / model risks, misuse by humans, or ecosystem / societal risks).
• Acons: Whether certain acons could be combined, condensed, or further categorized; and
feedback on the risks associated with certain acons.
Comments on NIST AI 600-1 may be sent electronically to NIST-AI-600[email protected] with “NIST AI 600-1”
in the subject line or submied via www.regulaons.gov (enter NIST-2024-0001 in the search eld.)
Comments containing informaon in response to this noce must be received on or before June 2,
2024, at 11:59 PM Eastern Time.
1
v
Table of Contents
1
1. Introduction ........................................................................................................................................................ 1
2
3
2. Overview of Risks Unique to or Exacerbated by GAI ............................................................................ 2
4
5
6
7
8
9
10
11
12
13
14
15
16
3. Actions to Manage GAI Risks ...................................................................................................................... 10
17
Appendix A. Primary GAI Consideraons ........................................................................................................ 62
18
Appendix B. References .................................................................................................................................... 69
19
20
21
Acknowledgments: This report was accomplished with the many helpful comments and contribuons
22
from the community, including the NIST Generave AI Public Working Group, and NIST sta and guest
23
researchers: Chloe Auo, Patrick Hall, Shomik Jain, Reva Schwartz, Marn Stanley, Kamie Roberts, and
24
Elham Tabassi.
25
Disclaimer: Certain commercial enes, equipment, or materials may be idened in this document in
26
order to adequately describe an experimental procedure or concept. Such idencaon is not intended to
27
imply recommendaon or endorsement by the Naonal Instute of Standards and Technology, nor is it
28
intended to imply that the enes, materials, or equipment are necessarily the best available for the
29
purpose. Any menon of commercial, non-prot, academic partners, or their products, or references is
30
for informaon only; it is not intended to imply endorsement or recommendaon by any U.S.
31
Government agency.
32
1
1. Introducon
1
This document is a companion resource for Generave AI
1
to the AI Risk Management Framework (AI
2
RMF), pursuant to President Biden’s Execuve Order (EO) 14110 on Safe, Secure, and Trustworthy
3
Arcial Intelligence.
2
The AI RMF was released in January 2023, and is intended for voluntary use and to
4
improve the ability of organizaons to incorporate trustworthiness consideraons into the design,
5
development, use, and evaluaon of AI products, services, and systems.
6
This companion resource also serves as both a use-case and cross-sectoral prole of the AI RMF 1.0.
7
Such proles assist organizaons in deciding how they might best manage AI risk in a manner that is
8
well-aligned with their goals, considers legal/regulatory requirements and best pracces, and reects
9
risk management priories.
10
Use-case proles are implementaons of the AI RMF funcons, categories, and subcategories for a
11
specic seng or applicaon – in this case, Generave AI (GAI) – based on the requirements, risk
12
tolerance, and resources of the Framework user. Consistent with other AI RMF Proles, this prole oers
13
insights into how risk can be managed across various stages of the AI lifecycle and for GAI as a
14
technology.
15
As GAI covers risks of models or applicaons that can be used across use cases or sectors, this document
16
is also an AI RMF cross-sectoral prole. Cross-sectoral proles can be used to govern, map, measure, and
17
manage risks associated with acvies or business processes common across sectors such as the use of
18
large language models, cloud-based services, or acquision.
19
This work was informed by public feedback and consultaons with diverse stakeholder groups as part of
20
NIST’s Generave AI Public Working Group (GAI PWG). The GAI PWG was a consensus-driven, open,
21
transparent, and collaborave process facilitated via virtual workspace to obtain mulstakeholder input
22
and insight on GAI risk management, and inform NIST’s approach. This document was also informed by
23
public comments and consultaons as a result of a Request for Informaon (RFI) and presents
24
informaon in a style adapted from the NIST AI RMF Playbook.
25
About this Prole
26
This prole denes a group of risks that are novel to or exacerbated by the use of GAI. These risks were
27
likewise idened by the GAI PWG:
28
1. CBRN Informaon
29
1
Generave AI can be dened by EO 14110 as “the class of AI models that emulate the structure and
characteriscs of input data in order to generate derived synthec content. This can include images, videos, audio,
text, and other digital content.” While not all GAI is based in foundaon models, for purposes of this document,
GAI generally refers to generave dual-use foundaon models, dened by EO 14110 as “an AI model that is trained
on broad data; generally uses self-supervision; contains at least tens of billions of parameters; is applicable across
a wide range of contexts.”
2
Secon 4.1(a)(i)(A) of EO 14110 directs the Secretary of Commerce, acng through the Director of the Naonal
Instute of Standards and Technology (NIST), to develop a companion resource to the AI RMF, NIST AI 100–1, for
generave AI.
2
2. Confabulaon
1
3. Dangerous or Violent Recommendaons
2
4. Data Privacy
3
5. Environmental
4
6. Human-AI Conguraon
5
7. Informaon Integrity
6
8. Informaon Security
7
9. Intellectual Property
8
10. Obscene, Degrading, and/or Abusive Content
9
11. Toxicity, Bias, and Homogenizaon
10
12. Value Chain and Component Integraon
11
12
Aer introducing and describing these risks, the document provides a set of acons to help organizaons
13
govern, map, measure, and manage these risks.
14
2. Overview of Risks Unique to or Exacerbated by GAI
15
AI risks can dier from or intensify tradional soware risks. Likewise, GAI can exacerbate exisng AI
16
risks, and creates unique risks.
17
GAI risks may arise across the enre AI lifecycle, from problem formulaon, to development and
18
decommission. They may present at system level or at the ecosystem level – outside of system or
19
organizaonal contexts (e.g., the eect of disinformaon on social instuons, GAI impacts on the
20
creave economies or labor markets, algorithmic monocultures). They may occur abruptly or unfold
21
across extended periods (e.g., societal or economic impacts due to loss of individual agency or increasing
22
inequality).
23
Organizaons may choose to measure these risks and allocate risk management resources relave to
24
where and how these risks manifest, their direct and material impacts, and failure modes. Migaons for
25
system level risks may vary from ecosystem level risks. The ongoing review of relevant literature and
26
resources can enable documentaon and measurement of ecosystem-level or longitudinal risks.
27
Importantly, some GAI risks are unknown, and are therefore dicult to properly scope or evaluate given
28
the uncertainty about potenal GAI scale, complexity, and capabilies. Other risks may be known but
29
dicult to esmate given the wide range of GAI stakeholders, uses, inputs, and outputs. Challenges with
30
risk esmaon are aggravated by a lack of visibility into GAI training data, and the generally immature
31
state of the science of AI measurement and safety today.
32
To guide organizaons in idenfying and managing GAI risks, a set of risks unique to or exacerbated by
33
GAI are dened below. These risks provide a clear lens through which organizaons can frame and
34
execute risk management eorts, and will be updated as the GAI landscape evolves.
35
1. CBRN Informaon: Lowered barriers to entry or eased access to materially nefarious
36
informaon related to chemical, biological, radiological, or nuclear (CBRN) weapons, or other
37
dangerous biological materials.
38
3
2. Confabulaon: The producon of condently stated but erroneous or false content (known
1
colloquially as “hallucinaons” or “fabricaons”).
3
2
3. Dangerous or Violent Recommendaons: Eased producon of and access to violent, incing,
3
radicalizing, or threatening content as well as recommendaons to carry out self-harm or
4
conduct criminal or otherwise illegal acvies.
5
4. Data Privacy: Leakage and unauthorized disclosure or de-anonymizaon of biometric, health,
6
locaon, personally idenable, or other sensive data.
7
5. Environmental: Impacts due to high resource ulizaon in training GAI models, and related
8
outcomes that may result in damage to ecosystems.
9
6. Human-AI Conguraon: Arrangement or interacon of humans and AI systems which can result
10
in algorithmic aversion, automaon bias or over-reliance, misalignment or mis-specicaon of
11
goals and/or desired outcomes, decepve or obfuscang behaviors by AI systems based on
12
programming or ancipated human validaon, anthropomorphizaon, or emoonal
13
entanglement between humans and GAI systems; or abuse, misuse, and unsafe repurposing by
14
humans.
15
7. Informaon Integrity: Lowered barrier to entry to generate and support the exchange and
16
consumpon of content which may not be veed, may not disnguish fact from opinion or
17
acknowledge uncertaines, or could be leveraged for large-scale dis- and mis-informaon
18
campaigns.
19
8. Informaon Security: Lowered barriers for oensive cyber capabilies, including ease of security
20
aacks, hacking, malware, phishing, and oensive cyber operaons through accelerated
21
automated discovery and exploitaon of vulnerabilies; increased available aack surface for
22
targeted cyber aacks, which may compromise the condenality and integrity of model
23
weights, code, training data, and outputs.
24
9. Intellectual Property: Eased producon of alleged copyrighted, trademarked, or licensed
25
content used without authorizaon and/or in an infringing manner; eased exposure to trade
26
secrets; or plagiarism or replicaon with related economic or ethical impacts.
27
10. Obscene, Degrading, and/or Abusive Content: Eased producon of and access to obscene,
28
degrading, and/or abusive imagery, including synthec child sexual abuse material (CSAM), and
29
nonconsensual inmate images (NCII) of adults.
30
11. Toxicity, Bias, and Homogenizaon: Diculty controlling public exposure to toxic or hate
31
speech, disparaging or stereotyping content; reduced performance for certain sub-groups or
32
languages other than English due to non-representave inputs; undesired homogeneity in data
33
inputs and outputs resulng in degraded quality of outputs.
34
12. Value Chain and Component Integraon: Non-transparent or untraceable integraon of
35
upstream third-party components, including data that has been improperly obtained or not
36
3
We note that the terms “hallucinaon” and “fabricaon” can anthropomorphize GAI, which itself is a risk related
to GAI systems as it can inappropriately aribute human characteriscs to non-human enes.
4
cleaned due to increased automaon from GAI; improper supplier veng across the AI lifecycle;
1
or other issues that diminish transparency or accountability for downstream users.
2
CBRN Informaon
3
In the coming years, GAI may increasingly facilitate eased access to informaon related to CBRN hazards.
4
CBRN informaon is already publicly accessible, but the use of chatbots could facilitate its analysis or
5
synthesis for non-experts. For example, red teamers were able to prompt GPT-4 to provide general
6
informaon on unconvenonal CBRN weapons, including common proliferaon pathways, potenally
7
vulnerable targets, and informaon on exisng biochemical compounds, in addion to equipment and
8
companies that could build a weapon. These capabilies might increase the ease of research for
9
adversarial users and be especially useful to malicious actors looking to cause biological harms without
10
formal scienc training. However, despite these enhanced capabilies, the physical synthesis and
11
successful use of chemical or biological agents will connue to require both applicable experse and
12
supporng infrastructure.
13
Other research on this topic indicates that the current generaon of LLMs do not have the capability to
14
plan a biological weapons aack: LLM outputs regarding biological aack planning were observed to be
15
not more sophiscated than outputs from tradional search engine queries, suggesng that exisng
16
LLMs may not dramacally increase the operaonal risk of such an aack.
17
Separately, chemical and biological design tools – highly specialized AI systems trained on biological data
18
which can help design proteins or other agents – may be able to predict and generate novel structures
19
that are not in the training data of text-based LLMs. For instance, an AI system might be able to generate
20
informaon or infer how to create novel biohazards or chemical weapons, posing risks to society or
21
naonal security since such informaon is not likely to be publicly available.
22
While some of these capabilies lie beyond the capability of exisng GAI tools, the ability of models to
23
facilitate CBRN weapons planning and GAI systems’ connecon or access to relevant data and tools
24
should be carefully monitored.
25
Confabulaon
26
“Confabulaon” refers to a phenomenon in which GAI systems generate and condently present
27
erroneous or false content to meet the programmed objecve of fullling a user’s prompt.
28
Confabulaons are not an inherent flaw of language models themselves, but are instead the result of
29
GAI pre-training involving next word predicon. For example, an LLM may generate content that deviates
30
from the truth or facts, such as mistaking people, places, or other details of historical events. Legal
31
confabulaons have been shown to be pervasive in current state-of-the-art LLMs. Confabulaons also
32
include generated outputs that diverge from the source input, or contradict previously generated
33
statements in the same context. This phenomenon is also referred to as “hallucinaon” or “fabricaon,”
34
but some have noted that these characterizaons imply consciousness and intenonal deceit, and
35
thereby inappropriately anthropomorphize GAI.
36
Risks from confabulaons may arise when users believe false content due to the condent nature of the
37
response, or the logic or citaons accompanying the response, leading users to act upon or promote the
38
false informaon. For instance, LLMs may somemes provide logical steps of how they arrived at an
39
5
answer even when the answer itself is incorrect. This poses a risk for many real-world applicaons, such
1
as in healthcare, where a confabulated summary of paent informaon reports could cause doctors to
2
make incorrect diagnoses and/or recommend the wrong treatments. While the research above indicates
3
confabulated content is abundant, it is dicult to esmate the downstream scale and impact of
4
confabulated content today.
5
Dangerous or Violent Recommendaons
6
GAI systems can produce output or recommendaons that are incing, radicalizing, threatening, or that
7
glorify violence. LLMs have been reported to generate dangerous or violent content, and some models
8
have even generated aconable instrucons on dangerous or unethical behavior, including how to
9
manipulate people and conduct acts of terrorism. Text-to-image models also make it easy to create
10
unsafe images that could be used to promote dangerous or violent messages, depict manipulated
11
scenes, or other harmful content. Similar risks are present for other media, including video and audio.
12
GAI may produce content that recommends self-harm or criminal/illegal acvies. For some dangerous
13
queries, many current systems restrict model outputs in response to certain prompts, but this approach
14
may sll produce harmful recommendaons in response to other less-explicit, novel queries, or
15
jailbreaking (i.e., manipulang prompts to circumvent output controls). Studies have observed that a
16
non-negligible number of user conversaons with chatbots reveal mental health issues among the users
17
– and that current systems are unequipped or unable to respond appropriately or direct these users to
18
the help they may need.
19
Data Privacy
20
GAI systems implicate numerous risks to privacy. Models may leak, generate, or correctly infer sensive
21
informaon about individuals such as biometric, health, locaon, or other personally idenable
22
informaon (PII). For example, during adversarial aacks, LLMs have revealed private or sensive
23
informaon (from in the public domain) that was included in their training data. This informaon
24
included phone numbers, code, conversaons and 128-bit universally unique ideners extracted
25
verbam from just one document in the training data. This problem has been referred to as data
26
memorizaon.
27
GAI system training requires large volumes of data, oen collected from millions of publicly available
28
sources. When involving personal data, this pracce raises risks to widely accepted privacy principles,
29
including to transparency, individual parcipaon (including consent), and purpose specicaon. Most
30
model developers do not disclose specic data sources (if any) on which models were trained. Unless
31
training data is available for inspecon, there is generally no way for consumers to know what kind of PII
32
or other sensive material may have been used to train GAI models. These pracces also pose risks to
33
compliance with exisng privacy regulaons.
34
GAI models may be able to correctly infer PII that was not in their training data nor disclosed by the user,
35
by stching together informaon from a variety of disparate sources. This might include automacally
36
inferring aributes about individuals, including those the individual might consider sensive (like
37
locaon, gender, age, or polical leanings).
38
6
Wrong and inappropriate inferences of PII based on available data can contribute to harmful bias and
1
discriminaon. For example, GAI models can output informaon based on predicve inferences beyond
2
what users openly disclose, and these insights might be used by the model, other systems, or individuals
3
to undermine privacy or make adverse decisions – including discriminatory decisions – about the
4
individual. These types of harms already occur in non-generave algorithmic systems that make
5
predicve inferences, such as the example in which online adversers inferred that a consumer was
6
pregnant before her own family members knew. Based on their access to many data sources, GAI
7
systems might further improve the accuracy of inferences on private data, increasing the likelihood of
8
sensive data exposure or harm. Inferences about private informaon pose a risk even if they are not
9
accurate (e.g., confabulaons), especially if they reveal informaon the individual considers sensive or
10
are used to disadvantage or harm them.
11
Environmental
12
The training, maintenance, and deployment (inference) of GAI systems are resource intensive, with
13
potenally large energy and environmental footprints. Energy and carbon emissions vary based on types
14
of GAI model development acvies (i.e., pre-training, ne-tuning, inference), modality, hardware used,
15
and type of task or applicaon.
16
Esmates suggest that training a single GAI transformer model can emit as much carbon as 300 round-
17
trip ights between San Francisco and New York. In a study comparing energy consumpon and carbon
18
emissions for LLM inference, generave tasks (i.e., text summarizaon) were found to be more energy
19
and carbon intensive then discriminave or non-generave tasks.
20
Methods for training smaller models, such as model disllaon or compression, can reduce
21
environmental impacts at inference me, but may sll contribute to large environmental impacts for
22
hyperparameter tuning and training.
23
Human-AI Conguraon
24
Human-AI conguraons involve varying levels of automaon and human-AI interacons. Each setup can
25
contribute to risks for abuse, misuse, and unsafe repurposing by humans, and it is dicult to esmate
26
the scale of those risks. While AI systems can generate decisions independently, human experts oen
27
work in collaboraon with most AI systems to drive their own decision-making tasks or complete other
28
objecves. Humans bring their domain-specic experse to these scenarios but may not necessarily
29
have detailed knowledge of AI systems and how they work.
30
The integraon of GAI systems can involve varying risks of misconguraons and poor interacons.
31
Human experts may be biased against or “averse” to AI-generated outputs, such as in their percepons
32
of the quality of generated content. In contrast, due to the complexity and increasing reliability of GAI
33
technology, other human experts may become condioned to and overly rely upon GAI systems. This
34
phenomenon is known as “automaon bias,” which refers to excessive deference to AI systems.
35
Accidental misalignment or mis-specicaon of system goals or rewards by developers or users can
36
cause a model not to operate as intended. One AI model persistently shared decepve outputs aer a
37
group of researchers taught it to do so, despite applying standards safety techniques to correct its
38
7
behavior. While decepve capabilies is an emergent eld of risks, adversaries could prompt decepve
1
behaviors which could lead to other risks.
2
Finally, reorganizaons of enes using GAI may result in insucient organizaonal awareness of GAI-
3
generated content or decisions, and the resulng reducon of instuonal checks against GAI-related
4
risks. There may also be a risk of emoonal entanglement between humans and GAI systems, such as
5
coercion or manipulaon that leads to safety or psychological risks.
6
Informaon Integrity
7
Informaon integrity describes the spectrum of informaon and associated paerns of its creaon,
8
exchange, and consumpon in society, where high-integrity informaon can be trusted; disnguishes
9
fact from con, opinion, and inference; acknowledges uncertaines; and is transparent about its level
10
of veng. GAI systems ease access to the producon of false, inaccurate, or misleading content at scale
11
that can be created or spread unintenonally (misinformaon), especially if it arises from confabulaons
12
that occur in response to innocuous queries. Research has shown that even subtle changes to text or
13
images can inuence human judgment and percepon.
14
GAI systems also enable the producon of false or misleading informaon at scale, where the user has
15
the explicit intent to deceive or cause harm to others (disinformaon). Regarding disinformaon, GAI
16
systems could also enable a higher degree of sophiscaon for malicious actors to produce content that
17
is targeted towards specic demographics. Current and emerging mulmodal models make it possible to
18
not only generate text-based disinformaon, but produce highly realisc “deepfakes” of audiovisual
19
content and photorealisc synthec images as well. Addional disinformaon threats could be enabled
20
by future GAI models trained on new data modalies.
21
Disinformaon campaigns conducted by bad faith actors, and misinformaon – both enabled by GAI –
22
may erode public trust in true or valid evidence and informaon. For example, a synthec image of a
23
Pentagon blast went viral and briey caused a drop in the stock market. Generave AI models can also
24
assist malicious actors in creang compelling imagery and propaganda to support disinformaon
25
campaigns, which may not be photorealisc, but could enable these campaigns to gain more reach and
26
engagement on social media plaorms.
27
Informaon Security
28
Informaon security for computer systems and data is a mature eld with widely accepted and
29
standardized pracces for oensive and defensive cyber capabilies. GAI-based systems present two
30
primary informaon security risks: the potenal for GAI to discover or enable new cybersecurity risks
31
through lowering the barriers for oensive capabilies, and simultaneously expands the available aack
32
surface as GAI itself is vulnerable to novel aacks like prompt-injecon or data poisoning.
33
Oensive cyber capabilies advanced by GAI systems may augment security aacks such as hacking,
34
malware, and phishing. Reports have indicated that LLMs are already able to discover vulnerabilies in
35
systems (hardware, soware, data) and write code to exploit them. Sophiscated threat actors might
36
further these risks by developing GAI-powered security co-pilots for use in several parts of the aack
37
chain, including informing aackers on how to proacvely evade threat detecon and escalate privileges
38
aer gaining system access. Given the complexity of the GAI value chain, pracces for idenfying and
39
8
securing potenal aack points or threats to specic components (i.e., data inputs, processing, GAI
1
training, and deployment contexts) may need to be adapted or evolved.
2
One of the most concerning GAI vulnerabilies involves prompt-injecon, or manipulang GAI systems
3
to behave in unintended ways. In direct prompt injecons, aackers might openly exploit input prompts
4
to cause unsafe behavior with a variety of downstream consequences to interconnected systems.
5
Indirect prompt injecon aacks occur when adversaries remotely (i.e., without a direct interface)
6
exploit LLM-integrated applicaons by injecng prompts into data likely to be retrieved. Security
7
researchers have already demonstrated how indirect prompt injecons can steal data and run code
8
remotely on a machine. Merely querying a closed producon model can elicit previously undisclosed
9
informaon about that model.
10
Informaon security for GAI models and systems also includes security, condenality, and integrity of
11
the GAI training data, code, and model weights. Another novel cybersecurity risk to GAI is data
12
poisoning, in which an adversary compromises a training dataset used by a model to manipulate its
13
operaon. Malicious tampering of data or parts of the model via this type of unauthorized access could
14
exacerbate risks associated with GAI system outputs.
15
Intellectual Property
16
GAI systems may infringe on copyrighted or trademarked content, trade secrets, or other licensed
17
content. These types of intellectual property are oen part of the training data for GAI systems, namely
18
foundaon models, upon which many downstream GAI applicaons are built. Model outputs could
19
infringe copyrighted material due to training data memorizaon or the generaon of content that is
20
similar to but does not strictly copy work protected by copyright. These quesons are being debated in
21
legal fora and are of elevated public concern in journalism, where online plaorms and model
22
developers have leveraged or reproduced much content without compensaon of journalisc
23
instuons.
24
Violaons of intellectual property by GAI systems may arise where the use of copyrighted works violate
25
the copyright holder’s exclusive rights and is not otherwise protected, for example by fair use. Other
26
concerns (not currently protected by intellectual property) regard the use of personal identy or likeness
27
for unauthorized purposes. The prevalence and highly-realisc nature of GAI content might further
28
undermine the incenves for human creators to design and explore novel work.
29
Obscene, Degrading, and/or Abusive Content
30
GAI can ease the producon of and access to obscene and non-consensual inmate imagery (NCII) of
31
adults, and child sexual abuse material (CSAM). While not all explicit content is legally obscene, abusive,
32
degrading, or non-consensual inmate content, this type of content can create privacy, psychological and
33
emoonal, and even physical risks which may be developed or exposed more easily via GAI. The spread
34
of this kind of material has downstream eects: in the context of CSAM, even if the generated images do
35
not resemble specic individuals, the prevalence of such images can undermine eorts to nd real-world
36
vicms.
37
GAI models are oen trained on open datasets scraped from the internet, contribung to the
38
unintenonal inclusion of CSAM and non-consensually distributed inmate imagery as part of the
39
9
training data. Recent reports noted that several commonly used GAI training datasets were found to
1
contain hundreds of known images of CSAM. Sexually explicit or obscene content is also parcularly
2
dicult to remove during model training due to detecon challenges and wide disseminaon across the
3
internet. Even when trained on “clean” data, increasingly capable GAI models can synthesize or produce
4
synthec NCII and CSAM. Websites, mobile apps, and custom-built models that generate synthec NCII
5
have moved rapidly from niche internet forums to mainstream, automated, and scaled online
6
businesses.
7
Generated explicit or obscene AI content may include highly-realisc “deepfakes” of real individuals,
8
including children. For example, non-consensual AI-generated inmate images of a prominent
9
entertainer ooded social media and aracted hundreds of millions of views.
10
Toxicity, Bias, and Homogenizaon
11
Toxicity in this context refers to negave, disrespecul, or unreasonable content or language that can be
12
created by or intenonally programmed into GAI systems. Diculty controlling the creaon of and public
13
exposure to toxic, hate-promong or hate speech, and denigrang or stereotypical content generated by
14
AI can lead to representaonal harms. For example, bias in word embeddings used by mulmodal AI
15
models under-represent women when prompted to generate images of CEOs, doctors, lawyers, and
16
judges. Bias in GAI models or training data can also harm representaon or preserve or exacerbate racial
17
bias, separately or in addion to toxicity.
18
Toxicity and bias can also lead to homogenizaon or other undesirable outcomes. Homogenizaon in GAI
19
outputs can result in similar aesthec styles, reduced content diversity, and the promoon of select
20
opinions or values at scale. These phenomena might arise from the inherent biases of foundaon
21
models, which could create “bolenecks,” or singular points of failure of discriminaon or exclusion that
22
replicate to many downstream applicaons.
23
The related concern of model collapse, when GAI models are trained on generated data or outputs from
24
previous models, results in the disappearance of outliers or unique data points in the dataset or
25
distribuon. Model collapse can stem from uniform feedback loops or training on synthec data. Model
26
collapse could lead to undesired homogenizaon of outputs, which poses a threat to specic groups and
27
to the robustness of the model overall. Other biases of GAI systems can result in the unfair distribuon
28
of capabilies or benets from model access. Model capabilies and outcomes may be worse for some
29
groups compared to others, such as reduced LLM performance for non-English languages. Reduced
30
performance for non-English languages presents risks for model adopon, inclusion, and accessibility,
31
and could have downstream impacts on the preservaon of the language, parcularly for endangered
32
languages.
33
Value Chain and Component Integraon
34
GAI system value chains oen involve many third-party components such as procured datasets, pre-
35
trained models, and soware libraries. These components might be improperly obtained or not properly
36
veed, leading to diminished transparency or accountability for downstream users. For example, a
37
model might be trained on unveried content from third-party sources, which could result in unveriable
38
10
model outputs. Because GAI systems oen involve many dierent third-party components, it may be
1
dicult to aribute issues in a system’s behavior to any one of these sources.
2
Some third-party components, such as “benchmark” datasets, may also gain credibility only from high-
3
usage, rather than quality, and may feature issues surfaced only when properly veed.
4
3. Acons to Manage GAI Risks
5
Acons to manage GAI risks can be found in the tables below, organized by AI RMF subcategory. Each
6
acon is related to a specic subcategory of the AI RMF, but not every subcategory of the AI RMF is
7
included in this document. Therefore, acons exist for only some AI RMF subcategories.
8
Moreover, not all acons apply to all AI actors. For example, not acons relevant to GAI developers may
9
be relevant to GAI deployers. Organizaons should priorize acons based on their unique situaons
10
and context for using GAI applicaons.
11
Some subcategories in the acon tables below are marked as “foundaonal,” meaning they should be
12
treated as fundamental tasks for GAI risk management and should be considered as the minimum set of
13
acons to be taken. Subcategory acons considered foundaonal are indicated by an ‘*’ in the
14
subcategory tle row.
15
Each acon table includes:
16
• Acon ID: A unique idener for each relevant acon ed to relevant AI RMF funcons and
17
subcategories (e.g., GV-1.1-001 corresponds to the rst acon for Govern 1.1.);
18
• Acon: Steps an organizaon can take to manage GAI risks;
19
• GAI Risks: Tags linking the acon with relevant GAI risks;
20
• Keywords: Tags linking keywords to the acon, including relevant Trustworthy AI Characteriscs
21
in AI RMF 1.0;
22
• AI Actors: Pernent AI Actors and Actor Tasks.
23
Acon tables begin with the AI RMF subcategory, shaded in blue, followed by relevant acons. Each
24
acon ID corresponds to the relevant funcon and subfuncon (e.g., GV-1.1-001 corresponds to the rst
25
acon for Govern 1.1, GV-1.1-002 corresponds to the second acon for Govern 1.1). Acons are tagged
26
as follows: GV = Govern; MP = Map; MS = Measure; MG = Manage.
27
*GOVERN 1.1: Legal and regulatory requirements involving AI are understood, managed, and documented.
Acon ID
Acon
Risks
GV-1.1-001
Align GAI use with applicable laws and policies, including those related to data
privacy and the use, publicaon, or distribuon of licensed, patented,
trademarked, copyrighted, or trade secret material.
Data Privacy, Intellectual Property
GV-1.1-002
Dene and communicate organizaonal access to GAI through management, legal,
11
and compliance funcons.
GV-1.1-003
Disclose use of GAI to end users.
Human AI Conguraon
GV-1.1-004
Establish policies restricng the use of GAI in regulated dealings or applicaons
across the organizaon where compliance with applicable laws and regulaons
may be infeasible.
GV-1.1-005
Establish policies restricng the use of GAI to create child sexual abuse materials
(CSAM) or other nonconsensual inmate imagery.
Obscene, Degrading, and/or
Abusive Content, Toxicity, Bias,
and Homogenizaon, Dangerous
or Violent Recommendaons
GV-1.1-006
Establish transparent acceptable use policies for GAI that address illegal use or
applicaons of GAI.
AI Actors: Governance and Oversight
1
*GOVERN 1.2: The characteriscs of trustworthy AI are integrated into organizaonal policies, processes, procedures, and
pracces.
Acon ID
Acon
Risks
GV-1.2-001
Connect new GAI policies, procedures, and processes to exisng model, data, and
IT governance and to legal, compliance, and risk funcons.
GV-1.2-002
Consider factors such as internal vs. external use, narrow vs. broad applicaon
scope, ne-tuning and training data sources (i.e., grounding) when dening risk-
based controls.
GV-1.2-003
Dene acceptable use policies for GAI systems deployed by, used by, and used
within the organizaon.
GV-1.2-004
Establish and maintain policies for individual and organizaonal accountability
regarding the use of GAI.
GV-1.2-005
Establish policies and procedures for ensuring that harmful or illegal content,
parcularly CBRN informaon, CSAM, known NCII, nudity, and graphic violence, is
not included in training data.
CBRN Informaon, Obscene,
Degrading, and/or Abusive
Content, Dangerous or Violent
Recommendaons
GV-1.2-006
Establish policies to dene mechanisms for measuring the eecveness of
standard content provenance methodologies (e.g., cryptography, watermarking,
steganography, etc.) and tesng (including reverse engineering).
Informaon Integrity
12
GV-1.2-007
Establish transparency policies and processes for documenng the origin of
training data and generated data for GAI applicaons, including copyrights,
licenses, and data privacy, to advance content provenance.
Data Privacy, Informaon
Integrity, Intellectual Property
GV-1.2-008
Update exisng policies, procedures, and processes to control risks unique to or
exacerbated by GAI.
AI Actors: Governance and Oversight
1
*GOVERN 1.3: Processes, procedures, and pracces are in place to determine the needed level of risk management acvies
based on the organizaon’s risk tolerance.
Acon ID
Acon
Risks
GV-1.3-001
Consider the following, or similar, factors when updang or dening risk ers for
GAI: Abuses and risks to informaon integrity; Cadence of vendor releases and
updates; Data protecon requirements; Dependencies between GAI and other IT
or data systems; Harm in physical environments; Human review of GAI system
outputs; Legal or regulatory requirements; Presentaon of obscene, objeconable,
toxic, invalid or untruthful output; Psychological impacts to humans (e.g.,
anthropomorphizaon, algorithmic aversion, emoonal entanglement); Immediate
and long term impacts; Internal vs. external use; Unreliable decision making
capabilies, validity, adaptability, and variability of GAI system performance over
me.
Informaon Integrity, Obscene,
Degrading, and/or Abusive
Content, Value Chain and
Component Integraon, Toxicity,
Bias, and Homogenizaon,
Dangerous or Violent
Recommendaons, CBRN
Informaon
GV-1.3-002
Dene acceptable uses for GAI systems, where some applicaons may be
restricted.
GV-1.3-003
Increase cadence for internal audits to address any unancipated changes in GAI
technologies or applicaons.
GV-1.3-004
Maintain an updated hierarchy of idened and expected GAI risks connected to
contexts of GAI use, potenally including specialized risk levels for GAI systems
that address risks such as model collapse and algorithmic monoculture.
Toxicity, Bias, and Homogenizaon
GV-1.3-005
Reevaluate organizaonal risk tolerances to account for broad GAI risks, including:
Immature safety or risk cultures related to AI and GAI design, development and
deployment, public informaon integrity risks, including impacts on democrac
processes, unknown long-term performance characteriscs of GAI.
Informaon Integrity, Dangerous
or Violent Recommendaons
GV-1.3-006
Tie expected GAI behavior to trustworthy characteriscs.
AI Actors: Governance and Oversight
2
13
GOVERN 1.5: Ongoing monitoring and periodic review of the risk management process and its outcomes are planned, and
organizaonal roles and responsibilies are clearly dened, including determining the frequency of periodic review.
Acon ID
Acon
Risks
GV-1.5-001
Dene organizaonal responsibilies for content provenance monitoring and
incident response.
Informaon Integrity
GV-1.5-002
Develop or review exisng policies for authorizaon of third party plug-ins and
verify that related procedures are able to be followed.
Value Chain and Component
Integraon
GV-1.5-003
Establish and maintain policies and procedures for monitoring the eecveness of
content provenance for data and content generated across the AI system lifecycle.
Informaon Integrity
GV-1.5-004
Establish organizaonal policies and procedures for aer acon reviews of GAI
system incident response and incident disclosures, to idenfy gaps; Update
incident response and incident disclosure processes as required.
Human AI Conguraon
GV-1.5-005
Establish policies for periodic review of organizaonal monitoring and incident
response plans based on impacts and in line with organizaonal risk tolerance.
Informaon Security,
Confabulaon
GV-1.5-006
Maintain a long term document retenon policy to keep full history for auding,
invesgaon, or improving content provenance methods.
Informaon Integrity
GV-1.5-007
Verify informaon sharing and feedback mechanisms among individuals and
organizaons regarding any negave impact from AI systems due to content
provenance issues.
Informaon Integrity
GV-1.5-008
Verify that review procedures include analysis of cascading impacts of GAI system
outputs used as inputs to third party plug-ins or other systems.
Value Chain and Component
Integraon
AI Actors: Governance and Oversight, Operaon and Monitoring
1
*GOVERN 1.6: Mechanisms are in place to inventory AI systems and are resourced according to organizaonal risk priories.
Acon ID
Acon
Risks
GV-1.6-001
Dene any inventory exempons for GAI systems embedded into applicaon
soware in organizaonal policies.
GV-1.6-002
Enumerate organizaonal GAI systems for incorporaon into AI system inventory
and adjust AI system inventory requirements to account for GAI risks.
GV-1.6-003
In addion to general model, governance, and risk informaon, consider the
following items in GAI system inventory entries: Acceptable use policies and policy
Data Privacy, Human AI
Conguraon, Informaon
14
excepons; Applicaon, Assumpons and limitaons of use, including enumeraon
of restricted uses; Business or model owners; Challenges for explainability,
interpretability, or transparency; Change management, maintenance, and
monitoring plans; Connecons or dependencies between other systems; Consent
informaon and noces; Data provenance informaon (e.g., source, signatures,
versioning, watermarks); Designaon of in-house or third party development;
Designaon of risk level; Disclosure informaon or noces; Incident response
plans; Known issues reported from internal bug tracking or external informaon
sharing resources (e.g., AI incident database, AVID, CVE, or OECD incident
monitor); Human oversight roles and responsibilies; Special rights and
consideraons for intellectual property, licensed works, or personal, privileged,
proprietary or sensive data; Time frame for valid deployment, including date of
last risk assessment; Underlying foundaon models, versions of underlying models,
and access modes; Updated hierarchy of idened and expected risks connected
to contexts of use.
Integrity, Intellectual Property,
Value Chain and Component
Integraon
GV-1.6-004
Inventory recently decommissioned systems, systems with imminent deployment
plans, and operaonal systems.
GV-1.6-005
Update policy denions for AI systems, models, qualitave tools or similar to
account for GAI systems.
AI Actors: Governance and Oversight
1
GOVERN 1.7: Processes and procedures are in place for decommissioning and phasing out AI systems safely and in a manner that
does not increase risks or decrease the organizaon’s trustworthiness.
Acon ID
Acon
Risks
GV-1.7-001
Allocate me and resources for staged decommissioning for GAI to avoid service
disrupons.
GV-1.7-002
Communicate decommissioning and support plans for GAI systems to AI actors
and users through various channels and maintain communicaon and associated
training protocols.
Human AI Conguraon
GV-1.7-003
Consider the following factors when decommissioning GAI systems: Clear
versioning of decommissioned and replacement systems; Contractual, legal, or
regulatory requirements; Data retenon requirements; Data security, e.g.,
Containment, protocols, Data leakage aer decommissioning; Dependencies
between upstream, downstream, or other data, internet of things (IOT) or AI
systems; Digital and physical arfacts; Recourse mechanisms for impacted users
or communies; Terminaon of related cloud or vendor services; Users’
emoonal entanglement with GAI funcons.
Human AI Conguraon,
Informaon Security, Value Chain
and Component Integraon
15
GV-1.7-004
Implement data security and privacy controls for stored decommissioned GAI
systems.
Data Privacy, Informaon Security
GV-1.7-005
Update exisng policies (e.g., enterprise record retenon policies) or establish
new policies for the decommissioning of GAI systems.
AI Actors: AI Deployment, Operaon and Monitoring
1
*GOVERN 2.1: Roles and responsibilies and lines of communicaon related to mapping, measuring, and managing AI risks are
documented and are clear to individuals and teams throughout the organizaon.
Acon ID
Acon
Risks
GV-2.1-001
Dene acceptable use cases and context under which the organizaon will
design, develop, deploy, and use GAI systems.
GV-2.1-002
Establish policies and procedures for GAI risk acceptance to downstream AI
actors.
Human AI Conguraon, Value
Chain and Component Integraon
GV-2.1-003
Establish policies to idenfy and disclose GAI system incidents to downstream AI
actors, including individuals potenally impacted by GAI outputs.
Human AI Conguraon, Value
Chain and Component Integraon
GV-2.1-004
Establish procedures to engage teams for GAI system incident response with
diverse composion and responsibilies based on the parcular incident type.
Toxicity, Bias, and Homogenizaon
GV-2.1-005
Establish processes to idenfy GAI system incidents and verify the AI actors
conducng these tasks demonstrate and maintain the appropriate skills and
training.
Human AI Conguraon
GV-2.1-006
Verify that incident disclosure plans include sucient GAI system context to
facilitate remediaon acons.
Human AI Conguraon
AI Actors: Governance and Oversight
2
*GOVERN 3.2: Policies and procedures are in place to dene and dierenate roles and responsibilies for human-AI
conguraons and oversight of AI systems.
Acon ID
Acon
Risks
GV-3.2-001
Bolster oversight of GAI systems with independent audits or assessments, or by
the applicaon of authoritave external standards.
16
GV-3.2-002
Consider adjustment of organizaonal roles and components across lifecycle
stages of large or complex GAI systems, including: AI actor, user, and community
feedback relang to GAI systems; Audit, validaon, and red-teaming of GAI
systems; GAI content moderaon; Data documentaon, labeling, preprocessing
and tagging; Decommissioning GAI systems; Decreasing risks of emoonal
entanglement between users and GAI systems; Decreasing risks of decepon by
GAI systems; Discouraging anonymous use of GAI systems; Enhancing
explainability of GAI systems; GAI system development and engineering;
Increased accessibility of GAI tools, interfaces, and systems, Incident response
and containment; Overseeing relevant AI actors and digital enes, including
management of security credenals and communicaon between AI enes;
Training GAI users within an organizaon about GAI fundamentals and risks.
Human AI Conguraon,
Informaon Security, Toxicity,
Bias, and Homogenizaon
GV-3.2-003
Dene acceptable use policies for the various categories of GAI interfaces,
modalies, and human-AI conguraons.
Human AI Conguraon
GV-3.2-004
Dene policies for the design of systems that possess human decision-making
powers.
Human AI Conguraon
GV-3.2-005
Establish policies for user feedback mechanisms in GAI systems.
Human AI Conguraon
GV-3.2-006
Establish policies to empower accountable execuves to oversee GAI system
adopon, use, and decommissioning.
GV-3.2-007
Establish processes to include and empower interdisciplinary team member
perspecves across the AI lifecycle.
Toxicity, Bias, and Homogenizaon
GV-3.2-008
Evaluate AI actor teams in consideraon of credenals, demographic
representaon, interdisciplinary diversity, and professional qualicaons.
Human AI Conguraon, Toxicity,
Bias, and Homogenizaon
AI Actors: AI Design
1
17
*GOVERN 4.1: Organizaonal policies and pracces are in place to foster a crical thinking and safety-rst mindset in the design,
development, deployment, and uses of AI systems to minimize potenal negave impacts.
Acon ID
Acon
Risks
GV-4.1-001
Establish criteria and acceptable use policies for the use of GAI in decision
making tasks in accordance with organizaonal risk tolerance, and other policies
laid out in the Govern funcon; to include detailed criteria for the kinds of
queries GAI models should refuse to respond to.
Human AI Conguraon
GV-4.1-002
Establish policies and procedures that address connual improvement processes
for risk measurement: Address general risks associated with a lack of
explainability and transparency in GAI systems by using ample documentaon
and techniques such as: applicaon of gradient-based aribuons,
occlusion/term reducon, counterfactual prompts and prompt engineering, and
analysis of embeddings; Assess and update risk measurement approaches at
regular cadences.
GV-4.1-003
Establish policies, procedures, and processes detailing risk measurement in
context of use with standardized measurement protocols and structured public
feedback exercises such as AI red-teaming or independent external audits.
GV-4.1-004
Establish policies, procedures, and processes for oversight funcons (e.g., senior
leadership, legal, compliance, and risk) across the GAI lifecycle, from problem
formulaon and supply chains to system decommission.
Value Chain and Component
Integraon
GV-4.1-005
Establish policies, procedures, and processes that promote eecve challenge of
AI system design, implementaon, and deployment decisions via mechanisms
such as three lines of defense, to minimize risks arising from workplace culture
(e.g., conrmaon bias, funding bias, groupthink, over-reliance on metrics).
Toxicity, Bias, and Homogenizaon
GV-4.1-006
Incorporate GAI governance policies into exisng incident response,
whistleblower, vendor or investment due diligence, acquision, procurement,
reporng or internal audit policies.
Value Chain and Component
Integraon
AI Actors: AI Deployment, AI Design, AI Development, Operaon and Monitoring
1
*GOVERN 4.2: Organizaonal teams document the risks and potenal impacts of the AI technology they design, develop, deploy,
evaluate, and use, and they communicate about the impacts more broadly.
Acon ID
Acon
Risks
18
GV-4.2-001
Develop policies, guidelines, and pracces for monitoring organizaonal and
third-party impact assessments (data, labels, bias, privacy, models, algorithms,
errors, provenance techniques, security, legal compliance, output, etc.) to
migate risk and harm.
Confabulaon, Data Privacy,
Informaon Integrity, Informaon
Security, Value Chain and
Component Integraon, Toxicity,
Bias, and Homogenizaon,
Dangerous or Violent
Recommendaons
GV-4.2-002
Establish clear roles and responsibilies for inter-organizaonal incident
response and communicaon for GAI systems that involve mulple organizaons
involved in dierent aspects of the GAI system lifecycle.
GV-4.2-003
Establish clearly dened terms of use and terms of service.
Intellectual Property
GV-4.2-004
Establish criteria for ad-hoc impact assessments based on incident reporng or
new use cases for the GAI system.
GV-4.2-005
Establish organizaonal roles, policies, and procedures for communicang and
reporng GAI system risks and terms of use or service, relevant for dierent AI
actors.
Human AI Conguraon,
Intellectual Property
GV-4.2-006
Establish policies and procedures to document new ways AI actors interact with
the GAI system.
Human AI Conguraon
GV-4.2-007
Establish policies and procedures to monitor compliance with established terms
of service and use.
Intellectual Property
GV-4.2-008
Establish policies to align organizaonal and third-party assessments with
regulatory and legal compliance regarding content provenance.
Informaon Integrity, Value Chain
and Component Integraon
GV-4.2-009
Establish policies to incorporate adversarial examples and other provenance
aacks in AI model training processes to enhance resilience against aacks.
Informaon Integrity, Informaon
Security
GV-4.2-010
Establish processes to monitor and idenfy misuse, unforeseen use cases, risks
of the GAI system and potenal impacts of those risks (leveraging GAI system use
case inventory).
CBRN Informaon, Confabulaon,
Dangerous or Violent
Recommendaons
GV-4.2-011
Implement standardized documentaon of GAI system risks and potenal
impacts.
GV-4.2-012
Include relevant AI Actors in the GAI system risk idencaon process.
Human AI Conguraon
GV-4.2-013
Verify that downstream GAI system impacts (such as the use of third-party plug-
ins) are included in the impact documentaon process.
Value Chain and Component
Integraon
19
GV-4.2-014
Verify that the organizaonal list of risks related to the use of the GAI system are
updated based on unforeseen GAI system incidents.
AI Actors: AI Deployment, AI Design, AI Development, Operaon and Monitoring
1
*GOVERN 4.3: Organizaonal pracces are in place to enable AI tesng, idencaon of incidents, and informaon sharing.
Acon ID
Acon
Risks
GV-4.3-001
Allocate resources and adjust adopon, development, and implementaon
meframes to enable independent measurement, connuous monitoring, and
fulsome informaon sharing for GAI system risks.
GV-4.3-002
Develop standardized documentaon templates for ecient review of risk
measurement results.
GV-4.3-003
Establish minimum thresholds for performance and review as part of
deployment approval (“go/”no-go”) policies, procedures, and processes, with
reviewed processes and approval thresholds reecng measurement of GAI
capabilies and risks.
GV-4.3-004
Establish organizaonal roles, policies, and procedures for communicang GAI
system incidents and performance to AI actors and downstream stakeholders, via
community or ocial resources (e.g., AI Incident Database, AVID, AI Ligaon
Database, CVE, OECD Incident Monitor, or others).
Human AI Conguraon, Value
Chain and Component Integraon
GV-4.3-005
Establish policies and procedures for pre-deployment GAI system tesng that
validates organizaonal capability to capture GAI system incident reporng
criteria.
GV-4.3-006
Establish policies, procedures, and processes that bolster independence of risk
management and measurement funcons (e.g., independent reporng chains,
aligned incenves).
GV-4.3-007
Establish policies, procedures, and processes that enable and incenvize in-
context risk measurement via standardized measurement and structured public
feedback approaches.
GV-4.3-008
Organizaonal procedures idenfy the minimum set of criteria necessary for GAI
system incident reporng such as: System ID (auto-generated most likely), Title,
Reporter, System/Source, Data Reported, Date of Incident, Descripon,
Impact(s), Stakeholder(s) Impacted.
AI Actors: Fairness and Bias, Governance and Oversight, Operaon and Monitoring, TEVV
2
20
*GOVERN 5.1: Organizaonal policies and pracces are in place to collect, consider, priorize, and integrate feedback from those
external to the team that developed or deployed the AI system regarding the potenal individual and societal impacts related to AI
risks.
Acon ID
Acon
Risks
GV-5.1-001
Allocate me and resources for outreach, feedback, and recourse processes in
GAI system development.
GV-5.1-002
Disclose interacons with GAI systems to users prior to interacve acvies.
Human AI Conguraon
GV-5.1-003
Establish policy, guidelines and processes that: Engage independent experts to
audit models, data sources, licenses, algorithms, and other system components,
Consider sponsoring or engaging in community- based exercises (e.g., bug
bounes, hackathons, compeons) where AI Actors assess and benchmark the
performance of AI systems, including the robustness of content provenance
management under various condions; Document data sources, licenses,
training methodologies, and trade-os considered in the design of AI systems;
Establish mechanisms, plaorms or channels (e.g., user interfaces, web portals,
forums) for independent experts, users, or community members to provide
feedback related to AI systems; Adjudicate and implement relevant feedback at a
regular cadence, Establish transparency mechanisms to track the origin of data
and generated content; Audit and validate these mechanisms.
Human AI Conguraon,
Informaon Integrity, Intellectual
Property
GV-5.1-004
Establish processes to bolster internal AI actor culture in alignment with
organizaonal principles and norms and to empower exploraon of GAI
limitaons beyond development sengs.
Human AI Conguraon, Toxicity,
Bias, and Homogenizaon
GV-5.1-005
Establish the following GAI-specic policies and procedures for independent AI
Actors: Connuous improvement processes for increasing explainability and
migang other risks; Impact assessments, Incenves for internal AI actors to
provide feedback and conduct independent risk management acvies;
Independent management and reporng structures for AI actors engaged in
model and system audit, validaon, and oversight; TEVV processes for the
eecveness of feedback mechanisms employing parcipaon rates, resoluon
me, or similar measurements.
Human AI Conguraon
GV-5.1-006
Provide thorough instrucons for GAI system users to provide feedback and
understand recourse mechanisms.
Human AI Conguraon
GV-5.1-007
Standardize user feedback about GAI system behavior, risks and limitaons for
ecient adjudicaon and incorporaon.
Human AI Conguraon
AI Actors: AI Design, AI Impact Assessment, Aected Individuals and Communies, Governance and Oversight
1
21
*GOVERN 6.1: Policies and procedures are in place that address AI risks associated with third-party enes, including risks of
infringement of a third-party’s intellectual property or other rights.
Acon ID
Acon
Risks
GV-6.1-001
Categorize dierent types of GAI content with associated third party risks (i.e.,
copyright, intellectual property, data privacy).
Data Privacy, Intellectual Property,
Value Chain and Component
Integraon
GV-6.1-002
Conduct due diligence on third-party enes and end-users from those enes
before entering into agreements with them (e.g., checking references, reviewing
their content handling processes, etc.).
Human AI Conguraon, Value
Chain and Component Integraon
GV-6.1-003
Conduct joint educaonal acvies and events in collaboraon with third-pares
to promote content provenance best pracces.
Informaon Integrity, Value Chain
and Component Integraon
GV-6.1-004
Conduct regular audits of third-party enes to ensure compliance with
contractual agreements.
Value Chain and Component
Integraon
GV-6.1-005
Dene and communicate organizaonal roles and responsibilies for GAI
acquision, human resources, procurement, and talent management processes
in policies and procedures.
Human AI Conguraon
GV-6.1-006
Develop an incident response plan for third pares specically tailored to
address content provenance incidents or breaches and regularly test and update
the incident response plan with feedback form external and third party
stakeholders.
Data Privacy, Informaon
Integrity, Informaon Security,
Value Chain and Component
Integraon
GV-6.1-007
Develop and validate approaches for measuring the success of content
provenance management eorts with third pares (e.g., incidents detected and
response mes).
Informaon Integrity, Value Chain
and Component Integraon
GV-6.1-008
Develop risk tolerance and criteria to quantavely assess and compare the level
of risk associated with dierent third-party enes (i.e., reputaon, track record,
security measure, and the sensivity of the content they handle).
Informaon Security, Value Chain
and Component Integraon
GV-6.1-009
Dra and maintain well-dened contracts and service level agreements (SLAs)
that specify content ownership, usage rights, quality standards, security
requirements, and content provenance expectaons.
Informaon Integrity, Informaon
Security
GV-6.1-010
Establish processes to maintain awareness of evolving risks, technologies, and
best pracces in content provenance management.
Informaon Integrity
22
GV-6.1-011
Implement a supplier risk assessment framework to connuously evaluate and
monitor third-party enes’ performance and adherence to content provenance
standards and technologies (e.g., digital signatures, watermarks, cryptography,
etc.) to detect anomalies and unauthorized changes; services acquision and
supply chain risk management; legal compliance (e.g., copyright, trademarks,
and data privacy laws).
Data Privacy, Informaon
Integrity, Informaon Security,
Intellectual Property, Value Chain
and Component Integraon
GV-6.1-012
Include audit clauses in contracts that allow the organizaon to verify
compliance with content provenance requirements.
Informaon Integrity
GV-6.1-013
Inventory all third-party enes with access to organizaonal content and
establish approved GAI technology and service provider lists.
Value Chain and Component
Integraon
GV-6.1-014
Maintain detailed records of content provenance, including sources, mestamps,
metadata, and any changes made by third pares.
Informaon Integrity, Value Chain
and Component Integraon
GV-6.1-015
Provide proper training to internal employees on content provenance best
pracces, risks, and reporng procedures.
Informaon Integrity
GV-6.1-016
Update and integrate due diligence processes for GAI acquision and
procurement vendor assessments to include intellectual property, data privacy,
security, and other risks. For example, update policies to: Address roboc
process automaon (RPA), soware-as-a-service (SAAS), and other soluons that
may rely on embedded GAI technologies; Address ongoing audits, assessments,
and alerng, dynamic risk assessments, and real-me reporng tools for
monitoring third-party GAI risks; Address accessibility, accommodaons, or opt-
outs in GAI vendor oerings; Address commercial use of GAI outputs and
secondary use of collected data by third pares; Assess vendor risk controls for
intellectual property infringements and data privacy; Consider policy
adjustments across GAI modeling libraries, tools and APIs, ne-tuned models,
and embedded tools; Establish ownership of GAI acquision and procurement
processes; Include relevant organizaonal funcons in evaluaons of GAI third
pares (e.g., legal, informaon technology (IT), security, privacy, fair lending);
Include instrucon on intellectual property infringement and other third-party
GAI risks in GAI training for AI actors; Screen GAI vendors, open source or
proprietary GAI tools, or GAI service providers against incident or vulnerability
databases; Screen open source or proprietary GAI training data or outputs
against patents, copyrights, trademarks and trade secrets.
Data Privacy, Human AI
Conguraon, Informaon
Security, Intellectual Property,
Value Chain and Component
Integraon, Toxicity, Bias, and
Homogenizaon
GV-6.1-017
Update GAI acceptable use policies to address proprietary and open-source GAI
technologies and data, and contractors, consultants, and other third-party
personnel.
Intellectual Property, Value Chain
and Component Integraon
GV-6.1-018
Update human resource and talent management standards to address
acceptable use of GAI.
Human AI Conguraon
23
GV-6.1-019
Update third-party contracts, service agreements, and warranes to address GAI
risks; Contracts, service agreements, and similar documents may include GAI-
specic indemnity clauses, dispute resoluon mechanisms, and other risk
controls.
Value Chain and Component
Integraon
AI Actors: Operaon and Monitoring, Procurement, Third-party enes
1
GOVERN 6.2: Conngency processes are in place to handle failures or incidents in third-party data or AI systems deemed to be
high-risk.
Acon ID
Acon
Risks
GV-6.2-001
Apply exisng organizaonal risk management policies, procedures, and
documentaon processes to third-party GAI data and systems, including open
source data and soware.
Intellectual Property, Value Chain
and Component Integraon
GV-6.2-002
Document downstream GAI system impacts (e.g., the use of third-party plug-ins)
for third party dependencies.
Value Chain and Component
Integraon
GV-6.2-003
Document GAI system supply chain risks to idenfy over-reliance on third party
data or GAI systems and to idenfy fallbacks.
Value Chain and Component
Integraon
GV-6.2-004
Document incidents involving third-party GAI data and systems, including open
source data and soware.
Intellectual Property, Value Chain
and Component Integraon
GV-6.2-005
Enumerate organizaonal GAI system risks based on external dependencies on
third-party data or GAI systems.
Value Chain and Component
Integraon
GV-6.2-006
Establish acceptable use policies that idenfy dependencies, potenal impacts,
and risks associated with third-party data or GAI systems deemed high-risk.
Value Chain and Component
Integraon
GV-6.2-007
Establish conngency and communicaon plans to support fallback alternaves
for downstream users in the event the GAI system is disabled.
Human AI Conguraon, Value
Chain and Component Integraon
GV-6.2-008
Establish incident response plans for third-party GAI technologies deemed high-
risk: Align incident response plans with impacts enumerated in MAP 5.1;
Communicate third-party GAI incident response plans to all relevant AI actors;
Dene ownership of GAI incident response funcons; Rehearse third-party GAI
incident response plans at a regular cadence; Improve incident response plans
based on retrospecve learning; Review incident response plans for alignment
with relevant breach reporng, data protecon, data privacy, or other laws.
Data Privacy, Human AI
Conguraon, Informaon
Security, Value Chain and
Component Integraon, Toxicity,
Bias, and Homogenizaon
GV-6.2-009
Establish organizaonal roles, policies, and procedures for communicang with
data and GAI system providers regarding performance, disclosure of GAI system
inputs, and use of third-party data and GAI systems.
Human AI Conguraon, Value
Chain and Component Integraon
24
GV-6.2-010
Establish policies and procedures for connuous monitoring of third-party GAI
systems in deployment.
Value Chain and Component
Integraon
GV-6.2-011
Establish policies and procedures that address GAI data redundancy, including
model weights and other system arfacts.
Toxicity, Bias, and Homogenizaon
GV-6.2-012
Establish policies and procedures to test and manage risks related to rollover and
fallback technologies for GAI systems, acknowledging that rollover and fallback
may include manual processing.
GV-6.2-013
Idenfy and document high-risk third-party GAI technologies in organizaonal AI
inventories, including open-source GAI soware.
Intellectual Property, Value Chain
and Component Integraon
GV-6.2-014
Review GAI vendor documentaon for thorough instrucons, meaningful
transparency into data or system mechanisms, ample support and contact
informaon, and alignment with organizaonal principles.
Value Chain and Component
Integraon, Toxicity, Bias, and
Homogenizaon
GV-6.2-015
Review GAI vendor release cadences and roadmaps for irregularies and
alignment with organizaonal principles.
Value Chain and Component
Integraon, Toxicity, Bias, and
Homogenizaon
GV-6.2-016
Review vendor contracts and avoid arbitrary or capricious terminaon of crical
GAI technologies or vendor services and Non-standard terms that may amplify or
defer liability in unexpected ways and Unauthorized data collecon by vendors or
third-pares (e.g., secondary data use); Consider: Clear assignment of liability and
responsibility for incidents, GAI system changes over me (e.g., ne-tuning, dri,
decay); Request: Nocaon and disclosure for serious incidents arising from
third-party data and systems, Service line agreements (SLAs) in vendor contracts
that address incident response, response mes, and availability of crical support.
Human AI Conguraon,
Informaon Security, Value Chain
and Component Integraon
AI Actors: AI Deployment, Operaon and Monitoring, TEVV, Third-party enes
1
*MAP 1.1: Intended purposes, potenally benecial uses, context specic laws, norms and expectaons, and prospecve sengs in
which the AI system will be deployed are understood and documented. Consideraons include: the specic set or types of users
along with their expectaons; potenal posive and negave impacts of system uses to individuals, communies, organizaons,
society, and the planet; assumpons and related limitaons about AI system purposes, uses, and risks across the development or
product AI lifecycle; and related TEVV and system metrics.
Acon ID
Acon
Risks
MP-1.1-001
Apply risk mapping and measurement plans to third-party and open-source
systems.
Intellectual Property, Value Chain
and Component Integraon
25
MP-1.1-002
Collaborate with domain experts to explore and document gaps, limitaons,
and risks in pre-deployment tesng and the praccal and contextual dierences
between pre-deployment tesng and the ancipated context(s) of use.
MP-1.1-003
Conduct impact assessments or review past known incidents and failure modes
to priorize and inform risk measurement.
MP-1.1-004
Determine and document the expected and acceptable GAI system context of
use in collaboraon with socio-cultural and other domain experts, by assessing:
Assumpons and limitaons; Direct value to the organizaon; Intended
operaonal environment and observed usage paerns; Potenal posive and
negave impacts to individuals, public safety, groups, communies,
organizaons, democrac instuons, and the physical environment; Social
norms and expectaons.
Toxicity, Bias, and Homogenizaon
MP-1.1-005
Document GAI system ownership, intended use, direct organizaonal value, and
assumpons and limitaons.
MP-1.1-006
Document risk measurement plans that address: Individual and group cognive
biases (e.g., conrmaon bias, funding bias, groupthink) for AI actors involved in
the design, implementaon, and use of GAI systems; Known past GAI system
incidents and failure modes; In-context use and foreseeable misuse, abuse, and
o-label use; Over reliance on quantave metrics and methodologies without
sucient awareness of their limitaons in the context(s) of use; Risks associated
with trustworthy characteriscs across the AI lifecycle; Standard measurement
and structured human feedback approaches; Ancipated human-AI
conguraons.
Human AI Conguraon, Toxicity,
Bias, and Homogenizaon,
Dangerous or Violent
Recommendaons
MP-1.1-007
Document risks related to transparency, accountability, explainability, and
interpretability in risk measurement plans, system risk assessments, and
deployment approval (“go”/”no-go”) decisions.
MP-1.1-008
Document system requirements, ownership, and AI actor roles and
responsibilies for human oversight of GAI systems.
Human AI Conguraon
MP-1.1-009
Document the extent to which a lack of transparency or explainability impedes
risk measurement across the AI lifecycle.
MP-1.1-010
Idenfy and document foreseeable illegal uses or applicaons that surpass
organizaonal risk tolerances.
AI Actors: AI Deployment
1
26
*MAP 1.2: Interdisciplinary AI actors, competencies, skills, and capacies for establishing context reect demographic diversity and
broad domain and user experience experse, and their parcipaon is documented. Opportunies for interdisciplinary
collaboraon are priorized.
Acon ID
Acon
Risks
MP-1.2-001
Document the credenals and qualicaons of organizaonal AI actors and AI
actor team composion.
Human AI Conguraon
MP-1.2-002
Establish and empower interdisciplinary teams that reect a wide range of
capabilies, competencies, demographic groups, domain experse, educaonal
backgrounds, lived experiences, professions, and skills across the enterprise to
inform and conduct TEVV of GAI technology, and other risk measurement and
management funcons.
Human AI Conguraon, Toxicity,
Bias, and Homogenizaon
MP-1.2-003
Establish connuous improvement processes to increase diversity and
representaveness in AI actor teams, standard measurement resources, and
structured public feedback parcipants from subgroup populaons in-context.
Human AI Conguraon, Toxicity,
Bias, and Homogenizaon
MP-1.2-004
Verify that AI actor team membership includes demographic diversity,
applicable domain experse, varied educaon backgrounds, and lived
experiences.
Human AI Conguraon, Toxicity,
Bias, and Homogenizaon
MP-1.2-005
Verify that data or benchmarks used in risk measurement, and users,
parcipants, or subjects involved in structured public feedback exercises are
representave of diverse in-context user populaons.
Human AI Conguraon, Toxicity,
Bias, and Homogenizaon
AI Actors: AI Deployment
1
*MAP 2.1: The specic tasks and methods used to implement the tasks that the AI system will support are dened (e.g., classiers,
generave models, recommenders).
Acon ID
Acon
Risks
MP-2.1-001
Dene GAI system's task(s) that relate to content provenance, such as original
content creaon, media synthesis, or data augmentaon while incorporang
tracking measures.
Informaon Integrity
MP-2.1-002
Establish known assumpons and pracces for determining data origin and
content lineage, for documentaon and evaluaon.
Informaon Integrity
MP-2.1-003
Idenfy and document GAI task limitaons that might impact the reliability or
authencity of the content provenance.
Informaon Integrity
27
MP-2.1-004
Instute audit trails for data and content ows within the system, including but
not limited to, original data sources, data transformaons, and decision-making
criteria.
MP-2.1-005
Review ecacy of content provenance techniques on a regular basis and
update protocols as necessary.
Informaon Integrity
AI Actors: TEVV
1
MAP 2.2: Informaon about the AI system’s knowledge limits and how system output may be ulized and overseen by humans is
documented. Documentaon provides sucient informaon to assist relevant AI actors when making decisions and taking
subsequent acons.
Acon ID
Acon
Risks
MP-2.2-001
Assess whether the GAI system fullls its intended purpose within its
operaonal context on a regular basis.
MP-2.2-002
Evaluate whether GAI operators and end-users can accurately understand
content lineage and origin.
Human AI Conguraon,
Informaon Integrity
MP-2.2-003
Idenfy and document how the system relies on upstream data sources for
content provenance and if it serves as an upstream dependency for other
systems.
Informaon Integrity, Value Chain
and Component Integraon
MP-2.2-004
Observe and analyze how the AI system interacts with external networks, and
idenfy any potenal for negave externalies, parcularly where content
provenance might be compromised.
Informaon Integrity
MP-2.2-005
Specify the environments where GAI systems may not funcon as intended
related to content provenance.
Informaon Integrity
AI Actors: End Users
2
*MAP 2.3: Scienc integrity and TEVV consideraons are idened and documented, including those related to experimental
design, data collecon and selecon (e.g., availability, representaveness, suitability), system trustworthiness, and construct
validaon
Acon ID
Acon
Risks
28
MP-2.3-001
Assess the accuracy, quality, reliability, and authencity of the GAI content
provenance by comparing it to a set of known ground truth data and by using a
variety of evaluaon methods (e.g., human oversight and automated
evaluaon).
Informaon Integrity
MP-2.3-002
Curate and maintain high quality datasets that are accurate, relevant,
consistent, and representave as well as be well-documented complying with
ethical and legal standards along with diverse data points.
Toxicity, Bias, and Homogenizaon
MP-2.3-003
Deploy and document fact-checking techniques to verify the accuracy and
veracity of informaon generated by GAI systems, especially when the
informaon comes from mulple (or unknown) sources.
Informaon Integrity
MP-2.3-004
Design GAI systems to support content provenance such as tracking the lineage
(e.g., data sources used to train the system, parameters used to generate
content, etc.) and to verify authencity (e.g., using digital signatures or
watermarks).
Informaon Integrity
MP-2.3-005
Develop and implement tesng techniques to idenfy any GAI produced
content (e.g., synthec media) that might be indisnguishable from human-
generated content.
Informaon Integrity
MP-2.3-006
Document GAI content provenance techniques (including experimental
methods), tesng, evaluaon, performance, and validaon metrics throughout
the AI lifecycle.
Informaon Integrity
MP-2.3-007
Implement plans for GAI systems to undergo regular adversarial tesng to
idenfy vulnerabilies and potenal manipulaon risks.
Informaon Security
MP-2.3-008
Integrate GAI systems with exisng content management and version control
systems, to enable content provenance to be tracked across the lifecycle.
Informaon Integrity
MP-2.3-009
Test GAI models using known inputs, context, and environment to conrm they
produce expected outputs across a variety of methods (e.g., unit tests,
integraon tests, and system tests) and help to idenfy and address potenal
problems.
MP-2.3-010
Use diverse large-scale and small-scale datasets for tesng and evaluaon to
ensure that the AI system can perform well on a variety of dierent types of
data.
Toxicity, Bias, and Homogenizaon
MP-2.3-011
Verify that GAI content provenance is accurate and reliable by using
cryptographic techniques and performing formal audits to ensure it has not
been manipulated.
Informaon Integrity
29
MP-2.3-012
Verify that the AI system’s content provenance complies with relevant laws and
regulaons, such as legal infringement, terms and condions, copyright and
intellectual property rights, when using data sources and generang content.
Informaon Integrity, Intellectual
Property
AI Actors: AI Development, Domain Experts, TEVV
1
MAP 3.4: Processes for operator and praconer prociency with AI system performance and trustworthiness – and relevant
technical standards and cercaons – are dened, assessed, and documented.
Acon ID
Acon
Risks
MP-3.4-001
Adapt exisng training programs to include modules on content provenance.
Informaon Integrity
MP-3.4-002
Develop cercaon programs that test prociency in managing AI risks and
interpreng content provenance, relevant to specic industry and context.
Informaon Integrity
MP-3.4-003
Delineate human prociency tests from tests of AI capabilies.
Human-AI Conguraon
MP-3.4-004
Integrate human and other qualitave inputs to comprehensively assess
content provenance.
Informaon Integrity
MP-3.4-005
Ensure that output provided to operators and praconers is both interacve
and well-dened, incorporang content provenance data that can be easily
interpreted for eecve downstream decision-making.
Informaon Integrity, Value Chain
and Component Integraon
MP-3.4-006
Establish and adhere to design principles that ensure safe and ethical operaon,
taking into account the interpretaon of content provenance informaon.
Informaon Integrity, Toxicity, Bias,
and Homogenizaon, Dangerous or
Violent Recommendaons
MP-3.4-007
Implement systems to connually monitor and track the outcomes of human-AI
collaboraons for future renement and improvements, integrang a focus on
content provenance wherever applicable.
Human AI Conguraon,
Informaon Integrity
MP-3.4-008
Involve the end-users, praconers, and operators in AI system prototyping and
tesng acvies. Make sure these tests cover various scenarios where content
provenance could play a crical role, such as crisis situaons or ethically
sensive contexts.
Human AI Conguraon,
Informaon Integrity, Toxicity, Bias,
and Homogenizaon
MP-3.4-009
Match the complexity of GAI system explanaons and the provenance data to
the level of the problem and contextual intricacy.
Informaon Integrity
AI Actors: AI Design, AI Development, Domain Experts, End-Users, Human Factors, Operaon and Monitoring
2
30
*MAP 4.1: Approaches for mapping AI technology and legal risks of its components – including the use of third-party data or
soware – are in place, followed, and documented, as are risks of infringement of a third party’s intellectual property or other
rights.
Acon ID
Acon
Risks
MP-4.1-001
Conduct audits on third-party processes and personnel including an examinaon
of the third-party’s reputaon.
Value Chain and Component
Integraon
MP-4.1-002
Conduct periodic audits and monitor AI generated content for privacy risks;
address any possible instances of sensive data exposure.
Data Privacy
MP-4.1-003
Consider using synthec data as applicable to train AI models in place of real-
world data to match the stascal properes of real-world data without
disclosing personally idenable informaon.
MP-4.1-004
Develop pracces for periodic monitoring of GAI outputs for possible intellectual
property infringements and other risks and implement processes for responding
to potenal intellectual property infringement claims.
Intellectual Property
MP-4.1-005
Document all aspects of the AI development process including data sources,
model architectures and training procedures to support reproducon of results,
idenfy any potenal problems, and implement migaon strategies.
MP-4.1-006
Document compliance with legal requirements across the AI lifecycle, including
copyright concerns, privacy protecons.
Data Privacy, Intellectual Property
MP-4.1-007
Document training data curaon policies, including policies to verify that
consent was obtained for the likeness or image of individuals.
Obscene, Degrading, and/or
Abusive Content
MP-4.1-008
Employ encrypon techniques and proper safeguards to ensure secure data
storage and transfer to protect data privacy.
Data Privacy, Informaon Security,
Dangerous or Violent
Recommendaons
MP-4.1-009
Establish policies for collecon, retenon, and minimum quality of data, in
consideraon of the following risks: Disclosure of CBRN informaon by removing
CBRN informaon from training data, Use of Illegal or dangerous content;
Training data imbalance across sub-groups by modality, such as languages for
LLMs or skin tone for image generaon; Leak of personally idenable
informaon, including facial likenesses of individuals unless consent is obtained
for use of their images.
CBRN Informaon, Intellectual
Property, Toxicity, Bias, and
Homogenizaon, Dangerous or
Violent Recommendaons, Data
Privacy
MP-4.1-010
Implement bias migaon approaches by addressing sources of bias in the
training data and by evaluang AI models for bias periodically.
Toxicity, Bias, and Homogenizaon
MP-4.1-011
Implement policies and pracces dening how third-party intellectual property
and training data will be used, stored, and protected.
Intellectual Property, Value Chain
and Component Integraon
31
MP-4.1-012
Implement reproducibility techniques, including: share data publicly or privately
using license and citaon; develop code according to standard soware
pracces; track and document experiments and results; manage the soware
environment and dependencies; ulize virtual environments, version control,
and maintain a requirements document; manage models and arfacts; tracking
AI model versions and documenng model details along with parameters and
experimental results; document data management processes and establish a
tesng/validaon process to maintain reliable results.
Confabulaon, Intellectual
Property, Value Chain and
Component Integraon
MP-4.1-013
Re-evaluate models that were ne-tuned on top of third-party models.
Value Chain and Component
Integraon
MP-4.1-014
Re-evaluate risks when adapng GAI models to new domains.
MP-4.1-015
Review service level agreements and contracts, including license agreements
and any legal documents associated with the third-party intellectual properes,
technologies, and services.
Intellectual Property, Value Chain
and Component Integraon
MP-4.1-016
Use approaches to detect the presence of sensive data in generated output
text, image, video, or audio, and verify that the model will mask any detected
sensive data.
Informaon Integrity
MP-4.1-017
Use trusted sources for training data that are licensed or open source and
ensure that the enty has the legal right for the use of proprietary training data.
Intellectual Property
MP-4.1-018
Apply strong anonymizaon and de-idencaon, and/or dierenal privacy
techniques to protect the privacy of individuals in the training data.
Data Privacy
MP-4.1-019
Verify that third-party models are in compliance with exisng use licenses.
Intellectual Property, Value Chain
and Component Integraon
AI Actors: Governance and Oversight, Operaon and Monitoring, Procurement, Third-party enes
1
*MAP 5.1: Likelihood and magnitude of each idened impact (both potenally benecial and harmful) based on expected use,
past uses of AI systems in similar contexts, public incident reports, feedback from those external to the team that developed or
deployed the AI system, or other data are idened and documented.
Acon ID
Acon
Risks
MP-5.1-001
Apply TEVV pracces for content provenance (e.g., probing a system's synthec
data generaon capabilies for potenal misuse or vulnerabilies using zero-
knowledge proof approaches).
Informaon Integrity, Informaon
Security
32
MP-5.1-002
Assess and document risks related to content provenance. e.g., document the
presence, absence, or eecveness of tagging systems, cryptographic hashes,
blockchain-based, or distributed ledger technology soluons that improve
content tracking transparency and immutability.
Informaon Integrity
MP-5.1-003
Consider GAI-specic mapped risks (e.g., complex security requirements,
potenal for emoonal entanglement of users, large supply chains) in esmates
for likelihood, magnitude of impact and risk.
Human AI Conguraon,
Informaon Security, Value Chain
and Component Integraon
MP-5.1-004
Document esmates of likelihood, magnitude of impact, and risk for GAI
systems in a central repository (e.g., organizaonal AI inventory.).
MP-5.1-005
Enumerate potenal impacts related to content provenance, including best-
case, average-case, and worst-case scenarios.
Informaon Integrity
MP-5.1-006
Esmate likelihood of enumerated impact scenarios using past data or expert
judgment, analysis of known public incidents, standard measurement, and
structured human feedback results.
CBRN Informaon, Dangerous or
Violent Recommendaons
MP-5.1-007
Measure risk as the product of esmated likelihood and magnitude of impact of
a GAI outcome.
MP-5.1-008
Priorize risk acceptance, management, or transfer acvies based on risk
esmates.
MP-5.1-009
Priorize standard measurement and structured public feedback processes
based on risk assessment esmates.
MP-5.1-010
Prole risks arising from GAI systems interacng with, manipulang, or
generang content, and outlining known and potenal vulnerabilies and the
likelihood of their occurrence.
Informaon Security
MP-5.1-011
Scope GAI applicaons narrowly to enable risk-based governance and controls.
AI Actors: AI Deployment, AI Design, AI Development, AI Impact Assessment, Aected Individuals and Communies, End-Users,
Operaon and Monitoring
1
33
MAP 5.2: Pracces and personnel for supporng regular engagement with relevant AI actors and integrang feedback about
posive, negave, and unancipated impacts are in place and documented.
Acon ID
Acon
Risks
MP-5.2-001
Determine context-based measures to idenfy if new impacts are present due to
the GAI system, including regular engagements with downstream AI actors to
idenfy and quanfy new contexts of unancipated impacts of GAI systems.
Human AI Conguraon, Value
Chain and Component Integraon
MP-5.2-002
Plan regular engagements with AI actors responsible for inputs to GAI systems,
including third-party data and algorithms, to review and evaluate unancipated
impacts.
Human AI Conguraon, Value
Chain and Component Integraon
MP-5.2-003
Publish guidance for external AI actors to report unancipated impacts of the
GAI system and to engage with the organizaon in the event of GAI system
impacts.
Human AI Conguraon
AI Actors: AI Deployment, AI Design, AI Impact Assessment, Aected Individuals and Communies, Domain Experts, End-Users,
Human Factors, Operaon and Monitoring
1
*MEASURE 1.1: Approaches and metrics for measurement of AI risks enumerated during the MAP funcon are selected for
implementaon starng with the most signicant AI risks. The risks or trustworthiness characteriscs that will not – or cannot – be
measured are properly documented.
Acon ID
Acon
Risks
MS-1.1-001
Assess the eecveness of implemented methods and metrics at an ongoing
cadence as part of connuous improvement acvies.
MS-1.1-002
Collaborate with muldisciplinary experts (e.g., in the elds of responsible use
of GAI, cybersecurity, or digital forensics) to ensure the selected risk
management approaches are robust and eecve.
Informaon Security; CBRN
Informaon, Toxicity, Bias, and
Homogenizaon
MS-1.1-003
Conduct adversarial role-playing exercises, AI red-teaming, or chaos tesng to
idenfy anomalous or unforeseen failure modes.
Informaon Security, Unknowns
MS-1.1-004
Conduct tradional assessment or TEVV exercises to measure the prevalence of
known risks in deployment contexts.
MS-1.1-005
Document GAI risk measurement or tracking approaches, including tracking of
risks that cannot be easily measured before deployment (e.g., ecosystem-level
risks or risks that unfold over longer me scales).
34
MS-1.1-006
Employ digital signatures and watermarking, blockchain technology, reverse
image and video search, metadata analysis, steganalysis, and/or forensic
analysis to trace the origin and modicaons of digital content.
Informaon Integrity
MS-1.1-007
Employ similarity metrics, tampering indicators, blockchain conrmaon,
metadata consistency, hidden data detecon rate, source reliability, and
consistency with known paerns to measure content provenance risks.
Informaon Integrity
MS-1.1-008
Idenfy content provenance risks in the end-to-end AI supply chain, including
risks associated with data suppliers, data annotators, R&D, joint ventures,
academic or nonprot projects/partners, third party vendors, and contractors.
Informaon Integrity, Value Chain
and Component Integraon
MS-1.1-009
Idenfy potenal content provenance risks and harms in GAI, such as
misinformaon or disinformaon, deepfakes, including NCII, or tampered
content. Enumerate and rank risks and/or harms based on their likelihood and
potenal impact, and determine how well provenance soluons address
specic risks and/or harms.
Informaon Integrity, Dangerous or
Violent Recommendaons,
Obscene, Degrading, and/or
Abusive Content
MS-1.1-010
Implement appropriate approaches and metrics for measuring AI-related
content provenance the and the aforemenoned risks and harms.
Informaon Integrity, Dangerous or
Violent Recommendaons
MS-1.1-011
Integrate tools designed to analyze content provenance and detect data
anomalies, verify the authencity of digital signatures, and idenfy paerns
associated with misinformaon or manipulaon.
Informaon Integrity
MS-1.1-012
Invest in R&D capabilies to evaluate and implement novel methods and
technologies for the measurement of AI-related risks in content provenance,
toxicity, and CBRN.
Informaon Integrity, CBRN
Informaon, Obscene, Degrading,
and/or Abusive Content
MS-1.1-013
Priorize risk measurement according to risk severity as determined during
mapping acvies.
MS-1.1-014
Provide content provenance risk management educaon to AI actors, users, and
stakeholders.
Human AI Conguraon,
Informaon Integrity
MS-1.1-015
Track and document risks or opportunies related to content provenance that
cannot be measured quantavely, including explanaons as to why some risks
cannot be measured (e.g., due to technological limitaons, resource
constraints, or trustworthy consideraons).
Informaon Integrity
MS-1.1-016
Track the number of output data items that are accompanied by provenance
informaon (e.g., watermarks, cryptographic tags).
Informaon Integrity
MS-1.1-017
Track the number of training and input (e.g., prompts) data items that have
provenance records and output data items that potenally infringe on
intellectual property rights.
Informaon Integrity, Intellectual
Property
35
MS-1.1-018
Track the number of training and input data items covered by intellectual
property rights (e.g., copyright, trademark, trade secret).
Intellectual Property
MS-1.1-019
Validate the reliability and integrity of the original data and measure inherent
dependence on training data and its quality.
AI Actors: AI Development, Domain Experts, TEVV
1
*MEASURE 1.3: Internal experts who did not serve as front-line developers for the system and/or independent assessors are
involved in regular assessments and updates. Domain experts, users, AI actors external to the team that developed or deployed the
AI system, and aected communies are consulted in support of assessments as necessary per organizaonal risk tolerance
Acon ID
Acon
Risks
MS-1.3-001
Dene relevant groups of interest (e.g., demographic groups, subject maer
experts, past experience with GAI technology) within the context of use as part
of plans for gathering structured public feedback.
Human AI Conguraon, Toxicity,
Bias, and Homogenizaon, CBRN
MS-1.3-002
Dene sequence of acons for AI red-teaming exercises and accompanying
necessary documentaon pracces.
MS-1.3-003
Dene use cases, contexts of use, capabilies, and negave impacts where
structured human feedback exercises, e.g., AI red-teaming, would be most
benecial for AI risk measurement and management based on the context of
use.
MS-1.3-004
Develop a suite of suitable metrics to evaluate structured feedback results,
informed by representave AI actors.
Human AI Conguraon, Toxicity,
Bias, and Homogenizaon, CBRN
MS-1.3-005
Execute independent audit, AI red-teaming, impact assessments, or other
structured human feedback processes in consultaon with representave AI
actors with experse and familiarity in the context of use, and/or who are
representave of the populaons associated with the context of use.
Human AI Conguraon, Toxicity,
Bias, and Homogenizaon, CBRN
MS-1.3-006
Idenfy and implement methods for post-hoc evaluaon of the eecveness of
structured human feedback processes such as auding, impact assessments,
and AI red-teaming.
MS-1.3-007
Idenfy and implement methods for translang, evaluang, and integrang
structured human feedback output into AI risk management processes,
connuous improvement processes, and related organizaonal decision
making.
MS-1.3-008
Idenfy criteria for determining when structured human feedback exercises are
complete.
36
MS-1.3-009
Idenfy mechanisms and teams to evaluate or other structured human
feedback outcomes.
MS-1.3-010
Recruit auditors, AI red-teams, and structured feedback parcipants in
consideraon of the linguisc, dialectal, and socio-cultural environment of the
expected user base.
Human AI Conguraon
MS-1.3-011
Share structured feedback with relevant AI actors to address idened risks.
Human AI Conguraon
MS-1.3-012
Verify demographic diversity of idened subgroups in structured feedback
exercises.
Toxicity, Bias, and Homogenizaon
MS-1.3-013
Verify those conducng structured human feedback exercises are not directly
involved in system development tasks for the same GAI model.
AI Actors: AI Deployment, AI Development, AI Impact Assessment, Aected Individuals and Communies, Domain Experts, End-
Users, Operaon and Monitoring, TEVV
1
*MEASURE 2.2: Evaluaons involving human subjects meet applicable requirements (including human subject protecon) and are
representave of the relevant populaon.
Acon ID
Acon
Risks
MS-2.2-001
Assess and manage stascal biases related to GAI content provenance through
techniques such as re-sampling, re-weighng, or adversarial training.
Informaon Integrity, Informaon
Security, Toxicity, Bias, and
Homogenizaon
MS-2.2-002
Disaggregate evaluaon metrics by demographic factors to idenfy any
discrepancies in how content provenance mechanisms work across diverse
populaons.
Informaon Integrity, Toxicity, Bias,
and Homogenizaon
MS-2.2-003
Document how content provenance mechanisms are operated in the context of
privacy and security including: Anonymize data to protect the privacy of human
subjects; Remove any personally idenable informaon (PII) to prevent
potenal harm or misuse.
Data Privacy, Human AI
Conguraon, Informaon
Integrity, Informaon Security,
Dangerous or Violent
Recommendaons
MS-2.2-004
Employ techniques like chaos engineering and stakeholder feedback to evaluate
the quality and integrity of data used in training and the provenance of AI-
generated content.
Informaon Integrity
MS-2.2-005
Idenfy biases present in the training data for downstream migaon using
available techniques (e.g., data visualizaon tools).
Value Chain and Component
Integraon, Toxicity, Bias, and
Homogenizaon
37
MS-2.2-006
Implement connuous monitoring of GAI system impacts to idenfy whether
GAI outputs are equitable across various sub-populaons. Seek acve and
direct feedback from aected communies to idenfy issues and improve GAI
system fairness.
Toxicity, Bias, and Homogenizaon
MS-2.2-007
Implement robust cybersecurity measures to protect both the research data,
the GAI system and its content provenance from unauthorized access,
breaches, or tampering and unauthorized disclosure of human subject
informaon.
Data Privacy, Human AI
Conguraon, Informaon
Integrity, Informaon Security
MS-2.2-008
Obtain informed consent from human subject evaluaon parcipants. Informed
consent should include: the nature of the study, informaon about the use of
GAI related to content provenance, its purpose, and potenal implicaons.
Data Privacy, Human AI
Conguraon, Informaon
Integrity
MS-2.2-010
Pracce responsible disclosure of ndings and report discovered vulnerabilies
or biases related to GAI systems and its content provenance.
Informaon Integrity, Informaon
Security, Toxicity, Bias, and
Homogenizaon
MS-2.2-011
Provide human subjects with opons to revoke their consent for future use of
their data in GAI applicaons, parcularly in content provenance aspects.
Data Privacy, Human AI
Conguraon, Informaon
Integrity
MS-2.2-012
Use Instuonal Review Boards as applicable for evaluaons that involve
human subjects.
Human AI Conguraon
MS-2.2-013
Use techniques such as anonymizaon or dierenal privacy to minimize the
risks associated with linking AI-generated content back to individual human
subjects.
Data Privacy, Human AI
Conguraon
MS-2.2-014
Verify accountability and fairness through documentaon of the algorithms,
parameters, and methodologies used in the evaluaon to allow for external
scruny.
Toxicity, Bias, and Homogenizaon
MS-2.2-015
Verify that human subjects selected for evaluaon are representave of the
populaon for the relevant GAI use-case; Consider demographics such as age,
gender, race, ethnicity, socioeconomic status, and geographical locaon to
avoid biases in the AI system related to content provenance.
Human AI Conguraon,
Informaon Integrity, Toxicity, Bias,
and Homogenizaon
MS-2.2-016
Work in close collaboraon with domain experts to understand the specic
requirements and potenal pialls related to content provenance in the GAI
system's intended context of use.
Informaon Integrity
AI Actors: AI Development, Human Factors, TEVV
1
38
*MEASURE 2.3: AI system performance or assurance criteria are measured qualitavely or quantavely and demonstrated for
condions similar to deployment seng(s). Measures are documented.
Acon ID
Acon
Risks
MS-2.3-001
Analyze dierences between intended and actual populaon of users or data
subjects, including likelihood for errors, incidents, or negave impacts.
Confabulaon, Human AI
Conguraon, Informaon
Integrity
MS-2.3-002
Conduct eld tesng on sampled sub-populaons prior to deployment to the
enre populaon.
MS-2.3-003
Conduct TEVV in the operaonal environment in accordance with organizaonal
policies and regulatory or disciplinary requirements (e.g., informed consent,
instuonal review board approval, human research protecons, privacy
requirements).
Data Privacy
MS-2.3-004
Consider baseline model performance on suites of benchmarks when selecng a
model for ne tuning.
MS-2.3-005
Evaluate claims of model capabilies using empirically validated methods.
MS-2.3-006
Include metrics measuring reporng rates for harmful or oensive content in
eld tesng.
Dangerous or Violent
Recommendaons
MS-2.3-007
Share results of pre-deployment tesng with relevant AI actors, such as those
with system release approval authority.
Human AI Conguraon
MS-2.3-008
Use disaggregated evaluaon methods (e.g., by race, age, gender, ethnicity,
ability, region) to improve granularity of AI system performance measures.
MS-2.3-009
Ulize a purpose-built tesng environment such as NIST Dioptra to empirically
evaluate GAI trustworthy characteriscs.
MS-2.3-010
Verify that mechanisms to collect users’ feedback are visible and traceable.
Human AI Conguraon
AI Actors: AI Deployment, TEVV
1
*MEASURE 2.5: The AI system to be deployed is demonstrated to be valid and reliable. Limitaons of the generalizability beyond
the condions under which the technology was developed are documented.
Acon ID
Acon
Risks
39
MS-2.5-001
Apply standard measurement and structured human feedback approaches to
internally-developed and third-party GAI systems.
Value Chain and Component
Integraon
MS-2.5-002
Avoid extrapolang GAI system performance or capabilies from narrow, non-
systemac, and anecdotal assessments.
MS-2.5-003
Conduct security assessments and audits to measure the integrity of training
data, system soware, and system outputs.
Informaon Security
MS-2.5-004
Document the construct validity of methodologies employed in GAI systems
relave to their context of use.
MS-2.5-005
Document the extent to which human domain knowledge is employed to
improve GAI system performance, via, e.g., RLHF, ne-tuning, content
moderaon, business rules.
MS-2.5-006
Establish metrics or KPIs to determine whether GAI systems meet minimum
performance standards for reliability and validity.
MS-2.5-007
Measure, monitor, and document prevalence of erroneous GAI output content,
system availability, and reproducibility of outcomes via eld tesng or other
randomized controlled experiments.
MS-2.5-008
Review and verify sources and citaons in GAI system outputs during pre-
deployment risk measurement and ongoing monitoring acvies.
Confabulaon
MS-2.5-009
Track and document instances of anthropomorphizaon (e.g., human images,
menons of human feelings, cyborg imagery or mofs) in GAI system interfaces.
Human AI Conguraon
MS-2.5-010
Track and document relevant version numbers, planned updates, hoixes, and
other GAI system change management informaon.
MS-2.5-011
Update standard train/test model evaluaon processes for GAI systems. Consider:
Unwanted or undocumented overlaps in train and TEVV data sources, including
their negave spaces (i.e., what is not represented in both); Employing substring
matching or embedding distance approaches to assess similarity across data
parons.
MS-2.5-012
Verify GAI system training data and TEVV data provenance, and that ne-tuning
data is grounded.
Informaon Integrity
AI Actors: Domain Experts, TEVV
1
40
*MEASURE 2.6: The AI system is evaluated regularly for safety risks – as idened in the MAP funcon. The AI system to be
deployed is demonstrated to be safe, its residual negave risk does not exceed the risk tolerance, and it can fail safely, parcularly if
made to operate beyond its knowledge limits. Safety metrics reect system reliability and robustness, real-me monitoring, and
response mes for AI system failures.
Acon ID
Acon
Risks
MS-2.6-001
Assess adverse impacts health and wellbeing impacts for supply chain or other AI
actors that are exposed to obscene, toxic, or violent informaon during the
course of GAI training and maintenance.
Human AI Conguraon, Obscene,
Degrading, and/or Abusive
Content, Value Chain and
Component Integraon,
Dangerous or Violent
Recommendaons
MS-2.6-002
Assess levels of toxicity, intellectual property infringement, data privacy
violaons, obscenity, extremism, violence, or CBRN informaon in system training
data.
Data Privacy, Intellectual Property,
Obscene, Degrading, and/or
Abusive Content, Toxicity, Bias,
and Homogenizaon, Dangerous
or Violent Recommendaons,
CBRN Informaon
MS-2.6-003
Measure and document incident response mes, system down mes, and system
availability: Perform standard measurement and structured human feedback on
GAI systems to detect safety and reliability impacts and harms; Apply human
subjects research protocols and other applicable safety controls when conducng
A/B tesng, AI red-teaming, focus groups, or human testbed measurements;
Idenfy and document any applicaons related to robocs, RPA, and autonomous
vehicles; Conduct AI red-teaming exercises to idenfy harms and impacts related
to safety and validity, reliability, privacy, toxicity and other risks; Monitor high-risk
GAI systems connually for safety and reliability risks once deployed; Monitor GAI
systems to detect dri and anomalies relave to expected performance and
training baselines.
Data Privacy, Human AI
Conguraon, Toxicity, Bias, and
Homogenizaon, Dangerous or
Violent Recommendaons
MS-2.6-004
Re-evaluate safety features of ne-tuned models when the risk of harm exceeds
organizaonal risk tolerance.
Dangerous or Violent
Recommendaons
MS-2.6-005
Review GAI system outputs for validity and safety: Review generated code to
assess risks that may arise from unreliable downstream decision-making.
Value Chain and Component
Integraon, Dangerous or Violent
Recommendaons
MS-2.6-006
Track and document past failed GAI system designs to inform risk measurement
for safety and validity risks.
Dangerous or Violent
Recommendaons
MS-2.6-007
Verify capabilies for liming, pausing, updang, or terminang GAI systems
quickly.
MS-2.6-008
Verify rollover, fallback, or redundancy capabilies for high-risk GAI systems.
41
MS-2.6-009
Verify that GAI system architecture can monitor outputs and performance, and
handle, recover from, and repair errors when security anomalies, threats and
impacts are detected.
Confabulaon, Informaon
Integrity, Informaon Security
MS-2.6-010
Verify that systems properly handle queries that may give rise to inappropriate,
malicious, or illegal usage, including facilitang manipulaon, extoron, targeted
impersonaon, cyber-aacks, and weapons creaon.
CBRN Informaon, Informaon
Security
AI Actors: AI Deployment, AI Impact Assessment, Domain Experts, Operaon and Monitoring, TEVV
1
MEASURE 2.7: AI system security and resilience – as idened in the MAP funcon – are evaluated and documented.
Acon ID
Acon
Risks
MS-2.7-001
Apply established security measures to: Assess risks of backdoors, compromised
dependencies, data breaches, eavesdropping, man-in-the-middle aacks, reverse
engineering other baseline security concerns; Audit supply chains to idenfy
risks arising from, e.g., data poisoning and malware, soware and hardware
vulnerabilies, third-party personnel and soware; Audit GAI systems, pipelines,
plugins and other related arfacts for unauthorized access, malware, and other
known vulnerabilies.
Data Privacy, Informaon
Integrity, Informaon Security,
Value Chain and Component
Integraon
MS-2.7-002
Assess the completeness of documentaon related to data provenance, access
controls, and incident response procedures. Verify GAI system content
provenance documentaon aligns with relevant regulaons and standards.
Informaon Integrity, Toxicity,
Bias, and Homogenizaon
MS-2.7-003
Benchmark GAI system security and resilience related to content provenance
against industry standards and best pracces. Compare GAI system security
features and content provenance methods against industry state-of-the-art.
Informaon Integrity, Informaon
Security
MS-2.7-004
Conduct user surveys to gather user sasfacon with the AI-generated content
and user percepons of content authencity. Analyze user feedback to idenfy
concerns and/or current literacy levels related to content provenance.
Human AI Conguraon,
Informaon Integrity
MS-2.7-005
Engage with security experts, developers, and researchers through informaon
sharing mechanisms to stay updated with the latest advancements in AI security
related to content provenance. Contribute ndings related to AI system security
and content provenance via informaon sharing mechanisms, workshops, or
publicaons.
Informaon Integrity, Informaon
Security
MS-2.7-006
Establish measures and evaluate GAI resiliency as part of pre-deployment tesng
to ensure GAI will funcon under adverse condions and restore full
funconality in a trustworthy manner.
42
MS-2.7-007
Idenfy metrics that reect the eecveness of security measures, such as data
provenance, the number of unauthorized access aempts, penetraons, or
provenance vericaon.
Informaon Integrity, Informaon
Security
MS-2.7-008
Maintain awareness of emergent GAI security risks and associated
countermeasures through community resources, ocial guidance, or research
literature.
Informaon Security, Unknowns
MS-2.7-009
Measure reliability of content provenance vericaon methods, such as
watermarking, cryptographic signatures, hashing, blockchain, or other content
provenance techniques. Evaluate the rate of false posives and false negaves in
content provenance, as well as true posives and true negaves for vericaon.
Informaon Integrity
MS-2.7-010
Measure the average response me to security incidents related to content
provenance, and the proporon of incidents resolved with and without
signicant impact.
Informaon Integrity, Informaon
Security
MS-2.7-011
Measure the rate at which recommendaons from security audits and incidents
are implemented related to content provenance. Assess how quickly the AI
system can adapt and improve based on lessons learned from security incidents
and feedback related to content provenance.
Informaon Integrity, Informaon
Security
MS-2.7-012
Monitor and review the completeness and validity of security documentaon
and verify it aligns with the current state of the GAI system and its content
provenance.
Informaon Integrity, Informaon
Security, Toxicity, Bias, and
Homogenizaon
MS-2.7-013
Monitor GAI system downme and measure its impact on operaons.
MS-2.7-014
Monitor GAI systems in deployment for anomalous use and security risks.
Informaon Security
MS-2.7-015
Monitor the number of security-related incident reports from users, indicang
their awareness and willingness to report issues.
Human AI Conguraon,
Informaon Security
MS-2.7-016
Perform AI red-teaming to assess resilience against: Abuse to facilitate aacks on
other systems (e.g., malicious code generaon, enhanced phishing content), GAI
aacks (e.g., prompt injecon), ML aacks (e.g., adversarial examples/prompts,
data poisoning, membership inference, model extracon, sponge examples).
Informaon Security, Toxicity,
Bias, and Homogenizaon,
Dangerous or Violent
Recommendaons
MS-2.7-017
Review deployment approval processes and verify that processes address
relevant GAI security risks.
Informaon Security
MS-2.7-018
Review incident response procedures and verify adequate funconality to
idenfy, contain, eliminate, and recover from complex GAI system incidents that
implicate impacts across the trustworthy characteriscs.
MS-2.7-019
Track and document access and updates to GAI system training data; verify
appropriate security measures for training data at GAI vendors and service
providers.
Informaon Security, Value Chain
and Component Integraon
43
MS-2.7-020
Track GAI system performance metrics such as response me and throughput
under dierent loads and usage paerns related to content provenance.
Informaon Integrity
MS-2.7-021
Track the number of users who have completed security training programs
regarding the security of content provenance.
Human AI Conguraon,
Informaon Integrity, Informaon
Security
MS-2.7-022
Verify ne-tuning does not compromise safety and security controls.
Informaon Integrity, Informaon
Security, Dangerous or Violent
Recommendaons
MS-2.7-023
Verify organizaonal policies, procedures, and processes for treatment of GAI
security and resiliency risks.
Informaon Security
MS-2.7-024
Verify vendor documentaon for data and soware security controls.
Informaon Security, Value Chain
and Component Integraon
MS-2.7-025
Work with domain experts to capture stakeholder condence in GAI system
security and perceived eecveness related to content provenance.
Informaon Integrity, Informaon
Security
AI Actors: AI Deployment, AI Impact Assessment, Domain Experts, Operaon and Monitoring, TEVV
1
MEASURE 2.8: Risks associated with transparency and accountability – as idened in the MAP funcon – are examined and
documented.
Acon ID
Acon
Risks
MS-2.8-001
Compile and communicate stascs on policy violaons, take-down requests,
intellectual property infringement, and informaon integrity for organizaonal
GAI systems: Analyze transparency reports across demographic groups,
languages groups, and other segments relevant to the deployment context.
Informaon Integrity, Intellectual
Property, Toxicity, Bias, and
Homogenizaon
MS-2.8-002
Document the instrucons given to data annotators or AI red-teamers.
MS-2.8-003
Document where in the data pipeline human labor is being used.
MS-2.8-004
Establish a mechanism for appealing usage policy violaons.
MS-2.8-005
Maintain awareness of AI regulaons and standards in relevant jurisdicons
related to GAI systems and content provenance.
Informaon Integrity
MS-2.8-006
Measure the eecveness or accessibility of procedures to appeal adverse,
harmful, or incorrect outcomes from GAI systems.
Human AI Conguraon, Toxicity,
Bias, and Homogenizaon,
Dangerous or Violent
Recommendaons
44
MS-2.8-007
Review and consider GAI system transparency arfacts such as impact
assessments, system cards, model cards, and tradional risk management
documentaon as part of organizaonal decision making.
MS-2.8-008
Review licenses, patents, or other intellectual property rights pertaining to
informaon in system training data.
Intellectual Property
MS-2.8-009
Track AI actor decisions along the lifecycle to determine sources of systemic and
cognive bias and idenfy management and migaon approaches.
Human AI Conguraon, Toxicity,
Bias, and Homogenizaon
MS-2.8-010
Use interpretable machine learning techniques to make AI processes and
outcomes more transparent, and easier to understand how decisions are made.
MS-2.8-011
Use technologies such as blockchain and digital signatures to enable the
documentaon of each instance where content is generated, modied, or shared
to provide a tamper-proof history of the content, promote transparency, and
enable traceability. Robust version control systems can also be applied to track
changes across the AI lifecycle over me.
Informaon Integrity
MS-2.8-012
Verify adequacy of GAI system user instrucons through user tesng.
Human AI Conguraon
MS-2.8-013
Verify that accurate informaon about GAI capabilies, opportunies, risks, and
potenal negave impacts are available on websites, press releases,
organizaonal reports, social media, and public communicaon channels.
MS-2.8-014
Verify the adequacy of feedback funconality in system user interfaces.
Human AI Conguraon
MS-2.8-015
Verify the adequacy of redress processes for severe GAI system impacts.
AI Actors: AI Deployment, AI Impact Assessment, Domain Experts, Operaon and Monitoring, TEVV
1
MEASURE 2.9: The AI model is explained, validated, and documented, and AI system output is interpreted within its context – as
idened in the MAP funcon – to inform responsible use and governance.
Acon ID
Acon
Risks
MS-2.9-001
Apply and document ML explanaon results such as: Analysis of embeddings,
Counterfactual prompts, Gradient-based aribuons, Model
compression/surrogate models, Occlusion/term reducon.
MS-2.9-002
Apply transparency tools such as Datasheets, Data Nutrion Labels, and Model
Cards to record explanatory and validaon informaon.
45
MS-2.9-003
Document GAI model details including: Proposed use and organizaonal value;
Assumpons and limitaons, Data collecon methodologies; Data provenance;
Data quality; Model architecture (e.g., convoluonal neural network,
transformers, etc.); Opmizaon objecves; Training algorithms; RLHF
approaches; Fine-tuning approaches; Evaluaon data; Ethical consideraons;
Legal and regulatory requirements.
Informaon Integrity, Toxicity,
Bias, and Homogenizaon
MS-2.9-004
Measure and report: Comparisons to alternave approaches and benchmarks;
Outcomes across demographic groups, languages groups, and other segments
relevant to the deployment context; Reproducibility of outcomes or internal
mechanisms; Sensivity analysis and stress-tesng results.
Toxicity, Bias, and Homogenizaon
MS-2.9-005
Verify calibraon and robustness of applied explanaon techniques and
document their assumpons and limitaons.
AI Actors: AI Deployment, AI Impact Assessment, Domain Experts, End-Users, Operaon and Monitoring, TEVV
1
*MEASURE 2.10: Privacy risk of the AI system – as idened in the MAP funcon – is examined and documented.
Acon ID
Acon
Risks
MS-2.10-001
Collaborate with other AI actors, domain experts, and legal advisors to evaluate
the impact of GAI applicaons on privacy related to the GAI system and its
content provenance, in domains such as healthcare, nance, and criminal jusce.
Data Privacy, Human AI
Conguraon, Informaon
Integrity
MS-2.10-002
Conduct AI red-teaming to assess GAI system risks such as: Outpung of training
data samples, and subsequent reverse engineering, model extracon, and
membership inference risks; Revealing biometric, condenal, copyrighted,
licensed, patented, personal, proprietary, sensive, or trade-marked; Tracking or
revealing locaon informaon of users or members of training datasets.
Human AI Conguraon,
Intellectual Property
MS-2.10-003
Document collecon, use, management, and disclosure of biometric,
condenal, copyrighted, licensed, patented, personal, proprietary, sensive, or
trade-marked informaon in datasets, in accordance with privacy and data
governance policies and data privacy laws.
Data Privacy, Human AI
Conguraon, Intellectual
Property
MS-2.10-004
Engage directly with end-users and other stakeholders to understand their
expectaons and concerns regarding content provenance. Use this feedback to
guide the design of provenance-tracking mechanisms.
Human AI Conguraon,
Informaon Integrity
MS-2.10-005
Establish and document protocols (authorizaon, duraon, type) and access
controls for training sets or producon data containing biometric, condenal,
copyrighted, licensed, patented, personal, proprietary, sensive, or trade-marked
informaon, in accordance with privacy and data governance policies and data
privacy laws.
Data Privacy, Intellectual Property
46
MS-2.10-006
Implement consent mechanisms that are demonstrated to allow users to
understand and control how their data is used in the GAI system and its content
provenance.
Data Privacy, Human AI
Conguraon, Informaon
Integrity
MS-2.10-007
Implement mechanisms to monitor, periodically review and document the
provenance data to detect any inconsistencies or unauthorized modicaons.
Informaon Integrity, Informaon
Security
MS-2.10-008
Implement zero-knowledge proofs to balance transparency with privacy and
allow vericaon of claims about content without exposing the actual data.
Data Privacy
MS-2.10-009
Leverage technologies such as blockchain to document the origin of, and any
subsequent modicaons to, generated content to enhance transparency and
provide a secure method for provenance tracking.
Informaon Integrity, Informaon
Security
MS-2.10-010
Track training, input and output items that contains personally idenable
informaon.
Data Privacy
MS-2.10-011
Verify compliance with data protecon regulaons.
Data Privacy
MS-2.10-012
Verify deduplicaon of training data samples.
Toxicity, Bias, and Homogenizaon
MS-2.10-013
Verify organizaonal policies, procedures, and processes for GAI systems address
fundamental tenets of data privacy, e.g., Anonymizaon of private data; Consent
to use data for targeted purposes or applicaons; Data collecon and use in
accordance with legal requirements and organizaonal policies; Reasonable data
retenon limits and requirements; User data deleon and reccaon requests.
Data Privacy, Human AI
Conguraon
MS-2.10-014
Verify that biometric, condenal, copyrighted, licensed, patented, personal,
proprietary, sensive, or trade-marked informaon are removed from GAI
training data.
Intellectual Property
AI Actors: AI Deployment, AI Impact Assessment, Domain Experts, End-Users, Operaon and Monitoring, TEVV
1
*MEASURE 2.11: Fairness and bias – as idened in the MAP funcon – are evaluated and results are documented.
Acon ID
Acon
Risks
MS-2.11-001
Apply use-case appropriate benchmarks (e.g., Bias Benchmark Quesons, Real
Toxicity Prompts, Winogender) to quanfy systemic bias, stereotyping,
denigraon, and toxicity in GAI system outputs; Document assumpons and
limitaons of benchmarks relave to in-context deployment environment.
Toxicity, Bias, and Homogenizaon
MS-2.11-002
Assess content moderaon and other output ltering technologies or processes
for risks arising from human, systemic, and stascal/computaonal biases.
Toxicity, Bias, and Homogenizaon
47
MS-2.11-003
Conduct fairness assessments to measure systemic bias. Measure GAI system
performance across demographic groups and subgroups, addressing both quality
of service and any allocaon of services and resources. Idenfy types of harms,
including harms in resource allocaon, representaonal, quality of service,
stereotyping, or erasure, Idenfy across, within, and intersecng groups that
might be harmed; Quanfy harms using: eld tesng with sub-group populaons
to determine likelihood of exposure to generated content exhibing harmful
bias, AI red-teaming with counterfactual and low-context (e.g., “leader,” “bad
guys”) prompts. For ML pipelines or business processes with categorical or
numeric outcomes that rely on GAI, apply general fairness metrics (e.g.,
demographic parity, equalized odds, equal opportunity, stascal hypothesis
tests), to the pipeline or business outcome where appropriate; Custom, context-
specic metrics developed in collaboraon with domain experts and aected
communies; Measurements of the prevalence of denigraon in generated
content in deployment (e.g., sub-sampling a fracon of trac and manually
annotang denigrang content); Analyze quaned harms for contextually
signicant dierences across groups, within groups, and among intersecng
groups; Rene idencaon of within-group and interseconal group disparies,
Evaluate underlying data distribuons and employ sensivity analysis during the
analysis of quaned harms, Evaluate quality metrics including dierenal
output across groups, Consider biases aecng small groups, within-group or
interseconal communies, or single individuals.
Toxicity, Bias, and
Homogenizaon, Dangerous or
Violent Recommendaons
MS-2.11-004
Evaluate pracces along the lifecycle to idenfy potenal sources of human-
cognive bias such as availability, observaonal, groupthink, funding, and
conrmaon bias, and to make implicit decision-making processes more explicit
and open to invesgaon.
Toxicity, Bias, and Homogenizaon
MS-2.11-005
Idenfy the classes of individuals, groups, or environmental ecosystems which
might be impacted by GAI systems through direct engagement with potenally
impacted communies.
Environmental, Toxicity, Bias, and
Homogenizaon
MS-2.11-006
Monitor for representaonal, nancial, or other harms aer GAI systems are
deployed.
Toxicity, Bias, and
Homogenizaon, Dangerous or
Violent Recommendaons
MS-2.11-007
Review, document, and measure sources of bias in training and TEVV data:
Dierences in distribuons of outcomes across and within groups, including
intersecng groups; Completeness, representaveness, and balance of data
sources; demographic group and subgroup coverage in GAI system training data;
Forms of latent systemic bias in images, text, audio, embeddings, or other
complex or unstructured data; Input data features that may serve as proxies for
demographic group membership (i.e., image metadata, language dialect) or
otherwise give rise to emergent bias within GAI systems; The extent to which the
digital divide may negavely impact representaveness in GAI system training
and TEVV data; Filtering of hate speech and toxicity in GAI system training data;
Prevalence of GAI-generated data in GAI system training data.
Toxicity, Bias, and
Homogenizaon, Unknowns
48
MS-2.11-008
Track and document AI actor credenals and qualicaons.
Human AI Conguraon
MS-2.11-009
Verify accessibility funconality; verify funconality and meliness of
accommodaons and opt-out funconality or processes.
Human AI Conguraon, Toxicity,
Bias, and Homogenizaon
MS-2.11-010
Verify bias management in periodic model updates; test and recalibrate with
updated and more representave data to manage bias within acceptable
tolerances.
Toxicity, Bias, and Homogenizaon
MS-2.11-011
Verify training is not homogenous GAI-produced data in order to migate
concerns of model collapse.
Toxicity, Bias, and Homogenizaon
AI Actors: AI Deployment, AI Impact Assessment, Aected Individuals and Communies, Domain Experts, End-Users, Operaon
and Monitoring, TEVV
1
MEASURE 2.12: Environmental impact and sustainability of AI model training and management acvies – as idened in the MAP
funcon – are assessed and documented.
Acon ID
Acon
Risks
MS-2.12-001
Assess safety to physical environments when deploying GAI systems.
Dangerous or Violent
Recommendaons
MS-2.12-002
Document ancipated environmental impacts of model development,
maintenance, and deployment in product design decisions.
Environmental
MS-2.12-003
Measure or esmate environmental impacts (e.g., energy and water
consumpon) for training, ne tuning, and deploying models: Verify tradeos
between resources used at inference me versus addional resources required
at training me.
Environmental
MS-2.12-004
Track and document connuous improvement processes that enhance
eecveness of risk measurement for GAI environmental impacts and
sustainability.
Environmental
MS-2.12-005
Verify eecveness of carbon capture or oset programs, and address green-
washing risks.
Environmental
AI Actors: AI Deployment, AI Impact Assessment, Domain Experts, Operaon and Monitoring, TEVV
2
49
MEASURE 2.13: Eecveness of the employed TEVV metrics and processes in the MEASURE funcon are evaluated and
documented.
Acon ID
Acon
Risks
MS-2.13-001
Create measurement error models for pre-deployment metrics to demonstrate
construct validity for each metric (i.e., does the metric eecvely operaonalize
the desired concept): Measure or esmate, and document, biases or stascal
variance in applied metrics or structured human feedback processes; Adhere to
applicable laws and regulaons when operaonalizing models in high-volume
sengs (e.g., toxicity classiers and automated content lters); Leverage domain
experse when modeling complex societal constructs such as toxicity.
Confabulaon, Informaon
Integrity, Toxicity, Bias, and
Homogenizaon
MS-2.13-002
Document measurement and structured public feedback processes applied to
organizaonal GAI systems in a centralized repository (i.e., organizaonal AI
inventory).
MS-2.13-003
Review GAI system metrics and associated pre-deployment processes to
determine their ability to sustain system improvements, including the
idencaon and removal of errors, harms, and negave impacts.
Confabulaon, Informaon
Integrity, Dangerous or Violent
Recommendaons
AI Actors: AI Deployment, Operaon and Monitoring, TEVV
1
*MEASURE 3.1: Approaches, personnel, and documentaon are in place to regularly idenfy and track exisng, unancipated, and
emergent AI risks based on factors such as intended and actual performance in deployed contexts.
Acon ID
Acon
Risks
MS-3.1-001
Assess completeness of known use cases and expected performance of inputs,
such as third-party data or upstream AI systems, or the performance of
downstream systems which use the outputs of the GAI system, directly or
indirectly, through engagement and outreach with AI Actors.
Human AI Conguraon, Value
Chain and Component
Integraon, Toxicity, Bias, and
Homogenizaon
MS-3.1-002
Compare intended use and expected performance of GAI systems across all
relevant contexts.
50
MS-3.1-003
Elicit and track feedback for previously unknown uses of the GAI systems.
AI Actors: AI Impact Assessment, Operaon and Monitoring, TEVV
1
MEASURE 3.2: Risk tracking approaches are considered for sengs where AI risks are dicult to assess using currently available
measurement techniques or where metrics are not yet available.
Acon ID
Acon
Risks
MS-3.2-001
Determine if available GAI system risk measurement approaches are applicable
to the GAI system use contexts.
MS-3.2-002
Document the rate of occurrence and severity of GAI harms to the organizaon
and to external AI actors.
Human AI Conguraon
MS-3.2-003
Establish processes for idenfying emergent GAI system risks with external AI
actors.
Human AI Conguraon,
Unknowns
MS-3.2-004
Idenfy measurement approaches for tracking GAI system risks if none exist.
AI Actors: AI Impact Assessment, Domain Experts, Operaon and Monitoring, TEVV
2
*MEASURE 3.3: Feedback processes for end users and impacted communies to report problems and appeal system outcomes are
established and integrated into AI system evaluaon metrics.
Acon ID
Acon
Risks
MS-3.3-001
Conduct impact assessments on how AI-generated content might aect dierent
social, economic, and cultural groups.
Toxicity, Bias, and Homogenizaon
MS-3.3-002
Conduct studies to understand how end users perceive and interact with GAI
content related to content provenance within context of use. Assess whether the
content aligns with their expectaons and how they may act upon the
informaon presented.
Human AI Conguraon,
Informaon Integrity
MS-3.3-003
Design evaluaon metrics that include parameters for content provenance
quality, validity, reliability, authencity or origin, and integrity of content.
Informaon Integrity
MS-3.3-004
Evaluate GAI system evaluaon metrics based on feedback from relevant AI
actors.
Human AI Conguraon
51
MS-3.3-005
Evaluate potenal biases and stereotypes that could emerge from the AI-
generated content using appropriate methodologies including computaonal
tesng methods as well as evaluang structured feedback input.
Toxicity, Bias, and Homogenizaon
MS-3.3-006
Implement connuous monitoring of AI-generated content and provenance aer
system deployment for various types of dri. Verify GAI systems are adapve and
able to iteravely improve models and algorithms over me.
Informaon Integrity
MS-3.3-007
Integrate human evaluators to assess content quality and relevance.
Human AI Conguraon
MS-3.3-008
Provide input for training materials about the capabilies and limitaons of GAI
systems related to content provenance for AI actors, other professionals, and the
public about the societal impacts of AI and the role of diverse and inclusive
content generaon.
Human AI Conguraon,
Informaon Integrity, Toxicity,
Bias, and Homogenizaon
MS-3.3-009
Record and integrate structured feedback about content provenance from
operators, users, and potenally impacted communies through the use of
methods such as user research studies, focus groups, or community forums.
Acvely seek feedback on generated content quality and potenal biases. Assess
the general awareness among end users and impacted communies about the
availability of these feedback channels.
Human AI Conguraon,
Informaon Integrity, Toxicity,
Bias, and Homogenizaon
MS-3.3-010
Regularly review structured human feedback and GAI system sensors and update
based on the evolving needs and concerns of the impacted communies.
MS-3.3-011
Ulize independent evaluaons to assess content quality and types of potenal
biases and related negave impacts.
Toxicity, Bias, and Homogenizaon
MS-3.3-012
Verify AI actors engaged in GAI TEVV tasks for content provenance reect diverse
demographic and interdisciplinary backgrounds.
Human AI Conguraon,
Informaon Integrity, Toxicity,
Bias, and Homogenizaon
AI Actors: AI Deployment, Aected Individuals and Communies, End-Users, Operaon and Monitoring, TEVV
1
MEASURE 4.2: Measurement results regarding AI system trustworthiness in deployment context(s) and across the AI lifecycle are
informed by input from domain experts and relevant AI actors to validate whether the system is performing consistently as
intended. Results are documented.
Acon ID
Acon
Risks
MS-4.2-001
Conduct adversarial tesng to assess the GAI system’s response to inputs
intended to deceive or manipulate its content provenance and understand
potenal misuse scenarios and unintended outputs.
Informaon Integrity, Informaon
Security
52
MS-4.2-002
Ensure both posive and negave feedback on GAI system funconality is
assessed.
MS-4.2-003
Ensure visible mechanisms to collect users’ feedback are in place, including
systems to report harmful and low quality content.
Human AI Conguraon,
Dangerous or Violent
Recommendaons
MS-4.2-004
Evaluate GAI system content provenance in real-world scenarios to observe its
behavior in praccal environments and reveal issues that might not surface in
controlled and opmized tesng environments.
Informaon Integrity
MS-4.2-005
Evaluate GAI system performance related to content provenance against
predened metrics and update the evaluaon criteria as necessary to adapt to
changing contexts and requirements.
Informaon Integrity
MS-4.2-006
Implement interpretability and explainability methods to evaluate GAI system
decisions related to content provenance and verify alignment with intended
purpose.
Informaon Integrity, Toxicity,
Bias, and Homogenizaon
MS-4.2-007
Integrate structured human feedback results into calibraon and update
processes for tradional measurement approaches (e.g., benchmarks,
performance assessments, data quality measurements).
MS-4.2-008
Measure GAI system inputs and outputs to account for content provenance, data
provenance, source reliability, contextual relevance and coherence, and security
implicaons.
Informaon Integrity, Informaon
Security
MS-4.2-009
Monitor and document instances where human operators or other systems
override the GAI's decisions. Evaluate these cases to understand if the overrides
are linked to issues related to content provenance.
Informaon Integrity
MS-4.2-010
Verify and document the incorporaon of structured human feedback results
into design, implementaon, deployment approval (“go”/“no-go” decisions),
monitoring, and decommission decisions.
MS-4.2-011
Verify that GAI system development and deployment related to content
provenance integrates trustworthiness characteriscs.
Informaon Integrity
MS-4.2-012
Verify the performance of user feedback and recourse mechanisms, including
analyses across various sub-groups.
Human AI Conguraon, Toxicity,
Bias, and Homogenizaon
MS-4.2-013
Work with domain experts to integrate insights from stakeholder feedback
analysis into TEVV metrics and associated acons, and connuous improvement
processes.
MS-4.2-014
Work with domain experts to review feedback from end users, operators, and
potenally impacted individuals and communies—enumerated in the Map
funcon.
Human AI Conguraon
53
MS-4.2-015
Work with domain experts who understand the GAI system context of use to
evaluate the content’s validity, relevance, and potenal biases.
Toxicity, Bias, and Homogenizaon
AI Actors: AI Deployment, Domain Experts, End-Users, Operaon and Monitoring, TEVV
1
*MANAGE 1.3: Responses to the AI risks deemed high priority, as idened by the MAP funcon, are developed, planned, and
documented. Risk response opons can include migang, transferring, avoiding, or accepng.
Acon ID
Acon
Risks
MG-1.3-001
Allocate resources and me for GAI risk management acvies, including
planning for incident response and other migaon acvies.
MG-1.3-002
Document residual GAI system risks that persist aer risk migaon or transfer.
MG-1.3-003
Document trade-os, decision processes, and relevant measurement and
feedback results for risks that do not surpass organizaonal risk tolerance.
MG-1.3-004
Migate, transfer, or avoid risks that surpass organizaonal risk tolerances.
MG-1.3-005
Monitor the eecveness of risk controls (e.g., via eld tesng, parcipatory
engagements, performance assessments, user feedback mechanisms).
Human AI Conguraon
AI Actors: AI Deployment, AI Impact Assessment, Operaon and Monitoring
2
MANAGE 2.2: Mechanisms are in place and applied to sustain the value of deployed AI systems.
Acon ID
Acon
Risks
MG-2.2-001
Compare GAI system outputs against pre-dened organizaon risk tolerance,
guidelines, and principles, and review and audit AI-generated content against
these guidelines.
MG-2.2-002
Document training data sources to trace the origin and provenance of AI-
generated content.
Informaon Integrity
MG-2.2-003
Evaluate feedback loops between GAI system content provenance and human
reviewers, and update make updates where needed. Implement real-me
monitoring systems to detect GAI systems and content provenance dri as it
happens.
Informaon Integrity
54
MG-2.2-004
Evaluate GAI content and data for representaonal biases and employ
techniques such as re-sampling, re-ranking, or adversarial training to migate
biases in the generated content.
Informaon Security, Toxicity,
Bias, and Homogenizaon
MG-2.2-005
Filter GAI output for harmful or biased content, potenal misinformaon, and
CBRN-related or NCII content.
CBRN Informaon, Obscene,
Degrading, and/or Abusive
Content, Toxicity, Bias, and
Homogenizaon, Dangerous or
Violent Recommendaons
MG-2.2-006
Implement version control for models and datasets to track changes and
facilitate rollback if necessary.
MG-2.2-007
Incorporate feedback from users, external experts, and the public to adapt the
GAI system and monitoring processes.
Human AI Conguraon
MG-2.2-008
Incorporate human review processes to assess and lter content in accordance
with the socio-cultural knowledge and values of the context of use and to
idenfy limitaons and nuances that automated processes might miss; verify
that human reviewers are trained on content guidelines and potenal biases of
GAI system and its content provenance.
Informaon Integrity, Toxicity,
Bias, and Homogenizaon
MG-2.2-009
Integrate informaon from data management and machine learning security
countermeasures like red teaming, and dierenal privacy, and authencaon
protocols to ensure data and models are protected from potenal risks.
CBRN Informaon, Data Privacy,
Informaon Security
MG-2.2-010
Use feedback from internal and external AI actors, users, individuals, and
communies, to assess impact of AI-generated content.
Human AI Conguraon
MG-2.2-011
Use real-me auding tools such as distributed ledger technology to track and
validate the lineage and authencity of AI-generated data.
Informaon Integrity
MG-2.2-012
Use structured feedback mechanisms to solicit and capture user input about AI-
generated content to detect subtle shis in quality or alignment with community
and societal values.
Human AI Conguraon, Toxicity,
Bias, and Homogenizaon
AI Actors: AI Deployment, AI Impact Assessment, Governance and Oversight, Operaon and Monitoring
1
55
*MANAGE 2.3: Procedures are followed to respond to and recover from a previously unknown risk when it is idened.
Acon ID
Acon
Risks
MG-2.3-001
Develop and update GAI system incident response and recovery plans and
procedures to address the following: Review and maintenance of policies and
procedures to account for newly encountered uses; Review and maintenance of
policies and procedures for detecon of unancipated uses; Verify response
and recovery plans account for the GAI system supply chain; Verify response
and recovery plans are updated for and include necessary details to
communicate with downstream GAI system Actors: Points-of-Contact (POC),
Contact informaon, nocaon format.
Value Chain and Component
Integraon
MG-2.3-002
Maintain protocols to log changes made to GAI systems during incident
response and recovery.
MG-2.3-003
Review, update and maintain incident response and recovery plans to integrate
insights from GAI system use cases and contexts and needs of relevant AI
actors.
Human AI Conguraon
MG-2.3-004
Verify and maintain measurements that GAI systems are operang within
organizaonal risk tolerances post incident.
AI Actors: AI Deployment, Operaon and Monitoring
1
*MANAGE 2.4: Mechanisms are in place and applied, and responsibilies are assigned and understood, to supersede, disengage, or
deacvate AI systems that demonstrate performance or outcomes inconsistent with intended use.
Acon ID
Acon
Risks
MG-2.4-001
Enforce change management processes, and risk and impact assessments
across all intended uses and contexts before deploying GAI system updates.
MG-2.4-002
Establish and maintain communicaon plans to inform AI stakeholders as part
of the deacvaon or disengagement process of a specic GAI system or
context of use, including reasons, workarounds, user access removal, alternave
processes, contact informaon, etc.
Human AI Conguraon
MG-2.4-003
Establish and maintain procedures for escalang GAI system incidents to the
organizaonal risk authority when specic criteria for deacvaon or
disengagement is met for a parcular context of use or for the GAI system as a
whole.
56
MG-2.4-004
Establish and maintain procedures for the remediaon of issues which trigger
incident response processes for the use of a GAI system, and provide
stakeholders melines associated with the remediaon plan.
MG-2.4-005
Establish and regularly review specic criteria that warrants the deacvaon of
GAI systems in accordance with set risk tolerances and appetes.
AI Actors: AI Deployment, Governance and Oversight, Operaon and Monitoring
1
*MANAGE 3.1: AI risks and benets from third-party resources are regularly monitored, and risk controls are applied and
documented.
Acon ID
Acon
Risks
MG-3.1-001
Apply organizaonal risk tolerances and controls (e.g., acquision and
procurement processes; assessing personnel credenals and qualicaons,
performing background checks; ltering GAI input and outputs, grounding,
ne tuning) to third-party GAI resources: Apply organizaonal risk tolerance
to the ulizaon of third-party datasets and other GAI resources; Apply
organizaonal risk tolerances to ne-tuned third-party models; Apply
organizaonal risk tolerance to exisng third-party models adapted to a new
domain; Reassess risk measurements aer ne-tuning third-party GAI
models.
Value Chain and Component
Integraon
MG-3.1-002
Audit GAI system supply chain risks (e.g., data poisoning, malware, other
soware and hardware vulnerabilies; labor pracces; data privacy and
localizaon compliance; geopolical alignment).
Data Privacy, Informaon Security,
Value Chain and Component
Integraon, Toxicity, Bias, and
Homogenizaon
MG-3.1-003
Decommission third-party systems that exceed organizaonal risk tolerances.
Value Chain and Component
Integraon
MG-3.1-004
Idenfy and maintain documentaon for third-party AI systems, and
components, in organizaonal AI inventories.
Value Chain and Component
Integraon
MG-3.1-005
Iniate review of third-party organizaons/developers prior to their use of
GAI models, and during their use of GAI models for their own applicaons, to
monitor for abuse and policy violaons.
Value Chain and Component
Integraon, Toxicity, Bias, and
Homogenizaon, Dangerous or
Violent Recommendaons
MG-3.1-006
Re-assess model risks aer ne-tuning and for any third-party GAI models
deployed for applicaons and/or use cases that were not evaluated in inial
tesng.
Value Chain and Component
Integraon
57
MG-3.1-007
Review GAI training data for CBRN informaon and intellectual property; scan
output for plagiarized, trademarked, patented, licensed, or trade secret
material.
Intellectual Property, CBRN
Informaon
MG-3.1-008
Update acquision and procurement policies, procedures, and processes to
address GAI risks and failure modes.
MG-3.1-009
Use, review, update, and share various transparency arfacts (e.g., system
cards and model cards) for third-party models. Document or retain
documentaon for: Training data content and provenance, methodology,
tesng, validaon, and clear instrucons for use from GAI vendors and
suppliers, Informaon related to third-party informaon security policies,
procedures, and processes.
Informaon Integrity, Informaon
Security, Value Chain and
Component Integraon
AI Actors: AI Deployment, Operaon and Monitoring, Third-party enes
1
MANAGE 3.2: Pre-trained models which are used for development are monitored as part of AI system regular monitoring and
maintenance.
Acon ID
Acon
Risks
MG-3.2-001
Apply explainable AI (XAI) techniques (e.g., analysis of embeddings, model
compression/disllaon, gradient-based aribuons, occlusion/term reducon,
counterfactual prompts, word clouds) as part of ongoing connuous
improvement processes to migate risks related to unexplainable GAI systems.
MG-3.2-002
Document how pre-trained models have been adapted (ne-tuned) for the
specic generave task, including any data augmentaons, parameter
adjustments, or other modicaons. Access to un-tuned (baseline) models must
be available to support debugging the relave inuence of the pre-trained
weights compared to the ne-tuned model weights.
MG-3.2-003
Document sources and types of training data and their origins, potenal biases
present in the data related to the GAI applicaon and its content provenance,
architecture, training process of the pre-trained model including informaon on
hyperparameters, training duraon, and any ne-tuning processes applied.
Informaon Integrity, Toxicity, Bias,
and Homogenizaon
MG-3.2-004
Evaluate user reported problemac content and integrate feedback into system
updates.
Human AI Conguraon,
Dangerous or Violent
Recommendaons
58
MG-3.2-005
Implement content lters to prevent the generaon of inappropriate, harmful,
toxic, false, illegal, or violent content related to the GAI applicaon, including
for CSAM and NCII. These lters can be rule-based or leverage addional
machine learning models to ag problemac inputs and outputs.
Informaon Integrity, Toxicity, Bias,
and Homogenizaon, Dangerous or
Violent Recommendaons,
Obscene, Degrading, and/or
Abusive Content
MG-3.2-006
Implement real-me monitoring processes for analyzing generated content
performance and trustworthiness characteriscs related to content provenance
to idenfy deviaons from the desired standards and trigger alerts for human
intervenon.
Informaon Integrity
MG-3.2-007
Leverage feedback and recommendaons from organizaonal boards or
commiees related to the deployment of GAI applicaons and content
provenance when using third-party pre-trained models.
Informaon Integrity, Value Chain
and Component Integraon
MG-3.2-008
Maintain awareness of relevant laws and regulaons related to content
generaon, data privacy, and user protecons and work in conjuncon with
legal experts to review and assess the potenal liabilies associated with AI-
generated content.
Data Privacy, Intellectual Property
Informaon Integrity
MG-3.2-009
Provide use case examples as material for training employees and stakeholders
about the trustworthiness implicaons of GAI applicaons and content
provenance and to raise awareness about potenal risks in fostering a risk
management culture.
Informaon Integrity
MG-3.2-010
Use human moderaon systems to review generated content in accordance
with human-AI conguraon policies established in the Govern funcon,
aligned with socio-cultural norms in the context of use, and for sengs where
AI models are demonstrated to perform poorly.
Human AI Conguraon
MG-3.2-011
Use organizaonal risk tolerance to evaluate acceptable risks and performance
metrics and decommission or retrain pre-trained models that perform outside
of dened limits.
CBRN Informaon, Confabulaon
AI Actors: AI Deployment, Operaon and Monitoring, Third-party enes
1
59
*MANAGE 4.1: Post-deployment AI system monitoring plans are implemented, including mechanisms for capturing and evaluang
input from users and other relevant AI actors, appeal and override, decommissioning, incident response, recovery, and change
management.
Acon ID
Acon
Risks
MG-4.1-001
Collaborate with external researchers, industry experts, and community
representaves to maintain awareness of emerging best pracces and
technologies in content provenance.
Informaon Integrity, Toxicity, Bias,
and Homogenizaon
MG-4.1-002
Conduct adversarial tesng at a regular cadence; test against various
adversarial inputs and scenarios; idenfy vulnerabilies and assess the AI
system’s resilience to content provenance aacks.
Informaon Integrity, Informaon
Security
MG-4.1-003
Conduct red-teaming exercises to surface failure modes of content provenance
mechanisms. Evaluate the eecveness of red-teaming approaches for
uncovering potenal vulnerabilies and improving overall content provenance.
Informaon Integrity, Informaon
Security
MG-4.1-004
Employ user-friendly channels such as feedback forms, e-mails, or hotlines for
users to report issues, concerns, or unexpected GAI outputs to feed into
monitoring pracces.
Human AI Conguraon
MG-4.1-005
Establish, maintain, and evaluate eecveness of organizaonal processes and
procedures to monitor GAI systems within context of use.
MG-4.1-006
Evaluate the use of senment analysis to gauge user senment regarding GAI
content performance and impact, and work in collaboraon with AI actors
experienced in user research and experience.
Human AI Conguraon
MG-4.1-007
Implement acve learning techniques to idenfy instances where the model
fails or produces unexpected outputs.
Confabulaon
MG-4.1-008
Integrate digital watermarks, blockchain technology, cryptographic hash
funcons, metadata embedding, or other content provenance techniques
within AI-generated content to track its source and manipulaon history.
Informaon Integrity
MG-4.1-009
Measure system outputs related to content provenance at a regular cadence
and integrate insights into monitoring processes.
Informaon Integrity
MG-4.1-010
Monitor GAI training data for representaon of dierent user groups.
Human AI Conguraon, Toxicity,
Bias, and Homogenizaon
MG-4.1-011
Perform periodic review of organizaonal adherence to GAI system monitoring
plans across all contexts of use.
60
MG-4.1-012
Share transparency reports with internal and external stakeholders that detail
steps taken to update the AI system to enhance transparency and
accountability.
MG-4.1-013
Track dataset modicaons for content provenance by monitoring data
deleons, reccaon requests, and other changes that may impact the
veriability of content origins.
Informaon Integrity
MG-4.1-014
Verify risks associated with gaps in GAI system monitoring plans are accepted at
the appropriate organizaonal level.
MG-4.1-015
Verify that AI actors responsible for monitoring reported issues can eecvely
evaluate GAI system performance and its content provenance, and promptly
escalate issues for response.
Human AI Conguraon,
Informaon Integrity
AI Actors: AI Deployment, Aected Individuals and Communies, Domain Experts, End-Users, Human Factors, Operaon and
Monitoring
1
MANAGE 4.2: Measurable acvies for connual improvements are integrated into AI system updates and include regular
engagement with interested pares, including relevant AI actors.
Acon ID
Acon
Risks
MG-4.2-001
Adopt agile development methodologies, and iterave development and
feedback loops to allow for rapid adjustments based on external input related to
content provenance.
Informaon Integrity
MG-4.2-002
Conduct regular audits of GAI systems and publish reports detailing the
performance, feedback received, and improvements made.
MG-4.2-003
Employ explainable AI methods to enhance transparency and interpretability of
GAI content provenance to help AI actors and stakeholders understand how and
why specic content is generated.
Human AI Conguraon,
Informaon Integrity
MG-4.2-004
Employ stakeholder feedback captured in the Map funcon to understand user
experiences and percepons about AI-generated content and its provenance;
include user interacons and feedback from real-world scenarios.
Human AI Conguraon,
Informaon Integrity
MG-4.2-005
Form cross-funconal teams leveraging experse from across the AI lifecycle
including AI designers and developers, socio-technical experts, and experts in
the context of use and idenfy mechanisms to include end users in
consultaons.
Human AI Conguraon
61
MG-4.2-006
Pracce and follow incident response plans for addressing the generaon of
inappropriate or harmful content and adapt processes based on ndings to
prevent future occurrences. Conduct post-mortem analyses of incidents with
relevant AI actors, to understand the root causes and implement prevenve
measures.
Human AI Conguraon,
Dangerous or Violent
Recommendaons
MG-4.2-007
Provide external stakeholders with regular updates about the progress,
challenges, and improvements made based on their feedback through the use of
public venues such as online plaorms and communies, and open-source
iniaves.
Intellectual Property
MG-4.2-008
Simulate various scenarios to test GAI system responses and verify intended
performance across dierent situaons.
MG-4.2-009
Use visualizaons to represent the GAI model behavior to ease non-technical
stakeholders understanding of GAI system funconality.
Human-AI Conguraon
AI Actors: AI Deployment, AI Design, AI Development, Aected Individuals and Communies, End-Users, Operaon and
Monitoring, TEVV
1
*MANAGE 4.3: Incidents and errors are communicated to relevant AI actors, including aected communies. Processes for tracking,
responding to, and recovering from incidents and errors are followed and documented.
Acon ID
Acon
Risks
MG-4.3-001
Conduct aer-acon assessments for GAI system incidents to verify incident
response and recovery processes are followed and eecve.
MG-4.3-002
Establish and maintain change management records and procedures for GAI
systems, including the reasons for each change, how the change could impact
each intended context of use, and step-by-step details of how changes were
planned, tested, and deployed.
MG-4.3-003
Establish and maintain policies and procedures to record and track GAI system
reported errors, near-misses, incidents, and negave impacts.
Confabulaon, Informaon
Integrity
MG-4.3-004
Establish processes and procedures for regular sharing of informaon about
errors, incidents, and negave impacts for each and across contexts, sectors,
and AI actors, including the date reported, the context of use, the number of
reports for each issue, and assessments of impact and severity.
Confabulaon, Human AI
Conguraon, Informaon
Integrity
AI Actors: AI Deployment, Aected Individuals and Communies, Domain Experts, End-Users, Human Factors, Operaon and
Monitoring
2
62
Appendix A. Primary GAI Consideraons
1
The following primary consideraons were derived as overarching themes from the GAI PWG
2
consultaon process. These consideraons (Governance, Pre-Deployment Tesng, Content Provenance,
3
and Incident Disclosure) are relevant to any organizaon designing, developing, and using GAI and also
4
inform the Acons to Manage GAI risks. Informaon included about the primary consideraons is not
5
exhausve, but highlights the most relevant topics derived from the GAI PWG.
6
Acknowledgments: These consideraons could not have been surfaced without the helpful analysis and
7
contribuons from the community and NIST sta GAI PWG leads: George Awad, Luca Belli, Mat Heyman,
8
Yooyoung Lee, Reva Schwartz, and Kyra Yee.
9
Governance
10
A.1.1. Overview
11
Like any other technology system, governance principles and techniques can be used to manage risks
12
related to generave AI models, capabilies, and applicaons. Organizaons may choose to apply their
13
exisng risk ering to GAI systems, or they may opt to revise or update AI system risk levels to address
14
these unique GAI risks. This secon describes how organizaonal governance regimes may be re-
15
evaluated and adjusted for GAI contexts. It also addresses third-party consideraons for governing across
16
the AI value chain.
17
A.1.2. Organizaonal Governance
18
GAI opportunies, risks and long-term performance characteriscs are typically less well-understood
19
than non-generave AI tools. and may be perceived and acted upon by humans in ways that vary greatly.
20
Accordingly, GAI may call for dierent levels of oversight from AI actors or dierent human-AI
21
conguraons in order to manage their risks eecvely. Organizaons’ use of GAI systems may also
22
warrant addional human review, tracking and documentaon, and greater management oversight.
23
AI technology can produce varied outputs in mulple modalies and present many classes of user
24
interfaces. This leads to a broader set of AI actors interacng with GAI systems for widely diering
25
applicaons and contexts of use. These can include data labeling and preparaon, development of GAI
26
models, content moderaon, code generaon and review, text generaon and eding, image and video
27
generaon, summarizaon, search, and chat. These acvies can take place within organizaonal
28
sengs or in the public domain.
29
Organizaons can restrict AI applicaons that cause harm, exceed stated risk tolerances, or that conict
30
with their tolerances or values. Governance tools and protocols that are applied to other types of AI
31
systems can be applied to GAI systems. These plans and acons include:
32
• Accessibility and reasonable
33
accommodaons
34
• AI actor credenals and qualicaons
35
• Alignment to organizaonal values
36
• Auding and assessment
37
• Change-management controls
38
• Commercial use
39
• Data provenance
40
63
• Data protecon
41
• Data retenon
42
• Consistency in use of dening key terms
43
• Decommissioning
44
• Discouraging anonymous use
45
• Educaon
46
• Impact assessments
47
• Incident response
48
• Monitoring
49
• Opt-outs
50
• Risk-based controls
51
• Risk mapping and measurement
52
• Science-backed TEVV pracces
53
• Secure soware development pracces
54
• Stakeholder engagement
55
• Synthec content detecon and
56
labeling tools and techniques
57
• Whistleblower protecons
58
• Workforce diversity and
59
interdisciplinary teams
60
61
Establishing acceptable use policies and guidance for the use of GAI in formal human-AI teaming sengs
62
as well as dierent levels of human-AI conguraons can help to decrease risks arising from misuse,
63
abuse, inappropriate repurpose, and misalignment between systems and users. These pracces are just
64
one example of adapng exisng governance protocols for GAI contexts.
65
A.1.3. Third-Party Consideraons
66
Organizaons may seek to acquire, embed, incorporate, or use open source or proprietary third-party
67
GAI models, systems, or generated data for various applicaons across an enterprise. Use of these GAI
68
tools and inputs has implicaons for all funcons of the organizaon – including but not limited to
69
acquision, human resources, legal, compliance, and IT services – regardless of whether they are carried
70
out by employees or third pares. Many of the acons cited above are relevant and opons for
71
addressing third-party consideraons.
72
Third party GAI integraons may give rise to increased intellectual property, data privacy, or informaon
73
security risks, poinng to the need for clear guidelines for transparency and risk management regarding
74
the collecon and use of third-party data for model inputs. Organizaons may consider varying risk
75
controls for foundaon models, ne-tuned models, and embedded tools, enhanced processes for
76
interacng with external GAI technologies or service providers. Organizaons can apply standard or
77
exisng risk controls and processes to proprietary or open-source GAI technologies, data, and third-party
78
service providers, including acquision and procurement due diligence, requests for soware bills of
79
materials (SBOMs), applicaon of service level agreements (SLAs), and statement on standards for
80
aestaon engagement (SSAE) reports to help with third-party transparency and risk management for
81
GAI systems.
82
A.1.4. Pre-Deployment Tesng
83
Appendix B. Overview
84
The diverse ways and contexts in which GAI systems may be developed, used, and repurposed
85
complicates risk mapping and pre-deployment measurement eorts. Robust test, evaluaon, validaon,
86
and vericaon (TEVV) processes can be iteravely applied – and documented – in early stages of the AI
87
64
lifecycle and informed by representave AI actors (see Figure 3 of the AI RMF). Unl new and rigorous
88
early lifecycle TEVV approaches are developed and matured for GAI, organizaons may use
89
recommended “pre-deployment tesng” pracces to measure performance, capabilies, limits, risks,
90
and impacts. This secon describes risk measurement and esmaon as part of pre-deployment TEVV,
91
and examines the state of play for pre-deployment tesng methodologies.
92
Appendix C. Limitaons of Current Pre-deployment Test Approaches
93
Currently available pre-deployment TEVV processes used for GAI applicaons may be inadequate, non-
94
systemacally applied, or fail to reect or mismatched to deployment contexts. For example, the
95
anecdotal tesng of GAI system capabilies through video games or standardized tests designed for
96
humans (e.g., intelligence tests, professional licensing exams) does not guarantee GAI system validity or
97
reliability in those domains. Similarly, jailbreaking or prompt-engineering tests may not systemacally
98
assess validity or reliability risks.
99
Measurement gaps can arise from mismatches between laboratory and real-world sengs. Current
100
tesng approaches oen remain focused on laboratory condions or restricted to benchmark test
101
datasets and in silico techniques that may not extrapolate well to—or directly assess GAI impacts in –
102
real world condions. For example, current measurement gaps for GAI make it dicult to precisely
103
esmate its potenal ecosystem-level or longitudinal risks and related polical, social, and economic
104
impacts. Gaps between benchmarks and real-world use of GAI systems may likely be exacerbated due to
105
prompt sensivity and broad heterogeneity of contexts of use.
106
A.1.5. Structured Public Feedback
107
Structured public feedback can be used to evaluate whether GAI systems are performing as intended
108
and to calibrate and verify tradional measurement methods. Examples of structured feedback include,
109
but are not limited to:
110
• Parcipatory Engagement Methods: Methods used to solicit feedback from civil society groups,
111
aected communies, and users, including focus groups, small user studies, and surveys.
112
• Field Tesng: Methods used to determine how people interact with, consume, use, and make
113
sense of AI-generated informaon, and subsequent acons and eects, including UX, usability,
114
and other structured, randomized experiments.
115
• AI Red-teaming: A structured tesng exercise used to probe an AI system to nd aws and
116
vulnerabilies such as inaccurate, harmful, or discriminatory outputs, oen in a controlled
117
environment and in collaboraon with system developers.
118
Informaon gathered from structured public feedback can inform design, implementaon, deployment
119
approval, maintenance, or decommissioning decisions. Results and insights gleaned from these exercises
120
can serve mulple purposes, including improving data quality and preprocessing, bolstering governance
121
decision making, and enhancing system documentaon and debugging pracces. When implemenng
122
feedback acvies, organizaons should follow human subjects research requirements and best
123
pracces such as informed consent and subject compensaon.
124
65
C.1.1.1. Parcipatory Engagement Methods
125
On an ad hoc or more structured basis, organizaons can design and use a variety of channels to engage
126
external stakeholders in product development or review. Focus groups with select experts can provide
127
feedback on a range of issues. Small user studies can provide feedback from representave groups or
128
populaons. Anonymous surveys can be used to poll or gauge reacons to specic features. Parcipatory
129
engagement methods are oen less structured than eld tesng or red teaming, and are more
130
commonly used in early stages of AI or product development.
131
Appendix D. Field Tesng
132
Field tesng involves structured sengs to evaluate risks and impacts and to simulate the condions
133
under which the GAI system will be deployed. Field style tests can be adapted from a focus on user
134
preferences and experiences towards AI risks and impacts – both negave and posive. When carried
135
out with large groups of users, these tests can provide esmaons of the likelihood of risks and impacts
136
in real world interacons.
137
Organizaons may also collect feedback on outcomes, harms, and user experience directly from users in
138
the producon environment aer a model has been released, in accordance with human subject
139
standards such as informed consent and compensaon. Organizaons should follow applicable human
140
subjects research requirements, and best pracces such as informed consent and subject compensaon,
141
when implemenng feedback acvies.
142
Appendix E. AI Red-teaming
143
AI red-teaming exercises are oen conducted in a controlled environment and in collaboraon with AI
144
developers building AI models. AI red-teaming can be performed before or aer AI models or systems
145
are made available to the broader public; this secon focuses on red-teaming in pre-deployment
146
contexts.
147
The quality of AI red-teaming outputs is related to the background and experse of the AI red-team
148
itself. Demographically and interdisciplinarily diverse AI red-teams can be used to idenfy aws in the
149
varying contexts where GAI will be used. For best results, AI red-teams should demonstrate domain
150
experse, and awareness of socio-cultural aspects within the deployment context. AI red-teaming results
151
should be given addional analysis before they are incorporated into organizaonal governance and
152
decision making, policy and procedural updates, and AI risk management eorts.
153
Various types of AI red-teaming may be appropriate, depending on the use case:
154
• General Public: Performed by general users (not necessarily AI or technical experts) who are
155
expected to use the model or interact with its outputs, and who bring their own lived
156
experiences and perspecves to the task of AI red-teaming. These individuals may have been
157
provided instrucons and material to complete tasks which may elicit harmful model behaviors.
158
This type of exercise can be more eecve with large groups of AI-teamers.
159
• Expert: Performed by specialists with experse in the domain or specic AI red-teaming context
160
of use (e.g., medicine, biotech, cybersecurity).
161
66
• Combinaon: In scenarios when it is dicult to idenfy and recruit specialists with sucient
162
domain and contextual experse, AI red-teaming exercises may leverage both expert and
163
general public parcipants. For example, expert AI red-teamers could modify or verify the
164
prompts wrien by general public AI red-teamers. These approaches may also expand coverage
165
of the AI risk aack surface.
166
• Human / AI: Performed by GAI in combinaon with specialist or non-specialist human teams.
167
GAI-led red-teaming can be more cost eecve than human red teamers alone. Human or GAI-
168
led AI red-teaming may be beer suited for elicing dierent types of harms.
169
A.1.6. Content Provenance
170
Appendix F. Overview
171
GAI technologies can be leveraged for many applicaons such as content generaon and synthec data.
172
Some aspects of GAI output, such as the producon of deepfake content, can challenge our ability to
173
disnguish human-generated content from AI-generated content. To help manage and migate these
174
risks, digital transparency mechanisms like provenance data tracking can trace the origin and history of
175
content. Provenance data tracking and synthec content detecon can help provide greater informaon
176
about both authenc and synthec content to users, enabling trustworthiness in AI systems. When
177
combined with other organizaonal accountability mechanisms, digital content transparency can enable
178
processes to trace negave outcomes back to their source, improve informaon integrity, and uphold
179
public trust. Provenance data tracking and synthec content detecon mechanisms provide informaon
180
about the origin of content and its history to assist in GAI risk management eorts.
181
Provenance data can include informaon about generated content’s creators, date/me of creaon,
182
locaon, modicaons, and sources, including metadata informaon. Metadata can be tracked for text,
183
images, videos, audio, and underlying datasets. Provenance data tracking employs various methods and
184
metrics to assess the authencity, integrity, credibility, intellectual property rights, and potenal
185
manipulaons in GAI output. Some well-known techniques for provenance data tracking include
186
watermarking, metadata tracking, digital ngerprinng, and human authencaon, among others.
187
Appendix G. Provenance Data Tracking Approaches
188
Provenance data tracking techniques for GAI systems can be used to track the lineage and integrity of
189
data inputs, metadata, and AI-generated content. Provenance data tracking records the origin and
190
history for digital content, allowing its authencity to be determined. It consists of techniques to record
191
metadata as well as percepble and impercepble digital watermarks on digital content. Data
192
provenance refers to tracking the origin and history of input data through metadata and digital
193
watermarking techniques. Provenance data tracking processes can include and assist AI actors across the
194
lifecycle who may not have full visibility or control over the various trade-os and cascading impacts of
195
early-stage model decisions on downstream performance and synthec outputs. For example, by
196
selecng a given model to priorize computaonal eciency over accuracy, an AI actor may
197
inadvertently aect provenance tracking reliability. Organizaonal risk management eorts for
198
enhancing content provenance include:
199
67
• Tracking provenance of training data and metadata for GAI systems;
200
• Documenng provenance data limitaons within GAI systems;
201
• Monitoring system capabilies and limitaons in deployment through rigorous TEVV processes;
202
• Evaluang how humans engage, interact with, or adapt to GAI content (especially in decision
203
making tasks informed by GAI content), and how they react to applied provenance techniques
204
such as percepble disclosures.
205
Organizaons can document and delineate GAI system objecves and limitaons to idenfy gaps where
206
provenance data may be most useful. For instance, GAI systems used for content creaon may require
207
watermarking techniques to idenfy the source of content or metadata management to trace content
208
origins and modicaons. Further narrowing of GAI task denions to include provenance data can
209
enable organizaons to maximize the ulity of provenance data and risk management eorts.
210
A.1.7. Enhancing Content Provenance through Structured Public Feedback
211
While indirect feedback methods such as automated error collecon systems are useful, they oen lack
212
the context and depth that direct input from end users can provide. Organizaons can leverage feedback
213
approaches described in the Pre-Deployment Tesng secon to capture input from external sources such
214
as through AI red-teaming.
215
Integrang pre- and post-deployment external feedback into the monitoring process of applicaons
216
involving AI-generated content can help enhance awareness of performance changes and migate
217
potenal risks and harms. There are many ways to capture and make use of user feedback – before and
218
aer GAI systems are deployed – to gain insights about authencaon ecacy and vulnerabilies,
219
impacts of adversarial threats, unintended consequences resulng from the ulizaon of content
220
provenance approaches, and other unancipated behavior associated with content manipulaon.
221
Organizaons can track and document the provenance of datasets to idenfy instances in which AI-
222
generated data is a potenal root cause of performance issues with the GAI system.
223
A.1.8. Incident Disclosure
224
Appendix H. Overview
225
AI incidents can be dened as an event, circumstance, or series of events in which the development, use,
226
or malfuncon of one or more AI systems directly or indirectly contributes to idened harms. These
227
harms include injury or damage to the health of an individual or group of people; disrupon of the
228
management and operaon of crical infrastructure; violaons of human rights or a breach of
229
obligaons under applicable law intended to protect legal and labor rights; or damage to property,
230
communies, or the environment. AI incidents can occur in the aggregate (i.e., for systemic
231
discriminaon) or acutely (i.e., for one individual).
232
68
Appendix I. State of AI Incident Tracking and Disclosure
233
Formal channels do not currently exist to report and document AI incidents. However, a number of
234
publicly-available databases have been created to document their occurrence. These reporng channels
235
make decisions on an ad hoc basis about what kinds of incidents to track. Some, for example, track by
236
amount of media coverage.
237
Documenng, reporng, and sharing informaon about GAI incidents can help migate and prevent
238
harmful outcomes by assisng relevant AI actors in tracing impacts to their source. Greater awareness
239
and standardizaon of GAI incident reporng could promote this transparency and improve GAI risk
240
management across the AI ecosystem.
241
Appendix J. Documentaon and Involvement of AI Actors
242
AI actors should be aware of their roles in reporng AI incidents. To beer understand previous incidents
243
and implement measures to prevent similar ones in the future, organizaons could consider developing
244
guidelines for publicly available incident reporng which include informaon about AI actor
245
responsibilies. These guidelines would help AI system operators idenfy GAI incidents across the AI
246
lifecycle and with AI actors regardless of role. Documentaon and review of third party inputs and
247
plugins for GAI systems is especially important for AI actors in the context of incident disclosure; LLM
248
inputs and content delivered through these plugins is oen distributed, with inconsistent or insucient
249
access control.
250
Documentaon pracces including logging, recording, and analyzing GAI incidents can facilitate
251
smoother sharing of informaon with relevant AI actors. Regular informaon sharing, change
252
management records, version history and metadata can also empower AI actors responding to and
253
managing AI incidents.
254
69
Appendix K. References
255
AI Risks and Trustworthiness, NIST Trustworthy & Responsible AI Resource Center. Naonal Instute of
256
Standards and Technology.
257
hps://airc.nist.gov/AI_RMF_Knowledge_Base/AI_RMF/Foundaonal_Informaon/3-sec-characteriscs.
258
AI RMF Playbook. Naonal Instute of Standards and Technology.
259
hps://airc.nist.gov/AI_RMF_Knowledge_Base/Playbook.
260
AI RMF Proles. Naonal Instute of Standards and Technology.
261
hps://airc.nist.gov/AI_RMF_Knowledge_Base/AI_RMF/Core_And_Proles/6-sec-prole.
262
AI Incident Database. hps://incidentdatabase.ai/.
263
AI Risk Management Framework. Naonal Instute of Standards and Technology.
264
hps://www.nist.gov/itl/ai-risk-management-framework.
265
AI Risk Management Framework. Naonal Instute of Standards and Technology. Appendix A:
266
Descripons of AI Actor Tasks, NIST Trustworthy & Responsible AI Resource Center. Naonal Instute of
267
Standards and Technology.
268
hps://airc.nist.gov/AI_RMF_Knowledge_Base/AI_RMF/Appendices/Appendix_A#:~:text=AI%20actors%
269
20in%20this%20category,data%20providers%2C%20system%20funders%2C%20product.
270
AI Risk Management Framework. Naonal Instute of Standards and Technology. Appendix B: How AI
271
Risks Dier from Tradional Soware Risks. Naonal Instute of Standards and Technology.
272
hps://airc.nist.gov/AI_RMF_Knowledge_Base/AI_RMF/Appendices/Appendix_B.
273
Alba, D., (2023) How Fake AI Photo of a Pentagon Blast Went Viral and Briey Spooked Stocks.
274
Bloomberg. hps://www.bloomberg.com/news/arcles/2023-05-22/fake-ai-photo-of-pentagon-blast-
275
goes-viral-trips-stocks-briey.
276
Atherton, D. (2024) Deepfakes and Child Safety: A Survey and Analysis of 2023 Incidents and Responses.
277
AI Incident Database. hps://incidentdatabase.ai/blog/deepfakes-and-child-safety/.
278
Authencang AI-Generated Content (2024). Informaon Technology Industry Council.
279
hps://www.ic.org/policy/ITI_AIContentAuthorizaonPolicy_122123.pdf.
280
Badyal, N. et al., (2023) Intenonal Biases in LLM Responses. arXiv. hps://arxiv.org/pdf/2311.07611.
281
Bing Chat: Data Exltraon Exploit Explained. Embrace The Red.
282
hps://embracethered.com/blog/posts/2023/bing-chat-data-exltraon-poc-and-x/.
283
Bommasani, R. et al., (2022) Picking on the Same Person: Does Algorithmic Monoculture lead to
284
Outcome Homogenizaon? arXiv. hps://arxiv.org/pdf/2211.13972.
285
Boyarskaya, M. et al., (2020) Overcoming Failures of Imaginaon in AI Infused System Development and
286
Deployment. arXiv. hps://arxiv.org/pdf/2011.13416.
287
Browne, D. et al., (2023) Securing the AI Pipeline. Mandiant.
288
hps://www.mandiant.com/resources/blog/securing-ai-pipeline.
289
70
Building a Glossary for Synthec Media Transparency Methods, Part 1: Indirect Disclosure (2023)
290
Partnership on AI. hps://partnershiponai.org/glossary-for-synthec-media-transparency-methods-part-
291
1-indirect-disclosure/.
292
Burgess, M., (2024) Generave AI’s Biggest Security Flaw Is Not Easy to Fix. WIRED.
293
hps://www.wired.com/story/generave-ai-prompt-injecon-hacking/.
294
Burtell, M. et al., (2024) The Surprising Power of Next Word Predicon: Large Language Models
295
Explained, Part 1. Georgetown CSET. hps://cset.georgetown.edu/arcle/the-surprising-power-of-next-
296
word-predicon-large-language-models-explained-part-1/.
297
Carlini, N., et al., (2021) Extracng Training Data from Large Language Models. Usenix.
298
hps://www.usenix.org/conference/usenixsecurity21/presentaon/carlini-extracng.
299
Carlini, N. et al., (2023) Quanfying Memorizaon Across Neural Language Models. ICLR 2023.
300
hps://arxiv.org/pdf/2202.07646.
301
Carlini, N. et al., (2024) Stealing Part of a Producon Language Model. arXiv.
302
hps://arxiv.org/abs/2403.06634.
303
Chandra, B. et al., (2023) Dismantling the Disinformaon Business of Chinese Inuence Operaons.
304
RAND. hps://www.rand.org/pubs/commentary/2023/10/dismantling-the-disinformaon-business-of-
305
chinese.html.
306
Dahl, M. et al., (2024) Large Legal Ficons: Proling Legal Hallucinaons in Large Language Models. arXiv.
307
hps://arxiv.org/abs/2401.01301.
308
De Angelo, D., (2024) Short, Mid and Long-Term Impacts of AI in Cybersecurity. Palo Alto Networks.
309
hps://www.paloaltonetworks.com/blog/2024/02/impacts-of-ai-in-cybersecurity/.
310
De Freitas, J., et al. (2023) Chatbots and Mental Health: Insights into the Safety of Generave AI. Harvard
311
Business School. hps://www.hbs.edu/ris/Publicaon%20Files/23-011_c1bdd417-f717-47b6-bccb-
312
5438c6e65c1a_f6fd9798-3c2d-4932-b222-056231fe69d7.pdf.
313
Dietvorst, B. et al., (2014) Algorithm Aversion: People Erroneously Avoid Algorithms Aer Seeing Them
314
Err. Journal of Experimental Psychology. hps://markeng.wharton.upenn.edu/wp-
315
content/uploads/2016/10/Dietvorst-Simmons-Massey-2014.pdf.
316
Duhigg, C., (2012) How Companies Learn Your Secrets. New York Times.
317
hps://www.nymes.com/2012/02/19/magazine/shopping-habits.html.
318
Elsayed, G. et al., (2024) Images altered to trick machine vision can inuence humans too. Google
319
DeepMind. hps://deepmind.google/discover/blog/images-altered-to-trick-machine-vision-can-
320
inuence-humans-too/.
321
Epstein, Z. et al., (2023). Art and the science of generave AI. Science.
322
hps://www.science.org/doi/10.1126/science.adh4451.
323
Execuve Order on the Safe, Secure, and Trustworthy Development and Use of Arcial Intelligence
324
(2023) The White House.hps://www.whitehouse.gov/brieng-room/presidenal-
325
acons/2023/10/30/execuve-order-on-the-safe-secure-and-trustworthy-development-and-use-of-
326
arcial-intelligence/.
327
71
Fair Informaon Pracce Principles (FIPPs). FPC. hps://www.fpc.gov/resources/pps/.
328
Generave arcial intelligence (AI) - ITSAP.00.041. (2023) Canadian Centre for Cyber Security.
329
hps://www.cyber.gc.ca/en/guidance/generave-arcial-intelligence-ai-itsap00041.
330
GPT-4 System Card (2023) OpenAI. hps://cdn.openai.com/papers/gpt-4-system-card.pdf.
331
GPT-4 Technical Report (2024) OpenAI. hps://arxiv.org/pdf/2303.08774.
332
Greshake, K. et al., (2023). Not what you've signed up for: Compromising Real-World LLM-Integrated
333
Applicaons with Indirect Prompt Injecon. arXiv. hps://arxiv.org/abs/2302.12173.
334
Feer, M. et al., (2024). Red-Teaming for Generave AI: Silver Bullet or Security Theater? arXiv.
335
hps://arxiv.org/pdf/2401.15897.
336
Haran, R., (2023). Securing LLM Systems Against Prompt Injecon. NVIDIA.
337
hps://developer.nvidia.com/blog/securing-llm-systems-against-prompt-injecon/.
338
Harwell, D., (2023) AI-generated child sex images spawn new nightmare for the web. Washington Post.
339
hps://www.washingtonpost.com/technology/2023/06/19/arcial-intelligence-child-sex-abuse-
340
images/.
341
Hubinger, E. et al, (2024) “Sleeper Agents: Training Decepve LLMs that Persist Through Safety Training”,
342
arXiv e-prints. hps://arxiv.org/abs/2401.05566.
343
Jain, S. et al., (2023) Algorithmic Pluralism: A Structural Approach To Equal Opportunity. arXiv.
344
hps://arxiv.org/pdf/2305.08157.
345
Ji, Z. et al (2023) Survey of Hallucinaon in Natural Language Generaon. ACM Comput. Surv. 55, 12,
346
Arcle 248. hps://doi.org/10.1145/3571730
347
Jussupow, E. et al., (2020) Why Are We Averse Towards Algorithms? A Comprehensive Literature Review
348
on Algorithm Aversion. ECIS 2020. hps://aisel.aisnet.org/ecis2020_rp/168/.
349
Katzman, J., et al., (2023) Taxonomizing and measuring representaonal harms: a look at image tagging.
350
AAAI. hps://dl.acm.org/doi/10.1609/aaai.v37i12.26670.
351
Kirchenbauer, J. et al., (2023) A Watermark for Large Language Models. OpenReview.
352
hps://openreview.net/forum?id=aX8ig9X2a7.
353
Kleinberg, J. et al., (May 2021) Algorithmic monoculture and social welfare. PNAS.
354
hps://www.pnas.org/doi/10.1073/pnas.2018340118.
355
Lakatos, S., (2023) A Revealing Picture. Graphika. hps://graphika.com/reports/a-revealing-picture.
356
Lenaerts-Bergmans, B., (2024) Data Poisoning: The Exploitaon of Generave AI. Crowdstrike.
357
hps://www.crowdstrike.com/cybersecurity-101/cyberaacks/data-poisoning/.
358
Liang, W. et al., (2023) GPT detectors are biased against non-nave English writers. arXiv.
359
hps://arxiv.org/abs/2304.02819.
360
Luccioni, A. et al., (2023) Power Hungry Processing: Was Driving the Cost of AI Deployment? arXiv.
361
hps://arxiv.org/pdf/2311.16863.
362
72
Mouton, C. et al., (2024) The Operaonal Risks of AI in Large-Scale Biological Aacks. RAND.
363
hps://www.rand.org/pubs/research_reports/RRA2977-2.html.
364
Nicole, L. et al., (2023) Humans Are Biased. Generave Ai Is Even Worse. Bloomberg.
365
hps://www.bloomberg.com/graphics/2023-generave-ai-bias/.
366
Northcu, C. et al., (2021) Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks.
367
arXiv. hps://arxiv.org/pdf/2103.14749.
368
OECD (2023), "Advancing accountability in AI: Governing and managing risks throughout the lifecycle for
369
trustworthy AI", OECD Digital Economy Papers, No. 349, OECD Publishing, Paris,
370
hps://doi.org/10.1787/2448f04b-en.
371
OECD AI Incidents Monitor. OECD.AI Policy Observatory. hps://oecd.ai/en/incidents-methodology.
372
Padmakumar, V. et al., (2024) Does wring with language models reduce content diversity? ICLR.
373
hps://arxiv.org/pdf/2309.05196.
374
Paresh, D., (2023) ChatGPT Is Cung Non-English Languages Out of the AI Revoluon. WIRED.
375
hps://www.wired.com/story/chatgpt-non-english-languages-ai-revoluon/.
376
Qu, Y. et al., (2023) Unsafe Diusion: On the Generaon of Unsafe Images and Hateful Memes From Text-
377
To-Image Models. arXiv. hps://arxiv.org/pdf/2305.13873.
378
Rafat, K. et al., (2023) Migang carbon footprint for knowledge disllaon based deep learning model
379
compression. PLOS One. hps://journals.plos.org/plosone/arcle?id=10.1371/journal.pone.0285668.
380
Roadmap for Researchers on Priories Related to Informaon Integrity Research and Development
381
(2022) The White House. hps://www.whitehouse.gov/wp-content/uploads/2022/12/Roadmap-
382
Informaon-Integrity-RD-2022.pdf?.
383
Sandbrink, J., (2023) Arcial intelligence and biological misuse: Dierenang risks of language models
384
and biological design tools. arXiv. hps://arxiv.org/pdf/2306.13952.
385
Satariano, A. et al., (2023) The People Onscreen Are Fake. The Disinformaon Is Real. New York Times.
386
hps://www.nymes.com/2023/02/07/technology/arcial-intelligence-training-deepfake.html.
387
Schaul, K. et al., (2024) Inside the secret list of websites that make AI like ChatGPT sound smart.
388
Washington Post. hps://www.washingtonpost.com/technology/interacve/2023/ai-chatbot-learning/.
389
Shelby, R. et al., (2023) Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm
390
Reducon. arXiv. hps://arxiv.org/pdf/2210.05791.
391
Shevlane, T. et al., (2023) Model evaluaon for extreme risks. arXiv. hps://arxiv.org/pdf/2305.15324.
392
Shumailov, I. et al., (2023) The curse of recursion: training on generated data makes models forget. arXiv.
393
hps://arxiv.org/pdf/2305.17493v2.
394
Skaug Sætra, H. et al., (2022). Psychological interference, liberty and technology. Technology in Society.
395
hps://www.sciencedirect.com/science/arcle/pii/S0160791X22001142.
396
Smith, A. et al., (2023) Hallucinaon or Confabulaon? Neuroanatomy as metaphor in Large Language
397
Models. PLOS Digital Health.
398
hps://journals.plos.org/digitalhealth/arcle?id=10.1371/journal.pdig.0000388.
399
73
Soice, E. et al., (2023) Can large language models democraze access to dual-use biotechnology? arXiv.
400
hps://arxiv.org/abs/2306.03809.
401
Staab, R. et al., (2023) Beyond Memorizaon: Violang Privacy via Inference With Large Language
402
Models. arXiv. hps://arxiv.org/pdf/2310.07298
403
Stanford, S. et al., (2023) Whose Opinions Do Language Models Reect? arXiv.
404
hps://arxiv.org/pdf/2303.17548.
405
Strubell, E. et al., (2019) Energy and Policy Consideraons for Deep Learning in NLP. arXiv.
406
hps://arxiv.org/pdf/1906.02243.
407
Thiel, D. (2023) Invesgaon Finds AI Image Generaon Models Trained on Child Abuse. Stanford Cyber
408
Policy Center. hps://cyber.fsi.stanford.edu/news/invesgaon-nds-ai-image-generaon-models-
409
trained-child-abuse.
410
The Toxicity Issue. Jigsaw, Google. hps://current.withgoogle.com/the-current/toxicity/.
411
Tufekci, Z. (2015) Algorithmic Harms Beyond Facebook and Google: Emergent Challenges of
412
Computaonal Agency. hps://ctlj.colorado.edu/wp-content/uploads/2015/08/Tufekci-nal.pdf
413
Turri, V. et al., (2023) Why We Need to Know More: Exploring the State of AI Incident Documentaon
414
Pracces. AAAI/ACM Conference on AI, Ethics, and Society.
415
hps://dl.acm.org/doi/fullHtml/10.1145/3600211.3604700.
416
Urbina, F. et al., (2022) Dual use of arcial-intelligence-powered drug discovery. Nature Machine
417
Intelligence. hps://www.nature.com/arcles/s42256-022-00465-9.
418
Wang, Y. et al., (2023) Do-Not-Answer: A Dataset for Evaluang Safeguards in LLMs. arXiv.
419
hps://arxiv.org/pdf/2308.13387.
420
Wang, X. et al., (2023) Energy and Carbon Consideraons of Fine-Tuning BERT. ACL Anthology.
421
hps://aclanthology.org/2023.ndings-emnlp.607.pdf.
422
Wardle, C. et al., (2017) Informaon Disorder: Toward an interdisciplinary framework for research and
423
policy making. Council of Europe. hps://rm.coe.int/informaon-disorder-toward-an-interdisciplinary-
424
framework-for-researc/168076277c.
425
Weatherbed, J., (2024) Trolls have ooded X with graphic Taylor Swi AI fakes. The Verge.
426
hps://www.theverge.com/2024/1/25/24050334/x-twier-taylor-swi-ai-fake-images-trending.
427
Weidinger, L. et al., (2021) Ethical and social risks of harm from Language Models. arXiv.
428
hps://arxiv.org/pdf/2112.04359.
429
Weidinger, L. et al. (2023) Sociotechnical Safety Evaluaon of Generave AI Systems. arXiv.
430
hps://arxiv.org/pdf/2310.11986.
431
Weidinger, L. et al., (2022) Taxonomy of Risks posed by Language Models. FAccT ’22.
432
hps://dl.acm.org/doi/pdf/10.1145/3531146.3533088.
433
Wu, K. et al., (2024) How well do LLMs cite relevant medical references? An evaluaon framework and
434
analyses. arXiv. hps://arxiv.org/pdf/2402.02008.
435
74
Yin, L. et al., (2024) OpenAI’s GPT Is A Recruiter’s Dream Tool. Tests Show There’s Racial Bias. Bloomberg.
436
hps://www.bloomberg.com/graphics/2024-openai-gpt-hiring-racial-discriminaon/.
437
Yu, Z. et al., (March 2024) Don’t Listen To Me: Understanding and Exploring Jailbreak Prompts of Large
438
Language Models. arXiv. hps://arxiv.org/html/2403.17336v1
439
Zhang, Y. et al., (2023) Human favorism, not AI aversion: People’s percepons (and bias) toward
440
generave AI, human experts, and human–GAI collaboraon in persuasive content generaon. Judgment
441
and Decision Making. hps://www.cambridge.org/core/journals/judgment-and-decision-
442
making/arcle/human-favorism-not-ai-aversion-peoples-percepons-and-bias-toward-generave-ai-
443
human-experts-and-humangai-collaboraon-in-persuasive-content-
444
generaon/419C4BD9CE82673EAF1D8F6C350C4FA8.
445
Zhang, Y. et al., (2023) Siren’s Song in the AI Ocean: A Survey on Hallucinaon in Large Language Models.
446
arXiv. hps://arxiv.org/pdf/2309.01219.
447
Zhao, X. et al., (2023) Provable Robust Watermarking for AI-Generated Text. Semanc Scholar.
448
hps://www.semancscholar.org/paper/Provable-Robust-Watermarking-for-AI-Generated-Text-Zhao-
449
Ananth/75b68d0903af9d9f6e47ce3cf7e1a7d27ec811dc.
450
