Govern the designing and building of the AI system

At a glance

This competency covers the front half of the AI life cycle — everything from “should we even build this?” to a system that is ready for training and testing. The governance professional’s job here is not to write the model but to make sure the right questions are asked and answered, in the right order, and documented.

Five moves, in sequence:

1. III.A.1 — Frame it. Define the business context and use case (NIST AI RMF MAP).
2. III.A.2 — Assess it. Perform or review an impact assessment (ISO/IEC 42005; EU AI Act FRIA).
3. III.A.3 — Build it right. Apply policies, procedures, best practice and ethics to design and build.
4. III.A.4 — De-risk it. Identify and manage internal and external risks (probability/severity matrix, mitigation hierarchy, pilots).
5. III.A.5 — Write it down. Document the design-and-build process for compliance and risk management.

A useful anchor: this whole competency lives mostly inside NIST’s MAP function (establish context and identify risks), spilling into MEASURE as testing begins. ISO/IEC 42001 supplies the management-system wrapper; the EU AI Act supplies the legal obligations a provider must satisfy before placing a high-risk system on the market.

III.A.1 — Define the business context and use case

Before any building, you establish what the system is for and who it touches. This mirrors NIST AI RMF MAP: understand the context, intended purpose, deployment setting, and the people involved.

A complete framing captures:

• Purpose and intended use — the decision the AI informs and the value it should create.
• Intended users and affected stakeholders — operators, subjects of the decision, and affected non-users.
• Deployment context — domain (e.g., credit, hiring, healthcare), jurisdiction, and whether the use is regulated.
• Success criteria and constraints — performance targets, budget, legal limits, ethical red lines.
• Reasonably foreseeable misuse — how the system could be used outside its intended purpose.
• Whether AI is even the right tool — sometimes the governed answer is “don’t build it.”

Getting the use case right also drives legal classification: under the EU AI Act, the intended purpose is what determines whether a system is prohibited, high-risk, limited-risk or minimal-risk — so the use-case definition feeds directly into risk obligations.

III.A.2 — Perform or review an impact assessment

An AI impact assessment (AIA) is a documented evaluation of how the system could affect individuals, groups and society — and what will be done about it. ISO/IEC 42005 is the dedicated standard for conducting and documenting one across the life cycle.

What a good impact assessment covers:

• System description and intended use (carried over from III.A.1).
• Potential impacts on individuals, groups, society and the environment — benefits and harms.
• Affected stakeholders and how their rights/interests are implicated.
• Likelihood and severity of each potential harm.
• Mitigations and the residual risk that remains after them.
• Sign-off and review triggers (re-assess on material change).

Several adjacent assessments are easy to confuse:

These overlap and can be combined, but they are not the same. A DPIA is about personal data; an AIA/FRIA is broader (safety, fairness, autonomy, societal effects).

III.A.3 — Apply policies, procedures, best practice and ethics to design and build

This PI is the practical core: translate the organisation’s policies and the responsible-AI principles into concrete design decisions. The BOK enumerates the activities — treat each as a checkpoint:

• Purpose of AI — keep design tied to the use case; resist scope creep into uses you didn’t assess.
• Requirements gathering — capture functional, performance, legal, ethical and data requirements explicitly.
• Architecture and model selection — choose the model type and system design to fit requirements and risk. A higher-stakes, regulated use leans toward more interpretable models, stronger data controls and tighter oversight; “build vs. buy” and “classic vs. generative” are decided here.
• Human oversight — design the intervention points up front (review queues, override, confidence thresholds, escalation). EU AI Act Art. 14 requires oversight that is effective, not nominal.
• Data analysis — examine the training data for representativeness, quality and bias before relying on it (links to III.B).
• Metric and threshold evaluation — decide in advance which metrics define success and the thresholds that must be met (accuracy, precision/recall, fairness gaps, latency, robustness) so go/no-go is objective.
• Stakeholder engagement and feedback — bring affected groups and cross-functional experts in during design, not after.
• Operational controls — bake in guardrails: access control, input/output filtering, rate limits, fallback behaviour, logging and a kill-switch.

III.A.4 — Identify and manage internal and external risks

Now you systematically surface and treat the risks of this build. The BOK names the toolkit:

Probability/severity harms matrix. Score each potential harm on how likely it is against how severe it would be, then prioritise the top-right cells.

Risk mitigation hierarchy — apply treatments in order of preference:

1. Avoid / eliminate — change or drop the design so the risk can’t occur (e.g., don’t collect the sensitive attribute).
2. Reduce / mitigate — controls that lower probability or severity (better data, guardrails, oversight).
3. Transfer — shift residual risk via insurance, contracts or vendor indemnities.
4. Accept — knowingly retain low residual risk with documented sign-off.

Internal vs. external risks — internal: data quality, model brittleness, integration faults, skills gaps, mis-set objectives. External: regulatory change, adversarial attack (poisoning, prompt injection), third-party/supply-chain dependencies, reputational and societal harm.

Other named tools:

• Stakeholder mapping — who is affected, so no impact is missed.
• Use-case evaluation — pressure-test the use against ethical and legal limits.
• Benchmarking — compare against baselines/standard datasets to judge adequacy.
• Pre-deployment pilots and testing — shadow mode, sandboxes and limited rollouts to find problems on real conditions before full launch.

III.A.5 — Document the designing and building process

Documentation is the evidence layer. It establishes compliance, enables risk management, and creates the record auditors, regulators and incident responders rely on later.

What to capture during design and build:

• The use-case definition and intended purpose (III.A.1).
• The impact assessment and its sign-off (III.A.2).
• Design decisions and rationale — architecture, model choice, data sources, oversight measures (III.A.3).
• The risk register — identified risks, ratings, treatments and residual risk (III.A.4).
• Requirements, metrics and acceptance thresholds.

This feeds directly into the EU AI Act’s technical documentation for high-risk systems (Annex IV) and into ISO/IEC 42001’s management-system records. Document as you go — reconstructed documentation is weaker evidence and usually incomplete.

How this shows up later

• The use case, impact assessment and risk register built here are the inputs to data governance and testing in III.B.
• Oversight, metrics and operational controls defined now become the monitoring baselines in III.C.
• The technical documentation started here is completed for release and conformity (III.C.1) and the public disclosures (III.C.6).
• The same design-time discipline reappears from the deployer’s side in Domain IV when an organisation adopts a third-party model.