Govern the collection and use of data in training and testing

At a glance

Modern AI is data-dependent, so governing the data is governing the model. This competency covers the data and testing middle of the life cycle — mostly NIST AI RMF MEASURE, with data governance running through MAP and MANAGE.

Five moves:

1. III.B.1 — Govern the data. Lawful rights to use it; quality, quantity, integrity, fit-for-purpose.
2. III.B.2 — Trace the data. Establish and document lineage and provenance.
3. III.B.3 — Plan and perform testing. Unit, integration, validation, performance, security, bias, interpretability.
4. III.B.4 — Manage issues. Identify and treat risks that surface during training and testing.
5. III.B.5 — Document it. Record the training/testing process to validate results, prove compliance, manage risk.

The two phrases the exam keeps returning to: “garbage in, garbage out” (data quality bounds model quality) and the validation vs. verification distinction.

III.B.1 — Establish and follow data governance requirements

Two pillars: lawful rights and data quality.

Lawful rights to collect and use data — assessed and documented before training:

• A valid legal basis for personal data (under GDPR: consent, contract, legitimate interests, etc.) and respect for purpose limitation — data collected for one purpose may not be freely repurposed to train a model.
• Special-category data (biometrics, health, race) needs an Art. 9 condition.
• IP and licensing clearance for scraped or third-party data (copyright in training data is contested — see the U.S. Copyright Office work).
• Contractual permission for vendor/partner data.

Data quality dimensions the BOK and EU AI Act Art. 10 emphasise:

• Quality — accurate, complete, correctly labelled, free of errors.
• Quantity — enough data to learn the task without overfitting; more is not automatically better.
• Integrity — unaltered, uncorrupted, protected from tampering (e.g., poisoning).
• Fit-for-purpose — relevant and representative of the population and deployment context. Data can be clean and plentiful yet unfit if it represents the wrong population.

III.B.2 — Establish and document data lineage and provenance

Provenance answers where did this data come from and who owns it — origin, collection method, rights, chain of custody. Lineage answers what happened to it — every transformation from raw collection through cleaning, labelling, feature engineering, into the training set.

Why governance cares:

• Legal defensibility — proving you had rights to the data and can honour deletion / data-subject requests.
• Reproducibility — rebuilding a model or explaining a result requires knowing exactly what went in.
• Bias and quality tracing — when a problem appears in output, lineage lets you trace it back to a source or transformation.
• Incident root-cause — feeds the drift and data-quality analysis in III.C.5.

This is increasingly demanded externally too: GPAI providers under the EU AI Act must publish a summary of training-content sources.

III.B.3 — Plan and perform training and testing

Testing AI blends classic software testing with ML-specific evaluation. The BOK lists the types — know what each one checks:

Data splitting underpins all of it: train on the training set, tune on the validation set, and report on a held-out test set used only once. Comparing training vs. validation/test performance is how you catch overfitting (memorising) and underfitting (too simple). Before any training, feature engineering — selecting and transforming raw data into the input variables (features) the model learns from — is itself a governance point: which attributes are included (or proxy for protected ones) is decided here.

NIST wraps this whole discipline in one acronym worth knowing: TEVV — Test, Evaluation, Verification and Validation — which the AI RMF treats not as a phase but as a continuous activity across the entire lifecycle (design-time checks, pre-deployment evaluation, and post-deployment monitoring all count as TEVV).

III.B.4 — Identify and manage issues and risks during training and testing

Problems surface once you have data and a model. Common issues and their treatments:

• Poor data quality / mislabelling → fix labels, improve collection, re-clean; trace via lineage.
• Bias / disparate impact → rebalance or augment data, reweight, adjust thresholds, or change features.
• Overfitting → more/varied data, regularisation, simpler model, cross-validation.
• Underfitting / inadequate performance → richer features, more capable model, more data.
• Security findings (poisoning, adversarial weakness) → harden pipeline, sanitise inputs, adversarial training.
• Data leakage — target or test information bleeding into training, inflating results → fix the split.
• Insufficient or unrepresentative data → collect more, expand coverage, or reconsider the use case.

Manage these with the same discipline as III.A.4: rate by probability/severity, apply the mitigation hierarchy, and escalate residual risk for sign-off. Issues that cannot be mitigated to within tolerance are a no-go signal.

III.B.5 — Document the training and testing process

Documentation here serves three jobs the BOK names: validate results, establish compliance, manage risk.

Capture:

• Datasets used — sources, provenance/lineage, splits, preprocessing (III.B.1–B.2).
• Test plans and results — which tests, on what data, with what metrics and thresholds, pass/fail (III.B.3).
• Issues found and resolutions — the training/testing risk log (III.B.4).
• Final performance, fairness and security evaluation against the acceptance thresholds set in III.A.3.

This is the raw material for data sheets / dataset documentation, the EU AI Act technical documentation (Annex IV) and the eventual model card (III.C.1). Documenting as you test makes results reproducible and the eventual conformity claim defensible.

How this shows up later

• The model’s tested performance, fairness and security profile becomes the release-readiness evidence and model card content in III.C.1.
• Test thresholds set here become the monitoring alarms that detect drift in III.C.2.
• Lineage and provenance captured here are what root-cause analysis (III.C.5) uses when an incident traces back to data.
• The deployer in Domain IV relies on this documentation to assess a third-party model it did not train.