Model Origin Risk: Open-Source, Commercial, and Sovereign Models

The provenance of a model carries material risk that the AI System Assessor must assess and document. Where a model came from, who trained it, on what data, and under what governance all determine whether the organisation can satisfy the EU AI Act's documentation and conformity requirements. Any organisation using an external model as a component of a high-risk system inherits the documentation gaps of that model's development process and must either fill them through its own due diligence or accept the resulting non-conformity risk.

The Model Selection pillar establishes model selection as one of the most consequential early decisions in building a compliant AI system. C-03-01 examines the three origin categories, open-source, proprietary third-party, and in-house, along with the fine-tuning boundary that can shift an organisation from deployer to provider under Article 25.

Open-source models from repositories such as Hugging Face, GitHub, or academic publications offer accessibility and community validation, but they introduce several distinct risks. Training data provenance may be unknown or poorly documented. The model may have been trained on data that includes copyrighted material, personal data processed without consent, or data that is unrepresentative of the intended deployment population. The development process may not have included the bias testing, adversarial evaluation, or record-keeping that the EU AI Act requires.

Any organisation using an open-source model as a component of a high-risk system inherits these documentation gaps and must either fill them through its own due diligence or accept the resulting non-conformity risk. Module 3 of the AISDP records which open-source components are incorporated, the due diligence performed on each component, the licensing terms and their compatibility with the system's commercial and regulatory context, and the residual risks arising from gaps in the component's provenance documentation.

Models licensed from commercial providers present a different risk profile centred on disclosure limitations. A vendor may refuse to disclose the training data composition, model architecture details, or fairness evaluation results, citing trade secrets. This creates a documentation gap in the AISDP that the organisation must address. The vendor due diligence questionnaire from the Discovery phase should have captured the vendor's willingness to provide the information required by Annex IV.

Where the vendor's disclosures are insufficient, the AI System Assessor records the gaps as non-conformities, and the organisation must assess whether it can compensate through its own testing and evaluation of the model's outputs. The risk register captures these gaps alongside their potential regulatory and reputational impact, enabling the organisation to make an informed decision about whether the vendor relationship is viable for a high-risk deployment.

Models developed internally offer the greatest control over documentation and governance. The risk profile centres on process discipline: whether the development team followed the documented methodology, whether the training data governance requirements were satisfied, and whether the testing was rigorous. In-house models developed informally, by a data scientist working independently or as part of a hackathon or innovation sprint, may lack the documentation infrastructure that the AISDP demands.

The discovery process should have identified these models, and the risk assessment must evaluate their documentation readiness. Organisations with strong MLOps practices and version-controlled training pipelines will find this assessment straightforward. Those with ad hoc development processes face a remediation exercise to bring existing models up to the documentation standard required by the EU AI Act before they can be deployed in high-risk contexts.

Organisations that fine-tune a GPAI model for use in a high-risk system occupy an ambiguous position in the AI Act's provider and deployer framework. Article 25(1)(b) provides that a deployer of a high-risk AI system is considered a provider if it places on the market or puts into service a high-risk AI system under its own name or trademark, or if it makes a substantial modification to a high-risk AI system. Fine-tuning a general-purpose foundation model for a specific high-risk application, such as adapting a general LLM for use in a credit assessment system or a recruitment screening tool, will in most cases change the model's intended purpose from general purpose to a specific high-risk use case. This modification triggers Article 25(1)(b) and places the fine-tuning organisation in the role of provider, with the full set of provider obligations under Article 16.

The Legal and Regulatory Advisor should assess the fine-tuning activity against three criteria. First, does the fine-tuning change the model's intended purpose as documented by the GPAI provider? Second, does the fine-tuning alter the model's risk profile, introducing new bias patterns, failure modes, or accuracy characteristics not present in the base model? Third, does the fine-tuning affect the model's compliance with the GPAI provider's own obligations under Articles 51 to 56, potentially voiding safety evaluations or alignment testing conducted on the base model?

Where the Legal and Regulatory Advisor determines that fine-tuning triggers provider status, the AISDP must document the fine-tuning process with the same rigour applied to in-house model development. This includes the fine-tuning data governance per Section 4, the fine-tuning methodology and hyperparameters per Module 3, the evaluation against the system's declared performance and fairness thresholds, and a clear delineation between the base model's characteristics inherited from the GPAI provider and the fine-tuned model's characteristics that fall under the organisation's responsibility.

The Model Selection Record in Section 3.12 should document the base model selection as a GPAI integration decision and the fine-tuning as a development decision, with separate risk assessments for each. For parameter-efficient fine-tuning methods such as LoRA, QLoRA, prefix tuning, and adapters, the same analysis applies. The compliance boundary does not depend on the volume of parameters modified; it depends on whether the modification changes the model's intended purpose or risk profile.

SLSA (Supply-chain Levels for Software Artifacts) provides a framework originally designed for software supply chain security that adapts well to ML artefacts. SLSA defines four levels of increasing assurance: Level 1 requires that the build process is documented, Level 2 requires automation with verifiable provenance metadata, Level 3 requires a hardened and tamper-resistant build environment, and Level 4 requires two-person review of all changes. For AI systems under the EU AI Act, Level 2 is the minimum practical target. The Technical Owner automates the model training pipeline, and the pipeline produces a provenance record linking the trained model to its training code, training data, hyperparameters, and execution environment.

Model signing provides tamper evidence. Sigstore and its cosign tool allow the organisation to cryptographically sign model artefacts after training, creating a verifiable record that the model has not been modified since the signature was applied. The signature is stored alongside the model in the model registry. Before deploying a model to production, the deployment pipeline verifies the signature, ensuring that the model artefact is identical to the one that was evaluated and approved. This prevents a class of supply chain attacks where a model is replaced with a modified version between evaluation and deployment.

For models downloaded from public repositories such as Hugging Face, TensorFlow Hub, and PyTorch Hub, the provenance discipline differs from in-house training. Best practice is to download the model once, compute a cryptographic hash using SHA-256, store the model and its hash in the internal model registry, and ensure that all subsequent references use the internal copy rather than re-downloading from the external hub. This prevents silent changes if the hub updates the model under the same identifier. Hugging Face's transformers library supports this through the revision parameter, which pins to a specific Git commit SHA.

The provenance record for each model version should capture, at minimum, the model's origin (whether in-house trained, open-source downloaded, or proprietary licensed), the training data version linked to the data versioning system, the training code commit linked to the code repository, the hyperparameters and training configuration, the experiment tracking run ID, the pipeline execution ID and timestamp, the evaluation metrics from the approval gate, the content hash of the model artefact, and the digital signature. This record is attached as structured metadata in the model registry and is referenced by AISDP Module 3.

For Software and ML Bill of Materials, SPDX and CycloneDX are the two dominant standards that support listing the model's dependencies. These dependencies include ML framework versions, third-party libraries, and pre-trained model components with their licence terms. The SBOM serves two purposes: it enables vulnerability scanning, identifying whether any dependency has a known security vulnerability, and it enables licence compliance checking, identifying whether any dependency's licence terms conflict with the system's commercial or regulatory context.

SBOM generation is automated as part of the CI pipeline, ensuring that every model release includes a current and complete bill of materials. The cybersecurity requirements of the EU AI Act intersect directly with SBOM practices, as Annex IV requires documentation of the measures adopted to address cybersecurity threats. An up-to-date SBOM provides auditable evidence that the organisation has visibility into its model's supply chain and can respond promptly when vulnerabilities are disclosed in any component.