I want to use an AI system for work. How do I manage the risks? (Part 2)

In our first article, we discussed 10 notable risks that may arise when a business implements or procures an AI system. In this follow-up article, we look at some of the practical steps your business could take to try to mitigate some of those risks. Throughout this article, we will refer to the previous risks, and follow up with some mitigation solutions for you…

1. How does the AI system actually work? Do your due diligence!

What’s the risk again? Can an operator explain how the AI system actually works, and why it produces particular outcomes? What happens if they change or update it?

So, what’s the solution then? The starting point is robust pre-contract due diligence. You should ask why the AI is being used, whether it is necessary and proportionate, who is providing it, where that provider is located, and consider whether there are any data protection, confidentiality, or IP concerns associated with the proposed deployment.

Alongside that, you should develop an internal AI governance framework incorporating due diligence questionnaires, acceptable use policies, AI playbooks or model clauses, and should build in contractual requirements around transparency and change notification so you know what you’re using and when it changes. You should also conduct an AI risk assessment. Things to think about here include:

• What legal right do you have to process personal data with an AI system?
• How do you comply with applicable laws & regulations that impact on what you’re doing?
• Will decisions be based on AI output and, if so, what is the potential impact of those decisions?
• Will personal data be transferred to countries with less robust data protection laws?
• Are you at risk of infringing IP rights or client/customer confidentiality?
• Could use of the AI system have implications for human rights?

The language used in any contract with an operator of an AI system is important, but the due diligence you carry out before you get to that point is vital for your understanding and subsequent AI risk assessment.

2. Data use and training

What’s the risk again? Businesses need to understand whether data inputted into the AI system will be retained and used to train the model, because that creates risks around confidentiality, data protection and potential leakage of commercially sensitive information.

So, what’s the solution then? Contracts should clearly state how the AI system may use your data and should expressly prohibit the supplier from using your data to train its own (or third-party!) models unless you expressly agree otherwise.

Also, check that inputs, outputs and any top-up training data are treated as your confidential information. When reviewing your contract with an AI system operator, you should check the confidentiality, data protection/security, and post-termination provisions to make sure your data is actually protected as intended. Where possible, you should also consider local deployment (such as having a “closed” version of the AI system for use only by your business) or other technical arrangements that ensure your data never leaves your own environment.

3. Intellectual property rights and ownership of AI outputs

What’s the risk again? Where AI-generated outputs are commercially valuable, businesses need to understand whether they can actually own, reuse, or licence them – Otherwise they may invest in AI-generated outputs without having the rights they need to actually benefit from them.

So, what’s the solution then? The contract should expressly address the allocations of rights with respect to outputs generated by the AI system, whether that involves ownership of the outputs or rights to use, modify, licence and commercialise them. Intellectual property rights ownership of the following should be dealt with separately:

• The AI system operator’s rights in the pre-existing model;
• Your inputs and prompts;
• Your training data; and
• The outputs.

Depending on the use case, you may require full ownership, but a sufficiently broad licence may be adequate… The point is that the position must be fit for your purposes.

You should also consider seeking indemnities for third-party IP claims, particularly where there is uncertainty over the origins of training data or the possibility that outputs may reproduce protected material. That is especially important where open-source (sometimes referred to as “copyleft”) material may have been used in training, since that can create additional downstream licensing risk.

4. On what data was the AI system trained?

What’s the risk again? If the training data used by the AI provider includes copyrighted works, personal data, or confidential information obtained without proper permission, users of the system and its outputs may face infringement claims or other actions being brought against them.

So, what’s the solution then? You should seek express warranties from the AI system operator that the training data has been lawfully obtained and that they have the necessary rights, licences, and permissions to use it – See the indemnity option above!

In addition, you should ensure responsibility is clearly allocated if something goes wrong, including where the AI system operator seeks to argue that poor customer-provided training data is an “excusing cause” of a problem. If personal data is involved, you should consider whether a Data Protection Impact Assessment (DPIA) is required (remembering that current ICO guidance indicates that a DPIA is often required where AI is involved) and ask the AI system operator for sufficient information to support that assessment. Sometimes the AI system operator may supply you with a template DPIA but, if you are the data controller, remember that it is still your obligation to carry out your own DPIA, rather than accepting assurances at face value.

5. Automated decision-making

What’s the risk again? Where AI outputs influence or determine decisions affecting individuals, particularly without meaningful human involvement, businesses need safeguards to address bias, transparency and compliance, particularly if such decisions meet the threshold for “automated decision-making”.

So, what’s the solution then? You should first identify where AI is involved in decision-making processes and ensure that transparency, human intervention and other appropriate safeguards are built into the relevant processes. Where such decisions amount to “automated decision-making”, additional safeguards may be needed. You should also ensure that there is a genuine “human in the loop”, decision-making processes are appropriately documented, that outcomes can be explained where necessary, and that responsibility for decisions remains clearly allocated and understood.

This should also be documented in an AI risk assessment (see above), so you can evidence why the technology is being used, whether it is proportionate, and what accountability sits with your business and what accountability sits with the provider.

6. Is the AI system safe and secure?

What’s the risk again? AI systems can create safety and security risks including inaccurate outputs, data leakage, model manipulation, cyber misuse, and deepfakes. These risks are particularly acute where a business remains the data controller.

So, what’s the solution then? You should put in place both governance measures and AI-specific contractual controls. These could include acceptable use restrictions, staff guidance, content verification, monitoring, and breach notification/response procedures. You should also investigate the supplier’s technical and organisational security measures, require evidence of standards, and insist on appropriate breach reporting obligations in your contract with them.

It is also sensible to check how vulnerabilities will be monitored and patched, whether sub-processors are involved, and whether the supplier’s liability and insurance position provides a realistic mechanism of redress for you if something goes wrong.

7. “Model drift”

What’s the risk again? AI systems can become less reliable over time as underlying data patterns shift, meaning that a system which initially performed well may later cease to meet needs.

So, what’s the solution then? You should require mechanisms to monitor the AI system’s performance throughout the contract term to ensure that it continues to meet agreed standards. That may include ongoing testing, reporting, performance thresholds, and specific remedies if performance starts to go downhill.

Depending on the use case, it may also be appropriate to require the provider to carry out regular audits, retraining, testing, and fine-tuning on a continuing basis, with service credits, remediation obligations or termination rights available in the contract if the required standard is not maintained. Fortunately, this is not an entirely new concept. Software and technology providers are often expected to maintain service levels, remedy defects and manage updates, and similar principles should apply to AI systems.

8. AI systems used by suppliers to your business

What’s the risk again? Even if a business uses AI responsibly, risk may still arise where its suppliers use AI systems in performing services, especially if they use public AI tools to process the business’ confidential or customer information.

So, what’s the solution then? This is another area where due diligence matters. Before contracting with a supplier, you should ask whether they use AI systems in delivering their services, what type of systems they use, how those systems work, whether data is retained for training, and what policies they have around employee use of AI.

Once these are identified, they should be addressed expressly in the contract through confidentiality obligations, restrictions on the supplier’s use of your data in AI systems, and subcontracting/third party controls. Depending on the risk profile, you may also want consent rights over subcontracting or change of control, direct NDAs with subcontractors, or clear flow-down obligations for confidentiality and data protection set out too.

9. Using AI systems for marketing

What’s the risk again? AI-generated marketing material can create particular risk in the consumer context if it is misleading, inaccurate, contains hallucinated claims, or fails to disclose AI-generated content where required.

So, what’s the solution then? You should ensure that consumer-facing AI outputs are subject to your review (by a human!) before publication and that the use of AI in marketing is assessed against applicable consumer, advertising and transparency requirements. That includes checking the factual accuracy of content, avoiding unsupported claims, and considering whether disclosure of AI-generated content is required in the relevant context.

This is also an area where internal training can make a real difference. Staff should understand that AI-generated outputs cannot simply be treated as ready for publication – You have to actually check it personally for accuracy, IP, confidentiality, and compliance issues!

10. AI systems usage, audit and logging

What’s the risk again? Because AI systems can operate as “black boxes”, inadequate logging and auditability can make it difficult for a business to prove compliance with applicable laws, investigate incidents, or explain how an outcome was reached.

So, what’s the solution then? Transparency and accountability are central to maintaining trust with your clients/customers, but also vital to demonstrate regulatory compliance. You should ensure from the outset that you have adequate logs, audit rights, and information rights so you can investigate incidents, evidence compliance, and respond effectively if challenged by a regulator or client/customer.

In practice, this may mean requiring supplier reporting on security incidents, bias, inaccurate performance, hallucinations, or infringing outputs, as well as “human in the loop” review and testing, so that your business is not left unable to reconstruct what happened after the event if something goes wrong.

What are the next steps?

Whilst AI technology continues to develop, many of the protections needed to address AI risk are familiar. Your business will undoubtedly already undertake due diligence on technology suppliers, negotiate contractual protections, manage data protection and cyber security risks, and monitor supplier performance. AI introduces additional considerations, but many of the underlying principles remain the same. Before implementing or procuring an AI system, you should consider:

• Careful due diligence;
• Clear contractual drafting;
• Effective governance;
• Sensible risk allocation; and
• Ongoing monitoring.

Businesses that establish appropriate governance, contractual protections and oversight now will be far better placed to realise the benefits of AI while managing the associated legal and operational risks.

If you’re thinking of engaging an operator for the use of AI systems in your business, would like advice on what protections to ask for, or have questions about an agreement you are considering signing, please get in touch with Kiran Chita at kiran.chita@fsp-law.com or please contact our Commercial, IP & Technology Team.