I want to use an AI system for work. What are the risks? (Part 1)

Although the use of AI systems is becoming increasingly normalised in commercial uses, these innovations come with new risks and responsibilities to which businesses should pay attention.

It’s crucial for businesses to understand the opportunities that these technologies can bring, as well as the legal, operational, and (occasionally) ethical issues that they can introduce into workflows.

On the back of these considerations, there are also updates to regulations to bear in mind, for example, the ICO clarifying that a Data Protection Impact Assessment (DPIA) is required when introducing new AI systems – Which means that businesses should in turn consider developing firm-wide policies, implement staff training, and embedding AI system use risk assessments in their processes.

Below, we explore some of the specific risks that may arise for businesses when engaging with an operator for the use of an AI system. Our (non-exhaustive!) list below hopefully provides some guidance on some easy mistakes, so that you can keep an eye out for exposures relevant to your business:

1. How does the AI system actually work? Do your due diligence!

When engaging an operator for the use of an AI system, it is of course a good idea to carry out some due diligence as to how the tool is going to work for your business. You should consider the following risks:

1. Explainability: Can the operator of the AI system explain how it operates and why it produces a particular outcome? This will give you an idea as to how well the operator knows their product, and accordingly, whether they know it is the right product for you.
2. Updates & Change Notification: Similarly, AI systems are typically updated frequently, meaning the version you procure may be very different a few months later. Without a mechanism for change notification in respect of these updates, you may receive and rely on outputs generated by a materially different product.
3. Opacity: Whilst opacity is a risk that is particularly pronounced with AI systems, it is not a particularly new issue, and is similar to traditional software licensing concerns around updates and modifications. However, the consequences may be greater due to the potential impact of even minor model changes on the outputs that your business may use.
4. Data use and training

Your business should understand whether input data is retained by the AI system and so used to further train the model. If data belonging to your business is allowed to be used to improve the AI system, risk may arise around confidentiality, data protection, or disclosure of commercial or competitively sensitive data. Especially where output of the AI system provided to a third party or competitor during their use of it has been based on data provided by your business.

3. Intellectual property rights and ownership of AI outputs

Depending on the way in which your business needs to rely on the output of the AI system, you should consider the extent to which any intellectual property rights arising in such outputs can be protected. Where outputs are commercially valuable, a business may invest heavily in deliverables containing AI generated elements but lack the proper rights to those deliverables, limiting its ability to reuse or licence them.

4. On what data was the AI system trained?

AI systems are trained on huge amounts of data, but the provenance of that data is often opaque. If the data used to train an AI system included works subject to copyright, personal data, or confidential materials obtained without the proper permissions, users of that AI system and the outputs it generates may find themselves exposed to infringement claims brought by a third party. Reports of a recent case highlight this risk – An action has been brought against Google in New York by a group of publishers, including Cengage Learning, Elsevier, and Hachette Book Group, in which it is alleged that Google has used millions of books subject to copyright in order to train the Gemini AI systems.

5. Automated decision-making

Where your business uses the outputs from AI systems internally, you should pay attention to whether those outputs influence or directly determine business decisions, including those affecting individuals. Where decisions that affect individuals arise from solely automated decision-making (namely those without meaningful human involvement) there are various safeguards that must be in place. For example, output from an AI system used in recruitment could automatically filter candidates in a way that introduces bias. It is therefore important to understand whether and where any automated decision-making may be present in your business, and ensure transparency, human intervention and other appropriate safeguards are in place.

6. Is the AI system safe and secure?

AI systems may create safety and security risks, including inaccurate or misleading outputs, data leakage (especially where confidential information is involved), model manipulation, cyber misuse, impersonation, and the creation of deepfakes or other synthetic content. If you are using AI systems in your business, you must consider appropriate risk management strategies to mitigate these. This is particularly critical when your business remains the data controller, as liability sits with you even where processing is outsourced.

7. “Model drift”

AI systems can degrade over time due to “model drift”, this is where outputs become less reliable over time as the underlying data patterns shift or change. For example, an AI-based fraud detection system that is trained on historical data may become less reliable as fraud methodologies evolve leading to an increasing number of missed threats as the underlying data becomes increasingly outdated.

8. AI systems used by suppliers to your business

In addition to use of AI systems internally, businesses may also be exposed to risk when suppliers with whom they engage also use AI systems. For example, if an outsourced services provider uses a “public” AI system to process customer information on behalf of a business, it may cause that business to be in breach of the confidentiality or data protection obligations contained in its contracts with its customers.

9. Using AI systems for marketing

If your business uses an AI system to create marketing materials, this may create a risk in the context of consumers. AI-generated marketing output will not comply with the applicable laws if it is misleading or inaccurate, or includes hallucinated claims, unfounded product representations or failures to disclosure AI-generated content where required.

10. AI systems usage, audit, and logging

As mentioned, key issues arising in the use of AI systems are the lack of transparency and the difficulties in explaining how outputs are generated. AI systems are often said to operate as “black boxes”, but such operation is often not compatible with the requirements imposed by the variety of laws applicable to businesses. This is especially notable in the context of maintaining records and logs for auditing purposes. The final risk on our list is to consider your use of AI systems and how it affects the ability of your business to maintain adequate records of compliance and accountability – If the way in which your AI system operates affects your ability to do this to a satisfactory degree, you may fall short in proving compliance, or find that you are not able to investigate incidents effectively.

In Part 2, we will discuss some of the things you can do to mitigate some of these risks, but in the meantime: If you’re thinking of engaging an operator for the use of their AI system(s) in your business, would like some more detailed advice on things you should look out for, or have questions about an agreement you’re thinking of signing, please get in touch with Kiran Chita at kiran.chita@fsp-law.com or please contact our Commercial, IP & Technology Team.