Out-Law / Your Daily Need-To-Know
Financial services providers can turn to transparency, good governance and testing to reduce the risk of things going wrong with artificial intelligence (AI) systems, but should be prepared to take responsibility for issues even if they are not at fault.
Liability can arise in a number of areas, such as intellectual property rights (IPR) ownership and infringement, data protection claims, discrimination and bias issues, and financial loss to customers.
The challenge of identifying responsibility for fault
In the UK, there is no overarching legal framework in relation to AI, but some issue-specific laws on automated processes have been implemented, such as the Automated and Electric Vehicles Act 2018, which attributes liability for damage caused by insured automated vehicles to the insurer, and specific provisions on automated decision making in the Data Protection Act 2018.
As AI is increasingly being used across a number of sectors and for a variety of purposes, the development of sector specific laws regulating AI development and use may be more appropriate than a general AI legal regime. A sector-focused approach will enable the implementation of laws which not only address the issues associated with AI but may also ensure that such laws are relevant and applicable at a practical level for the sectors seeking to rely on them. This would also allow the relevant regulator, such as the FCA in the financial service sector, to have oversight and give more specific guidance and direction on AI adoption and use.
As AI becomes increasingly autonomous and removed from human decision making, it can be difficult to establish who will be at fault if AI causes damage, particularly given the number of parties typically involved in the development and operation of AI tools – from developers, to data providers, manufacturers, and users. Further difficulties arise where loss results from machine learning systems which learn to make decisions independently and without human intervention.
Risks of liability for AI in financial services
It is important for financial services businesses to consider how AI is to be implemented into business processes and to assess the potential liabilities that may arise. Businesses should look to future-proof contracts with technology providers and customers as much as possible. Contracts should be thoroughly considered to ensure appropriate apportionment of liability.
Luke Scanlon
Head of Fintech Propositions
Algorithmic errors, insufficient or inaccurate data, and lack of training of systems and AI users, could result in bad decisions leading to financial loss for both the financial services businesses themselves and their customers
When future-proofing contracts, the various types of loss and claims which may arise through AI adoption should be considered. The use of AI can give rise to losses in a number of areas including, for example, in circumstances where it is embedded in drones or machines, for personal injury and product liability. However, these scenarios are less likely to be of significance to many financial services businesses which use AI technology as part of digital solutions.
In a digital financial services context, the following may be of particular concern:
Economic loss
From chatbots and insurance quotes, to assessing investment portfolios and analysing market trends, businesses are increasingly relying on AI technology in back office operations and to provide services to its customers and assist with making financial decisions. Algorithmic errors, insufficient or inaccurate data, and lack of training of systems and AI users, could result in bad decisions leading to financial loss for both the financial services businesses themselves and their customers.
Data protection claims
An AI solution may rely on data, whether that is personal, non-personal or a combination of both. Businesses need to ensure that they lawfully collect and use data, and in particular, personal data, in a way which is compliant with data protection laws – including the General Data Protection Regulation (GDPR) and Data Protection Act 2018 – to minimise the risk of customers raising data protection claims and regulatory fines.
Where data collection and use is seen to be unlawful, financial services businesses using AI to collect data from customers may face enforcement action from the Information Commissioner's Office (ICO). Infringement could result in fines under the GDPR of up to €20 million or 4% of a firm's worldwide annual revenue, whichever is higher.
Security breaches and data loss
A failure to implement adequate security measures to protect data can lead to corruption, leaks and losses of significant volumes of customer data which in turn may lead to customer complaints and a right to compensation. The ICO in its draft AI auditing framework highlighted two security concerns which may be heightened in the AI context.
The first is the extent to which AI is dependent on the use of third party frameworks and code libraries and the supply chain security issues this creates. Machine learning technologies often require access to large third party code repositories with the ICO's study finding that one popular machine learning development framework included "887,000 lines of code" and relied on "137 external dependencies".
The second is around the use of open source code. While open source is a necessary and valid option, consideration must be given to the liability implications of finding security vulnerabilities when they are used.
IPR ownership and infringement
Disputes over ownership of IP generated by AI technologies may arise along with claims in relation to infringement by AI of third party IPR.
Rights of ownership may be not be clear in respect of the standard legal categories of protection for confidential information, know-how and copyright where one party provides the data and the other the algorithm, and in relation to patent rights where machine learning is used to achieve a novel or innovative step.
In respect of IP infringement, a starting position may be an expectation that liability would rest with the legal entity that controls or directs the AI system. However, the position may not be so clear cut where multiple businesses are involved in the process and where AI systems develop to make decisions independently and without human supervision. Where AI technology evolves or improves its processes beyond the original purpose for which a business creates it, it may risk infringing another business’s copyright by using that business’s data as one of its inputs. Attributing liability may be less challenging where a business is able to trace back how decisions were made by AI but this is complicated by the existence of 'black box AI' where AI decision making cannot be explained.
Bias and discrimination
Decisions which are based on insufficient or low quality data may produce biased outcomes. These outcomes can lead to complaints from consumers and requests for decisions to be retaken, particularly where the decisions put individuals at a disadvantage. This might include, for instance, where customers face paying higher insurance premiums due to their location or gender. Where this is the case, financial services businesses should consider how to minimise the risk of discriminatory outputs. Considerations such as whether the biased outputs could have been prevented through testing of algorithms and quality checks on data may be relevant when attributing the level of liability.
Reducing AI risk
While an AI technology provider may be willing to stand behind the performance of its algorithm, it may not be willing to give its customers comfort that the AI system as a whole, after learning from the customer's data, will not result in outputs that cause loss. As regulatory accountability will ultimately lie with the financial services business, so risk control measures other than the transfer of risk to the AI technology provider may need to be put in place.
Examples include:
Explainable AI
The use of explainable AI can assist with tracing back to how a decision was made and where the process went wrong. Businesses should weigh the benefits of using AI systems which produce unexplainable decisions against the risks which arise from doing so, including ultimately being liable for losses even where it is unclear how it was caused.
Keeping clear, accurate and up to date records of the data used, in accordance with legal requirements, how AI systems are trained and tested, and perhaps the decision making processes themselves where possible, might enable businesses to not only trace back how a decision was made through assessing decision making patterns and an AI's behaviour, but also to step in and resolve issues before they arise, and determine at which point something went wrong. Traceability facilitates accountability and auditability.
Data protection
Businesses are required to carry out data protection impact assessments (DPIA) prior to using AI to ensure that the processing of personal data is lawful and complies with data protection law. DPIAs may also assist with pinpointing whether systems are 'high risk' and therefore can help businesses prepare for the level of data protection risk and liability that they could be taking on. Businesses should also ensure that all necessary consents are obtained and fair processing notices are issued prior to processing personal data.
Security
Businesses should ensure that adequate technical and organisational security measures are implemented to protect customer data, and address the risk of data loss and AI algorithms being hacked and tampered with. Security should be built into the technology and the overall process at the outset rather than considered as a secondary measure. A review of existing organisational security processes should be conducted to test whether measures already in place are sufficient for AI use.
Human intervention
Ensuring that there is always human input in the AI lifecycle, and that it is the appropriate or right level of human input, is important. Exercising control over the AI system can help mitigate issues which may give rise to liability, such as poor or inaccurate decisions, and discrimination. A European Commission's expert group has recommended asking:
- did you consider the task allocation between the AI system and humans for meaningful interactions?;
- are there safeguards to prevent overconfidence in or overreliance on the AI system for work processes?; and
- which detection and response mechanisms did you establish to assess whether something could go wrong?
Audit
Financial services businesses should ensure that they have in place a robust process for auditing AI technologies. AI audits may focus on quality checking AI outputs, assessing the risks with using data, and continuous monitoring of AI technologies and those operating the technology. Developing an AI audit framework may entail:
- re-assessing existing audit processes to check that they remain fit for purpose and adequately address the specific risks AI poses. Using AI can create new risks or make it more difficult to identify risks, generally;
- reviewing and amending (where necessary) access and audit provisions within contracts to ensurethe real risks that AI poses are covered; and
- engaging with external auditors to allow for independent assessments of the AI technology.
Insurance
Customers will want to know that the businesses that they are dealing with will have sufficient resources to make good any claims they have against them. Questions arise as to whether insurers have developed products to cover risks that the use of AI can give rise to.
In a similar way to which businesses obtain traditional insurance products, there is potential for insurance products to be developed against the consequences of AI and taken out either by the developer of the AI on behalf of its clients, or directly by those using AI within their businesses. This may be particularly important where the failure could have a significant impact on customers.
Bespoke AI insurance policies are not currently commercially available, and AI losses are not explicitly covered by traditional forms of insurance. This means firms using AI should explore and carefully analyse the extent to which existing types of insurance, such as business interruption, cyber, and professional indemnity insurance, cover specific AI related risks.
Complaint-handling
There are varying types of liability which can arise from AI use. Measures can be implemented to mitigate the risks associated with adopting AI, from keeping records, to ensuring systems are subject to regular and thorough testing, and future- proofing contracts.
Unexplainable 'black box' AI creates various challenges and so financial services businesses should think about whether these systems should be used in customer facing environments where there are various duties to customers – including the principle of treating customers fairly and right to be informed under data protection legislation – or if at all.
Ultimately, however, financial services businesses as the first point of contact for the customer and the holder of the customer relationship, must ensure that adequate processes are in place to deal with customer complaints and claims, irrespective of who is responsible for any loss or damage.
Luke Scanlon and Priya Jhakra are experts in fintech law at Pinsent Masons, the law firm behind Out-Law.
Written by
