Risk-based regulation of AI proposed by EU policy makers

Under the proposed new Artificial Intelligence Act (AI Act), AI that poses an “unacceptable risk” to people will be prohibited, while the bulk of the regulatory requirements will apply to ‘high-risk’ AI systems, including obligations around the quality of data sets used, record-keeping, transparency and the provision of information to users, human oversight, and robustness, accuracy and cybersecurity. ‘Low-risk’ AI systems would be subject to limited transparency obligations.

High-risk AI would be subject to both conformity assessment prior to those systems being sold or put into service, as well as a system of post-market monitoring that each systems provider would need to put in place and adhere to.

Compliance with the requirements of the AI Act would also be assessed by national regulators under the Commission’s plans, with companies responsible for the most serious breaches subject to fines of up to €20 million or 4% of their annual global turnover, whichever is higher.

The Commission has proposed to define ‘high-risk’ AI within the AI Act. AI systems that are stand-alone products or used as safety components within a product could fall within the definition.

One of the factors relevant to whether an AI system is characterised as high-risk or not would be the extent of its “adverse impact” on EU fundamental rights.

“Those rights include the right to human dignity, respect for private and family life, protection of personal data, freedom of expression and information, freedom of assembly and of association, and non-discrimination, consumer protection, workers’ rights, rights of persons with disabilities, right to an effective remedy and to a fair trial, right of defence and the presumption of innocence, right to good administration,” according to the Commission.

Requirements around prior conformity assessments are to be linked to the level of oversight AI systems will have in passing existing regulatory hurdles. If a high-risk AI system would already be considered by “professional pre-market certifiers in the field of product safety”, such as is the case with medical devices or vehicles, then providers of those systems would be free to carry out their own conformity assessment prior to the AI system being sold or being put into service in the EU. All other high-risk AI systems would be subject to third party conformity assessment prior to being put on the market or into service.

According to the proposals, a new conformity assessment would need to be undertaken “whenever a change occurs which may affect the compliance of the system with [the AI Act] or when the intended purpose of the system changes”. No reassessment of conformity would be necessary where the only changes made to the algorithm and its performance are those that have been pre-determined by the provider and which are assessed at the original conformity assessment.

“The beauty of AI is that it can learn autonomously,” said data expert Nils Rauer of Pinsent Masons, the law firm behind Out-Law. “The concept of pre-determination therefore runs, at least to certain extent, counter to the idea of artificial intelligence. This means the scope of the conformity assessment exception might be rather limited in practice.”

A new European Artificial Intelligence Board would be established under the draft AI Act and tasked with producing opinions, recommendations and guidance in relation to the regulation, while the creation of national AI regulatory sandboxes is also envisaged “to facilitate the development and testing of innovative AI systems under strict regulatory oversight before these systems are placed on the market or otherwise put into service”.

The AI systems that will be banned under the AI Act include those that deploy “subliminal techniques beyond a person’s consciousness in order to materially distort a person’s behaviour in a manner that causes or is likely to cause that person or another person physical or psychological harm”. Other AI systems which would be prohibited include those designed to exploit and manipulate the behaviour of children or disabled people to cause harm – including “toys using voice assistance encouraging dangerous behaviour of minors”, according to the Commission.

The use of ‘real-time’ remote biometric identification systems in public by law enforcement agencies will also be subject to a general ban, though exceptions apply to enable its use for specific purposes, like combatting a risk of terrorism, and under strict conditions – including judicial oversight. There would also be a ban on certain uses of AI systems by governments and other public bodies for the purposes of “social scoring”.