Activities of Patrick BREYER related to 2020/2016(INI)
Shadow reports (1)
REPORT on artificial intelligence in criminal law and its use by the police and judicial authorities in criminal matters
Shadow opinions (1)
OPINION on artificial intelligence in criminal law and its use by the police and judicial authorities in criminal matters
Amendments (38)
Amendment 1 #
Draft opinion
Recital A
Recital A
A. whereas the right to fair trial is a fundamental right which also applies tofundamental rights and freedoms enshrined in the Charter of Fundamental Rights of the European Union, such as the respect for private life, the protection of personal data, the freedom of thought, conscience and religion, the freedom of assembly and of association, but also equality before the law, such as non-discrimination, as well as citizen's rights, such as the freedom of movement and of residence, the right to a fair trial and the presumption of innocence and right of defence should not be undermined by the use of AI and related technologies and apply to the enforcement of the law in all circumstances;
Amendment 1 #
Motion for a resolution
Citation 1
Citation 1
— having regard to the Treaty of the European Union, in particular Articles 2 and 6, and the Treaty on the Functioning of the European Union, in particular Article 16,
Amendment 2 #
Motion for a resolution
Citation 2
Citation 2
— having regard to the Charter of Fundamental Rights of the European Union, in particular Article 6, Article 7, Article 8, Article 11, Article 12, Article 13, Article 20, Article 21, Article 24, and Article 47 thereof;
Amendment 4 #
Motion for a resolution
Citation 4
Citation 4
— having regard to the Council of Europe Convention for the Protection of Individuals with regard to Automatic Processing of Personal Data (ETS 108), and its amending protocol (“Convention 108+”);
Amendment 9 #
Motion for a resolution
Citation 12 a (new)
Citation 12 a (new)
- having regard to the hearing in the Committee on Civil Liberties, Justice and Home Affairs (LIBE) on 20 February 2020 on Artificial Intelligence in criminal law and its use by the police and judicial authorities in criminal matters;
Amendment 10 #
Draft opinion
Recital A a (new)
Recital A a (new)
A a. Whereas AI and related technologies, including their self-learning abilities, always involve a certain level of human intervention;
Amendment 11 #
Motion for a resolution
Citation 12 b (new)
Citation 12 b (new)
- having regard to the report of the LIBE mission to the United States in February 2020;
Amendment 12 #
Motion for a resolution
Citation 12 c (new)
Citation 12 c (new)
- having regard to its Resolution of 19 June 2020 on the anti-racism protests following the death of George Floyd1a; _________________ 1a P9_TA(2020)0173
Amendment 13 #
Draft opinion
Recital B
Recital B
Amendment 13 #
Motion for a resolution
Citation 12 d (new)
Citation 12 d (new)
- having regard to its Resolution of 14 March 2017 on fundamental rights implications of big data: privacy, data protection, non-discrimination, security and law-enforcement1a; _________________ 1a P8_TA(2017)0076
Amendment 16 #
Motion for a resolution
Recital A
Recital A
A. whereas digital technologies in general and artificial intelligence (AI) in particular bring with them extraordinary promise; whereas AI is one of the strategic technologies of the 21st century, generating substantial benefits in efficiency, accuracy, and convenience, and thus bringing positive change to the European economys and risks;; whereas AI should not be seen as an end in itself, but as a tool for serving people, with the ultimate aim of increasing human well-being;
Amendment 27 #
Motion for a resolution
Recital B
Recital B
B. whereas the development of AI must respect the values on which the Union is founded, in particular human dignity, freedom, democracy, equality, the rule of law, and human and fundamental rightfundamental rights and freedoms enshrined in the Charter of Fundamental Rights of the European Union, such as the respect for private life, the protection of personal data, the freedom of thought, conscience and religion, the freedom of assembly and of association, but also equality before the law, such as non-discrimination, as well as citizen's rights, such as the freedom of movement and of residence, the right to a fair trial and the presumption of innocence and right of defence should not be undermined by the use of AI and related technologies and apply to the enforcement of the law in all circumstances;
Amendment 33 #
Draft opinion
Paragraph 1
Paragraph 1
1. Emphasises the high importance of duly assessing the risks and considering all the ethical and operational implications of the use of AI and related technologies in our society, in particular by State authorities, the police and judicial authorities in criminal matters, as well as within criminal justice systems;
Amendment 35 #
Draft opinion
Paragraph 1 a (new)
Paragraph 1 a (new)
1 a. Stresses the need to work on the most efficient way of reducing bias in AI systems, in line with ethical and non- discrimination standards; underlines that the outputs should be reviewed to avoid all forms of stereotypes, discrimination and biases and when appropriate, make use of AI to identify and correct human biases when that might exist; calls on the Commission to encourage and facilitate the sharing of de-biasing strategies for data, in particular in the field of law enforcement in criminal matters; in the light of these risks of bias in AI systems, further calls on the Commission to declare a ban on the use of AI and related technologies for the assistance of judicial systems and of judicial decisions;
Amendment 37 #
Motion for a resolution
Recital D
Recital D
D. whereas the Union together with the Member States bear a critical responsibility for ensuring that policy choices surrounding the development, deployment and use of AI applications in the field of the judiciary and law enforcement are made in a transparent manner, respect the principles of necessity and proportionality, and guarantee that the policies and measures adopted will fully safeguard fundamental rights within the Union, in particular that any AI applications do not perpetuate existing discriminations, biases or prejudices;
Amendment 38 #
Draft opinion
Paragraph 1 b (new)
Paragraph 1 b (new)
1 b. Calls on the Commission to further clarify the rules on the protection and sharing of the data collected through AI and related technologies by authorised authorities habilitated to collect and/or process such data, including non-personal and anonymised data that directly or indirectly identify persons, in full respect of the GDPR and of the ePrivacy Directive; further underlines that the right to a fair trial should involve the ability for citizens and litigants to access these data, especially when the latter are collected from their personal devices or equipment, in accordance with the GDPR, but also for the purpose of their right of defence as soon as their legal liability is engaged;
Amendment 41 #
Draft opinion
Paragraph 2
Paragraph 2
2. Underlines the importance of being able tofor the developers of AI and related technologies to provide for a sufficient level of transparency of algorithms and of algorithmic decisions towards competent authorities and citizens, and to enable access to AI-produced or AI-assisted outputs, particularly for notification procedures and the role of AI and related technologies in criminal law enforcement and crime prevention; recalls the importance of questions related to governance, transparency and accountain criminal matters; recalls the importance of questions related to governance, transparency and accountability, while ensuring human oversight at all times; insists on the obligation of the judicial authorities to motivate their decisions, including when using elements of proof provided by AI assisted technologies,the latter requiring the a high level of judicial scrutiny and of admissibility;
Amendment 46 #
Motion for a resolution
Recital E
Recital E
E. whereas AI applications may offer great opportunities in the field of law enforcement, in particular in improving the working methods of law enforcement agencies and judicial authorities, and combating certain types of crime more efficiently, in particular financial crime, money laundering and terroris; whereas any blanket application of AI to all financial or other transactions without financing, as well as certain types of cybercrimitial suspicion would be disproportionate;
Amendment 49 #
Motion for a resolution
Recital F
Recital F
F. whereas a clear model for assigning legal responsibility for the potential harmful effects of AI systems in the field of criminal law is imperative; whereas the first and foremost aim must be to avoid that any harmful effects materialise to begin with;
Amendment 51 #
Draft opinion
Paragraph 2 a (new)
Paragraph 2 a (new)
2 a. Calls on all competent public authorities, especially law enforcement authorities like the police and the judiciary, to inform the public and provide sufficient transparency as to their use of AI and related technologies when implementing their powers, especially in criminal law matters, including public access to the source code as well as disclosure of false positive and false negative rates of the technology at hand;
Amendment 54 #
Motion for a resolution
Recital G
Recital G
G. whereas AI applications in use by law enforcement include applications such as facial recognition technologies, automated number plate recognition, speaker identification, speech identification, lip-reading technologies, aural surveillance (i.e. gunshot detection algorithms), autonomous research and analysis of identified databases , forecasting (predictive policing and crime hotspot analytics), behaviour detection tools, autonomous tools to identify financial fraud and terrorist financing, social media monitoring (scraping and data harvesting for mining connections), international mobile subscriber identity (IMSI) catchers, and automated surveillance systems incorporating different detection capabilities (such as heartbeat detection and thermal cameras); whereas the aforementioned applications have vastly varying degrees of reliability and accuracy, accuracy, and effectiveness; whereas many of these tools are used in third countries, but would be illegal under the Unions data protection legislative framework and case-law; whereas the routine deployment of algorithms, even with a small false positive rate, can result in false alerts by far outnumbering correct alerts;
Amendment 55 #
Draft opinion
Paragraph 3
Paragraph 3
Amendment 60 #
Motion for a resolution
Recital H
Recital H
H. whereas AI tools and applications are also used by the judiciary worldwide, including in sentencing, calculating probabilities for reoffending and in determining probation; whereas this has led to distorted and diminished chances for people of colour and other minorities;
Amendment 63 #
Draft opinion
Paragraph 3 a (new)
Paragraph 3 a (new)
3 a. Calls on the Commission to implement, through legislative and non- legislative means and if necessary through infringement proceedings, a ban on any biometric processing of personal data for law enforcement purposes that leads to mass surveillance in public spaces; Calls on the Commission to stop funding biometric research or deployment which could contribute to mass surveillance in public spaces;
Amendment 66 #
Draft opinion
Paragraph 4
Paragraph 4
Amendment 68 #
Motion for a resolution
Recital I
Recital I
I. whereas use of AI in law enforcement entails a number of potential risks, such as opaque decision-making, a chilling effect, different types of discrimination, and risks to the protection of privacy and personal data, the protection of freedom of expression and information, and the presumption of innocence, and the right to an effective remedy and a fair trial;
Amendment 75 #
Motion for a resolution
Recital J
Recital J
J. whereas AI systems used by law enforcement are also vulnerable to AI- empowered attacksattacks against information systems ; whereas in these situations the resulting damage is potentially even more significant, and can result in exponentially greater levels of harm to both individuals and groups;
Amendment 76 #
Draft opinion
Paragraph 4 a (new)
Paragraph 4 a (new)
4 a. In the light of all the risks exposed, as long as fundamental rights and freedoms, as well as human review are not fully guaranteed, calls on the Commission to declare a ban on the use of AI and related technologies for the assistance of judicial systems and of judicial decisions;
Amendment 96 #
Motion for a resolution
Paragraph 2 a (new)
Paragraph 2 a (new)
2 a. Considers that AI applications used by police and judicial authorities have to be categorised as high-risk in all cases given their public role and responsibility and the impact of decisions taken by such authorities; considers furthermore that the use of AI system by such authorities can occur in a way that might have legal consequences or may significantly affect people’s lives;
Amendment 104 #
Motion for a resolution
Paragraph 3 a (new)
Paragraph 3 a (new)
3 a. Underlines that any decision about a natural person that is based solely on automated processing, including profiling, and which produces an adverse legal effect concerning the data subject or significantly affects him or her, is prohibited under Union law, unless authorised by Union or Member State law which at least provides for the right to obtain human intervention; reminds that decisions in the field of law enforcement are almost always decisions that have a legal effect on the person affected, due to the executive nature of law enforcement authorities and their actions; calls on the Commission, the European Data Protection Board and other independent supervisory authorities to propose legislation or at least issue guidelines, recommendations and best practices in order to further specify the criteria and conditions for decisions based on profiling and the use of AI for law enforcement purposes;
Amendment 111 #
Motion for a resolution
Paragraph 4
Paragraph 4
4. Highlights the importance of preventing mass surveillance by means of AI technologies, and of banning applications that would result in it;Considers data collected and monitoring of individuals shall be limited to suspects of criminal acts, or suspects in preparation of committing a crime as opposed to automated, and indiscriminate data collection; Highlights the importance of preventing mass surveillance by means of AI technologies, and of banning applications that would result in it; calls on the Commission to implement, through legislative and non-legislative means and if necessary infringement proceedings, a ban on any biometric processing of personal data for law enforcement purposes that leads to mass surveillance in public spaces; calls on the Commission to stop funding biometric research or deployment which could contribute to mass surveillance in public spaces; reminds that mass surveillance by third countries’ authorities implies that their level of data protection is not adequate, as confirmed by the Court of Justice in its judgment on Schrems I1a; _________________ 1aJudgment in case C-362/14, 6 October 2015
Amendment 120 #
Motion for a resolution
Paragraph 5
Paragraph 5
5. Stresses the potential for bias and discrimination arising from the use of machine learning and AI applications; notes that biases can be inherent in underlying datasets, especially when historical data is being used, introduced by the developers of the algorithms, or generated when the systems are implemented in real world settings; reminds of its Resolution of of 19 June 2020 on the anti-racism protests following the death of George Floyd; points out that wide-spread real-world racism in the police forces is still prevalent; underlines that such racism will inevitably lead to racist bias in AI-generated findings, scores, and recommendations; therefore reiterates its call on Member States to promote anti-discrimination policies in all areas and to develop national action plans against racism, including in policing and in the justice system; in close cooperation with civil society and the communities concerned;, and to step up measures to increase diversity within police forces and to establish frameworks for dialogue and cooperation between police and communities;
Amendment 125 #
Motion for a resolution
Paragraph 5 a (new)
Paragraph 5 a (new)
5 a. Points out that several cities in the United States have ended their predictive policing systems, after audits of programs that attempted to predict individuals’ behaviors in Chicago and Los Angeles proved their discriminatory impact and practical failure; points out that place- based predictive systems have been shut down in Los Angeles and other cities that initially had adopted the technology; reminds that during the LIBE Committee’s mission to the United States in February 2020, Members were informed by the police departments of New York City and Cambridge/Mass that they had phased out their predictive policing programmes due to a lack of effectiveness and have turned to community policing instead; reminds that this lead to a decline in crime rates;
Amendment 144 #
Motion for a resolution
Paragraph 9
Paragraph 9
9. Considers it necessary to create a clear and fair regime for assigning legal responsibility for the potential adverse consequences produced by these advanced digital technologies; underlines however that the first and foremost aim must be to avoid that any such consequences materialise to begin with; calls for the consequent application of the precautionary principle for all applications of AI in the law enforcement context;
Amendment 154 #
Motion for a resolution
Paragraph 10
Paragraph 10
10. Underlines that in judicial and law enforcement contexts, the final decision always needs to be taken by a human, who can be held accountable for the decisions made, and include the possibility of a recourse for a remedy; points out however that if humans only rely on the data, profiles and recommendations generated by machines, they will not be able to do an independent assessment; is of the opinion that the sovereign discretion of judges, and decision-making on a case-by-case basis has to be upheld; calls on the Commission to declare a ban on the use of AI and related technologies for the assistance of judicial systems and of judicial decisions;
Amendment 161 #
Motion for a resolution
Paragraph 11
Paragraph 11
11. Calls for algorithmic explainability and transparency in order to ensure that the development, deployment and use of AI systems for judiciary and law enforcement comply with fundamental rights, and are trusted by citizens, as well as in order to ensure that results generated by AI algorithms can be rendered intelligible to users and to those subject to these systems, and that there is transparency on the source data and how the system arrived at a certain conclusion; points out that in order to ensure technical transparency, robustness, and accuracy, such tools should be released in open source software and should provide documentation in clear intelligible language about the nature of the service, the tools developed, the performance and conditions under which they can be expected to function and the risks that they might cause;
Amendment 188 #
Motion for a resolution
Paragraph 15
Paragraph 15
15. Calls for a moratorium on the deployment of facial recognition systems for law enforcement, until the technical standards can be considered fully fundamental rights compliant, results derived are non-discriminatory, and there is public trust in the necessity and proportionality for the deployment of such technologiesReminds of the scandal around the revelations that law enforcement agencies in several countries are using ClearView face recognition technology where the database was filled with faces of persons scraped illegally from social networks and other parts of the internet;
Amendment 192 #
Motion for a resolution
Paragraph 16
Paragraph 16
16. Calls for greater overall transparency from Member States, and for a comprehensive understanding of the use of AI applications in the Union, broken down by Member State law enforcement and judicial authority, the type of tool in use, the types of crime they are applied to, and the companies whose tools are being used; calls on all competent public authorities, especially law enforcement authorities like the police and the judiciary, to inform the public and provide sufficient transparency as to their use of AI and related technologies when implementing their powers, especially in criminal law matters, including public access to the source code as well as disclosure of false positive and false negative rates of the technology at hand;