An interesting review report from Algorithm Watch points out the complete failure of 3 automated systems in use by the Swiss police and judiciary to predict anomalous behaviour. As an example, one of the systems tries to predicts burglaries from past data, based on the assumption that burglars often operate in small areas. Hence, if a cluster of burglaries is detected in a neighbourhood, the police should patrol it more often to put an end to it. It seems that this analysis is inspired by one of the first big data analysis that revealed a cluster of infected people around infected water fountains in London at the beginning of the last century. Another system in use in Switzerland aims to predict the likelihood that a person will assault his/her intimate partner. The cost of each system is close to 100.000 euros.
According to Algorithm Watch this failure is hardly a surprise because it has been anticipated by previous analyses. As an example, a 2019 report by the University of Hamburg has reported the lack of any evidence of the efficacy of predictive policing solutions. Furthermore, the false positive rate and the false negative one of these predictive systems both lie in the range 70–80%. Notice this holds even if fully accurate data on the problem of interest is used. Imagine what could happen if some database is poisoned with some carefully designed data.
The failure of predictive AI points out a too optimistic attitude towards AI techniques and it should ring a bell for all those that believe that anomaly detection per se suffices to discover and signal attacks against an ICT system. Almost all of these systems are deployed under the assumption (hope??) that they can work without information on the target system, the applications and the security policies that have been adopted. The price to pay for the lack of this information are unacceptable rates of false positive and false negative. Experience confirms that these rates are dramatically reduced by taking into account constraints arising because of the system architecture, the interconnection topology and available applications.
Link
https://algorithmwatch.org/en/story/swiss-predictive-policing/
