The challenges of using AI in critical systems
Artificial intelligence (AI) is playing an increasingly important role in society, not only in our day-to-day lives, but also in the way critical systems and infrastructure is managed.
Ahead of the AI Action Summit being organised in Paris on 10 and 11 February 2025, Thales has published a position paper on artificial intelligence for critical systems, which details how it is implementing AI and striving to find the right balance between autonomy, performance and safety and security.
We spoke with Juliette Mattioli, Senior Expert in Artificial Intelligence and one of the co-authors of the paper, three questions to help better understand the approach.
We are used to hearing about the use of AI in civilian applications, but how is it being used within critical systems?
AI can be used within critical systems as a complement to human experience and expertise. The technology can analyse huge quantities of data from a variety of sources such as sensors and effectors, combined with business knowledge, much quicker than humans can. Reducing the mental workload of human operators through the delegation of non-critical and – in some carefully controlled instances – even critical functions, enables them to make quicker, better-informed decisions.
What are the challenges of using AI in these environments?
Although AI can greatly improve the speed and efficiency of many processes by suggesting actions to operators, it is vital that humans remain at the centre of the decision-making process.
We also need to ensure that the actions suggested by the AI we implement are valid and explainable, not just after the fact, but also in real time. If a digital co-pilot in the cockpit tells the pilot to make a manoeuvre, the pilot has to understand – quickly, and in easily-comprehensible terms – why the suggestion is being made, in order to be able to decide whether to act on it or not.
Finally, there are the physical challenges related to embeddability. AI incorporated into critical systems, particularly in the military field, often has to meet stringent requirements in terms of size, weight and power consumption. Thales’s engineers have both the technical expertise and the operational knowledge to design AI systems that incorporate those constraints right from the design phase.
In critical systems, security and cybersecurity are fundamental…
AI opens up a whole range of capabilities in the management of critical systems, but it also brings with it new vulnerabilities. We are working constantly to ensure that our AI systems are secure and protected from cyberattacks, be they on the systems themselves or on the data that is used to train the artificial intelligence.
Thales leverages its vast expertise in the cybersecurity field to protect the AI it deploys in critical systems and environments, and we have a team of ethical hackers that work on identifying vulnerabilities in the AI systems we deploy. Their work ensures that the systems are resilient to cyberattacks, thus guaranteeing that our government and enterprise customers implement artificial intelligence that they can trust.
