Explainability and interpretability have become some of the hottest topics in artificial intelligence in recent years, even following the term Explainable AI (XAI). Not surprisingly, after the boom in interest in the uses of AI solutions in almost every industry in the world, there has been a need to understand how such AI solutions work. This is not only to validate their performance, but also to build trust between models and users, especially when used in critical decision-making tasks. If you search the web on this subject, you will find many articles that talk about explainability and many others that talk about interpretability in artificial intelligence. In many occasions, these terms are wrongly used as synonyms. But the reality is that there is an important difference between them and in today’s blogpost I intend to make a brief introduction to understand what each one covers.
Interpretability. We can define interpretability as the ability of a model to be understood by a human. The ability to understand the cause and effect within it. To be able to look at the parameters of the model or a summary of the model and understand exactly why it has made a certain prediction. It is important to note here that this is not a question of whether or not a model is interpretable or not, but rather the interpretability of a model on a spectrum—some models are, by their nature, more interpretable than others. In general, the main problem in developing efficient and highly interpretable models is that they often involve a high cost in computational effort and depend heavily on the availability of accurate domain knowledge.
Explainability. In artificial intelligence, we often use the term black box model to refer to models that are too complex and whose internal behaviour is almost impossible to understand. In this way we are dealing with models with very low interpretability. These are cases where we need to use additional techniques to try to open the black box and understand how the model works. While these solutions help us look inside these models, it is important to keep in mind that explainability can, in some occasions, not be entirely describe how the original model works. Often explanations can be difficult to understand or may not provide enough detail to understand what the black box is doing. This not only adds to the complexity of the whole process, having to explain the explanation of the model, but can also reduce the overall trust in the solution.
What do I need? With a better understanding in mind of interpretability and explainability, the question may arise: what do I need my model to be? And, as an engineer by training, my answer would be: it depends. Of course, the ideal goal would always be to move towards more interpretable models. However, as we have seen, in general, models with a higher degree of interpretability tend to be less powerful. So the particularities of each use case need to be assessed. This problem is often referred to as Interpretability-Accuracy Trade-off. The widely held belief is that for greater interpretability, there may be a trade-off of lower accuracy/performance: in some cases, simpler models may make less accurate predictions. As always this really depends on the problem you are trying to solve. Although much progress has been made in the field of explainability, there is a growing momentum in the belief that a well-defined set of features can make the performance between a highly interpretable model and a very complex one minimal.
As we have seen, explainability is a good thing when it comes to understanding how a black model works, though it has its disadvantages. Of course, in cases where the prediction of a model is not high risk or high cost, such as which movie Netflix recommends you, black box models are perfectly valid. The problem arises when we approach situations where the implications of a prediction can be more severe, such as in the medical field or in Air Traffic Management. It is in these cases where the difference between the two can be critical. In the end, there is no single ideal solution, and the importance of understanding the needs of the problem from the beginning is key to developing a viable solution. At Innaxis, since our beginnings in data-related projects, we have understood this importance and that is why in projects such as Safecloudsand SafeOPS or in the development of the digital assistant for air traffic controllers by DataBeacon, we have always made a special effort to understand the operational problem and to establish the necessary requirements of interpretability and/or explainability of the solutions sought.
This blog may not have answered all your questions, but I hope that at least I have been able to broaden your knowledge in two very important areas of artificial intelligence. Interpretability and explainability are two growing and continuously developing areas, so keep an eye on what the future holds. For more interesting posts on data science in aviation, don’t forget to visit our blog!
