From the perspective of technology companies, one of the greatest challenges to developing cutting edge technology is ensuring that the innovation budget receives significant recognition among users, regulators and other stakeholders.

Receiving recognition sometimes means publishing papers in peer-reviewed magazines, presenting in conferences and preparing press releases that showcase the advancements in technology. Though companies can sometimes grow to be very “optimistic” of perceived progress, some claims of progress lack the necessary proofs to ensure that developments are relevant.

While AI potential applications can be varied and diverse, making sure that a system is sound, scalable, uses available datasets and can perform in a real-world environment is a daunting task. Sometimes the proposed approaches fail in their methodology as the data is simply not available to support industrialisation of the system; or, the system is not safe and confidentiality concerns remain about how the data will be used.

Verifying AI system development claims is important from all perspectives. For users, it is very important to understand how new technology fits into existing processes, how those processes need to be changed and/or how the roadmap helps with that transition. For regulators in aviation, safety and security concerns should be prominent, including security of data and cybersecurity. For academics, raising and improving upon the state-of-the-art is key to understanding how future research should be driven towards new scientific goals. Finally, for AI companies, understanding if a competitor is progressing the state-of-the-art ahead of them is simply not enough; said companies could help save significant funding and reroute work to make sure they are making the best use of their resources.

Open AI has published the most relevant document on the subject – Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims. A large group of authors at 30 organisations have prepared this very relevant 78 page report as a way of encouraging system users, customers, civil society, governments, and other stakeholders to trust AI developers.