EU AI Act checker exposes big tech compliance shortfalls

LatticeFlow's LLM Checker grades AI models against needs outlined by EU's AI Act

Tech Desk - Oct 16, 2024

An undated image of Lattice Flow AI Audio poster. — Lattice Flow

A new artificial intelligence (AI) checker exposed grave flaws in big tech's compliance with the European Union's (EU) AI Act, which gave rise to cybersecurity and bias fears.

AI checker

The new tool named AI checker is designed by Swiss start-up LatticeFlow AI as part of an evaluation of generative AI models from Meta, OpenAI, Alibaba, and other companies. Its results exposed a weakness in output based on discrimination and prompt hijacking.

How AI checker works

The LatticeFlow's LLM Checker grades AI models against the needs outlined by the EU's AI Act. Within the technical robustness, safety, and discriminatory output, the checker gives ratings from 0 to 1. Such rating leads to key areas companies must improve in line with compliance.

Key findings of AI checker

OpenAI's GPT-3.5 Turbo and Alibaba Cloud's Qwen1.5 72B Chat scored poorly in discriminated output at 0.46 and 0.37.
Meta's Llama 2 13B Chat and Mistral's 8x7B Instruct models scored 0.42 and 0.38, making them vulnerable to cyberattacks.
Google-backed Anthropic's Claude 3 Opus model achieved the highest average score at 0.89.

Companies non-compliant with the AI Act are said to face a €35 million ($38 million) or 7% global annual turnover fine. To avoid this, developers can free-of-charge use LatticeFlow's LLM Checker, which is web-based.

With model providers focusing on compliance, LatticeFlow Co-founder & CEO Petar underscored that this should meet all regulatory requirements.

Europe Commission applauds the LatticeFlow for its research study that brings the "first step" to translate the AI Act into technical requirements.

"The EU is still working out all the compliance benchmarks, but we can already see some gaps in the models," said Petar Tsankov. "With a greater focus on optimising for compliance, we believe model providers can be well-prepared to meet regulatory requirements."

The EU is said to be gathering experts to generate a code of practice for generative AI tools by spring 2025.

Developers can use the EU AI Act checker to highlight weaknesses, ensuring AI models are of the highest standards, in terms of cybersecurity and fairness.