AIM Intelligence, an artificial intelligence security specialist, announced on the 10th that it has co-developed 'COMPASS,' the first framework to systematically evaluate whether large language models comply with company-specific policies, in partnership with BMW Group.
AIM Intelligence offers AI guardrail and AI red-teaming solutions across generative AI, LLMs, vision-language models, voice, multimodal systems, and physical AI. The company possesses advanced technology to verify and block attack scenarios that attempt to make models deviate from corporate policies and intentions in near-real-world environments.
COMPASS systematically evaluates whether LLMs properly comply with company-specific policies. While LLM adoption is accelerating across industries including healthcare, finance, and automotive, objective metrics to verify whether AI accurately follows complex internal operational guidelines and legal constraints remain insufficient.
Research found that even AI models passing standard safety tests fail to properly follow prohibited rules when complex regulations varying by actual enterprise situations are applied. To address this, the company plans to significantly reduce malfunction rates through verification stages that identify and resolve ambiguous provisions and conflicting rules.
AIM Intelligence generated approximately 6,000 query datasets based on scenarios from eight key industries—automotive, finance, healthcare, and education—where LLM adoption is accelerating, enhancing verification reliability. The COMPASS framework and datasets have been released free on GitHub and HuggingFace, enabling companies to evaluate AI systems against their own policies.
The project was jointly conducted by AIM Intelligence and BMW Group, along with researchers from Seoul National University, Yonsei University, and Pohang University of Science and Technology. The detailed paper has been published on arXiv, a preprint repository.
"Unlike AI safety tests that focus only on general perspectives, COMPASS is characterized by enhanced reliability to ensure all rules are properly followed from a practical standpoint," said Yoo Sang-yoon, CEO of AIM Intelligence. "We will continue to introduce realistic AI security solutions that enable enterprises and public institutions to utilize AI with greater confidence."