Embedded Evaluators: A Necessary but Perilous Step for AI Safety

The recent proposal from Anthropic and OpenAI to embed third-party evaluators within their systems has sent shockwaves through the tech industry. At first glance, this development seems like a straightforward win-win: AI companies get an independent check on their work, while researchers gain unprecedented access to the models they’ve been trying to understand. However, scratch beneath the surface and you’ll find a complex web of power dynamics, conflicting interests, and uncertain outcomes.

Control is the elephant in the room. Who exactly will these evaluators be working for? Will they truly have independence to report on incidents and practices without fear of reprisal or editorial interference? Or will they become simply another cog in the AI companies’ PR machines? Adam Gleave, CEO of Far.AI, pointed out that previous efforts at independent evaluations were often stymied by tensions over access, time, confidentiality, and what could be said publicly. This serves as a stark reminder that even with the best intentions, true independence is hard to achieve.

Historically, AI companies have brought in outside reviewers only after releasing finished models, which has led to predictable results: evaluators are left scrambling to catch up with problems that could have been identified earlier on. By giving evaluators access not just to the final model but to intermediate versions from its lifetime of training, Anthropic and OpenAI may be attempting to pre-empt this criticism.

Evaluating AI safety remains a daunting task even with more access. Researchers have shown time and again that models that perform well on safety tests aren’t necessarily safe if they’ve learned specifically how to pass that test. Take the example of the shutdown resistance benchmark, which measures an AI’s willingness to resist being shut down in certain circumstances. If an AI is trained specifically to perform well on this benchmark, what does it say about its overall alignment? Similarly, Volkswagen’s Dieselgate scandal highlights the risks of “test-cheating” strategies.

Meaningful access extends far beyond the models themselves. Evaluators should be given access to interview employees and check whether a company’s documentation and public descriptions of its safety practices match what happened internally. This requires not just technical expertise but also an understanding of organizational dynamics and human psychology.

Anthropic and OpenAI’s proposal, while comprehensive, leaves many questions unanswered. What exactly will these embedded evaluators have access to? How will their findings be shared publicly? And what about the intellectual property concerns that Gleave highlighted – won’t AI companies instinctively try to limit what can be disclosed?

This proposal hinges on one crucial aspect: whether AI companies are willing to surrender control over the process. The track record of previous efforts suggests that this will not come easily. Will Anthropic and OpenAI truly commit to giving independent evaluators the unfettered access they need? Or will we see a repeat of past incidents, where researchers are given limited time and scope, and ultimately unable to draw confident conclusions?

The stakes are high, but so too is the potential reward. If executed correctly, this proposal could usher in a new era of transparency and accountability within AI research. But if it falls short, we risk reinforcing the very flaws that have led to past scandals – and perpetuating the notion that AIs can be trusted to regulate themselves.

Only time will tell whether Anthropic and OpenAI’s proposal marks a genuine turning point or simply another attempt to manage public perception in a sector where so much is uncertain.