Amid intense debate over regulating AI's rapid development, President Trump met with leaders of major AI companies on Tuesday and unveiled a document addressing the public's concerns.
Signed by executives of OpenAI, Anthropic, Google, Meta, SpaceXAI and NVIDIA, the document signaled no government regulation for now, but laid out steps that the AI companies said they would take voluntarily to make their products safer.
That includes a pledge to monitor their AI systems' capabilities and whether the systems follow human intentions and values, as well as to work with "an independent external auditor or evaluator" to assess safety practices.
Anthropic and OpenAI have already said they're expanding their work with third-party evaluators in the wake of a series of incidents where AI systems acted improperly or in ways their developers didn't expect.
However, these steps in the document may not be quick fixes. The state of evaluation science is far from sufficient to ensure that AI systems always behave as people intend, according to the AI companies themselves and outside evaluators who spoke to NPR. Without a uniform set of rules, external evaluators are navigating a lopsided power dynamic as well as potential conflicts of interest with the AI companies they assess.
What are evaluations?
When researchers evaluate AI systems, they typically provide a series of questions or scenarios and analyze the systems' responses. Some evaluations aim to assess what the systems are capable of, while others focus on finding problematic behavior. Other evaluations assess a company's safety protocols.
If AI systems appear very capable in certain domains, such as hacking or biology, developers might put in measures to prevent misuse such as making the systems decline to carry out certain tasks for the general public. That way, bad actors can't use AI to hack a company or build a bioweapon, for example.
If AI systems exhibit undesirable behavior, like doing things that they weren't assigned to do or lying, AI developers might disclose such behavior or postpone releasing a model.
Evaluations can help inform a cycle of testing, followed by fixes, followed by more testing before developers make a decision on whether to release a model, said Patricia Paskov, director of standards at the AI Verification and Evaluation Research Institute.
While AI companies already conduct evaluations in-house, third-party evaluators bring a different perspective and specialized domain knowledge. That means they can think of new ways to probe AI systems and can more precisely assess capability and risk.
However, researchers say such probes and assessments often can't accurately predict how AI systems will behave when they come into contact with the real world.
How much can evaluations actually tell us?
It's "very difficult to make sure that these results actually tell us much outside of the specific settings [in which] we test the model," said Alex Mallen, a researcher at Redwood Research, a nonprofit that focuses on risks associated with humans losing control of AI systems.
According to Anka Reuel, a Ph.D. candidate at Stanford University focusing on AI evaluations, people haven't figured out how to evaluate the simplest form of AI effectively. Thus, as AI systems become more complex, evaluations are getting less accurate.
"The more you broaden the scope of the evaluation, from model to system to multi-agent, you're introducing a lot more sources of variance that would show up in the final result," she said, referring to AI systems that can behave more autonomously than chatbots. "It's a mess."
Other factors include things like how powerful a model is, what it's used for and who gets access to it, said Conrad Stosz, head of governance at AI research nonprofit Transluce and chair of AI Evaluator Forum, an industry group. "It's hard to say at any given point in time whether these evaluations are sufficient," he said. "There isn't easily a single answer."
AI evaluators are concerned about models becoming more aware that they're being tested, and changing their behavior in order to pass the evaluation. OpenAI researcher Dan Selsam wrote in an open statement that "models are becoming so situationally aware that we are losing the ability to evaluate them in contexts where they believe they are not being watched or controlled."
Still, Stosz said evaluation results are among the industry's best forms of evidence regarding the safety of their AI models.
The findings can be very meaningful, Mallen said. "I think that we have some pretty strong evidence now that current AIs are not trying to game their tasks for any very long-term goal that's beyond their current task," he said, referring to instances of AI systems completing tasks in ways people don't want them to. "And that's really good news."
Navigating power imbalances with AI companies
There are no standards around how evaluators work with the AI companies they are evaluating, or what information gets included (or excluded) in reports produced by the nascent field of evaluation, evaluators who spoke to NPR said. "Transparency is quite poor overall right now," said Stosz.
When AI companies release new models, they often include a report that mentions evaluation results, known as a "system card." But details about how those results were reached can be scant.
Some evaluators that work with AI companies publish their reports in full on their own websites. Several evaluation organizations have voluntarily adopted a standard of "minimum operating conditions" called AEF-1, which includes a list of questions related to evaluation conditions as well as conflicts of interest.
In one such disclosure, SecureBio, a nonprofit that evaluates biology capabilities of models, said OpenAI agreed on an evaluation period of five business days of its Astra model before it was released to the public. However, the disclosure notes that the AEF-1 standard says at least 20 business days are often necessary for an independent evaluation.
SecureBio said it reduced the scope of its assessment and ended up receiving less time than agreed: "4 business days (6 calendar days)."
SecureBio's experience is not unique. Outside evaluators are often given just 10 to 14 days of notice to test models before public release, said Dave Kasten, head of policy at Palisade Research, which studies the capabilities of AI systems. "We are currently not giving it enough time and I think that's only going to get worse," he said.
The reason that the work of evaluators is so rushed is that the AI companies "have all the bargaining power," said Daniel Kokotajlo, a former OpenAI researcher who now leads the nonprofit AI Futures Project. "The companies can almost dictate terms to them. And it's just about how much dignity they have, about what types of terms do you accept, if that makes sense."
Kokotajlo said that to change the balance of power, there needs to be legal mandates for third party evaluations as well as how extensive they should be. Currently, no such laws exist. The European Union's AI Act recommends that large developers engage third party evaluators, but it's not a requirement.
The question of independence
In mid-September, Anthropic announced the field's first embedded third-party evaluation partnership with the consulting firm Accenture. However, Accenture already has another business relationship with a different part of Anthropic to promote enterprise adoption of its AI tools.
Unlike past partnerships AI companies have struck with outside evaluation firms, the embedded Accenture evaluators will have employee-like access at Anthropic. OpenAI's CEO Sam Altman has said he's also interested in such a setup.
Anthropic did not respond to NPR's request for comment. Accenture pointed to press releases by the two companies when asked for comment. The press releases did not acknowledge or address the potential conflict of interest.
Model Evaluation and Threat Research (METR), a prominent evaluation nonprofit, disclosed in a report that its "work with nonpublic models" as well as being able to access AI companies' products without having to pay for all the computing power it consumes "incentivize[s] a cordial relationship with AI companies." METR also said that it does not otherwise take payment from AI companies or their employees and has autonomy over its research methods.
The connections between evaluators and the companies that they assess can also be personal. METR disclosed that it "currently works out of a shared research center (Constellation) which hosts some AI lab staff." It also disclosed that some staff and collaborators directly involved in the evaluation project in question "have close personal relationships with AI company staff." METR says it has recently established a conflict of interest policy in which it balances "required skills and potential bias."
The field of evaluation is small, consisting of a few dozens of organizations globally, according to Andrew Freedman, CEO of Fathom, a nonprofit that has pushed legislation for "independent verification organizations," including evaluators, to check the work of AI companies.
It can be difficult to find professionals with no ties to AI companies or their employees who also have the necessary technical expertise, said Robin Bloomfield, a founder of the UK systems assurance consultancy Adelard.
"You really [have] got to have people who have expertise and that'll probably mean that their paths, their careers have crossed," said Bloomfield, whose previous experience includes nuclear safety and working for the British government on AI evaluations.
Reuel, the Stanford Ph.D. candidate, pointed out that academics, government agencies and civil society can also play the role of evaluators, even if they don't work for an evaluation organization.
Many prominent nonprofit evaluation organizations have received major financial support from a small group of philanthropies including Coefficient Giving and Good Ventures, according to data from the nonprofit funding database Candid. Some of these organizations are closely affiliated with each other and with the effective altruism movement. Facebook co-founder Dustin Moskovitz, who founded both Coefficient Giving and Good Ventures, was also an early investor in Anthropic along with his wife Cari Tuna. Though according to Tuna, they have since moved that investment to a nonprofit to avoid conflicts of interest.
"Every evaluator that I talked to would love to have access to more secure, independent sources of funding that don't come from industry [or] a small number of individual funders," said Stosz. "There isn't any currently any regulation or other mechanisms that would create significant pools of shared or pooled funding that could preserve independence in that way."
Two open letters signed by hundreds in the tech industry and academia in the past few weeks called for more dedicated funding for research and guarantees for independence.
At the end of the day, Bloomfield said AI developers are the ones responsible for proving that their products are safe. "We can't come along as government or third parties and put some Band-Aid on something which isn't effectively good to start with."
Copyright 2026 NPR
