By Aditya Soni
NEW YORK, Oct 13 (Reuters) – Anthropic and OpenAI have sought significant control over what outside evaluators can publish, concerned that findings could reveal trade secrets, according to two people familiar with the discussions.
Anthropic CEO Dario Amodei said last month that external evaluators should have the right to publish key findings “without editorial control.” OpenAI agreed days later, saying assessors should have “editorial independence.”
The AI companies are worried that evaluator staff working inside the companies’ walls as part of so-called embedded evaluation could take proprietary information to rivals, according to the people, who declined to be named as the matter was private.
Measures under discussion to prevent intellectual-property leaks include having lab employees observe embedded teams as they work and preventing evaluators embedded at one lab from working at another simultaneously, one of the people added.
But what labs consider crucial intellectual property can overlap with information relevant to judging a model’s risk, which can lead to arguments and redactions, said a separate person with knowledge of past dealings with the labs.
Experts have said independence of evaluators is a concern as Anthropic and OpenAI choose who gets to carry out the reviews. Reuters is reporting details of these negotiations between the evaluators and the AI labs for the first time.
The discussions highlight the tension between a need to make the world aware of powerful and potentially dangerous AI and demands to safeguard trade secrets by AI companies seeking trillion-dollar valuations in what could be major IPOs.
The outcome of the talks will help determine how the world scrutinizes a technology that some of its creators warn may be advancing beyond humanity’s ability to manage.
Without agreed standards and legislation for independent reviews, some experts said the companies building the most powerful AI models could dictate how their systems are examined, leaving evaluators at the mercy of the labs themselves.
NEGOTIATIONS OVER PUBLICATION RIGHTS
The nature of the discussions changed quickly. The two people with knowledge of the matter said that as recently as October 5, Anthropic and OpenAI were seeking strict editorial control over what the evaluators could publish. By October 8, talks between Anthropic and some evaluators appeared to have evolved, according to one person familiar with the matter.
In response to questions from Reuters, Anthropic said that seeking significant control over embedded evaluators’ publications did not accurately characterize all of its discussions and work with independent evaluators.
The company behind the Claude AI models said that, in current conversations with evaluators it previously worked with, it had made clear that the outside reviewers control their methods, evaluation criteria, conclusions and what to publish.
For new evaluator firms, Anthropic said it reserved the right to comment on and approve reports as the reviewers learn how its models operate and establish a working relationship.
OpenAI, for its part, said it was not seeking approval rights over the full text of reports produced by evaluators. It said its standard practice is to redact confidential information, and evaluators can disclose whether major redactions were made.
The ChatGPT maker said last month that AI labs and assessors should agree on the scope and access before testing, disclose methods and conflicts, and let assessors control their findings, while allowing labs to protect sensitive information.
The language AI labs already use in certain contracts with firms that evaluate their models can bar those firms from publishing confidential information in their reports without the companies’ consent, according to a contract seen by Reuters.
PROMISES OF INDEPENDENCE
Anthropic last month said that Faculty, Accenture’s specialist AI business, would be its first embedded evaluator, but said details on what evaluators can access and how they report findings were still being worked out.
“Evaluators are only useful if people can trust what they report, so independence and transparency are critical,” Faculty CEO Marc Warner told Reuters.
Technology companies typically bar the firms they work with from sharing such information with others without their consent.
But experts said the stakes are higher with AI, technology that, according to its developers, will have major implications for humans.
The labs could control the access evaluators depend on in the absence of standards, giving them leverage over what gets reported, Rumman Chowdhury, former US Science Envoy for AI under the Biden administration and CEO of Humane Intelligence PBC, told Reuters.
“Evaluators have this very privileged access, but whether explicitly or implicitly, they know that it comes at a cost,” Chowdhury said.
Independent evaluation was one of four safety measures in the one-page White House Accord on Super Intelligence, signed by President Donald Trump and leading AI executives late last month. But the accord, which Trump has called “morally binding,” did not say how those reviews should be conducted.
Evaluators also issued their own minimum conditions last month in a public letter that has been signed by more than 200 researchers, including Geoffrey Hinton, one of the godfathers of AI. The letter calls for independent control of findings, broad access to AI labs and protection from retaliation.
REDACTIONS AND RESTRICTED ACCESS
OpenAI redacted nonpublic information from a report by Model Evaluation and Threat Research, or METR, and Redwood Research on the Hugging Face incident, in which autonomous agents powered by OpenAI’s advanced models went rogue and set off a hack that compromised Hugging Face’s infrastructure.
The AI safety nonprofits’ investigation, first planned as two days at OpenAI, stretched to six days over three visits, with a more complete set of transcripts arriving only on the last one, after the researchers flagged gaps in the data.
METR and Redwood said they edited the report’s structure, emphasis, clarity and tone based on OpenAI’s feedback, but that nothing else important to their conclusions was removed. They stood by their findings and praised the access OpenAI ultimately provided, but said the risk of deterring developers from bringing in outside evaluators after serious incidents shaped their judgment calls in drafting, editing and redacting the report.
(Reporting by Aditya Soni in New York; Editing by Kenneth Li and Matthew Lewis)
