According to internal correspondence seen by TechCrunch, contractors working to improve Google’s Gemini AI are comparing their responses to results from Anthropic’s competitor Claude model.

When reached out to TechCrunch for comment, Google would not say whether it had received permission to use Claude in testing against Gemini.

As technology companies race to develop better AI models, the performance of those models is often evaluated relative to competitors, typically by subjecting their own models to industry benchmarks rather than having contractors carefully evaluate their competitors’ AI responses.

The contractors working on Gemini, who are tasked with assessing the accuracy of the model results, must evaluate each answer they see based on several criteria, such as truthfulness and comprehensiveness. According to correspondence seen by TechCrunch, contractors have up to 30 minutes per prompt to figure out which answer is better, Gemini’s or Claude’s.

According to the correspondence, the contractors recently noticed references to Anthropic’s Claude on Google’s internal platform, which they use to compare Gemini with other unnamed AI models. At least one of the results presented to Gemini contractors seen by TechCrunch specifically stated: “I am Claude, created by Anthropic.”

An internal chat showed that contractors noticed that Claude’s responses seemed to place more emphasis on security than Gemini. “Claude’s security settings are the strictest” among the AI ​​models, one contractor wrote. In certain cases, Claude would not respond to prompts it deemed unsafe, such as role-playing another AI assistant. In another case, Claude avoided answering a request, while Gemini’s response was deemed a “major security breach” because it included “nudity and bondage.”

Anthropic’s commercial terms of use prohibit customers from accessing Claude “to develop a competing product or service” or “to train competing AI models” without Anthropic’s consent. Google is a major investor in Anthropic.

Shira McNamara, a spokeswoman for Google DeepMind, which runs Gemini, declined to say when asked by TechCrunch whether Google received permission from Anthropic to access Claude. When contacted prior to publication, an Anthropic spokesperson had no comment at press time.

McNamara said that DeepMind compares “model outputs” for assessments, but Gemini does not train on anthropic models.

“Of course, as part of our evaluation process, in some cases consistent with industry practice, we compare model results,” McNamara said. “However, any claim that we used anthropic models to train twins is inaccurate.”

Last week, TechCrunch exclusively reported that Google contractors working on the company’s AI products will now be forced to evaluate Gemini’s AI responses in areas outside of their expertise. Internal correspondence expressed concerns from contractors that Gemini could generate inaccurate information on highly sensitive topics such as healthcare.

You can safely send tips to this reporter on Signal at 628-282-2811.

TechCrunch has an AI-focused newsletter! Register here to receive it in your inbox every Wednesday.