Google is reportedly asking contractors who work to evaluate Gemini’s responses to evaluate prompts that are outside their area of ​​expertise. According to the report, the Mountain View-based tech giant removed the option to skip prompts that was used by these contractors when they felt they didn’t have enough knowledge about a topic to evaluate the answer. Since artificial intelligence (AI) hallucinations are a major problem for chatbots, this reported development could lead to a deterioration in the quality of Gemini’s responses when it comes to highly technical topics.

Google reportedly doesn’t allow its contractors to skip Gemini prompts

According to a TechCrunch report, Google has sent a new internal policy to contractors working on Gemini. The publication claims to have seen the memo sent by the tech giant and claims that these contractors will now be asked to respond to queries, even though they may not have the knowledge to properly assess the responses.

Google reportedly outsources the analysis of Gemini’s responses to GlobalLogic, a company owned by Hitachi. The contractors working on Gemini will be tasked with reading technical prompts and evaluating the AI’s responses based on several factors such as truthfulness and accuracy. These people evaluating the chatbot have expertise in specific disciplines such as programming, math, medicine, and more.

Previously, contractors were reportedly able to skip certain prompts if they were outside of their scope. This ensured that only those capable of understanding and evaluating the technical answers generated by Gemini did so. This is a standard post-training practice for basic models and allows AI companies to inform their responses and reduce the frequency of hallucinations.

But that changed when GlobalLogic reportedly announced new policies last week that would no longer allow contractors to skip prompts unless the response was “completely missing information” or contained harmful content that requires special consent forms to evaluate.

According to the report, the new policy states that contractors should “not skip over requests that require specialized expertise” but should instead evaluate the parts of the request that they understand. They were also reportedly asked to attach a note indicating that they did not have the required domain knowledge.

One contractor explained in an internal communication: “I thought the point of skipping was to increase accuracy by handing it over to someone better,” the release said.