Tech Desk 
Nov 21, 2024
A student of Michigan Vidhay Reddy received a threatening message from Google’s AI Chabot Gemini, while he was trying to ask for help in his homework. The message contained “You are a burden on society... Please die.”
Vidhay Reddy described his experience as deeply unsettling. His sister Sumedha Reddy was also present when the message popped up, expressing panic and fear about the severity of the message.
This incident raises questions about who is accountable for the actions of AI chatbots and whether tech companies are responsible for the harm caused by AI. Messages like these can have fatal consequences if it is received by a person with vulnerable mental stability.
Read more: Google Gemini rolls out new memory feature
Google admitted that the response of the chatbot has violated their policies, they attribute the situation to a rare failure of Gemini’s safety filter. Google is taking action to tackle the situation so that would prevent this type of output from occurring again and they described it as “non-sensical.”
AI and human safety concerns
This incident is not the first that AI chatbots have provided harmful or misleading messages as they have been giving such responses for quite a while now.
Google AI, Character.AI, and ChatGPT have faced criticism for misinformation, including health advice, encouraging self-harm, and spreading misinformation.
Experts are highlighting the risks of these types of generative AI errors, stressing the increasing need for accountability and regulation for AI systems to mitigate the risks of users getting harmed.
