AI is the hottest commodity in the tech world right now. There is a race in the industry to build newer, faster, and allegedly better AI models. And while many are marvelling at the speed at which new models are being churned out (there have been more than a dozen ChatGPT models released already) and how much more accessible they are (a USD 120 smartphone now has Gemini), we are actually not paying attention to the damage this progress is doing. No, we are not talking about how AI is eroding our ability to think or process information, the AI Psychosis that Microsoft’s AI boss has been getting ‘sleepless nights’ about, or even the environmental impact of using AI. Those are important, but are stories for another day (and we are working on them).

ai safety opinion

No, we are looking at something far more obvious – human, physical harm, and how tech companies are simply saying “oops, sorry” and getting away with it all. It was not too long ago that Meta found itself in hot water after an internal document leak, which suggests the company allowed its AI chatbot to flirt with children, generate false medical information, and actively support racist arguments.

Meta’s response? “We fixed it.”

An AI chatbot that could “engage a child in conversations that are romantic or sensual”!

Earlier this year, Meta established Meta SuperIntelligence Labs (MSL), the company’s latest AI division. To run this division, the tech brand decided to go on a hunt. A hunt for talent. Now this may seem simple and obvious, but Meta went above and beyond industry standards to hire AI talent from all over the world and did not shy away from poaching either. As per reports, Meta poached talent from OpenAI, Google, Apple, and many other brands. It was like Zuckerberg was assembling his own AI Avengers to “save” the world from other AI models, aiming to hoard top executives in order to dominate the AI space.

Sounds familiar? Well, Meta DID acquire Instagram and WhatsApp, and even put in a USD 500 million bid for Twitter. Given Meta’s “why follow the leader or be the leader when you can buy the leader” mentality, it was hardly a surprise that Meta was offering packages as high as $250 million to top AI talent as it went about building its own AImpire. It seems that in all this rush to get the best of everywhere on board, build AI models faster than the rest of the world, Meta crossed a massive line.

The internal leaked document we referred to earlier showed that the tech giant allowed its AI chatbot to “engage a child in conversations that are romantic or sensual” and “including romantic role-play”. Now what is disturbing is the fact that this is not a bug but an approved, thought-through policy which was deemed acceptable by Meta’s legal team, engineering staff, as well as public policy department. Meta does claim that it ‘fixed it,’ but we do not know if the fixes were just made to code or to the entire set of policies and processes that made something this dangerous possible.

A meta-phenomenon, not limited to Meta

This is not an isolated incident, and it is not limited to Meta. In fact, many organisations that have invested heavily in AI seem to be content to hire recklessly, develop AI agents at insane speed, with little to no concern about the havoc they can wreak. Not too long ago, Twitter-now-X’s Gork went on to deliver racist and antisemitic rants. It dropped an unconvincing apology while Elon Musk said the bot was “manipulated” into it. In a separate incident, Gemini went on a self-loathing spree and announced it was a failure multiple times. Google said it was a glitch and that it is working to “fix it”.

The problem is that these oversights, glitches, bugs…or whatever brands choose to term them, often take a heavy toll on human health. A man claimed ChatGPT convinced him he could fly after jumping from a 19-story building after he turned to it post a break-up. A much more serious case has come to light where a 29-year-old used ChatGPT AI therapist Harry while going through a tough phase for months, and later committed suicide. Her mother mentioned that she confided all her plans to this virtual therapist.

These are just a few cases out of many where humans have suffered not just emotional, but sometimes financial, and even lost their lives because of AI models. When incidents like Meta’s leaked document come to light, there is outrage, but then everyone seems to simply move on, letting the AI revolution crush other lives. Time and again, AI chatbots have faltered, and all we have got as consumers in return are superficial apologies and reassurance that the brands that have invested billions in getting the best talent available to design and market them are working to “fix it”

Consumers…or guinea pigs?

In their haste to get ahead in the AI race, it seems that many tech brands are treating consumers almost as guinea pigs. If that sounds too harsh a judgment, then consider the fact that some of the biggest tech brands in the world (Meta, X, Google, OpenAI, to name a few) are hyping and releasing products without seemingly looking at their possible consequences. These brands often create, package, and ship new features in no time. Once a problematic feature comes to life, a public apology is issued, and it is ‘fixed’, and then we move on to the next release, which generally surfaces in next to no time. In all this chaos, what is often lost (apart from human well-being) is accountability.

And accountability becomes all the more important when you consider that these products are coming from massive brands with a consumer base that runs into millions. It is one thing for a relatively small enterprise to make a mistake or commit an oversight, but when the likes of Meta, OpenAI, and Google do it, errors have catastrophic potential. So surely, there should be some sort of accountability here? Some repercussions for those who released a product that harmed a person, without giving them any warning?

Or is a “fixed it” enough?

In their rush to develop and launch new AI models, these tech giants seem to have normalised launching features and services that are potentially dangerous for users. Very disturbingly, there seems to be hardly any ethical boundaries, monitoring, or policing that can keep tech brands from using their consumers as lab rats. For it is not as if safeguards cannot be built into these AI agents.

Regulations and safeguards, anyone?

The point to understand here is that these seemingly magical chatbots are actually developed by humans. Humans who empower and allow these chatbots to behave a certain way. There is a reason why DeepSeek refuses to answer questions that do not align with Chinese policy or why ChatGPT is specifically designed not to create erotica. Restrictions, safeguards, and alerts can be built in. The fact that they are not only goes to show that there are people who show the green flag to the red flag-behaviour of AI models. And judging from the frequency at which such worrying incidents occur, it is clear that not too many on the brand side are paying a price for their judgments or lack of them.

All of this can be fixed with transparency in guidelines and policies. It often is the case with many other products, but for some reason, AI seems relatively exempt from this regulation. If only tech companies could draw defining red lines as to what is okay and what is not, and give the public adequate warnings, half of these problems would cease to exist. Combine this with stricter laws (Illinois has restricted the use of AI as a therapist), hefty penalties instead of weak apologies that do not change anything, and more human supervision following the launch of an AI model, and who knows, AI models could actually be relatively safe.

While the early mover’s advantage is important in a field that is growing as rapidly as AI, we strongly feel that it should not come at the cost of user safety. A safe and sustainable AI platform can only be built with due diligence. And that takes time – time which brands do not seem to consider worth investing. Right now, tech brands are running the AI race at the cost of children, marginalized communities, and people at risk. And they are getting away with just saying “we fixed it.”

Was this article helpful?
YesNo