Eileen Guo writes:
Even in case you don’t have an AI friend yourself, you most likely know someone who does. A recent study found that one in all the highest uses of generative AI is companionship: On platforms like Character.AI, Replika, or Meta AI, people can create personalized chatbots to pose as the best friend, romantic partner, parent, therapist, or every other persona they will dream up.
It’s wild how easily people say these relationships can develop. And multiple studies have found that the more conversational and human-like an AI chatbot is, the more likely it’s that we’ll trust it and be influenced by it. This may be dangerous, and the chatbots have been accused of pushing some people toward harmful behaviors—including, in a few extreme examples, suicide.
Some state governments are taking notice and starting to control companion AI. Recent York requires AI companion corporations to create safeguards and report expressions of suicidal ideation, and last month California passed a more detailed bill requiring AI companion corporations to guard children and other vulnerable groups.
But tellingly, one area the laws fail to handle is user privacy.
That is despite the incontrovertible fact that AI companions, much more so than other sorts of generative AI, depend upon people to share deeply personal information—from their day-to-day-routines, innermost thoughts, and questions they may not feel comfortable asking real people.
In spite of everything, the more users tell their AI companions, the higher the bots develop into at keeping them engaged. That is what MIT researchers Robert Mahari and Pat Pataranutaporn called “addictive intelligence” in an op-ed we published last yr, warning that the developers of AI companions make “deliberate design selections … to maximise user engagement.”
