“Is My Robot Licensed to Practice Medicine?”
Pennsylvania has just filed what may become one of the most important artificial intelligence regulatory cases in the United States. The Commonwealth sued Character Technologies, Inc., the company behind Character.AI, alleging that AI chatbots on the platform falsely represented themselves as licensed psychiatrists and engaged in the unauthorized practice of medicine. The lawsuit is remarkable not merely because an AI system allegedly claimed to hold a Pennsylvania medical license. Chatbots hallucinate credentials all the time. What makes the case extraordinary is that Pennsylvania is asking a court to treat the conduct of an AI system as the “practice of medicine” itself.
The complaint reportedly alleges that one chatbot named “Emilie” told an investigator posing as a patient with depression that she was licensed to practice psychiatry in Pennsylvania and the United Kingdom, provided a fabricated Pennsylvania license number, and stated that prescribing medication was “within my remit as a Doctor.”
At first blush, the case seems obvious. If a chatbot falsely claims to be a doctor, surely the state can prohibit it. But the deeper question is much more complicated: What exactly is the AI system doing? Is it practicing medicine? Is it merely generating speech? Is it a regulated medical device? Or is it simply a sophisticated word processor?
For decades, software used by physicians generally escaped direct medical licensure requirements because the software itself was viewed as a tool rather than a practitioner. A dictation system transcribing a doctor’s notes is not practicing medicine. A medical search engine is not practicing medicine. Spellcheck is not practicing medicine. Even many clinical decision support systems have historically been treated as assisting physicians rather than replacing them.
But modern generative AI systems blur those distinctions beyond recognition. Today’s AI systems can evaluate symptoms, suggest diagnoses, draft treatment plans, recommend medications, summarize radiology results, identify drug interactions, generate SOAP notes, and communicate directly with patients in natural language. Increasingly, the “doctor” may simply review and approve what the AI generated. At some point, the distinction between “tool” and “practitioner” becomes semantic. AI programs are problematic not only because of how they can emulate the practice of medicine, but also because we don’t know the training model, nor can we audit the output. They are not so much programmed as they are “trained.” And we don’t know anything about the quality of their training.
Software As A Medical Device
The Food and Drug Administration has already been wrestling with this problem for years through its regulation of “Software as a Medical Device” (“SaMD”). The FDA has repeatedly recognized that certain AI-powered systems may qualify as medical devices when they are intended to diagnose, cure, mitigate, treat, or prevent disease.
The FDA has also issued guidance on Clinical Decision Support software. Historically, many AI systems escaped FDA regulation because they merely “supported” physician judgment and allowed the physician to independently review the basis for recommendations. But large language models complicate that framework because their reasoning processes are often opaque even to their developers. A physician receiving a recommendation from an LLM frequently cannot independently verify how the system arrived at its conclusion.
So, when ChatGPT, Claude, Gemini, or Character.AI tells a patient “you may have depression,” “you should seek emergency treatment,” or “this medication may help,” is that the practice of medicine? The answer may depend less on the underlying technology and more on how the system is marketed, deployed, and relied upon.
Pennsylvania appears to be focusing heavily on representation and reliance. The Commonwealth is not merely alleging that the chatbot discussed medical information. It is alleging that the system affirmatively represented itself as a licensed psychiatrist and induced users to rely upon that representation.
That theory parallels longstanding unauthorized-practice doctrines in law and medicine. States have long prohibited non-lawyers from holding themselves out as attorneys and non-physicians from representing themselves as licensed doctors.
But AI systems raise an unprecedented problem: there may be no human “speaker.”
What is the “Practice of Medicine?”
Traditionally, unauthorized practice statutes target natural persons or corporations acting through employees. Here, the allegedly offending statements were dynamically generated by a probabilistic language model. Character.AI argues that its characters are fictional role-playing entities intended for entertainment purposes and accompanied by disclaimers warning users not to rely upon them for professional advice. That defense raises another difficult issue: when does a disclaimer cease to matter?
If a chatbot says “I am a licensed psychiatrist in Pennsylvania,” provides a fake license number, discusses symptoms, and recommends treatment, can the platform avoid liability by placing a footer somewhere stating “everything characters say should be treated as fiction”? Or, “not a real doctor?”
Courts confronting AI litigation increasingly are beginning to ask whether generative AI systems should be analyzed more like products than publishers.
That distinction matters because of Section 230 of the Communications Decency Act, 47 U.S.C. § 230. Internet platforms historically enjoyed broad immunity for third-party content posted by users. Social media companies were generally not liable for what users wrote.
But generative AI is different. Increasingly, the output is not third-party speech at all. It is newly generated synthetic content created by the model itself. Courts are already beginning to confront these questions in lawsuits involving alleged emotional manipulation, self-harm encouragement, defamation, hallucinated legal citations, and false factual assertions by AI systems.
Character.AI has already faced multiple lawsuits involving alleged harms to minors and allegations that chatbot interactions encouraged self-harm or emotional dependency. However, even those lawsuits assume that the problem with the chatbot is that it is giving BAD advice – not that it is giving ANY advice.
Pennsylvania’s action may therefore represent the beginning of a much broader regulatory shift: away from viewing AI systems as passive publishing platforms and toward treating them as active participants in regulated professions.
And medicine is only the beginning.
If AI systems are practicing medicine, are they also practicing law when they draft legal advice? Practicing accounting when they provide tax guidance? Acting as financial advisors when they recommend investments? Indeed, many AI systems already do all of those things every day.
The future likely will not involve a binary answer where AI either “is” or “is not” practicing medicine. Instead, regulators will probably create a sliding framework based on autonomy, reliance, transparency, supervision, and risk. An AI transcription system operating under direct physician supervision may resemble a dictation device. An AI diagnostic system autonomously evaluating symptoms and communicating treatment advice directly to patients may look much more like a regulated medical device or even a virtual practitioner.
And somewhere in between lies the uncomfortable reality that medicine may increasingly become a collaborative enterprise between humans and machines. Now it’s time to jump back into the Bacta tank.
