"Move Fast and Break Things" Does Not Work When the Thing You Break Is Patient Privacy

"Move Fast and Break Things" Does Not Work When the Thing You Break Is Patient Privacy and Physician Trust

Generative AI has walked into healthcare through the side door. A nurse summarizes a discharge note. A billing clerk drops a denied claim into a chatbot to draft an appeal. A pharmaceutical rep pastes a physician's prescribing history into an AI assistant to prepare talking points. A startup engineer uploads real patient data to test a model. Each of these feels like productivity. Each of them can be a breach of confidentiality, and only one of them involves a patient's chart.

Our last post covered how California courts treat protected health information after J.M. v. Illuminate Education, Inc. (Cal. May 14, 2026). This one is about how sensitive information leaves the building in the first place, why the physician on the other side of the sales call is a data subject too, and why AI governance in healthcare and pharma has to start with a vocabulary lesson.

First, the Definitions Every Employee Needs

Personally identifiable information, or PII, is any data that identifies a person or can reasonably be linked to one: name, address, date of birth, email, device identifiers, and the combinations that add up to a specific human being.

Protected health information, or PHI, is the subset of PII tied to a person's health, treatment, or payment for care, held by a HIPAA-covered entity or its vendors. A diagnosis, a lab result, a prescription, or an insurance claim connected to a person is PHI. HIPAA lists eighteen identifiers, and if any of them are attached to health data, the record is protected (45 C.F.R. § 164.514(b)(2)).

Then there is the third category, the one that trips up commercial teams: collateral information. This is data that is not technically PHI but is still confidential and consequential. Prescriber-level prescription data, dispensing trends by physician and territory, formulary strategy, key opinion leader profiles, call notes, and the market intelligence a company licenses from data vendors. In our earlier posts we called the professional slice of this PPII, professional private information. The physician is the data subject here, and the record of what a doctor prescribes, how often, and to how many patients is one of the most valuable commercial data sets in medicine.

If an employee cannot tell these three categories apart, no policy document will save the company.

The Physician's Prescribing Habits Are Collateral Information

A physician's prescribing history is bought and sold every day. Pharmacies sell dispensing records to data vendors, the vendors match them to prescriber identifiers, and pharmaceutical companies license the result to see which doctors write which drugs. The Supreme Court confirmed the commercial value of that data in Sorrell v. IMS Health Inc., 564 U.S. 552 (2011), striking down a Vermont law that tried to keep prescriber-identifiable data away from marketers. The practical result is that no federal privacy statute protects a doctor's prescribing habits the way HIPAA protects a patient's chart.

What protects that data instead is a web of contracts and professional norms. Vendors such as IQVIA license prescriber-level data for specified permitted uses only, prohibit repurposing it, and require the licensee to safeguard it against unauthorized disclosure (IQVIA, "Limit Use of IQVIA Data"). The American Medical Association's Physician Data Restriction Program, active since 2006 and open to all physicians, lets a doctor block sales representatives from seeing their individual prescribing data, enforced through contracts with the data companies (AMA, Physician Data Restriction Program). A doctor's prescribing profile turns out to be heavily restricted, just not by HIPAA.

Now put that profile in front of a chatbot. A rep who pastes a physician's prescribing history, PDRP status, call notes, and a competitor's market share into a consumer AI tool to draft a pre-call plan has disclosed licensed data to an unauthorized third party in violation of the vendor contract, potentially exposed a physician who opted out of rep access to exactly the use they opted out of, and handed a general-purpose model the raw material to generate promotional claims that no medical-legal-regulatory review ever approved. None of that is a HIPAA breach. All of it is a breach of confidence and a contract problem, and it erodes the one thing HCP engagement depends on: the physician's trust that the company across the desk handles their professional information responsibly.

Why Typing It Into an AI Is a Disclosure

When an employee enters PHI into a consumer AI tool, the information has been transmitted to a third party. Under HIPAA, a vendor that receives PHI on a covered entity's behalf is a business associate and must sign a business associate agreement before it touches a single record (45 C.F.R. § 164.502(e)). OpenAI, for example, offers a BAA for ChatGPT for Healthcare and eligible API customers and does not train on that content (OpenAI, "Introducing OpenAI for Healthcare"). The free and standard consumer versions come with no BAA, and by default OpenAI may use consumer content to train its models unless the user opts out (OpenAI Help Center). A patient's record can end up inside a system the covered entity does not control, cannot audit, and cannot delete from.

HIPAA does not care that it was convenient. An impermissible disclosure of PHI is presumed to be a breach unless the organization can show a low probability the data was compromised (45 C.F.R. § 164.402), and that showing is hard to make when the recipient is a general-purpose AI platform. The same logic applies to collateral information: pasting a prescriber-level trend report into a public model is, under most data licenses, a disclosure to an unauthorized third party. The breach happened at the keystroke, not when a hacker showed up.

The Scale of the Problem

This is not fringe behavior. Netskope Threat Labs, analyzing healthcare organizations' traffic from March 2024 through March 2025, found that 81 percent of data policy violations involved regulated healthcare data and that 71 percent of healthcare workers were still using personal generative AI accounts at work (Netskope Threat Labs, "Threat Labs Report: Healthcare 2025"). (Emphasis Added). A Wolters Kluwer survey of 518 healthcare professionals published January 22, 2026 found that 40 percent had encountered unauthorized AI tools in their organizations, nearly 20 percent admitted to using them, and roughly 10 percent had used unsanctioned AI for direct patient care (Wolters Kluwer, Jan. 22, 2026). About a third of those users said their employer offered no approved alternative (Healthcare Brew, Feb. 19, 2026).

The Regulators Are Not Waiting

There is no AI exemption. HIPAA applies to AI vendors as it applies to any other business associate, and the proposed HIPAA Security Rule update would require organizations to inventory and risk-assess every system that touches electronic PHI, AI included (90 Fed. Reg. 898 (Jan. 6, 2025)). The FTC has already penalized GoodRx ($1.5 million), BetterHelp ($7.8 million), and Cerebral ($7 million) for sharing health data with third parties through tracking tools. States are moving faster: Texas requires disclosure when AI is used in relation to healthcare services (HB 149), California requires clinics using generative AI for patient communications to say so (AB 3030), and Louisiana requires verbal disclosure before any recording that will be transcribed by AI (Act 649 of 2026). Every one of these rules assumes the company knows where its AI tools are and what they are being fed.

Are Startups Policing Themselves?

U.S. digital health startups raised $7.4 billion across 244 deals in the first half of 2026, with twenty megadeals accounting for 45 percent of all capital invested (Rock Health, H1 2026). That capital is chasing speed. Speed is where "move fast and break things" came from, and in consumer software the motto has a certain logic. A broken feature can be patched.

A disclosed medical record cannot be un-disclosed, and neither can a physician's prescribing profile. A startup that uses production patient data to train or test a model without a BAA, without de-identification that meets the HIPAA standard (45 C.F.R. § 164.514), and without patient authorization has not moved fast. It has generated a breach obligation, a potential CMIA claim in California at $1,000 per patient (Cal. Civ. Code § 56.36(b)), FTC exposure, and a due diligence problem that will surface at the next funding round. A startup that trains its HCP engagement model on licensed prescriber data outside the permitted use has a contract termination waiting for it. The founders who treat data classification and vendor agreements as launch requirements rather than post-Series-A cleanup are the ones who survive their first breach inquiry.

What Good Governance Looks Like

Governance does not mean banning AI. It means knowing which tools are sanctioned, what data may go into each, and who is accountable. That starts with an inventory of every AI tool in use, including the unofficial ones. It requires a written acceptable-use policy that names PII, PHI, and collateral information like prescriber data in plain language and says which tools may receive which category. It requires BAAs and data licenses that prohibit vendor model training on your data. It requires training that shows employees the real difference between a de-identified data set and a spreadsheet with the names deleted. And it requires giving people an approved tool, because employees turn to shadow AI when the company offers nothing.

This area is moving faster than courts and regulators can keep up, and the enforcement picture for AI-specific rules is still forming. What is not in doubt is the underlying principle. Entering a patient's record or a physician's prescribing profile into a system you do not control is a disclosure, and disclosures have consequences. Patient privacy and physician trust are both things that break. Neither one gets patched in the next release.

This post is for informational purposes only and is not legal advice. Reading it does not create an attorney-client relationship.