Listen and followApple PodcastsSpotifyYouTubeMore apps · Browse all episodes
A patient arrives already sure of what is wrong with them. The words they use came from somewhere else. Alan P. Feren is a retired surgeon, health care consultant, and patient advocate. This episode is based on his article “Clinical authorship: the missing principle in medical AI,” published on KevinMD. He walks through the three points where artificial intelligence enters a visit: the patient shaping their story beforehand, the clinician leaning on decision support, and the ambient tool interpreting it after. No single step is reckless, he says, and that is the problem, because a diagnosis can travel from encounter to encounter with no one standing behind it. You will hear the questions he asks patients about what fits their experience and what does not, why an ambient note is an interpretation rather than a transcript, and why he puts actual intelligence first. If you sign notes you did not write, this conversation will change what you check first.
Partner with me on the KevinMD platform. With over three million monthly readers and half a million social media followers, I give you direct access to the doctors and patients who matter most. Whether you need a sponsored article, email campaign, video interview, or a spot right here on the podcast, I offer the trusted space your brand deserves to be heard. Let’s work together to tell your story.
PARTNER WITH KEVINMD → https://kevinmd.com/influencer
SUBSCRIBE TO THE PODCAST → https://www.kevinmd.com/podcast
Transcript
Kevin Pho: Hi, and welcome to the show. Subscribe at KevinMD.com/podcast. Today we welcome back Alan P. Feren. He’s a retired surgeon, health care consultant, and patient advocate. Today’s KevinMD article is “Clinical authorship: the missing principle in medical AI.” Alan, welcome back to the show.
Alan P. Feren: Thanks, Kevin. Glad to be back.
Kevin Pho: All right, let’s start by telling us why you decided to write this latest article on KevinMD.
Alan P. Feren: Sure. As I began to think about the impact of AI in clinical medicine, and right now we have more and more of what I call AI-mediated clinical care, I thought it was important to take a closer look at the effects of using AI. One of the things that came to mind was that AI is not just present during the examination in the clinical encounter. It really strikes the clinical encounter in three distinct places. First of all, patients are now switching from just Google to AI tools and trying to identify their story using the AI. During the examination, we as physicians use it for decision support. And then with ambient AI tools, which continue to improve, we get an interpretation of what that encounter was.
So there are three possible areas where a problem can arise. The first is with the patient. If the patient is allowing the AI tool to shape his or her story, there’s a problem that can be propagated throughout the entire clinical encounter. Second, the physician needs to understand what the tool is saying: Does this really fit the patient’s story? The physician has to be able to say, “This fits,” or, “This doesn’t fit.” And then third, you have the ambient listening tool, which is really an interpretation. It’s not really a transcript.
So there has to be a safeguard to all this, and the safeguard is what I’m calling clinical authorship. Clinical authorship is not merely a signature. A signature says, “I approved what was in the encounter.” Authorship means, “I can account for this story throughout the entire process.” It includes knowing that the patient’s story is true for that patient. It includes being able to defend the decision-making, because I know what fits and what doesn’t fit, and I know what is true for this patient and what isn’t. And then lastly, it means looking at the interpretation to make sure that it still fits what was heard during the encounter.
Medical journals really understand this principle. AI can’t be an author, because it can’t stand behind its claims and it can’t defend them. We as clinicians must do that. So before the visit, patients need to remain the authors of their own story, and physicians can ask, “What did AI help you express in this visit? What part of this fits your experience, and what part doesn’t?”
Real physician voices, twice a week
Free, and one click to unsubscribe.
Kevin Pho: So from a clinical perspective, let’s say a patient uses AI to flesh out or interpret their story. Talk to us about the implications of that down the road.
Alan P. Feren: You have a cascade of effects. Let’s say a patient comes in complaining of multiple vague symptoms, and the tool is suggesting different things. It can suggest long COVID. It can suggest chronic fatigue syndrome, maybe brain fog. And then as patients think about this, they can reflect and say, “Well, maybe I’ve had some brain fog,” or, “Maybe I’ve noticed that I’ve been more tired lately.” That then gets factored in, and they come into the clinical encounter saying, “I think I’ve got chronic fatigue,” or, “I think I have symptoms consistent with long COVID.”
Then the tool itself picks up on all these things that have been said during the encounter, and it may surface long COVID, chronic fatigue syndrome, or chronic inflammatory syndrome as one of its top possibilities. And then as the physician is thinking and doing his or her clinical reasoning, they say, “Well, this kind of makes sense.” And then the ambient tool has interpreted it as saying that this is probably long COVID, or it’s probably chronic fatigue. That then becomes the story. It goes down the entire line and becomes the story that’s seen through multiple different encounters, and that can be a very dangerous thing.
Kevin Pho: So you’re saying that whenever a patient, let’s say, uses a commercial large language model and types in their symptoms or story, those models would, as you say, put words in that patient’s mouth, and in the worst case even put in diagnoses that the patient may not even have considered. And those diagnoses eventually become a permanent part of the record if they get translated into that physician encounter.
Alan P. Feren: Absolutely. Nothing in this chain that we’ve just described is reckless. Each stage seems pretty reasonable clinically, but no one has really authored the entire clinical story, and that’s what I mean by clinical authorship. Clinical authorship is really a safeguard. A signature basically says, “Yeah, I approved this.” But authorship, again, means I can account for this story and why it’s true, I can account for the clinical reasoning behind this story, and I can account for why the treatment plan is true for this patient.
The other thing, which we discussed the last time we were together, is the fact that these AI tools, as smart as they are, lack context. Social determinants of health are often not included in the context, and neither is treatment burden: whether this is a single parent, whether they have a transportation problem, whether their drugs are covered under their current insurance, whether they don’t have money, whether they’re uninsured. A patient is not trained medically to enter all the information that we as clinicians have and factor into our decision-making and clinical reasoning.
Kevin Pho: So let’s talk about AI-assisted notes from the physician’s perspective. You did mention ambient AI systems, and I think those are becoming more and more prevalent. Tell us about some of those notes that are authored by these ambient AI systems. Are they, again from the clinician side, putting words in clinicians’ mouths that they didn’t necessarily mean to write?
Alan P. Feren: That is a real possibility. I’m not saying that it’s something that occurs on a regular basis, but I think that’s one of the major risks. You really have to treat whatever is recommended or interpreted by the ambient listening tool as an interpretation. It’s not a dictation. It’s not word for word. It’s really not a transcript.
Kevin Pho: And in theory, aren’t physicians supposed to review whatever the ambient AI or AI-assisted note-writing software gives them before they actually sign off on a chart?
Alan P. Feren: Absolutely. I think you’ve had others on your podcast who have made this specific point, that the AI is really there to amplify the clinical method and the thinking. But as I’ve said before, it should always be actual intelligence first and then AI second.
Kevin Pho: You did mention a few things in terms of going forward and sussing out what part is AI-generated or AI thinking versus what comes directly from the physician or the patient. What would be a systematic way of doing that? What kinds of questions should physicians be asking to make sure that the patient’s story is truly theirs and not augmented by AI?
Alan P. Feren: Well, first, I think there’s an overlying principle, which is that you need to treat the three different entry points as one continuous clinical problem, not three separate ones. So when the patient enters, the first question is, “Have you used AI to fashion your story or to help you organize your thinking around your signs and symptoms?” If the answer is yes, then there are several questions that need to follow. The first is, “What did the AI tool help you with and help you express?” And then specifically, “What was suggested that seems to fit your experience, and what part does not fit your experience?”
Really, the reverse of this is exactly what clinicians need to ask as well. One statement that I would make, which I think is very important, is that part of clinical authorship is the discipline to disregard what AI has recommended. Guidelines do not always fit a clinical case. That’s a standard argument that we as clinicians have always used. Every patient is different and deserves to have us find what fits in the guideline and what doesn’t fit. And it’s the same with the algorithms. They are all quite accurate currently, but they have to fit the patient. So physicians really need to be able to say, “These are the things that don’t fit,” throw them out, and disregard what AI has recommended in its top two or three recommendations.
That, I think, is really the issue for the clinicians in the first two steps. For the third step, the clinicians need to take a very close look at what the interpretation was. Did it really capture the essence of the clinical encounter? Was it true for the patient? And does it support the clinical reasoning and the treatment plan that was prescribed?
Kevin Pho: Now, how feasible or realistic would that be? Because not only are physicians, of course, pressed for time, but now they have to determine what part is AI-augmented and filter out the AI-augmented part from the patient’s true story. How much of a cognitive load are you asking physicians to take on?
Alan P. Feren: I think it’s going to be more to start. I’ve listened to others on your podcast who are frequent users of ambient listening tools, and they feel that it’s just become part of their practice now. Rather than spending time after the clinical encounters working from home, they find that they can go through it quite quickly.
Kevin Pho: We’re talking to Alan P. Feren, retired surgeon, health care consultant, and patient advocate. Today’s KevinMD article is “Clinical authorship: the missing principle in medical AI.” Now, having followed this medical AI space, which is evolving and iterating so quickly, what would you say are the next steps? If you were to advise a medical AI company or an ambient scribe company on transcribing or augmenting the note, what kind of advice would you give these companies?
Alan P. Feren: Well, if there was a way to do this, and I’m not sure there is, it would be to act on my principle, not theory, of actual intelligence first. It’s about when that clinical decision-making tool shows itself in the medical record. If it comes up too early, you can have premature coherence, which is basically closing the story before it’s fully completed. I think that we as physicians still take in all this information, and we are constantly developing our differential diagnosis as the patient is speaking. The interruption by a clinical AI decision-making tool, if it comes too early in our thinking, can mold that thinking. So if there was a way to have the tool surface itself later, I think that would be very helpful.
Kevin Pho: And we’ll end, of course, with your take-home messages to the KevinMD audience.
Alan P. Feren: Before the visit begins, make sure that you understand whether your patient has used AI, and have that very brief discussion about what fits and what doesn’t fit. Treat the encounter as one continuous clinical problem, not three separate ones. Be sure that you become the author of the entire story: that you’ve helped the patient mold a story that fits for them, and that you’ve taken the discipline to either accept or disregard what the AI tool has suggested, so that you can take full responsibility for the patient’s story.
And I think the last thing that I would have to say is that the central risk here is not about hallucination, which we know AI tools do, or factual error. Most of them are quite accurate. The clinical story can look organized, but it has to be completed by you as the clinician before you sign off. And when you do sign off, you have assured yourself that you are the author of this patient’s clinical encounter and their story.
Kevin Pho: Alan, thank you so much again for coming back on the show and sharing your perspective and insight.
Alan P. Feren: Great. Thanks, Kevin.
Listen and followApple PodcastsSpotifyYouTubeMore apps · Browse all episodes
