The 2 a.m. Second Opinion

In a recent piece on AI and de-skilling, I buried a bullet in the "Where I Might Be Wrong" section that I've been chewing on:

I may be underestimating how much augmented intelligence will reshape communication itself. If patients arrive having already received AI-generated prognostic estimates and AI-drafted care plans, the starting point of the serious illness conversation changes.

I've been sitting with that sentence for a while now. It's bigger than a bullet point. It might be bigger than a single newsletter. But let me try.

Because here's what I think is happening—quietly, unevenly, and faster than our field is tracking: the foundational architecture of serious illness communication is being restructured by a force we didn't design, don't control, and in many cases don't even know is in the room.

The Invisible Consultant

Your patient received a prognosis last night. Not from you. Not from their oncologist. Not from a family member who talked to someone who talked to a nurse.

From ChatGPT. Or Gemini. Or Claude. Or whatever large language model they reached for at 2 a.m. when the terror hit and the clinic was closed and the patient portal had nothing but lab values they couldn't interpret.

This is not speculative. Hundreds of millions of people are now turning to AI chatbots for medical guidance. ECRI named AI chatbot misuse the number one health technology hazard for 2026. An Oxford study published this month warned that LLMs present direct risks to people seeking medical advice due to inaccurate and inconsistent information. And a recent piece in Science News noted that most physicians are now using chatbots in some fashion—while patients are simultaneously using those same tools to generate their own clinical narratives.

So when you walk into the family meeting, you may be the second—or third—source of prognostic information that family has encountered. You just don't know it. And they may not tell you, because the AI sounded confident, detailed, and definitive in a way that felt like a real answer to a terrifying question.

This is not an information problem. It is a clinical problem. And it is about to fundamentally reshape what the serious illness conversation is actually for.

The Prompt Quality Gap (or: Why Scared People Are the Worst Prompt Engineers)

Here's the thing that gets lost in the breathless coverage of AI outperforming physicians on board exams: the conditions under which AI performs well are nothing like the conditions under which patients use it.

The emerging evidence is consistent: AI achieves near-physician-level diagnostic accuracy when it receives structured, complete, board-question-style inputs—every relevant detail present, nothing extraneous, clinical framing precise. My own research shows that the LLMs fall down when the prompts look more like what palliative care teams see every day. A physician querying an augmented intelligence tool with "58-year-old, ECOG 3, metastatic NSCLC with brain mets, progressed through pembrolizumab, second-line options and median survival data" gets something useful. A terrified spouse typing "my husband has lung cancer that spread to his brain and the chemo stopped working how long does he have" at 2 a.m. gets something very different.

It will sound authoritative. It will be detailed. It will cite studies. And it will be calibrated to nothing—not to the specific histology, not to the performance status, not to the comorbidities, not to the goals and values that should shape how prognostic information is framed and delivered in the first place.

The gap between AI-with-perfect-input and AI-with-frightened-human-input is where the clinical harm lives. And that gap is not narrowing as fast as the accuracy headlines suggest, because the problem isn't the model. The problem is that real patients prompt with fear, not with structured clinical data. They ask the question shaped by what terrifies them most, not the question that would generate the most accurate answer.

The Fragmented Family System

This is where it gets genuinely dangerous for the work we do.

Picture this: A patient with advanced heart failure and worsening renal function is admitted after a third hospitalization in six months. The palliative care team is consulted. You prepare for the family meeting.

What you don't know:

• The patient asked an AI chatbot two weeks ago, "Can I live ten more years with heart failure?" and received an optimistic answer because the prompt didn't include the ejection fraction, the creatinine trajectory, or the recurrent hospitalizations.
• The patient's daughter—a nurse—queried a different AI with more clinical detail and received a much more sobering estimate, which she has not shared with her father because she doesn't want to "take away his hope."
• The patient's son asked ChatGPT "what happens when kidneys fail in heart failure patients" and has been quietly researching dialysis options he hasn't discussed with anyone.

Three family members. Three AI-mediated narratives. Three different emotional realities, each shaped by a different prompt, a different model, a different level of health literacy, and a different set of fears. Each person has already done emotional processing—grief, bargaining, research, planning—around information the clinical team didn't generate and doesn't know about.

You walk into the room and ask, "What do you understand about where things are with your illness?"

The patient says, "I know it's serious, but I've read that people can live a long time with this."

He's not wrong, exactly. He's not lying. He's reporting what he was told by a source he experienced as credible. He just didn't give that source the information it needed to be accurate for him.

And his daughter is sitting in the corner, holding a completely different number in her head, wondering whether to speak.

This is the clinical reality that's coming. Not in five years for everyone. But for enough families, soon enough, that we need to be thinking about it now.

Our Frameworks Weren't Built for This (and That's Not a Failure—It's a Signal)

Let me be direct about something: VitalTalk's approach to serious illness communication and the Serious Illness Conversation Guide developed by Ariadne Labs are among the most important contributions our field has produced. I teach with them. They have improved the quality of care for countless patients and families. Nothing I'm about to say diminishes that.

AND.

They were designed for a specific informational architecture—one where the clinician is the primary source of prognostic information, and the patient's "illness understanding" represents either accurate knowledge received from prior clinical encounters or a gap waiting to be filled.

The calibration step—"What do you understand about your illness?" or "What have you been told?"—was built to gauge how much the clinician needs to share. It works beautifully for that purpose. If the patient has an excellent baseline understanding, you skip the review and move to values and goals. If there are gaps, you fill them with skill and compassion. This is elegant, evidence-based design and it has served us well.

What this step was not designed to detect is confidently held, AI-generated misinformation that the patient has already emotionally processed. That is a categorically different clinical situation than a knowledge gap. A gap invites filling. A confident but inaccurate narrative—one the patient experienced as a personalized, authoritative answer to their most frightening question—requires something harder: correction without condescension, recontextualization without dismissal, and emotional re-integration of a story the patient has already begun to live inside.

And the "Tell" step—where the clinician shares prognostic information—was designed for a world where the clinician was the first or primary source. When the clinician is now the second or third source, after an LLM and possibly a family member who relayed a different LLM output, "Tell" becomes something more complex. The task isn't disclosure. It's re-contextualization. That requires skills we haven't named, haven't formalized, and aren't yet explicitly training for.

To be absolutely clear one more time: this is not a critique of VitalTalk or the SICG. It is a recognition that the world these tools were designed for is changing, and the tools need to evolve with it. The architects of these frameworks know this—they've always emphasized that serious illness communication is iterative, not formulaic. What I'm flagging is that the nature of the iteration is about to shift, and we should be ahead of it rather than behind it.

The Equity Fracture

This is where the disruption stops being an abstract communication challenge and becomes a justice problem.

The patients most likely to arrive at the goals-of-care conversation having already queried an AI are younger, English-speaking (in my corner of the world), digitally literate, and commercially insured. These patients are also—on average—more likely to receive higher-quality serious illness communication in the first place, because the documented disparities in goals-of-care conversations track along exactly these axes.

The patients most likely to prompt an AI poorly—and therefore receive the most dangerously miscalibrated outputs—are those with lower health literacy, for whom English is not a first language, or who lack the clinical vocabulary to construct a prompt that generates an accurate, contextualized answer. These are the same patients who already face the steepest barriers to high-quality serious illness communication.

And the patients least likely to have queried an AI at all—older, less digitally engaged, more reliant on family intermediaries—are not exempt from this disruption. Their family members are querying AI on their behalf, often without telling them, often with incomplete information, and often arriving at the bedside with an emotional narrative built on an AI output the patient hasn't seen and the clinical team doesn't know about.

The result is a bifurcated—and in some families, trifurcated—clinical encounter:

• Patient A arrives with a detailed, somewhat accurate AI-generated understanding that needs recontextualization and integration with the clinical picture. The clinician's task is nuanced correction.
• Patient B arrives with a confident, badly miscalibrated AI narrative shaped by a poorly constructed prompt and low health literacy. The clinician's task is gentle but substantive correction—without alienating someone who trusted a source they experienced as credible and accessible.
• Patient C has no AI-mediated narrative at all, but their daughter has one she hasn't shared. The clinician's task is the traditional one, except the family system is already fragmented by information asymmetry the clinician can't see.

Three different clinical tasks. The current frameworks treat them as one. The same clinician may encounter all three in a single afternoon.

This is not a theoretical concern. The disparities in serious illness communication quality are well-documented along race, ethnicity, language, and socioeconomic axes. AI-mediated preinformation will deepen those disparities unless we deliberately design for it. The patients with the greatest access to high-quality AI tools and the health literacy to use them effectively are the patients who least need the help. The patients who most need accurate, compassionate, contextualized prognostic information are the ones most likely to receive the worst version of it from an AI—or to have a family member receive a bad version on their behalf.

If we don't name this, it becomes another layer of the invisible architecture that externalizes suffering onto the people least able to bear it.

This Is an IDT Problem, Not Just a Physician Problem

Physicians—and I'm note immune—will want to make this about diagnostic accuracy. We'll want to debate whether AI-generated prognoses are good enough, cite the studies where they are and the studies where they aren't, and design systems to correct the medical facts.

That's necessary. It's also insufficient.

Because the clinical complexity of the AI-preinformed family meeting is not primarily a medical-accuracy problem. It is a relational problem, an emotional problem, and a systems problem. And those are the domains where the interdisciplinary team lives.

The chaplain who helps a patient re-process meaning when their AI-generated timeline collapses—when the "ten more years" they'd been holding onto becomes "we're talking about months"—is doing spiritual care that no physician is expert in and no algorithm can approximate.

The social worker who maps a fragmented family information landscape—who notices that the daughter and the son are operating from different AI-mediated narratives and helps them surface and integrate what they each know—is doing systems work that determines whether the family meeting produces alignment or fracture.

The nurse who notices that the patient has stopped asking questions—not because they're at peace, but because they believe they already have the answers—is detecting a clinical signal that the physician, focused on the medical conversation, might miss entirely.

I wrote in What Physicians Are (Actually) For on Specialist Palliative Care Teams that our role is to make space, translate, and selectively intervene—then get out of the way. The AI-preinformed encounter makes that argument sharper, not weaker. The physician corrects the medical facts. The team does the rest.

This is yet another reason why Tier 1 programs without a full interdisciplinary team will be most vulnerable to this disruption. When the serious illness conversation was purely clinician-driven, a single-provider model could approximate the core function. When the conversation requires navigating a family system pre-informed by multiple AI-mediated narratives, the absence of the full IDT becomes a patient safety issue.

Where I Might Be Wrong

• Maybe patients don't trust AI for prognosis. The NPR and ECRI data suggest otherwise, but it's possible that the emotional stakes of serious illness are high enough that most patients defer to their physicians. I'd like that to be true. I'm not confident it is—because people reach for AI precisely when they feel they can't reach their physician, which is often.
• Maybe AI accuracy with real-world prompts improves fast enough that the prompt-quality problem resolves. If LLMs learn to ask clarifying questions before generating prognostic estimates—"Can you tell me the specific type of cancer? Do you know the stage?"—the output quality for patient-generated prompts could improve substantially. This is technically feasible. Whether consumer-facing AI tools will implement it is a product-design question, not a technology question.
• Maybe the fragmented family system is self-correcting. Families talk to each other. They compare notes. It's possible that competing AI narratives surface and resolve before the clinical encounter, arriving as "we've been reading different things and we're confused," which is actually a workable starting point. I'd love that to be the modal case. I suspect it won't be, because families dealing with serious illness often don't talk to each other openly about prognosis—that's why we exist.
• Maybe I'm overweighting the near-term disruption. Perhaps this is a 15-year problem, not a 5-year one, and there's time to adapt gradually. But the trajectory of AI adoption in consumer health suggests otherwise, and palliative care has a well-documented tendency to arrive late to structural shifts—we spent a decade on the wrong problem once already.

What We Should Be Building

This piece would fail my own test if I stopped at the diagnosis. So here's what I think needs to happen:

1. Evolve the frameworks. VitalTalk and the SICG should add explicit guidance for the AI-preinformed encounter. The calibration step needs a companion: not just "What do you understand about your illness?" but "Where have you been getting your information, and what did it tell you?" That second question—asked with genuine curiosity, not judgment—opens the door to detecting AI-mediated narratives without shaming the patient for seeking them. This is a small technical addition with large clinical implications.

2. Name the new clinical skill. We don't have a word for what happens when a clinician must help a patient re-process prognostic information they've already received from a confident but poorly calibrated source. It's not "breaking bad news"—the news has already been broken, badly. It's not "correcting misinformation"—that frames the patient as wrong rather than underserved by a tool that failed them. I'd propose something like narrative co-creation: the clinical act of entering the story the patient has already begun to build—acknowledging what they were told, honoring the emotional processing they've already done—and then constructing, together, a more accurate and more complete version that can hold the weight of the decisions ahead. The patient's narrative isn't discarded. It's the raw material. The clinician brings the clinical picture. The patient brings the values, the fears, the meaning they've already made. What emerges is neither the AI's story nor the clinician's. It's theirs.

3. Train fellows for the encounter that's coming. This threads directly back to the argument I made in the de-skilling piece: we are not preparing the next generation for the AI-infused clinical environment. In the context of serious illness communication specifically, fellows need practice navigating conversations where the patient has already received AI-generated information—including practice with wrong AI-generated information, because that's the harder clinical scenario. Simulation, roleplay, observed encounters. Not a module. A posture.

4. Design for the equity implications now. If we know that AI-mediated preinformation will deepen disparities in serious illness communication, we have an obligation to design countermeasures before the harm compounds. That might mean patient-facing tools that are designed for low-health-literacy users—tools that ask clarifying questions, refuse to generate prognostic estimates without sufficient clinical data, and actively direct users to their clinical teams. It might mean institutional intake workflows that screen for AI-mediated information as part of the serious illness encounter. It most certainly means that equity audits of AI in palliative care—which I called for in the augmented intelligence piece—must include the patient-facing dimension, not just the clinician-facing one.

5. Research this. We need data. How many patients with serious illness are querying AI about their prognosis? What are they asking? What are they being told? How does AI-generated prognostic information compare to clinician-delivered information in terms of accuracy, emotional impact, and downstream decision-making? How do families manage competing AI-mediated narratives? None of these questions have been studied in the palliative care context. This is a research agenda hiding in plain sight.

The Conversation Is Already Changing. The Question Is Whether We'll Notice in Time.

In Augmented Intelligence in Palliative Care, I argued that the real story isn't scribes—it's structural redesign. In the de-skilling piece, I argued that palliative care is uniquely protected from AI-driven skill erosion because our highest-value competencies are relational and interpretive, not pattern-based.

Both of those arguments still hold. But this piece complicates them—because the tools we celebrate when they offload our clinical busywork are the same category of tools that patients are using to bypass our clinical conversations. We cannot champion augmented intelligence for the clinician and ignore what happens when the patient has the same technology, fewer guardrails, and more fear.

The serious illness conversation isn't going away. If anything, it becomes more important—because the clinical task is no longer just "share prognostic information with skill and compassion." It's "help this family integrate what they've already been told, by sources you didn't choose, into a coherent understanding that supports the decisions they actually need to make."

That's harder. It requires more skill, not less. It demands the full interdisciplinary team. And it will widen existing disparities if we don't design for equity from the start.

We've been here before. We watched the health system redefine palliative care without us. We watched hospice buckle under structural choices we made. We watched our communication frameworks become the gold standard—and then paused, as if the work of evolving them was someone else's job.

It isn't.

The patient already has a prognosis. The family already has three. The conversation is already happening—at 2 a.m., in the dark, between a terrified human and a machine that never hesitates.

Our job was never to be the only voice in the room. It was to be the one that brought context, compassion, and truth into the hardest conversations a family will ever have.

That job just got more complicated. And more necessary.

Time to evolve.

I am a palliative care physician, educator, and professional strategery expert. Known for turning rounds into rants and rants into teaching points. Rounds & Rants represents my views — not those of any institution or professional membership organization where I hold a role. I don't write on their behalf and they don't vet what I publish.