A Healthcare AI Case Study: What Response Times Show

Nine seconds. That's the entire time savings Memorial Hermann Health System measured when clinicians used an AI assistant to draft replies to patients asking about test results, according to a health IT trade report published this month by HealthSystemCIO.

It's an odd number to build a healthcare AI case study around, and that's the point. Vendors selling AI for patient messaging promise that response times will collapse.

The hospitals that actually measured it found something quieter. Children's Hospital of Philadelphia saw almost the same pattern.

Over a year-long pilot running from August 2023 to July 2024 across 13 ambulatory specialties, described in a study in Applied Clinical Informatics, AI-drafted messages saved clinicians about four seconds each. That gap was too small to call statistically significant.

Support staff did somewhat better, and behavioral health staff did best of anyone, since those replies tend to run long and repetitive to begin with. The exact split is below.

Why the Time Savings Look So Small on Paper

The numbers line up too closely across two unrelated health systems to be a fluke.

Typing was never the bottleneck. Reading the chart and deciding what to say was.

UC San Diego Health ran the same experiment with generative AI drafting inside Epic, on physicians who each field patient messages all week long. A JAMA Network Open study on that pilot reported that AI drafts did not reduce response time either.

What changed was the message itself. One researcher on the study noted that the AI doesn't get tired, so its drafts stayed longer and more compassionate even at the end of a long shift.

The Number Every Healthcare AI Case Study Skips

Back at Memorial Hermann, patient follow-up questions about test results dropped by nearly a third once clinicians started sending AI-drafted comments.

That's the actual business case, and it has nothing to do with how fast any single reply went out. A clearer first message means fewer second messages.

Stanford Health Care found a version of the same effect from the other direction. Across 162 primary care and gastroenterology clinicians over five weeks, a Stanford Medicine study found the AI drafts didn't objectively save clinicians time.

Yet clinicians still reported less cognitive burden and fewer feelings of burnout. Answering from a draft is a lighter task than answering from a blank box, even when the clock says otherwise.

That lines up with what NYU Langone found when primary care physicians rated hundreds of blinded pairs of AI and human replies. The AI versions scored better on tone and were rated as empathetic well over twice as often.

None of that shows up in a stopwatch. It shows up in whether a patient reads the reply and feels answered, or writes back with the same question in different words.

What This Means Before You Buy AI for Patient Messaging

Photo by erwinbosman on Pixabay

If a vendor pitches AI to your practice on shaved seconds, ask for the follow-up-message number instead.

That's the metric Memorial Hermann's own data says actually moves. It's the one that shows up in staffing costs and patient satisfaction scores months later, not on a stopwatch during the demo.

Getting the tone of an AI-drafted reply right, so it actually cuts repeat questions instead of just going out faster, is a design problem as much as an engineering one.

It's the kind of work our UI/UX team does alongside AI and automation builds. That was close to what we did in a recent healthcare accessibility project, where the win wasn't a faster system. It was one patients didn't need to contact twice.

Cover photo by Tima Miroshnichenko on Pexels