AI-generated medical responses need monitoring, study finds

To tackle the rising administrative and documentation responsibilities for healthcare professionals, electronic health record (EHR) vendors have adopted generative AI algorithms to aid in drafting responses to patients.

However, Mass General Brigham researchers said that the efficiency, safety and clinical impact of these algorithms had been unknown prior to this adoption.

In a new study, the researchers found that while LLMs may help reduce physician workload and improve patient education, limitations in the algorithm’s responses could affect patient safety, suggesting that ‘vigilant oversight’ is essential for safe usage.

The research team used OpenAI’s GPT-4 to generate 100 scenarios about patients with cancer and an accompanying patient question. No questions from actual patients were used for the study.

GPT-4 responded to the generated questions, as well as six radiation oncologists. The same radiation oncologists were then provided with the LLM-generated responses for review and editing.

The study found that the radiation oncologists did not know whether GPT-4 or a human had written the responses, and in 31 per cent of cases, believed that an LLM-generated response had been written by a human.