News

A new study led by investigators from Mass General Brigham demonstrated a vulnerability in that LLMs are designed to be sycophantic, or excessively helpful and agreeable, which leads them to overwhelmingly fail to appropriately challenge illogical medical queries despite possessing the information necessary to do so. As a community, we need to work on training both patients and clinicians to be safe users of LLMs, and a key part of that is going to be bringing to the surface the types of errors that these models make," said corresponding author Danielle Bitterman, MD, a faculty member in the Artificial Intelligence in Medicine (AIM) Program and Clinical Lead for Data Science/AI at Mass General Brigham. Next, the researchers sought to determine the effects of explicitly inviting models to reject illogical requests and/or prompting the model to recall medical facts prior to answering a question. Lastly, the researchers fine-tuned two of the models so that they correctly rejected 99–100% of requests for misinformation and then tested whether the alterations they had made led to over-rejecting rational prompts, thus disrupting the models' broader functionality."