Adding a Lookup Step Makes AI Better at Assigning Medical Diagnosis Codes
A new study from researchers at the Mount Sinai Health System suggests that a simple tweak to how artificial intelligence (AI) assigns diagnostic codes could significantly improve accuracy, even outperforming physicians. The findings, reported in the September 25 online issue of NEJM AI [DOI: 10.1056/AIcs2401161], could help reduce the time doctors spend on paperwork, cut billing errors, and improve the quality of patient records.
“Our previous study showed that even the most advanced AI could produce the wrong codes, sometimes nonsensical ones, when left to guess,” says co-corresponding senior author Eyal Klang, MD, Chief of Generative AI in the Windreich Department of Artificial Intelligence and Human Health at the Icahn School of Medicine at Mount Sinai. “This time, we gave the model a chance to reflect and review similar past cases. That small change made a big difference.”
Doctors in the United States spend hours every week assigning ICD codes—alphanumeric strings, used to describe everything from sprained ankles to heart attacks. But large language models, like ChatGPT, often struggle to assign these codes correctly. To address this, the researchers tried a “lookup-before-coding” method that first prompts the AI to describe a diagnosis in plain language and then choose the most fitting code from a list of real-world examples. The approach delivered greater accuracy, fewer mistakes, and performance on par with or better than humans.
The team utilized 500 Emergency Department patient visits at Mount Sinai Health System hospitals. For each case, they fed the physician’s note to nine different AI models, including small open-source systems. First, the models generated an initial ICD diagnostic description. Using a retrieval method, each description was matched to 10 similar ICD descriptions from a database of more than 1 million hospital records, along with how often those diagnoses occurred. In a second step, the model used this retrieved information to select the most accurate ICD description and code.
Emergency physicians and two independent AI systems evaluated the coding results independently, without information about whether the codes were generated by AI or clinicians.
Across the board, models that used the retrieval step outperformed those that didn’t, and even did better than physician-assigned codes in many cases. Surprisingly, even small open-source models performed well when allowed to “look up” examples.
“This is about smarter support, not automation for automation’s sake,” says co-corresponding senior author Girish N. Nadkarni, MD, MPH, Chair of the Windreich Department of Artificial Intelligence and Human Health, Director of the Hasso Plattner Institute for Digital Health, and Irene and Dr. Arthur M. Fishberg Professor of Medicine at the Icahn School of Medicine at Mount Sinai, and Chief AI Officer for the Mount Sinai Health System. “If we can cut the time our physicians spend on coding, reduce billing errors, and improve the quality of our data, all with an affordable and transparent system, that’s a big win for patients and providers alike.”
The authors emphasize that this retrieval-enhanced method is designed to support, not replace, human oversight. While it’s not yet approved for billing and was tested specifically on primary diagnosis codes from emergency visits discharged home, it shows encouraging potential for clinical use. The researchers see immediate uses, such as suggesting codes in electronic records or flagging errors before billing.
The investigators are now integrating the method into Mount Sinai’s electronic health records system for pilot testing. They hope to expand it to other clinical settings and to include secondary and procedural codes in future versions.
“The big picture here is AI’s potential to transform how we care for patients. When technology relieves the administrative burden of our physicians and other providers, they have more time for direct patient care. That’s good for clinicians, that's good for patients and it’s good for health systems of every size,” says David L. Reich MD, Chief Clinical Officer of the Mount Sinai Health System and President of The Mount Sinai Hospital. “Using AI in this way improves our ability to provide attentive and compassionate care by spending more time with patients. This strengthens the foundation of hospitals and health systems everywhere.”
The paper is titled “Assessing Retrieval-Augmented Large Language Models for Medical Coding.”
The study’s authors, as listed in the journal, are Eyal Klang, Idit Tessler, Donald U. Apakama, Ethan Abbott, Benjamin S Glicksberg, Arnold Monique, Akini Moses, Ankit Sakhuja, Ali Soroush, Alexander W. Charney, David L. Reich, Jolion McGreevy, Nicholas Gavin, Brendan Carr, Robert Freeman, and Girish N Nadkarni.
This work was supported by the Clinical and Translational Science Awards (CTSA) grant UL1TR004419 from the National Center for Advancing Translational Sciences. Research reported in this publication was also supported by the Office of Research Infrastructure of the National Institutes of Health under grant award numbers S10OD026880 and S10OD030463.
For more Mount Sinai artificial intelligence news, visit: https://icahn.mssm.edu/about/artificial-intelligence.
About Mount Sinai's Windreich Department of AI and Human Health
Led by Girish N. Nadkarni, MD, MPH—an international authority on the safe, effective, and ethical use of AI in health care—Mount Sinai’s Windreich Department of AI and Human Health is the first of its kind at a U.S. medical school, pioneering transformative advancements at the intersection of artificial intelligence and human health.
The Department is committed to leveraging AI in a responsible, effective, ethical, and safe manner to transform research, clinical care, education, and operations. By bringing together world-class AI expertise, cutting-edge infrastructure, and unparalleled computational power, the department is advancing breakthroughs in multi-scale, multimodal data integration while streamlining pathways for rapid testing and translation into practice.
The Department benefits from dynamic collaborations across Mount Sinai, including with the Hasso Plattner Institute for Digital Health at Mount Sinai—a partnership between the Hasso Plattner Institute for Digital Engineering in Potsdam, Germany, and the Mount Sinai Health System—which complements its mission by advancing data-driven approaches to improve patient care and health outcomes.
At the heart of this innovation is the renowned Icahn School of Medicine at Mount Sinai, which serves as a central hub for learning and collaboration. This unique integration enables dynamic partnerships across institutes, academic departments, hospitals, and outpatient centers, driving progress in disease prevention, improving treatments for complex illnesses, and elevating quality of life on a global scale.
In 2024, the Department's innovative NutriScan AI application, developed by the Mount Sinai Health System Clinical Data Science team in partnership with Department faculty, earned Mount Sinai Health System the prestigious Hearst Health Prize. NutriScan is designed to facilitate faster identification and treatment of malnutrition in hospitalized patients. This machine learning tool improves malnutrition diagnosis rates and resource utilization, demonstrating the impactful application of AI in health care.
For more information on Mount Sinai's Windreich Department of AI and Human Health, visit: ai.mssm.edu
About the Hasso Plattner Institute at Mount Sinai
At the Hasso Plattner Institute for Digital Health at Mount Sinai, the tools of data science, biomedical and digital engineering, and medical expertise are used to improve and extend lives. The Institute represents a collaboration between the Hasso Plattner Institute for Digital Engineering in Potsdam, Germany, and the Mount Sinai Health System.
Under the leadership of Girish Nadkarni, MD, MPH, who directs the Institute, and Professor Lothar Wieler, a globally recognized expert in public health and digital transformation, they jointly oversee the partnership, driving innovations that positively impact patient lives while transforming how people think about personal health and health systems.
The Hasso Plattner Institute for Digital Health at Mount Sinai receives generous support from the Hasso Plattner Foundation. Current research programs and machine learning efforts focus on improving the ability to diagnose and treat patients.
About the Icahn School of Medicine at Mount Sinai
The Icahn School of Medicine at Mount Sinai is internationally renowned for its outstanding research, educational, and clinical care programs. It is the sole academic partner for the seven member hospitals* of the Mount Sinai Health System, one of the largest academic health systems in the United States, providing care to New York City’s large and diverse patient population.
The Icahn School of Medicine at Mount Sinai offers highly competitive MD, PhD, MD-PhD, and master’s degree programs, with enrollment of more than 1,200 students. It has the largest graduate medical education program in the country, with more than 2,600 clinical residents and fellows training throughout the Health System. Its Graduate School of Biomedical Sciences offers 13 degree-granting programs, conducts innovative basic and translational research, and trains more than 560 postdoctoral research fellows.
Ranked 11th nationwide in National Institutes of Health (NIH) funding, the Icahn School of Medicine at Mount Sinai is among the 99th percentile in research dollars per investigator according to the Association of American Medical Colleges. More than 4,500 scientists, educators, and clinicians work within and across dozens of academic departments and multidisciplinary institutes with an emphasis on translational research and therapeutics. Through Mount Sinai Innovation Partners (MSIP), the Health System facilitates the real-world application and commercialization of medical breakthroughs made at Mount Sinai.
-------------------------------------------------------
* Mount Sinai Health System member hospitals: The Mount Sinai Hospital; Mount Sinai Brooklyn; Mount Sinai Morningside; Mount Sinai Queens; Mount Sinai South Nassau; Mount Sinai West; and New York Eye and Ear Infirmary of Mount Sinai.
About the Mount Sinai Health System
Mount Sinai Health System is one of the largest academic medical systems in the New York metro area, with 48,000 employees working across seven hospitals, more than 400 outpatient practices, more than 600 research and clinical labs, a school of nursing, and a leading school of medicine and graduate education. Mount Sinai advances health for all people, everywhere, by taking on the most complex health care challenges of our time—discovering and applying new scientific learning and knowledge; developing safer, more effective treatments; educating the next generation of medical leaders and innovators; and supporting local communities by delivering high-quality care to all who need it.
Through the integration of its hospitals, labs, and schools, Mount Sinai offers comprehensive health care solutions from birth through geriatrics, leveraging innovative approaches such as artificial intelligence and informatics while keeping patients’ medical and emotional needs at the center of all treatment. The Health System includes approximately 9,000 primary and specialty care physicians and 10 free-standing joint-venture centers throughout the five boroughs of New York City, Westchester, Long Island, and Florida. Hospitals within the System are consistently ranked by Newsweek’s® “The World’s Best Smart Hospitals, Best in State Hospitals, World Best Hospitals and Best Specialty Hospitals” and by U.S. News & World Report's® “Best Hospitals” and “Best Children’s Hospitals.” The Mount Sinai Hospital is on the U.S. News & World Report® “Best Hospitals” Honor Roll for 2025-2026.
For more information, visit https://www.mountsinai.org or find Mount Sinai on Facebook, Instagram, LinkedIn, X, and YouTube.
Like Humans, AI Can Jump to Conclusions, Mount Sinai Study Finds
Jul 22, 2025 View All Press Releases
Physical Activity Boosts Mental Health in Women With Chronic Pelvic Pain Disorders
Feb 26, 2025 View All Press Releases