Artificial intelligence (AI) has made huge strides in healthcare, from helping doctors make predictions about patient outcomes to streamlining administrative tasks. But recent findings suggest that these tools, intended to improve efficiency and reduce costs, may be requiring more human oversight than initially expected, raising concerns about their long-term viability.
At the University of Pennsylvania Health System, an AI algorithm was introduced to help oncologists have important end-of-life discussions with cancer patients. It predicted the likelihood of death and nudged doctors to have conversations about treatment preferences. However, a routine checkup revealed that the algorithm’s accuracy had decreased during the COVID-19 pandemic, with its predictions becoming 7% less reliable, leading to missed opportunities for doctors to advise patients about potential treatments like chemotherapy.
Dr. Ravi Parikh, an oncologist at Emory University and lead author of the study, said this AI tool failed hundreds of times to prompt necessary conversations, potentially putting patients at risk. But this wasn’t an isolated issue. Parikh believes that many medical AI systems weakened during the pandemic, with institutions failing to consistently monitor the performance of these technologies.
The problem isn’t just about flawed predictions. AI algorithms are complex tools that require regular attention and a significant amount of staffing to ensure they’re working properly. As healthcare institutions adopt more AI systems, they’re realizing that the cost of running these technologies might exceed the savings they bring in.
Nigam Shah, the chief data scientist at Stanford Health Care, noted that while AI is often seen as a way to improve care and reduce costs, it could, in reality, lead to higher healthcare expenses if not properly managed. According to the FDA’s Commissioner, Robert Califf, hospitals across the United States are struggling to validate AI algorithms for clinical use, highlighting the challenge of maintaining and testing these systems.
Despite these challenges, AI is already widespread in healthcare, used for tasks ranging from predicting patient deterioration to approving insurance claims. But evaluating whether these technologies are effective – and whether they remain reliable – is a difficult and ongoing process.
For example, a recent study at Yale Medicine showed that six early warning systems used to predict patient deterioration performed very differently. Without standard tools for comparing AI models, it becomes tough for hospitals to pick the best options for their patients.
In doctors’ offices, one common AI tool is ambient documentation, which listens to patient visits and summarizes the information for the doctor. But there’s no standard way to measure the quality of these summaries. Even small mistakes can lead to major consequences in medical care. For instance, a study at Stanford found that AI-powered models had a 35% error rate when summarizing patients’ medical histories, a concerning figure when precision is critical.
At Mass General Brigham in Boston, a test of a genetic counseling tool revealed another problem: “nondeterminism,” where the AI gave different answers to the same question. These inconsistencies further illustrate the unpredictable nature of AI in healthcare.
So, what can hospitals do to address these issues? It comes down to investing in people. At Stanford, it took months and hundreds of hours to audit just two models for fairness and reliability. The idea of monitoring AI systems with another layer of AI is gaining traction, but experts warn that this approach would require even more resources.
As healthcare institutions continue to adopt AI, it’s clear that they need to strike a balance between technological innovation and human oversight. Without proper monitoring and resources, AI could end up costing more than it saves.
Read more:
- The High Cost Of Healthcare In America: What’s Behind It?
- Obesity Treatments Drive Sharp Rise In Employer Healthcare Costs For 2025
- What The UHC Suspect Failed To See In His Healthcare Railing?
