“Artificial intelligence (AI) has been developed that outperform doctors.” Such kind of headline is commonly seen in news reports these days. If an AI system that outperforms physicians is out in the market, why don’t hospitals immediately use it inpatient care?
The U.S. Food and Drug Administration has approved more than 30 AI-using medical devices as of 2019. In Korea, the Ministry of Food and Drug Safety also has granted the nod for more than 15 AI devices until last year. A local AI developer went public recently.
However, the general public and patients might still feel somewhat distant to the recent changes in AI development.
|Choi Yoon Sup|
I will talk about what a competent AI doctor means and how to judge its accuracy and usefulness.
To develop an AI program and verify its performance, we need data. For example, let’s say a hospital develops an AI program to find lesions on X-ray scans. If the hospital has 10,000 X-ray images, the hospital uses 80 percent of the randomly selected data for machine learning. The hospital uses the rest of the data to verify the accuracy of the AI program.
We call the evaluation of the accuracy, “internal validation,” because both the 80 percent of data for machine learning and the 20 percent data for verification come from the same hospital. Most of the news articles saying, “AI outperformed doctors” are based on this internal verification. Numerous papers found that AI showed high accuracy in internal verification.
However, we cannot assess the accuracy of AI by internal verification alone because it does not reflect the real world sufficiently. For more rigorous checking, we could do external validation on data from different environments (such as data from another hospital or data taken with other X-ray equipment), or conduct a randomized and controlled trial. There are not many AI-based healthcare programs that went through such meticulous verification.
Furthermore, it is essential to understand that “AI is accurate” is one thing, and “being able to treat patients better” is another. Just because AI helped the reading with high accuracy does not necessarily mean AI will surely improve treatment results. The reading may not be confirmed. The diagnosis may not mean access to the right medicine. Or doctors might not have a problem to treat a patient without the help of AI.
Distinguishing “accuracy” from “better treatment for a patient” is not only medically but industrially significant because it will be the criteria of whether an AI-based healthcare device could get the national health insurance coverage. For a domestic medical device maker, it is crucial to get the reimbursement, businesswise.
According to the AI guidelines in radiology recently published by the Ministry of Health and Welfare, an AI device that is “accurate” will not get the national health insurance benefit, except for exceptional cases. Only AI programs that prove the improvement of the therapeutic effect will be considered for a review of the health insurance coverage. There could be one more step to add – to prove an improvement in treatment compared to cost-effectiveness.
The guidelines are meaningful because they provided a benchmark for the reimbursement of AI devices in healthcare that are rare in the world. However, the guidelines were somewhat conservative, taken from the industry’s point of view. The guidelines made it difficult for most developers of authorized AI technologies in radiology to get the insurance benefit. Companies will now have to conduct an additional clinical trial to prove the improved therapeutic effect for reimbursement or look for a new business model that does not need reimbursement. Also, even if an AI device is covered by national health insurance, it is important to monitor how many levels the government will set the reimbursement rate on AI devices.
AI in healthcare has been advancing remarkably to nearly reaching a full-scale adoption in patient care. The establishment of the guidelines on regulations and reimbursement means that AI technologies have come closer to doctors and patients. The guidelines on the reimbursement on AI devices may be unsatisfactory for the industry. However, there could be a positive side in terms of determining the direction of the business because the guidelines reduced uncertainty over-regulation.
Through reasonable evaluation and compensation for the development of AI technologies, I hope that there will be positive results for both developers and users of the technologies.
<© Korea Biomedical Review, All rights reserved.>