In January we provided results from a study evaluating the accuracy of a deep learning algorithm in detecting diabetic retinopathy. Recently, one of the researchers from that trial-- Ehsan Rahimy, MD, a surgical and medical vitreoretinal specialist at the Palo Alto Medical Foundation in California--offered additional details about his team’s work with Google.
Investigators asked 10 clinicians (5 general ophthalmologists, 4 retinal specialists, and 1 retinal fellow) to read nearly 1800 images and grade for diabetic retinopathy severity. Clinicians read the images unassisted; using grades only; and using grades along with explanation heatmaps.
Readers were able to grade diabetic retinopathy better with model assistance than without. For cases with disease, accuracy was 58% for the model, 48% unassisted; 50% with grades only; and 62% with grades plus explanation heatmaps. Reader performance improved with assistance across all levels of disease, including for severe and proliferative diabetic retinopathy.
Among the observations recently made by Dr Rahimy about the results:
“The take-home message with machine learning is that the doctor plus the machine is better than either component alone,” said Dr Rahimy. Most importantly, assistance prevents underdiagnosis during a time when diabetic retinopathy is becoming more and more prevalent, and accurate, early detection is crucial.
What to Expect in the Real World
Can you expect the same accuracy observed in controlled studies in the real world? “There are many companies working on this technology,” explained Dr Rahimy. “The key is how you train your algorithm.” He noted that some companies may be “overfitting their models, which generates very high numbers [with the algorithm]. But when you apply it to a real-world situation, you may not necessarily be getting the same numbers.”
Dr Rahimy added that he foresees the ability in practice to develop different algorithms based on the clinician’s level of expertise. “In our study, we asked general ophthalmologists and retinal specialists to read and grade images. In the real world, optometrists, endocrinologists, and internists will also be involved. They will need different levels of assistance,” he said.
So, beyond basic image recognition and diagnosis, what’s does the future hold? “The next, most exciting frontier is going to be predictive analytics,” he explained. “As you start to accumulate sequential data--clean sets of data on the same patients over time--that's where machine learning is going to come in and actually start to give us predictive factors on how patients will do. Who will progress without treatment or who will require which type of treatment?”
Rahimy E. Assisted reads for diabetic retinopathy using a deep learning algorithm and integrated gradient explanation. Talk presented at: AAO 2018 annual meeting; October 26-30, 2018; Chicago.