More on Deep Learning Algorithms to Detect Diabetic Retinopathy

In January we provided results from a study evaluating the accuracy of a deep learning algorithm in detecting diabetic retinopathy. Recently, one of the researchers from that trial– Ehsan Rahimy, MD, a surgical and medical vitreoretinal specialist at the Palo Alto Medical Foundation in California–offered additional details about his team’s work with Google.

Investigators asked 10 clinicians (5 general ophthalmologists, 4 retinal specialists, and 1 retinal fellow) to read nearly 1800 images and grade for diabetic retinopathy severity. Clinicians read the images unassisted; using grades only; and using grades along with explanation heatmaps.

Readers were able to grade diabetic retinopathy better with model assistance than without. For cases with disease, accuracy was 58% for the model, 48% unassisted; 50% with grades only; and 62% with grades plus explanation heatmaps. Reader performance improved with assistance across all levels of disease, including for severe and proliferative diabetic retinopathy.

Among the observations recently made by Dr Rahimy about the results:

• Not surprisingly, accuracy and improvement depend on the reader’s background and experience. “At baseline, general ophthalmologists did not perform to the level of the algorithm,” said Dr Rahimy. But when we added grades-only and grades plus heatmaps, they did get up to the level of the algorithm.” Meanwhile, retinal specialists were already grading at the level of the algorithm, and surpassed it once they received the extra assistance.
• Overall, sensitivity increased with assistance. For nonproliferative diabetic retinopathy, it went from 79% unassisted to 88% with grades only and 89% with grades plus heatmaps.
• “Equally important, the increased sensitivity was not met with a drop in specificity,” he noted. Specificity was 97%, 96%, and 96%, respectively.
• Graders were asked to rate their confidence in achieving the correct diagnosis. They became more confident when they had assistance versus when they were unassisted.
• There was a learning curve. As graders became familiar with the platform and continued to use it—particularly with the help of the heatmaps—they became more and more comfortable. The grading time decreased as accuracy increased.

“The take-home message with machine learning is that the doctor plus the machine is better than either component alone,” said Dr Rahimy. Most importantly, assistance prevents underdiagnosis during a time when diabetic retinopathy is becoming more and more prevalent, and accurate, early detection is crucial.

What to Expect in the Real World

Can you expect the same accuracy observed in controlled studies in the real world? “There are many companies working on this technology,” explained Dr Rahimy. “The key is how you train your algorithm.” He noted that some companies may be “overfitting their models, which generates very high numbers [with the algorithm]. But when you apply it to a real-world situation, you may not necessarily be getting the same numbers.”

Dr Rahimy added that he foresees the ability in practice to develop different algorithms based on the clinician’s level of expertise. “In our study, we asked general ophthalmologists and retinal specialists to read and grade images. In the real world, optometrists, endocrinologists, and internists will also be involved. They will need different levels of assistance,” he said.

So, beyond basic image recognition and diagnosis, what’s does the future hold? “The next, most exciting frontier is going to be predictive analytics,” he explained. “As you start to accumulate sequential data–clean sets of data on the same patients over time–that’s where machine learning is going to come in and actually start to give us predictive factors on how patients will do. Who will progress without treatment or who will require which type of treatment?”

Stay tuned.

Reference

Rahimy E. Assisted reads for diabetic retinopathy using a deep learning algorithm and integrated gradient explanation. Talk presented at: AAO 2018 annual meeting; October 26-30, 2018; Chicago.