Diabetic retinopathy is a major cause of blindness, but early diagnosis and treatment can prevent visual loss. Currently, diabetic patients are usually screened by expert readers who examine digital photographs of the retina. Although only 5 to 20 percent of recently diagnosed diabetic patients show signs of retinopathy,1 expert readers have been found to have high sensitivity and specificity at detecting those signs.2
The drawback to this approach is that it's time-consuming and involves multiple individuals. This may partly account for the number of diabetic patients who fail to get an examination; according to the government, almost 50 percent of the 23.4 million patients with diabetes in the United States don't receive a regular, documented, dilated eye exam.3 Simplifying the process could make a big difference in how many people are examined and treated, and one way to accomplish that would be to automate at least part of the process. 
With that in mind, multiple groups around the world have been working on computer software capable of determining whether retinal images show signs of early retinopathy lesions. Recently, one of the leading researchers in this area decided to maximize progress by inviting individuals and teams from around the world to work together, via the Retinopathy Online Challenge. Noting that several algorithms have been developed but few have been applied in clinical practice, this challenge 1) provides a training set of retinal images with reference standards from internationally accepted experts that groups can use to refine their algorithms; and 2) will evaluate algorithm output using a uniform test set of images so algorithms can be compared to each other, and to the experts. In addition, meetings are being organized at international conferences to compare the algorithms and discuss the progress being made.
The ROC is the brainchild of Michael D. Abràmoff, MD, PhD, who is associate professor of ophthalmology at the
Dr. Abràmoff notes that the ROC idea was inspired by other competitions such as the X Prize, awarded for breakthroughs in space travel or the development of a super-efficient automobile. "We don't have a big prize yet," he admits, "but we hope that will change."
Setting the Wheels in Motion
Dr. Abràmoff worked in the computer industry before getting his medical degree and starting his ophthalmology residency. Struck by the inefficiency of the diagnostic process, he turned to computers for help. However, he faced a shortage of retinal images. "No one had a large dataset with thousands of people from an appropriate population," he notes. He solved this problem by creating a system that allowed images to be read online, which also sped up the process of diagnosis enormously. Today, that system is still in operation. "Images sent to us are reviewed by three retinal experts," he explains. "We look at about 25,000 patients each year."
Dr. Abràmoff notes that while this computerized approach to analysis has been used in other specialties (to detect cancer, for example), detecting problematic lesions in the retina is a challenge. "In other medical situations, if the computer detects one type of lesion, you're done," he says. "But with diabetic retinopathy you have to look at multiple types of lesions: microaneurysms; lipid exudates; neovascularization; and so forth. To develop our algorithm, we treated each type of lesion as a separate problem to detect, and then incorporated the resulting algorithms into one overall process."
Challenges of Automation
The algorithm Dr. Abràmoff's group has created falls into the category of "learning" software: In other words, the software bases what it looks for on materials previously provided to it. While this obviously has advantages, Dr. Abràmoff notes that it has one major downside. "People started worrying years ago that if you have a learning system you can't prove it's accurate because it's learning—i.e., changing—all the time," he explains. "The way to avoid that problem is to have a set of data that you use to train the algorithm. Once it has learned from that data, you stop it from learning. That way you can test the resulting algorithm to show that it's safe and effective."
Another big concern with an automated system is the accuracy of the information it learns from. "Retinal specialists looking at images sometimes make the wrong call," notes Dr. Abràmoff. "You don't want to end up saying that the algorithm is correct, when it really is just agreeing with specialists who disagree. Even the professional reading centers can never be 100-percent certain that a spot indicates disease.
"Given this issue," he continues, "all we can do is aim to create the best possible training set. Ideally we could use analyses that 1,000 retinal specialists agreed on. But we'll have to rely on something like the system used in the Early Treatment of Diabetic Retinopathy Study, with three independent, trained experts who work according to a strict protocol."
Another issue that has arisen each time Dr. Abràmoff has created a new approach to early detection is concern from ophthalmologists (and optometrists) about the effect the new approach will have on practice referrals. "When I started using the Internet to allow experts to screen retinal images back in 2000, I was criticized on the grounds that this approach would take patients away from practices," he says. "Eventually, however, doctors realized that our screening encounters increased the number of patients being referred. After all, if more patients are screened and show signs of retinopathy, those additional patients need to be treated."
Dr. Abràmoff believes the same thing will happen if automated early diagnosis is proven to be safe and effective. "First of all, when such an option does become available, doctors will start using it gradually," he says. "For example, you might begin by just letting the computer eliminate cases that the software is 100-percent certain are healthy eyes. That alone would save an enormous amount of time and effort. Eventually, as confidence in the process grows, the system could be allowed to make more sophisticated calls.
"In any case, I believe the ultimate effect will not be to reduce the number of patients seen by doctors, but to shift the nature of the patients seen," he continues. "Instead of evaluating patients who for the most part do not have retinopathy, ophthalmologists will be seeing more patients who are showing the early signs of retinopathy and need treatment. If automated detection makes it possible to examine the 50 percent of people who are not being examined, there will be more than enough work for everyone."
Pooling Talent
Dr. Abràmoff says the Retinopathy Online Challenge should do a lot to ensure that the automated approach to diagnosis becomes a safe and effective option. "I know from my computer science background that there are many groups working on image analysis in China, Europe and India as well as in the United States, but they don't have a good link to ophthalmologists and clinicians," he says. "They may have very bright ideas but no way of testing them as a means to detect retinopathy. By giving these people the opportunity to try their algorithms and ideas on a known data set, we can find out which algorithms really work the best.
"In fact, we currently have about 25 groups participating in the competition, including
For more information about the Retinopathy Online Challenge, visit roc.healthcare.uiowa.edu.
1. Wilson C, Horton M, Cavallerano J, Aiello LM. Addition of primary care-based retinal imaging technology to an existing eye care professional referral program increased the rate of surveillance and treatment of diabetic retinopathy. Diabetes Care 2005;28:318–322.
2. Lin DY, Blumenkranz MS, Brothers RJ, Grosvenor DM. The sensitivity and specificity of single-field nonmydriatic monochromatic digital fundus photography with remote image interpretation for diabetic retinopathy screening: A comparison with ophthalmoscopy and standardized mydriatic color photography. Am J Ophthalmol 2002;134:204–213.
3. Centers for Disease Control and Prevention: Data & Trends: National Diabetes Surveillance System: Preventive Care Practices, 1994–2004. Available from http://www.cdc.gov/diabetes/statistics.
4. Abràmoff MD, Niemeijer M, Suttorp-Schulten MS, Viergever MA, Russell SR, van Ginneken B. Evaluation of a system for automated detection of diabetic retinopathy from color fundus photographs in a large population of patients with diabetes. Diabetes Care 2008;31:2:193-198.
5. Abràmoff, MD, Niemeijer, M., Suttorp-Schulten, M. S.,
