I welcome and appreciate Dr. Tennant's comments about the statistics related to the Advanced Glaucoma Intervention Study (AGIS), which Dr. Gaasterland and colleagues were kind enough to review. This complex, landmark research deserves multiple inquiries that facilitate debate. By publishing this exchange, I hope to stimulate readers to comment on the merits of the various landmark glaucoma series that we have covered in Glaucoma Today. If you are fortunate enough to have expertise in biostatistics, please let us know your thoughts on this debate.

—Ronald L. Fellman, MD, section editor

COMMENTS BY JEFFREY TENNANT, MD

A recent articlev1 by Herndon and Moore in Glaucoma Today repeated a conclusion of the AGIS that black and white patients respond differently to glaucoma treatment (argon laser trabeculoplasty [ALT] followed by trabeculectomy followed by another trabeculectomy vs trabeculectomy followed by ALT followed by a second trabeculectomy). I realize this statement has been made often, but it is based on faulty statistical analysis. Specifically, the endpoints of the study were not specified in advance. Data cannot be analyzed as if they were.

At the end of 6 years, the data showed no difference in response between the races. At the end of 7 years, there was a response that was supposedly significant at the 1% level. The authors calculated this P value by assuming the decisions to analyze the response between the races and to end the study at 7 years were made at random (which they clearly were not).

After an initial period of frequent observation, patients were examined every 6 months, meaning there were 14 observations over the 7 years. The chances that the results of each observation would fall within the 99% expectation are (0.99) to the power of 14, which equals 0.87. This means that, 13% of the time, a response would fall outside the 99th percentile due to chance alone. If factors such as age and gender (as well as race) were considered, the odds of finding such a response would be even greater. If enough data are analyzed, the chance of finding a statistical outlier approach 100%.

I made this point quite a few years ago and noted that this common error in statistics has a name: the Texas sharpshooter fallacy. 2,3 An observer sees a barn in Texas with a series of targets on it, each of which has an arrow right at its center. The person concludes that he or she has seen the handiwork of a sharpshooter. Then, the person discovers that someone shot the arrows first and drew the targets around them. The observer nonetheless concludes that he or she has seen the work of a sharpshooter. After all, if only the targets had been there when the arrows were shot, each would have hit its mark.

If only the decision to analyze the results between the races and to end the study at 7 years had been made in advance, the observation would have been statistically significant. Alas, that did not happen.

RESPONSE ON BEHALF OF THE AGIS INVESTIGATORS BY DOUGLAS E. GAASTERLAND, MD; E. KENNETH SULLIVAN, PhD; AND PAUL VANVELDHUISEN, PhD

Planning for the AGIS started in the mid-1980s, soon after clinicians had enthusiastically endorsed ALT as an attractive and noninvasive treatment for medically uncontrolled open-angle glaucoma. 4 Many physicians had endorsed the unproven belief that glaucoma surgery starting with ALT yielded better outcomes. Two of the study's leaders (a clinician and a biostatisticianepidemiologist) devised a plan to test that belief and produced (in the earliest days of word processors) an AGIS Manual of Operations (the AGIS MOP, where P indicates procedures) that grew to 17 chapters and nearly 200 pages. The investigation was planned to study long-term visual outcomes of glaucoma surgical management after medication alone inadequately controlled the disease (ie, advanced glaucoma), and it recognized that an individual undergoing initial surgery for advanced glaucoma might eventually experience failure of that intervention and need a second or third surgery. 3 Interested, willing, and able AGIS investigators at 12 academic centers included glaucomatologists and senior biostatisticians and epidemiologists with experience in randomized clinical trials. They refined the AGIS MOP. After the privately funded planning, the study received grants from the National Eye Institute in early 1988, with first enrollment of patients in mid- 1988. A Policy and Treatment Effects Monitoring Board (PATEMB), composed of senior academic glaucomatologists and senior biostatisticians and with lay representation, was formed to advise the study leadership and the National Eye Institute on all major aspects affecting the conduct and course of the trial, including review and approval of manuscripts.

The AGIS plan had clear, vigorously debated, reliable inclusion and exclusion criteria for the phakic glaucomatous participants. Definitions and instructions for study procedures, observations, and conclusions were rigorous and prospective. Enrolled participants had glaucomatous damage to visual function but not so severe as to inhibit its getting measurably worse. Eyes were randomized to one of two sequences of glaucoma surgery: ALT followed by trabeculectomy if necessary followed by another trabeculectomy if necessary versus trabeculectomy followed by ALT if necessary followed by a second trabeculectomy if necessary. After the initial surgery, study visits were at 3 and 6 months and every 6 months thereafter, with nonstudy visits as frequent as needed for patients' care, status documentation and management. The endpoints were clearly defined, verified, applied, and measured by trained and certified clinical center staff. We did not accept unconfirmed observations of visual change, only sustained change from baseline. The primary endpoint was a deterioration of visual function (visual field or visual acuity). Of course, there was interest as well in IOP, requirement of medication, and failure of interventions as secondary endpoints.

Of the 591 participants enrolled, 249 (42%) selfreported as white, 332 (56%) as of African American background, and 10 (2%) as other racial identifications. During analysis of follow-up data ranging from 4 to 7 years, a statistically significant interaction between race and intervention-sequence assignment was discovered— not one outcome at one point in time but all of the predefined main visual outcome variables and the outcome of intervention failure. Many of the P values for these interactions were less than .01. Initial skepticism on the part of the AGIS leadership and the PATEMB resulted in numerous analyses trying to understand the nature of the interaction and whether the interaction should be attributed to other causes. The existence of the interaction was validated. The finding of the interaction resulted in the separate presentation of AGIS results for black and white patients, with the results within these subgroups based on randomized comparisons of the two treatment regimens. As with all of the published AGIS articles, the PATEMB members reviewed and approved the primary findings with its conclusions presented in AGIS reports 4 and 13.5,6 Further, both the predictive and the associative analyses— in AGIS report 7—of the IOP level and the consistency of reduction during follow-up of the enrolled patients with high-pressure primary open-angle glaucoma provided “dose-response” evidence that IOP control matters in glaucoma management. 7 This predated the present-day concept of target pressure and the importance of the consistency of IOP reduction.

We appreciate Dr. Tennant's comments and his concern that the AGIS analysis investigating statistical interactions of treatment assignment with race that led to reporting results separately for black and white patients was not predefined. Statistical interactions in clinical trials are rare, and we did not anticipate this finding when the AGIS was designed. Based on the statistical evidence from the analyses and the careful scrutiny of the results by AGIS clinical and biostatistical investigators and the independent PATEMB, however, we continue to believe that the findings are real and important for the management of patients with advanced glaucoma. We also believe that race should be taken into account in the planning and analysis of future glaucoma treatment trials.

We note and appreciate Herndon and Moore's comments in response to the nine questions presented by Dr. Fellman about the AGIS. The trial is complex, and their review required careful, in-depth study of the AGIS reports. We believe their comments accurately represent the AGIS results presented in our publications.

Section Editor Ronald L. Fellman, MD, is a glaucoma specialist at Glaucoma Associates of Texas in Dallas and clinical associate professor emeritus in the Department of Ophthalmology at UT Southwestern Medical Center in Dallas. Dr. Fellman may be reached at (214) 360-0000; rfellman@glaucomaassociates.com.

Douglas E. Gaasterland, MD, has retired from private practice with Eye Doctors of Washington DC in Chevy Chase, Maryland. Dr. Gaasterland may be reached at (301) 215- 7100; degaasterland@edow.com.

E. Kenneth Sullivan, PhD, formerly with The EMMES Corporation in Rockville, Maryland, is head, biostatistics and statistical programming, Alcon Laboratories, Inc., in Fort Worth, Texas. Dr. Sullivan may be reached at (817) 615-2464;ken.sullivan@alcon.com.

Jeffrey Tennant, MD, is in practice in Pekin, Illinois. Dr. Tennant may be reached at eyeballs123@yahoo.com. 

Paul VanVeldhuisen, PhD, is a senior biostatistician at The EMMES Corporation in Rockville, Maryland. Dr. VanVeldhuisen may be reached at pvanveldhuisen@emmes.com.

  1. Herndon LW, Moore DB. Take-home messages from the Advanced Glaucoma Intervention Study. Glaucoma Today. May/June 2013;11(3):20-24.
  2. Texas sharpshooter fallacy. Wikipedia. http://en.wikipedia.org/wiki/Texas_sharpshooter_fallacy. Accessed September 4, 2013.
  3. Texas sharpshooter fallacy. Rational Wiki. http://rationalwiki.org/wiki/Texas_sharpshooter_fallacy. Accessed September 4, 2013.
  4. Wise JB. Ten year results of laser trabeculoplasty. Does the laser avoid glaucoma surgery or merely defer it? Eye. 1987;1:45-50.
  5. The Advanced Glaucoma Intervention Study (AGIS) 4. Comparison of treatment outcomes within race. Sevenyear results. Ophthalmology. 1998;105:1146-1164.
  6. The Advanced Glaucoma Intervention Study (AGIS) 13. Comparison of treatment outcomes within race: 10-year results. Ophthalmology. 2004;111:651-664.
  7. The Advanced Glaucoma Intervention Study (AGIS): 7. The relationship between control of intraocular pressure and visual field deterioration. Am J Ophthalmol. 2000;130:429-440.