Evaluating clinical literature


Fam Pract Manag. 2004 Oct;11(9):14-15.

To the Editor:

Physicians need reliable methods for appraising studies for validity and relevance; they also want something quick. While Dr. Robert Flaherty’s article on the “PP-ICONS” approach makes some excellent points, it is insufficient for determining validity. The PP-ICONS approach cannot guarantee that you have identified serious, and potentially fatal, threats to validity, such as improper randomization methods, bias and confounding. Relying on this approach could result in applying misleading results. Only by reviewing study methods are you likely to determine bias in controls and subject selection, along with a host of other critical considerations.

Knowing that a study is a randomized controlled trial (RCT) is insufficient. There are instances when an observational study can be labeled as an RCT but a close review of the methods shows that it is not.1 Authors have shown that inadequate concealment and blinding can bias a study in favor of an intervention.2,3 Furthermore, authors often mislabel what they are doing, as Kruse et al show in a review of reported intention-to-treat analyses.4

Dr. Flaherty states that you can use the PP-ICONS method for articles on diagnostic treatment and screening. Diagnostic testing is much more complex, requiring assessments of “gold standards,” blinded evaluators, subjects who have the condition along with those who do not, and evaluations of measures of test function. Limner et al show problems with diagnostic test articles in a systematic review of 218 studies where only 15 satisfied criteria for good studies.5 Screening adds other dimensions when assessing validity, including lead- and length-time bias, and early versus late diagnosis and treatment options.

Dr. Flaherty also states that when using PP-ICONS, the abstract alone is often sufficient. However, a study by Pitkin shows that 18 to 68 percent of abstracts in six of the most respected journals contain information not verifiable in the body of the text.6 Abstracts often help weed out problem studies, but they are not reliable to determine validity.

On a similar note, Brandi White’s article “Making Evidence-Based Medicine Doable in Everyday Practice” [February 2004, page 51] should warn readers to beware of poor or dated systematic reviews and clinical practice guidelines that could provide misleading, and possibly harmful, information. Unless you use trusted sources, such as Clinical Evidence (http://www.clinicalevidence.com), Cochrane (http://www.cochrane.org) or the Database of Abstracts of Reviews of Effects (DARE) (http://www.york.ac.uk/inst/crd/darehp.htm), you need to critically appraise the guidelines or the reviews and assess the usefulness of the results found to be valid. And for guidelines and reviews from any source, you need to search for new important studies that have been appraised. Guidelines and systematic reviews are difficult to do well, and many are flawed. Many guidelines, even when bearing the label “evidence-based,” lack systematic development and critical appraisal of included content. Numerous so-called systematic reviews lack many, if not all, of the criteria for a systematic review and may be nothing more than narrative reviews or overviews, which can be highly prone to bias. In fact, large, well-done RCTs may be superior to systematic reviews in many instances.

Unfortunately, until we have more trusted sources (and ideally, even those should be critically appraised), shortcuts will likely continue to run the risk of applying misleading clinical information. From our experiences in the health care field, and as university-affiliated educators and trainers of evidence-based principles through our consulting company, Delfini Group, we believe that all clinicians should have a basic understanding of the key concepts of critical appraisal so they are not “had” by misleading information. The good news is that these skills are not that difficult to acquire. They can save you time and, more important, result in better care and better use of resources.


show all references

1. Labrie F, Candas B, Dupont A, et al. Screening decreases prostate cancer death: first analysis of the 1998 Quebec prospective randomized controlled trial. Prostate. 1999;382:83-91....

2. Chalmers TC, Celano P, Sacks HS, Smith H Jr. Bias in treatment assignment in controlled clinical trials. N Engl J Med. 1983;30922:1358-1361.

3. Egger M, Juni P, Bartlett C, Holenstein F, Sterne J. How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. Health Technol Assess. 2003;71:1-76.

4. Kruse RL, Alper BS, Reust C, Stevermer JJ, Shannon S, Williams RH. Intention-to-treat analysis: Who is in? Who is out? J Fam Pract. 2002;5111:969-971.

5. Lijmer JG, Mol BW, Heisterkamp S, et al. Empirical evidence of design-related bias in studies of diagnostic tests. JAMA. 1999;28211:1061-1066.

6. Pitkin RM, Branagan MA, Burmeister LF. Accuracy of data in abstracts of published research articles. JAMA. 1999;28112:1110-1111.

Author’s response:

Ms. Strite and Dr. Stuart illuminate an important problem with the current application of evidence-based medicine (EBM) to real-life practice. I think we all agree that the purpose of EBM is to identify the best existing evidence – research that is relevant, valid and applicable to clinical questions that will improve patient care.

Most clinical questions have not yet been rigorously reviewed by reputable organizations such as Cochrane. Thus, busy practicing physicians must develop an efficient way to identify journal articles that are likely to be relevant and valid. However, many do not have the time or inclination to meticulously analyze each article they read. Ms. Strite and Dr. Stuart recommend a rigorous analytic method is described in a series by the Journal of the American Medical Association1 and by Miser.2 While highly effective, these techniques are simply too time-consuming and cumbersome to be applied in a busy practice.

Clinical medicine is a trade-off between academic rigor and practicality. A rigorous analytic technique may result in a high likelihood that the research is valid and relevant. But if that technique cannot be easily applied, it will not be used by physicians in a busy practice and will not help them provide better patient care. A simpler technique that can be applied quickly and easily, such as PP-ICONS, may not result in an academically solid assessment of validity and relevance, but physicians will still have a much better idea of the validity and relevance of an article than they would have by simply accepting the conclusions at face value.

I agree with Ms. Strite and Dr. Stuart that a simple assessment tool like PP-ICONS is not perfect. It is, however, a practical compromise that is much better than no assessment at all, and, by being quick and easy to apply, PP-ICONS can improve the literature review abilities of most busy physicians.


1. Guyatt GH, Sackett DL, Cook DJ. Users’ guides to the medical literature. II. How to use an article about therapy or prevention. A. Are the results of the study valid? Evidence-based medicine working group. JAMA. 1993;270(21):2598-2601.

2. Miser WF. Critical appraisal of the literature. J Am Board Fam Pract. 1999;12(4):315-333.


Send your comments to fpmedit@aafp.org. Submission of a letter will be construed as granting AAFP permission to publish the letter in any of its publications in any form. We cannot respond to all letters we receive. Those chosen for publication will be edited for length and style.


Copyright © 2004 by the American Academy of Family Physicians.
This content is owned by the AAFP. A person viewing it online may make one printout of the material and may use that printout only for his or her personal, non-commercial reference. This material may not otherwise be downloaded, copied, printed, stored, transmitted or reproduced in any medium, whether now known or later invented, except as authorized in writing by the AAFP. Contact fpmserv@aafp.org for copyright questions and/or permission requests.

Want to use this article elsewhere? Get Permissions


May-Jun 2022

Access the latest issue
of FPM journal

Read the Issue

FPM E-Newsletter

Sign up to receive FPM's free, weekly e-newsletter, "Quick Tips & Insights."

Sign Up Now



Measuring What Matters in Primary Care: Implementing the Person-Centered Primary Care Measure

Learn how family physicians are using the person-centered primary care measure and get tips for how to implement it in your practice.

Improving Adult Immunization Rates Within Racial and Ethnic Minority Communities

Part one of this two-part supplement series highlights QI processes to reduce vaccine disparities, identifies recommended adult vaccines, and discusses their importance among racial and ethnic minority communities.