October 2004 Table of Contents
Letters
Determining a proper sample size
To the Editor:
In the article "A Simple Method for Evaluating the Clinical Literature" [May 2004, page 47], Dr. Robert Flaherty suggests that 400 subjects is a reliable sample size for a study to have adequate statistical power. Using this approach, I recently evaluated the first 40 abstracts for the Journal of the American Medical Association that I found by searching Medline for "randomized controlled trials." Of the 27 empirical studies, 52 percent had sample sizes less than 400. Dr. Flaherty's approach would suggest ignoring the evidence from more than half of the randomized controlled trials published in JAMA.
Joshua Fogel, PhD
Brooklyn, N.Y.
Author's response:
My recommendation of a sample size of at least 400 refers to a sample size above which the power of a study can be safely assumed. I agree with Dr. Fogel that, depending on the study, a smaller sample size may prove adequate for valid statistical analysis. The key point of my article is that physicians need quick and easy "rules of thumb" which, while not perfect, allow reasonably effective evaluation of journal articles. The "Rule of 400" is just such an imperfect rule, yet it is still useful. Nevertheless, I would enthusiastically welcome an easily applied, more accurate guideline for sample size.
Evaluating clinical literature
To the Editor:
Physicians need reliable methods for appraising studies for validity and relevance; they also want something quick. While Dr. Robert Flaherty's article on the "PP-ICONS" approach makes some excellent points, it is insufficient for determining validity. The PP-ICONS approach cannot guarantee that you have identified serious, and potentially fatal, threats to validity, such as improper randomization methods, bias and confounding. Relying on this approach could result in applying misleading results. Only by reviewing study methods are you likely to determine bias in controls and subject selection, along with a host of other critical considerations.
|
Knowing that a study is a randomized controlled trial (RCT) is insufficient. There are instances when an observational study can be labeled as an RCT but a close review of the methods shows that it is not.1 Authors have shown that inadequate concealment and blinding can bias a study in favor of an intervention.2,3 Furthermore, authors often mislabel what they are doing, as Kruse et al show in a review of reported intention-to-treat analyses.4
Dr. Flaherty states that you can use the PP-ICONS method for articles on diagnostic treatment and screening. Diagnostic testing is much more complex, requiring assessments of "gold standards," blinded evaluators, subjects who have the condition along with those who do not, and evaluations of measures of test function. Limner et al show problems with diagnostic test articles in a systematic review of 218 studies where only 15 satisfied criteria for good studies.5 Screening adds other dimensions when assessing validity, including lead- and length-time bias, and early versus late diagnosis and treatment options.
Dr. Flaherty also states that when using PP-ICONS, the abstract alone is often sufficient. However, a study by Pitkin shows that 18 to 68 percent of abstracts in six of the most respected journals contain information not verifiable in the body of the text.6 Abstracts often help weed out problem studies, but they are not reliable to determine validity.
On a similar note, Brandi White's article "Making Evidence-Based Medicine Doable in Everyday Practice" [February 2004, page 51] should warn readers to beware of poor or dated systematic reviews and clinical practice guidelines that could provide misleading, and possibly harmful, information. Unless you use trusted sources, such as Clinical Evidence (http://www.clinicalevidence.com), Cochrane (http://www.cochrane.org) or the Database of Abstracts of Reviews of Effects (DARE) (http://www.york.ac.uk/inst/crd/darehp.htm), you need to critically appraise the guidelines or the reviews and assess the usefulness of the results found to be valid. And for guidelines and reviews from any source, you need to search for new important studies that have been appraised. Guidelines and systematic reviews are difficult to do well, and many are flawed. Many guidelines, even when bearing the label "evidence-based," lack systematic development and critical appraisal of included content. Numerous so-called systematic reviews lack many, if not all, of the criteria for a systematic review and may be nothing more than narrative reviews or overviews, which can be highly prone to bias. In fact, large, well-done RCTs may be superior to systematic reviews in many instances.
Unfortunately, until we have more trusted sources (and ideally, even those should be critically appraised), shortcuts will likely continue to run the risk of applying misleading clinical information. From our experiences in the health care field, and as university-affiliated educators and trainers of evidence-based principles through our consulting company, Delfini Group, we believe that all clinicians should have a basic understanding of the key concepts of critical appraisal so they are not "had" by misleading information. The good news is that these skills are not that difficult to acquire. They can save you time and, more important, result in better care and better use of resources.
Sheri Strite
San Diego
Michael E. Stuart, MD
Seattle
1. Labrie F, Candas B, Dupont A, et al., Screening decreases prostate cancer death: first analysis of the 1998 Quebec prospective randomized controlled trial. Prostate. 1999;38(2):83-91.
2. Chalmers TC, Celano P, Sacks HS, Smith H Jr. Bias in treatment assignment in controlled clinical trials. N Engl J Med. 1983;309(22):1358-1361.
3. Egger M, Juni P, Bartlett C, Holenstein F, Sterne J. How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. Health Technol Assess. 2003;7(1):1-76.
4. Kruse RL, Alper BS, Reust C, Stevermer JJ, Shannon S, Williams RH. Intention- to-treat analysis: Who is in? Who is out? J Fam Pract. 2002;51(11):969-971.
5. Lijmer JG, Mol BW, Heisterkamp S, et al. Empirical evidence of design-related bias in studies of diagnostic tests. JAMA. 1999;282(11):1061-1066.
6. Pitkin RM, Branagan MA, Burmeister LF. Accuracy of data in abstracts of published research articles. JAMA. 1999;281(12):1110-1111.
Author's response:
Ms. Strite and Dr. Stuart illuminate an important problem with the current application of evidence-based medicine (EBM) to real-life practice. I think we all agree that the purpose of EBM is to identify the best existing evidence - research that is relevant, valid and applicable to clinical questions that will improve patient care.
Most clinical questions have not yet been rigorously reviewed by reputable organizations such as Cochrane. Thus, busy practicing physicians must develop an efficient way to identify journal articles that are likely to be relevant and valid. However, many do not have the time or inclination to meticulously analyze each article they read. Ms. Strite and Dr. Stuart recommend a rigorous analytic method is described in a series by the Journal of the American Medical Association1 and by Miser.2 While highly effective, these techniques are simply too time-consuming and cumbersome to be applied in a busy practice.
Clinical medicine is a trade-off between academic rigor and practicality. A rigorous analytic technique may result in a high likelihood that the research is valid and relevant. But if that technique cannot be easily applied, it will not be used by physicians in a busy practice and will not help them provide better patient care. A simpler technique that can be applied quickly and easily, such as PP-ICONS, may not result in an academically solid assessment of validity and relevance, but physicians will still have a much better idea of the validity and relevance of an article than they would have by simply accepting the conclusions at face value.
I agree with Ms. Strite and Dr. Stuart that a simple assessment tool like PP-ICONS is not perfect. It is, however, a practical compromise that is much better than no assessment at all, and, by being quick and easy to apply, PP-ICONS can improve the literature review abilities of most busy physicians.
Robert J. Flaherty, MD
Bozeman,
Mont.
1. Guyatt GH, Sackett DL, Cook DJ. Users' guides to the medical literature. II. How to use an article about therapy or prevention. A. Are the results of the study valid? Evidence-based medicine working group. JAMA.1993;270(21):2598-2601.
2. Miser WF. Critical appraisal of the literature. J Am Board Fam Pract. 1999;12(4):315-333.
Putting the family in family medicine
| We want to hear from you |
|
Send your comments to FPM Letters Editor by e-mail, fpmedit@aafp.org; by mail, Family Practice Management, 11400 Tomahawk Creek Parkway, Leawood, KS 66211-2672; or by fax, 913-906-6010. Include your address, daytime phone number and fax number. Submission of a letter will be construed as granting AAFP permission to publish the letter in any of its publications in any form. We cannot respond to all letters we receive. Those chosen for publication will be edited for length and style. |
To the Editor:
Dr. Sanford Brown's article ["What's in a name?" Practice Diary, April 2004, page 60] is a good jumping-off point for discussing the Future of Family Medicine project. The use of the word "family" in what we do is hardly "arbitrary," as stated in the article. When Paul, the patient described in Dr. Brown's article, has provided well-child and sick care for a toddler, helped that toddler's father quit smoking, sat up all night with the mother during a difficult second labor, ridden in an ambulance with the new little brother who is being transported to a tertiary care center, reviewed the great-grandmother's hospice care plan with the nursing staff, and provided grief counseling and short-term anxiolytics to other family members, then he is welcome to reopen the discussion of who can have the word "family" in his or her job description.
Scott A. Murkin, MD
Asheboro, N.C.
|
Copyright © 2004 by the
American Academy of Family Physicians. |
MEDLINE:
• Citation
RELATED TOPICS:
• Quality issues (229)
• Family medicine issues (76)
• Evidence-Based Medicine (8)
• Family Practice (321)