AFP Journal Club

The Story Behind the Study

How Do Clinical Practice Guidelines Go Awry?



FREE PREVIEW Log in or buy this issue to read the full article. AAFP members and paid subscribers get free access to all articles. Subscribe now.


FREE PREVIEW Subscribe or buy this issue. AAFP members and paid subscribers get free access to all articles.

Am Fam Physician. 2012 Sep 15;86(6):514-516.

image

From left: Dr. Mark Graber, Dr. Andrea Darby-Stewart, and Dr. Robert Dachs

Purpose

In AFP Journal Club, three presenters review an interesting journal article in a conversational manner. These articles involve “hot topics” that affect family physicians or “bust” commonly held medical myths. The presenters give their opinions about the clinical value of the individual study discussed. The opinions reflect the views of the presenters, not those of AFP or the AAFP.

Article

Grossman DC, Moyer VA, Melnyk BM, Chou R, DeWitt TG; U.S. Preventive Services Task Force. The anatomy of a US Preventive Services Task Force Recommendation: lipid screening for children and adolescents. Arch Pediatr Adolesc Med. 2011;165(3):205–210.

Bob: Because journal clubs are designed to teach participants how to critically analyze a study or journal article, participants may often feel that the medical literature is nothing but a wastebasket of bad studies. To turn the tables, we chose to analyze an article that critiques the guideline development process. It discusses how two reputable organizations, the U.S. Preventive Services Task Force (USPSTF) and the American Academy of Pediatrics (AAP), came to completely different conclusions in their recommendations on lipid screening in children. That will give us a starting point to discuss the topic of guidelines.

What does this article say?

Bob: The family physician is likely to face this clinical dilemma: Should I screen a child/adolescent for hyperlipidemia? In 2008, the AAP recommended screening in children two to 10 years of age with risk factors for cardiovascular disease or a family history of premature cardiovascular disease or hyperlipidemia.1 Around the same time, the USPSTF noted that there was insufficient evidence to recommend routine screening.2 How do two organizations with the same data come to such different conclusions?

Table 1 summarizes the main differences between the two organizations' approaches to developing recommendations.3 The USPSTF has a standardized approach of forming a key question, collecting evidence, assessing and synthesizing data, developing the recommendation, and obtaining an external peer and public review. In this case, the USPSTF starts with the core question: Are there any controlled trials of lipid screening versus no screening with information on long-term health outcomes? Because there are no such studies, they formulated nine other questions looking for indirect evidence. With a lack of adequate indirect evidence, a USPSTF I recommendation (insufficient evidence) was given for lipid screening in children. Although the AAP recommendation lacks this level of rigor, the article notes that previous AAP guidelines have been thorough and organized.

Table 1.

Comparison of the USPSTF and AAP Guideline Development Process

Standard USPSTF AAP

Clear description of methods used to identify and analyze scientific data

Yes

No

Method of updated literature review that is referenced and outlined

Yes

No

Standard process for appraisal and grading the quality or sufficiency of the evidence

Yes

No

Conflicts of interest rigorously vetted

Yes

No

External peer and public review

Yes

No


AAP = American Academy of Pediatrics; USPSTF = U.S. Preventive Services Task Force. Information from reference 3.

Table 1.   Comparison of the USPSTF and AAP Guideline Development Process

View Table

Table 1.

Comparison of the USPSTF and AAP Guideline Development Process

Standard USPSTF AAP

Clear description of methods used to identify and analyze scientific data

Yes

No

Method of updated literature review that is referenced and outlined

Yes

No

Standard process for appraisal and grading the quality or sufficiency of the evidence

Yes

No

Conflicts of interest rigorously vetted

Yes

No

External peer and public review

Yes

No


AAP = American Academy of Pediatrics; USPSTF = U.S. Preventive Services Task Force. Information from reference 3.

What should we make of this article?

Bob: It should come as no surprise that this article was written by the authors responsible for the USPSTF's childhood lipid screening recommendation. They clearly lay out their process of guideline development and fairly point out differences with the AAP. If they sound a little defensive, I can't blame them after the heat they received with their recent, highly publicized recommendations on screening for breast and prostate cancers.

Mark: I like this article because it points out some of the problems with the deluge of guidelines that are released.

Let's talk about “evidence,” a term that is thrown around far too loosely. The USPSTF's technique of question development followed by a detailed, standardized analysis of the literature to answer a specific question is “best in class.” Far too often, recommendations that are purported as evidence-based are nothing more than a consensus recommendation from a group of experts without any well-defined data to support it. Case in point: A 2009 review of 14 American Heart Association/American College of Cardiology (AHA/ACC) guidelines revealed that only 11 percent were based on multiple randomized trials or meta-analyses (AHA/ACC level A evidence), whereas nearly one-half were based on consensus (AHA/ACC level C evidence).4 Simply put, a group of experts expressing their views is not evidence.

Andrea: And just who are these experts involved with guideline development? The USPSTF dismisses experts with conflicts of interest, and so does the United Kingdom's National Institute for Health and Clinical Excellence. This is a no-brainer in my book. However, a review of 17 ACC/AHA guidelines involving 498 contributors revealed that 56 percent of the guidelines had an author with a conflict of interest.5 And this ethical dilemma is not limited to ACC/AHA guidelines. A 2011 review of 14 U.S. and Canadian clinical practice guidelines on diabetes mellitus and hyperlipidemia revealed that there was no conflict-of-interest documentation in five of the guidelines (this does not mean there were no conflicts of interest), and of the nine with documentation, 48 percent had authors who disclosed conflicts of interest.6 Digging deeper, specialty society guidelines in particular seem to have strong ties to pharmaceutical manufacturers. A 2002 survey of 100 specialty society guidelines revealed that 87 percent had ties to the pharmaceutical industry.7

Bob: One has to wonder why an organization that is creating a guideline does not seek conflict-of-interest disclosure or recuse authors that have a conflict of interest. Perhaps the truth is that specialty society organizations themselves may sometimes have a conflict of interest (e.g., receiving income from pharmaceutical companies). This can lead to the organization consciously or unconsciously predetermining the outcome of a guideline. Don't believe this can happen? A British Medical Journal report revealed that a prominent “nonprofit” organization that received a total of $11 million from a pharmaceutical company subsequently placed six members with ties to this company on a nine-person committee that was to determine the role of the company's drug in a guideline.8

Many have called for transparency in the clinical practice guideline development process. Currently, final recommendations often appear as if there was unanimity among the contributors' recommendations. This is not always the case. It would be more appropriate to disclose how the committee members voted on the final recommendation, similar to Supreme Court rulings. Wouldn't a five-to-four decision reflect a level of concern (or conflict) that clinicians should be aware of?

Mark: The Institute of Medicine (IOM) has proposed eight standards that should be followed when developing clinical practice guidelines9:

  • Complete transparency in guideline creation

  • Conflict-of-interest disclosure

  • Member composition that is multidisciplinary

  • Systematic review of the literature that meets IOM standards

  • Clear and consistent rating and description of the evidence

  • Recommendations that are articulated in detail and in a standard form

  • External review by the full spectrum of stakeholders

  • Appropriate updating of the guideline

Although these proposed standards are laudable, currently there is no way to enforce them. Consequently, guidelines will continue to remain unregulated in the foreseeable future.

Andrea: I would like to comment on the IOM's recommendation for appropriate updating. A guideline is worthwhile only if it is up to date. An analysis by the RAND Corporation estimates that 90 percent of guidelines are relevant for 3.6 years, and that 50 percent become obsolete after 5.8 years.10 Here is a good example: In 1996, the AAP made a strong recommendation that children six to 18 months of age with a febrile seizure should have a diagnostic lumbar puncture. This reflected the prevalence of childhood meningitis before pneumococcal vaccines. Over the next couple of years, a number of reports demonstrated the changing epidemiology of the disease and the limited yield of lumbar puncture in these patients.1113 However, the AAP didn't change its recommendation until 2011.14

Bob: A visit to the National Guideline Clearinghouse (http://www.guideline.gov), a product of the Agency for Healthcare Research and Quality, puts this issue into perspective. As of July 12, 2012, there were 2,476 clinical practice guidelines listed. There can be 10, 20, or more guidelines on a single topic. The Web site has a link that allows users to compare various guidelines. Still, there is no documented oversight on how the guidelines were developed, and there are no comments on the quality of the guidelines—any group who wants to create a clinical practice guideline is free to do so.

What should the family physician do?

Andrea: When it comes to the question of lipid screening in children, I would go with the more rigorous, albeit conservative, USPSTF recommendation. There is no evidence that early identification of hyperlipidemia in children changes long-term outcomes. Let's focus our limited health care resources where they may do some good: encouraging physical activity and maintaining an appropriate body weight.

Mark: When it comes to trying to evaluate an organization's clinical practice guideline, I think it is better to side with a primary care organization versus a specialty society. A review of 33 guidelines on hypertension, hyperlipidemia, and cholesterol screening and cardiovascular prevention showed that those from specialty groups tended to be less methodologically sound and recommended more aggressive therapy compared with those from primary care organizations.15

Bob: I still believe there is a role for well-conceived, impartial guidelines. The IOM has made sound recommendations, and the World Health Organization has also proposed a rigorous 19-step process for guideline development.16 The American Cancer Society has stated that it will follow the IOM recommendations.17 Let's see how many other organizations take these lessons to heart.

Main Points

  • There are no controlled trials demonstrating that lipid screening in children improves long-term health outcomes.

  • Many guidelines are dominated by recommendations from experts and are not based on sound clinical studies. The development of clinical practice guidelines needs to be transparent and devoid of commercial, financial, and political influences.

  • Be wary of outdated guidelines.

EBM Points

  • Primary care guidelines tend to be more methodologically sound than specialty society guidelines.

  • The level of evidence of clinical practice guidelines should be reviewed before widespread implementation.

editor's note: The issue of universal screening for hyperlipidemia in children will be addressed in an upcoming Controversies in Family Medicine pair of pro/con editorials.

Address correspondence to Robert Dachs, MD, FAAFP, at dachsmd@aol.com. Reprints are not available from the authors.

Author disclosure: No relevant financial affiliations to disclose.

REFERENCES

1. Daniels SR, Greer FR; Committee on Nutrition. Lipid screening and cardiovascular health in childhood. Pediatrics. 2008;122(1):198–208.

2. U. S. Preventive Services Task Force. Screening for lipid disorders in children: U.S. Preventive Services Task Force recommendation statement. Pediatrics. 2007;120(1):e215–e219.

3. Grossman DC, Moyer VA, Melnyk BM, Chou R, DeWitt TG; U. S. Preventive Services Task Force. The anatomy of a US Preventive Services Task Force Recommendation: lipid screening for children and adolescents. Arch Pediatr Adolesc Med. 2011;165(3):205–210.

4. Tricoci P, Allen JM, Kramer JM, Califf RM, Smith SC Jr. Scientific evidence underlying the ACC/AHA clinical practice guidelines [published correction appears in JAMA. 2009;301(15):1544]. JAMA. 2009;301(8):831–841.

5. Mendelson TB, Meltzer M, Campbell EG, Caplan AL, Kirkpatrick JN. Conflicts of interest in cardiovascular clinical practice guidelines. Arch Intern Med. 2011;171(6):577–584.

6. Neuman J, Korenstein D, Ross JS, Keyhani S. Prevalence of financial conflicts of interest among panel members producing clinical practice guidelines in Canada and United States: cross sectional study. BMJ. 2011;343:d5621.

7. Choudhry NK, Stelfox HT, Detsky AS. Relationships between authors of clinical practice guidelines and the pharmaceutical industry. JAMA. 2002;287(5):612–617.

8. Lenzer J. Alteplase for stroke: money and optimistic claims buttress the “brain attack” campaign. BMJ. 2002;324(7339):723–729.

9. Institute of Medicine. Clinical practice guidelines we can trust. March 23, 2011. http://www.iom.edu/Reports/2011/Clinical-Practice-Guidelines-We-Can-Trust.aspx. Accessed February 24, 2012.

10. Shekelle PG, Ortiz E, Rhodes S, et al. Validity of the Agency for Health-care Research and Quality clinical practice guidelines: how quickly do guidelines become outdated? JAMA. 2001;286(12):1461–1467.

11. Hsu HE, Shutt KA, Moore MR, et al. Effect of pneumococcal conjugate vaccine on pneumococcal meningitis. N Engl J Med. 2009;360(3):244–256.

12. Nigrovic LE, Kuppermann N, Malley R; Bacterial Meningitis Study Group of the Pediatric Emergency Medicine Collaborative Research Committee of the American Academy of Pediatrics. Children with bacterial meningitis presenting to the emergency department during the pneumococcal conjugate vaccine era. Acad Emerg Med. 2008;15(6):522–528.

13. Kimia AA, Capraro AJ, Hummel D, Johnston P, Harper MB. Utility of lumbar puncture for first simple febrile seizure among children 6 to 19 months of age. Pediatrics. 2009;123(1):6–12.

14. Subcommittee on Febrile Seizures; American Academy of Pediatrics. Neurodiagnostic evaluation of the child with a simple febrile seizure. Pediatrics. 2011;127(2):389–394.

15. Fretheim A, Williams JW Jr, Oxman AD, Herrin J. The relation between methods and recommendations in clinical practice guidelines for hypertension and hyperlipidemia. J Fam Pract. 2002;51(11):963–968.

16. Schünemann HJ, Fretheim A, Oxman AD; WHO Advisory Committee on Health Research. Improving the use of research evidence in guideline development: 1. Guidelines for guidelines. Health Res Policy Syst. 2006(4):13.

17. Brawley O, Byers T, Chen A, et al. New American Cancer Society process for creating trustworthy cancer screening guidelines. JAMA. 2011;306(22):2495–2499.

If you conduct a journal club and would like to know the next article that will be discussed, please e-mail afpjournal@aafp.org with “AFP Journal Club notification” in the subject line.

For more information on evidence-based medicine (EBM) terms, see the EBM Toolkit at http://www.aafp.org/afp/ebmtoolkit.

A collection of AFP Journal Club published in AFP is available at http://www.aafp.org/afp/jc.


Copyright © 2012 by the American Academy of Family Physicians.
This content is owned by the AAFP. A person viewing it online may make one printout of the material and may use that printout only for his or her personal, non-commercial reference. This material may not otherwise be downloaded, copied, printed, stored, transmitted or reproduced in any medium, whether now known or later invented, except as authorized in writing by the AAFP. Contact afpserv@aafp.org for copyright questions and/or permission requests.

Want to use this article elsewhere? Get Permissions


Article Tools

  • Download PDF
  • Print page
  • Share this page
  • AFP CME Quiz

Information From Industry

More in Pubmed

Navigate this Article