Letters

Evaluating online EBM

To the Editor:

"Practical Evidence-Based Internet Resources" [July/August 2003, page 49] by Dr. Brian Alper highlights an important topic: How do physicians get answers to their questions at the point of care - and not just any answers, but answers based on the best available evidence? Unfortunately, the article and comparison table neglect the following important features of a good point-of-care, evidence-based source:

• Are formal criteria for relevance used to filter the information for primary care physicians, reducing the amount of "noise"?
• Is each item labeled with a "level of evidence," or are recommendations labeled with a "strength of recommendation"?
• Are tools provided to help clinicians apply clinical decision rules and interpret diagnostic tests?
• Is the information fully peer-reviewed?
• Does the source provide a tool for "foraging" (keeping up-to-date on a daily basis with new information) as well as "hunting" (searching for the answers to clinical questions)?

The article's table also incorrectly states that InfoRetriever does not cite the best available evidence where rigorous evidence is lacking. This is simply not true; much of the information in the POEMs, diagnostic test database, Cochrane Database, Griffith's 5-Minute Clinical Consult and practice guideline summaries (all part of InfoRetriever) fits this description. We agree that evidence-based medicine means basing decisions on the best available evidence, not the best possible evidence.

While we agree none of these tools is perfect, all provide valuable guidance to physicians at the point of care. As family physicians, we are not only in the health care business, we are in the knowledge business, and it is important that we become comfortable using advanced knowledge tools like those described in the article.

Mark H. Ebell, MD, MS
David Slawson, MD, MS
Allen Shaughnessy, PharmD
Henry Barry, MD, MS

Note: The authors are co-founders of InfoPOEMs (http://www.infopoems.com).

Author's response:

I appreciate the opportunity to address the concerns of the InfoPOEMs editors, who are pioneers in this area. I'd like to respond to several points:

• DynaMed uses formal criteria for relevance based on InfoPOEMs criteria (designed for clinical alerting) and modified to update a clinical reference;
• "Level of evidence" labels can be easily misinterpreted. The important feature for clinicians, stating methods and quality of evidence related to "facts" and recommendations, is done to varying degrees by all evidence-based resources;
• "Foraging" tools are provided by InfoPOEMs (weekdays), DynaMed (whenever new updates warrant mass alerting), and UpToDate ("what"s new" summary every four months). This function is separate from answering clinical questions, and it warrants its own article.

Other "neglected" features were in fact addressed in the text of the article.

I stand by my original interpretation of best available evidence, defined as "high likelihood of finding research citations when the best available evidence consists of less rigorous studies." Griffith's 5-Minute Clinical Consult, the InfoRetriever component with greatest breadth, doesn't typically provide research citations. The only evidence-based resources with evidence for answering a substantial proportion of clinical questions are DynaMed (55 percent) and UpToDate (34 to 45 percent).1,2,3

• Alper BS, Stevermer JJ, White DS, Ewigman BG. Answering family physicians' clinical questions using electronic medical databases. J Fam Pract. 2001;50:960-965.
• Blackman D, Cifu A, Levinson W. Can an electronic database help busy physicians answer clinical questions? J Gen Intern Med. 2002;17(suppl 1):220.
• Schilling L, Steiner J, Anderson R. Patient-specific clinical questions: ask, seek, and receive. J Gen Intern Med. 2003;18(suppl 1):255.

Note: The author is founder of DynaMed (http://www.dynamicmedical.com).

Clarifying EBM resources

To the Editor:

Thank you for including UpToDate in Dr. Brian Alper's article, "Practical Evidence-Based Internet Resources." We would like to address a few points about UpToDate's editorial processes.

The article states that UpToDate does not include the Cochrane Library; however, the data cited in UpToDate is derived from a number of resources, including The Cochrane Database, Clinical Evidence, consensus guidelines and over 270 peer-reviewed journals. The article also states that UpToDate does not describe any systematic process for selecting the research included in our reviews, but we do have a systematic method for identifying and analyzing relevant literature.

Each topic has an author who is an expert in the area and at least two other physician reviewers. This group works together to screen and select studies for presentation based upon the following hierarchy of evidence:

• Systematic reviews of randomized trials,
• Single randomized trial,
• Systematic review of observational studies,
• Single observational study,
• Unsystematic clinical observations.

The recommendations are carefully linked to the available evidence, and when there is insufficient evidence from controlled clinical trials and other sources of data, UpToDate explicitly states that the recommendations are the authors'.

All of the topic reviews in UpToDate are revised whenever important new information is published. Updates are carefully integrated into UpToDate, with specific statements as to how the new findings should be applied clinically. Each topic review has a date indicating the most recent author review, and approximately 30 percent are updated during each four-month cycle. The new material in each topic review is underlined for easy identification, and updated and new topic reviews can be recognized by choosing the "Show Updated Topics" option in the table of contents. In addition, UpToDate is currently working with Gordon Guyatt at McMaster University to standardize our presentation of evidence.

Frank J. Domino, MD
Larry Culpepper, MD

Note: The authors are editors for UpToDate (http://www.uptodate.com).

Author's response:

A systematic process is necessary to identify the best available evidence. Selecting evidence based on the systematic evaluation of study methodology reduces bias that occurs when expert authors select articles based on individual choices. One approach is systematic searching using explicit search protocols. Clinical Evidence and PDxMD (for some content) use this approach. DynaMed, InfoRetriever and UpToDate do not use this approach for most content because of the increased effort required.

Another approach is systematic literature surveillance using explicit protocols for article selection. With this approach, the authors and reviewers may modify the content, but article selection is protocol-driven. DynaMed and InfoRetriever use this approach, which allows coverage of a broader range of content yet systematically selects for the best available evidence. This is different from a process that identifies articles but allows authors to selectively choose what and how to cite, negating the systematic approach.

For example, a CDC position-paper states that antibiotics produce only a small benefit in acute sinusitis, based on five placebo-controlled randomized trials and two meta-analyses. It also states that broad-spectrum antibiotics are no more effective than narrow-spectrum antibiotics, based on three meta-analyses.1,2 However, the UpToDate review of acute sinusitis states there is solid evidence supporting antibiotics to reduce illness duration based on a single citation, which is neither a scientific study nor systematic review.3,4 It recommends broad-spectrum antibiotics, citing disagreement with the "consensus guidelines" without disclosing their underlying evidence.

Hopefully UpToDate's work with Gordon Guyatt will lead to standardization in the way evidence is selected and not just the way it is presented.

1. Snow V, Mottur-Pilson C, Hickner JM. Principles of appropriate antibiotic use for acute sinusitis in adults. Ann Intern Med. 2001;134:495-497.
2. Hickner JM, Bartlett JG, Besser RE, et al. Principles of appropriate antibiotic use for acute rhinosinusitis in adults: background. Ann Intern Med. 2001;134:498-505.
3. Gwaltney Jr JM. Acute sinusitis (rhinosinusitis). In: Rose BD, ed. UpToDate. Wellesley, Mass: 2003.
4. Gwaltney Jr JM. Acute community acquired bacterial sinusitis: to treat or not to treat. Can Respir J. 1999;6(suppl A):46A.

Note: The author is founder of DynaMed (http://www.dynamicmedical.com).

The root of the problem

To the Editor:

Thank you for your focus on errors in medical practice. We have been working on this issue locally but have found it difficult to effect change in the larger health care system. Let me share a case in point:

Recently, I read a nice article about error prevention in the physician newsletter of a large medical institution. The physician who wrote the article is now in a high managerial position, but I have known him for years to be an intelligent, reasonable and conscientious physician. In the article, he invited comments about this important issue. It so happened that the year before I had had a bad experience with this institution when it failed to report a patient's seriously abnormal test result. I had to call to request the result after discovering through my tracking process that we had not received it after a few weeks. I offered suggestions that they tighten their tracking process but received no reply.

WE WANT TO HEAR FROM YOU

Send your comments to fpmedit@aafp.org. Submission of a letter will be construed as granting AAFP permission to publish the letter in any of its publications in any form. We cannot respond to all letters we receive. Those chosen for publication will be edited for length and style.

My colleague had put his e-mail, phone and fax numbers in the article, so I decided I would follow up with him about the problematic tracking process. I began by e-mailing him, but the e-mail did not go through. I phoned and left a message with his assistant and later a detailed message on his private voice mail with my pager number, phone number and e-mail. He did not respond. I tried to fax a letter, but it also failed to go through even after I checked the fax number with his assistant. Finally, I wrote a letter in care of the institution. I was unsuccessful in reaching him and gave up. Clearly, our health care institutions have much work to do in the area of error prevention.

P. Schludermann, MD
Hillsboro, Ore.

Correction

The sample performance record in "Tracking Performance" [Ask FPM, September 2003, page 73] should have included a space for the employee's initials rather than the employer's. When employees initial the record, they acknowledge they have seen and understood their employers' evaluation, which is especially important if any future performance discrepancies arise.

---

Copyright © 2003 by the American Academy of Family Physicians.
This content is owned by the AAFP. A person viewing it online may make one printout of the material and may use that printout only for his or her personal, non-commercial reference. This material may not otherwise be downloaded, copied, printed, stored, transmitted or reproduced in any medium, whether now known or later invented, except as authorized in writing by the AAFP. Contact fpmserv@aafp.org for copyright questions and/or permission requests.