« FPs Can Help Curb Sp... | Main | Do No Harm: When Is ... »

Thursday Jul 30, 2015

Self-Evaluation: How Many Stars Do I Rate?

As a physician, how do I know when I'm doing a good job? Is it when my patients all say they love me and come to visit even when they aren't in need of medical attention?

Is it when my diabetic patients' hemoglobin A1c readings are all less than 7.5 percent and my hypertensive patients all achieve 130/90 mmHg?

Am I doing a good job when all of the ratings web sites and physician evaluation tools say I have five stars for patient care?

Or if I can go home at night and sleep soundly knowing I did my best, does that mean I'm a good doctor?

Metrics are important to medicine in general for quality improvement and payment issues, but how do I measure myself?

Obviously, no easy answer for these questions exists. Far from a yes or no answer, the ability to quantify physician performance stymies the best statisticians and government bureaucrats. Not only do these measures affect patient recruitment and retention, they frequently determine reimbursement. And CMS is attempting to inform consumers by publicly reporting physician and group quality and patient experience data via Physician Compare(www.medicare.gov).

As the health care environment is rapidly shifting to value-based payment and public reporting, it is imperative that metrics accurately reflect a physician''s quality and performance. No single metric has yet captured the essence of physician performance and perhaps no single metric ever will. Medicine and the physician-patient relationship are too complex to represent as a single number.

The advent of the patient satisfaction score has led to subjective quantization of physician performance. Instead of evidence-based measures of clinical success, satisfaction scores derive from measures of a patient's emotional state. Less a reflection of any measurable quantity, these numbers show patient feelings of contentment and satisfaction with care delivery. So what, exactly, are these ratings worth? Research indicates that patients who are most satisfied with their doctors have higher health care costs and increased mortality.(archinte.jamanetwork.com)

Aside from being entirely subjective, these measures are also highly susceptible to statistical biases. Most often, reporting bias, or the tendency to report only observations of a certain kind, interferes with accurate assessment. Hospitals and doctors' offices may not randomize their surveys, but instead send them only to patients they know will report high satisfaction. Conversely, even if randomized, there are statistical data to show a higher return on negative results(jpubhealth.oxfordjournals.org).

This may be because patients with a negative experience are more likely to leave feedback or because patients with a positive experience don't complete surveys. Whatever the reason, these measures by themselves lack necessary mathematical significance to provide consistent and useful physician performance metrics.

In the internet age, anonymity and open access allow anyone to say whatever they please, ensuring that third-party, online reporting tools --- such as Healthgrades or Yelp -- fall prey to similar bias. Although many of these sites use metrics that do not solely rely on patient satisfaction scores, the core of these online ratings is subjective reporting. Good and bad experiences are distilled into a numerical report card, highlighted by commentary that may or may not be helpful. Inherent bias aside, allowing commentary does improve the usefulness and potential applicability of these metrics, but they are still inadequate to convey the breadth of physician competence and capability.

Most evidence-based physician outcomes derive from meeting evidence-based goals, such as the hemoglobin A1c goal above. Performance metrics cover objective data like quantifiable minimum standards of care, but that process is also fraught with problems. Culling usable data from an electronic health records system is a full-time job, meaning significant labor costs. Analyzing these data takes even more time and money. Most of these metrics rely on chart audit as well, meaning any mischarted or unrecorded data points change the overall results. Measures like post-operative infections and surgical complications also involve some level of luck, or at the very least, variables over which the physician has limited control.

There's also the question of how much the performance scores accurately reflect the ability of the physician. Given that the medical relationship requires both physician and patient contributions, patient outcomes may not be solely a measure of the competence or ability of the physician. Rather, some argue that specific patient targets more closely represent patient compliance, while only loosely approximating physician proficiency. To truly reflect physician performance, metrics should acknowledge the shared responsibility between the physician and the patient.

One may even ask if using outcome-based metrics may be detrimental in the long run. While a hemoglobin A1c of 7.5 percent may be acceptable for performance metrics -- a passing grade, if you will -- most diabetics benefit from a target closer to 6 percent. The same is true for measures like blood pressure, cholesterol and weight. If we let "good enough" be our only goal, we may miss the opportunity to help patients improve further.

Recently, the website ProPublica -- self-described as "an independent, non-profit newsroom that produces investigative journalism in the public interest," -- published what it called a Surgeon Scorecard(projects.propublica.org). Based on publicly available data for elective surgeries covered by Medicare, this metric analyzed the complication and mortality rates for several surgeons and hospitals across the country. Upon publication, both patients and surgeons questioned the validity and utility of this scoring system(www.kevinmd.com) because several major hospitals are missing. The data set came from elective procedures and was limited to hospitals that performed a specific number of the procedures in question. The number of procedures for each hospital varies, so the data -- like most other metrics -- tell an incomplete story. Given that there is no standard for comparison, further study of these data is needed before they can be presumed accurate.

So, if none of these measures accurately reflect physician performance, how then shall we rate ourselves? How do we aggregate objective and subjective data into a useful approximation of physician performance? I honestly don't know what works best. If I did, I'd have shared it long before now. If any of you have ideas, sound off in the comments. In the meantime, I'll content myself with remembering the pleasure I derive from seeing patients making healthy choices.

Gerry Tolbert, M.D., is a board-certified family physician who practices in northern Kentucky. A lifelong technophile, his interests include the intersection of medicine and technology. You can follow him on Twitter @DrTolbert.

Posted at 10:31AM Jul 30, 2015 by Gerry Tolbert, M.D.

« FPs Can Help Curb Sp... | Main | Do No Harm: When Is ... »


Subscribe to receive e-mail notifications when the blog is updated.



Leader Voices Blog - A Forum for AAFP Leaders and Members

FPs on the Front Lines


The opinions and views expressed here are those of the authors and do not necessarily represent or reflect the opinions and views of the American Academy of Family Physicians. This blog is not intended to provide medical, financial, or legal advice. All comments are moderated and will be removed if they violate our Terms of Use.