Jul-Aug 2008 Table of Contents

Eight Steps to a Chart Audit for Quality



FREE PREVIEW Log in or buy this issue to read the full article. AAFP members and paid subscribers get free access to all articles. Subscribe now.


FREE PREVIEW Subscribe or buy this issue. AAFP members and paid subscribers get free access to all articles.

A simple chart review can help your group answer the question on everyone's mind: “How are we doing?”

Fam Pract Manag. 2008 Jul-Aug;15(7):A3-A8.

For many family physicians, the idea of a chart audit conjures up images of federal investigators or insurance company representatives descending on their offices to look for evidence of wrongdoing. For the most part, however, a chart audit is not so scary. A chart audit is simply a tool physicians can use to check their own performance, determine how they're doing and identify areas where they might improve. The purpose of this article is to describe some scenarios in which a chart audit might be helpful and to offer step-by-step instructions for doing one.

Why a chart audit?

Chart audits can serve many purposes, from compliance to research to administrative to clinical. You can conduct a chart audit on virtually any aspect of care that is ordinarily documented in the medical record. Practices frustrated with clinical processes that don't work well can use chart audits to document that something is wrong, find the defect in the process and fix it.

Perhaps the most beneficial use for a chart audit is to measure quality of care so that you can improve it. Chart audits are often used as part of a quality improvement initiative. For example, a practice might review charts to see how often a particular vaccine is offered, given or declined. If the audit determines that the vaccine is not being offered or given as recommended, then there is room for improvement. The same practice could review the panels of individual physicians within the group to see if they differ in performance on this measure and to give focus to their improvement efforts (for additional chart audit ideas, see below).

POTENTIAL TOPICS FOR QUALITY AUDITS

Preventive care

Percentage of women ages 21–64 who have had a Pap smear within the past three years

Percentage of adults ages 51–80 who have had colon cancer screening

Percentage of children age 2 who have completed all recommended immunizations

Percentage of elderly adults with documented fall risk assessment within the past year

Chronic disease management

Percentage of patients with hypertension whose last blood pressure reading was < 140/90

Percentage of patients with diabetes with an A1C level recorded in the last year

Percentage of patients with diabetes whose A1C is < 7.0

Percentage of patients with diabetes with a documented eye exam within the last year

Percentage of patients with persistent asthma who are on an anti-inflammatory agent


Note: Any of these metrics would have to be defined with greater specificity before use.

POTENTIAL TOPICS FOR QUALITY AUDITS

View Table

POTENTIAL TOPICS FOR QUALITY AUDITS

Preventive care

Percentage of women ages 21–64 who have had a Pap smear within the past three years

Percentage of adults ages 51–80 who have had colon cancer screening

Percentage of children age 2 who have completed all recommended immunizations

Percentage of elderly adults with documented fall risk assessment within the past year

Chronic disease management

Percentage of patients with hypertension whose last blood pressure reading was < 140/90

Percentage of patients with diabetes with an A1C level recorded in the last year

Percentage of patients with diabetes whose A1C is < 7.0

Percentage of patients with diabetes with a documented eye exam within the last year

Percentage of patients with persistent asthma who are on an anti-inflammatory agent


Note: Any of these metrics would have to be defined with greater specificity before use.

A chart audit is one of numerous data sources available for quality improvement efforts. Others include patient surveys, discharge summary reviews, billing/claims data and employee feedback.

How to do it

Below we describe eight steps to a formal chart audit. Although the process is not necessarily linear, we will discuss each step in the order it might typically occur, using the example of a breast cancer screening audit to illustrate each step. Because the audit will involve reviewing confidential data, it is important to check your institutional guidelines regarding patient confidentiality before you get too far into the planning process.

Step 1: Select a topic. The focus of your audit must be clear, neither too narrow nor too broad, and measurable using data available in the medical record. If possible, choose an area that interests you. You will find that you are more able to recognize nuances in your study when you have personal interest in the topic. Of course, your topic should also be of interest to the practice, perhaps a problem or aspect of care that the providers have identified as needing improvement. The Joint Commission recommends studying issues that are high frequency, high risk or both.

You should also consider early in the process how important external comparison is to your purpose. If it is quite important, then choose a topic that has an existing, well-defined measure and available benchmark data – even one you might not choose otherwise – because this will be more practical than developing your own standard for comparison.

Chart auditing is an iterative process – don't be discouraged if you change directions several times before settling on a topic.

Example: Your practice wants to measure how well it's doing on meeting recommendations for preventive care. Since the insurance carriers in the area are focusing heavily on women's health, the group decides to focus its chart review on screening for breast cancer (mammography).

Step 2: Identify measures. Once you're set on a topic, you need to define exactly what you will measure. Criteria must be outlined precisely, with specific guidelines as to what should be counted as a “yes” (criteria met) and what should be counted as a “no” (not met).

For example, if you decided to review the rate at which foot exams were performed on patients with diabetes in the last year, you would need to decide what qualifies as an adequate foot exam. Is it monofilament testing for sensation? Visual inspection? Palpation of pulses? Many would say all three are necessary for a complete foot exam. If only two of the three are documented, how will you count that?

It may be worthwhile to do a literature review to help you define your measures or consult measures used by insurers or accrediting bodies; adopting measures that have been used successfully in the past will make your work easier. A literature review may also help you identify benchmarks for comparison.

Once you've chosen measures that seem workable, it can be helpful to conduct a pilot audit. Just going through a few charts will help to identify issues that need to be clarified before starting a full audit.

Example: For your audit on breast cancer screening, the group considers several measures, including the following:

  • Time since last mammogram. This provides the most specific information but would require more analysis.

  • Mammogram completed within last year. This measure attempts to assess compliance with clinical guidelines. The U.S. Preventive Services Task Force recommends screening mammography every one to two years for women age 40 and older. However, the Healthcare Effectiveness Data and Information Set (HEDIS) measures, which most health plans use for National Committee for Quality Assurance (NCQA) accreditation purposes, require at least one mammogram completed within the past 24 months.

  • Mammogram ordered within last year. Do you want to measure only whether the study was done, or whether it was recommended or ordered by the provider? Should providers be held accountable when patients decline to have the test?

After considerable discussion, the group decides to measure whether a mammogram was completed or recommended within the last 24 months.

Step 3: Identify the patient population. To determine which records to review, you need to define the population you want to assess. Characteristics to consider may include age, gender, disease status and treatment status. In many cases, the focus of the audit and even the measure itself will help to define the population. You'll also need to develop specific inclusion or exclusion criteria.

Example: In keeping with the HEDIS breast cancer screening measure that your group decided to follow, your patient population will be women age 40 to 69. Because you'll be looking for evidence of a mammogram in the past 24 months, the lower age limit for the sample will be 42. Only those patients with at least three visits in the last two years and one in the last 13 months will be included. You decide to exclude women who have had bilateral mastectomies or are terminally ill.

Step 4: Determine sample size. A manual audit of all charts meeting your inclusion criteria will not be feasible in most situations. That's where sampling comes in. For an informal, or “quick and dirty,” audit designed to give you a sense of whether a more sophisticated audit is warranted, you may find it useful to sample a minimum of 20 charts. For better results, a common rule of thumb is to try for 10 percent of the eligible charts. Or you may choose to use a convenience sample: the patients from a single day or all the charts on a single shelf in the records room.

If you want to track a measure over time, or if you want your results to be statistically valid, your sample size is critical. If the sample is too small, the random variability will be too large, and the results will be limited in their applicability.

Example: Using the process outlined below, your group determines that its sample should total 81 charts.

DETERMINING SAMPLE SIZE

Calculating a statistically valid sample size for a chart review follows steps adapted from statistical techniques used for descriptive studies. The process uses a nomogram, or table, to identify the desired number:

1. Estimate the expected proportion within the population that will have the measure of interest.

If you have a benchmark from literature or prior studies, use it. Otherwise, consult with colleagues or experts in the field to determine an estimate. The tables generally require this proportion to be 50 percent or less. If more than 50 percent of the population is expected to have the characteristic, then base your sample size calculation on the proportion without the characteristic.

2. Specify the width of the confidence interval you wish to use.

All empirical estimates based on a sample have a certain degree of uncertainty associated with them. It is necessary, therefore, to specify the desired width of the confidence interval (W). This gives a range of values that you can be confident contains the true value. In most cases, an appropriate width is 0.20 (that is, plus or minus 10 percent).

3. Set the confidence level.

This is a measure of the precision or level of uncertainty. Typically 95 percent is used, meaning that we are 95 percent certain that the interval includes the true value. This is arbitrary, however, and other levels of confidence can be used. The table shown below is for a 95-percent confidence level. The narrower the width of the confidence interval and the higher the confidence level, the larger the sample size.

4. Use the nomogram (below) to estimate sample size.

Sample size for a descriptive study of a dichotomous variable 95-percent confidence interval

Width of the confidence interval (W) 0.10 0.15 0.20 0.25 0.30

Expected proportion (P)

0.10

138

61

0.15

196

87

49

31

0.20

246

109

61

39

27

0.25

288

128

72

46

32

0.30

323

143

81

52

36

0.40

369

164

92

59

41

0.50

384

171

96

61

43


Adapted with permission from Hulley SB, et al. Designing Clinical Research, 3rd ed. Philadelphia: Wolters Kluwer Health; 2006:91.

Sample size for a descriptive study of a dichotomous variable 95-percent confidence interval

View Table

Sample size for a descriptive study of a dichotomous variable 95-percent confidence interval

Width of the confidence interval (W) 0.10 0.15 0.20 0.25 0.30

Expected proportion (P)

0.10

138

61

0.15

196

87

49

31

0.20

246

109

61

39

27

0.25

288

128

72

46

32

0.30

323

143

81

52

36

0.40

369

164

92

59

41

0.50

384

171

96

61

43


Adapted with permission from Hulley SB, et al. Designing Clinical Research, 3rd ed. Philadelphia: Wolters Kluwer Health; 2006:91.

AN EXAMPLE

According to HEDIS 2007 Audit Means, Percentiles and Ratios, the NCQA's annual report of health plan performance data, 68.9 percent of women age 40 to 69 had a mammogram during 2006. This makes the expected proportion of those without screening 31.1 percent. We choose a width of the confidence interval of 0.20 (plus or minus 10 percent) and a confidence level of 95 percent. This means that we want to be 95 percent confident that the result falls between 58.9 percent and 78.9 percent. Using the nomogram to determine the sample size, we read down the left column of figures for the expected proportion without the characteristic (0.30 is the closest value to 31.1 percent) and then across to the chosen width of the confidence interval (0.20). When we follow the column down, we find the required sample size (81). If the number required is too large to be completed, we can recalculate with a lower confidence level or wider interval; this will produce a smaller sample size.

Step 5: Create audit tools. To complete your chart audit, you will need instruments on which to record your findings. How they are structured and the details they include will affect the analysis you can do and the eventual usability of your findings. Data should be collected in a format that keeps all individual records separate but allows for easy compiling.

Many chart audits involve the calculation of a rate, percentage, mean or other statistical measurement. An electronic spreadsheet format can be customized to do these calculations for you. For those more comfortable with paper-based systems, a preprinted form that lists the specific items to check in each chart serves well as an audit tool. One form is completed for each chart, and the forms can then be sorted and counted as desired. A separate form can be used to tabulate results.

Creating clear, simple audit tools will make it possible for nonclinical staff to perform many audits effectively. Once you've developed the forms, if someone other than you will be doing the actual chart reviews, go over a few examples together to be sure the reviewer understands the criteria exactly as you intend.

Example: Your group decides to use paper forms for the chart audit (see the completed forms below).

CHART AUDIT FOR BREAST CANCER SCREENING

Patient identification Inclusion criteria Exclusion criteria Mammogram in past 24 months No mammogram in past 24 months
Patient name MRN Age 42–69 as of 12/31/07 3 visits in past 3 years 1 visit in past 13 months Bilateral mastectomy Left practice, terminally ill, expired Locally Elsewhere No discussion documented Discussed patient declined Mammogram order not completed

Jane D

A2345

53

yes

yes

no

no

yes

no

Sue S

B2345

62

yes

yes

no

no

yes

no

Ann J

C2345

59

yes

yes

no

no

no

no

yes

no

no

Betty M

D2345

65

yes

yes

yes

no

Julie J

E2345

57

yes

yes

no

yes

Bonnie B

F2345

52

no

Alice G

G2345

55

yes

yes

no

no

yes

no

Kate H

H2345

61

yes

no

Dana T

I2345

63

yes

yes

right side only

no

no

yes

Doris B

J2345

40

Helen P

K2345

64

yes

yes

no

no

yes

no

Evelyn C

L2345

51

yes

yes

no

no

yes

no

Paula T

M2345

49

yes

yes

no

no

yes

no

Mary S

N2345

69

yes

yes

no

no

yes

no


Note: Shading indicates that the patient has not met the exclusion or inclusion criteria.

CHART AUDIT FOR BREAST CANCER SCREENING

View Table

CHART AUDIT FOR BREAST CANCER SCREENING

Patient identification Inclusion criteria Exclusion criteria Mammogram in past 24 months No mammogram in past 24 months
Patient name MRN Age 42–69 as of 12/31/07 3 visits in past 3 years 1 visit in past 13 months Bilateral mastectomy Left practice, terminally ill, expired Locally Elsewhere No discussion documented Discussed patient declined Mammogram order not completed

Jane D

A2345

53

yes

yes

no

no

yes

no

Sue S

B2345

62

yes

yes

no

no

yes

no

Ann J

C2345

59

yes

yes

no

no

no

no

yes

no

no

Betty M

D2345

65

yes

yes

yes

no

Julie J

E2345

57

yes

yes

no

yes

Bonnie B

F2345

52

no

Alice G

G2345

55

yes

yes

no

no

yes

no

Kate H

H2345

61

yes

no

Dana T

I2345

63

yes

yes

right side only

no

no

yes

Doris B

J2345

40

Helen P

K2345

64

yes

yes

no

no

yes

no

Evelyn C

L2345

51

yes

yes

no

no

yes

no

Paula T

M2345

49

yes

yes

no

no

yes

no

Mary S

N2345

69

yes

yes

no

no

yes

no


Note: Shading indicates that the patient has not met the exclusion or inclusion criteria.

Step 6: Collect data. Select the date or dates on which you will collect data. Be sure to coordinate the specifics (date, time and number of charts to be pulled) with the medical records staff. Review each chart to determine if the patient meets the selection criteria. The reviewer should complete one audit tool (paper form or row in the electronic spreadsheet) for each patient that meets the criteria. To protect patient confidentiality, patient names should not be included on the review forms.

Example: You instruct your office staff to pull the charts of roughly 100 adult female patients. Once you've identified 81 that meet the selection criteria, your nursing supervisor fills out the audit tool for each one, reserving questionable cases for physician review.

Step 7: Summarize results. Summarizing the data is a little more complex than just counting up all the data sheets. You must consider how the data will be used and make sure the information is presented in a way that will make it meaningful. Inconsistencies here can produce data that can't be interpreted.

Example: Your breast cancer screening audit results show that 57 percent of your sample received mammograms (see the results table below).

BREAST CANCER SCREENING RESULTS

# %

Total charts reviewed

100

Patients included in audit

81

Patients who received mammogram

46

57

Received mammogram locally

25

31

Received mammogram elsewhere

10

12

Patients with no documentation of completed mammogram

35

43

Documented declined mammography

6

17

Documented mammogram ordered, not completed

4

13

No documentation of discussion of mammography

25

71

BREAST CANCER SCREENING RESULTS

View Table

BREAST CANCER SCREENING RESULTS

# %

Total charts reviewed

100

Patients included in audit

81

Patients who received mammogram

46

57

Received mammogram locally

25

31

Received mammogram elsewhere

10

12

Patients with no documentation of completed mammogram

35

43

Documented declined mammography

6

17

Documented mammogram ordered, not completed

4

13

No documentation of discussion of mammography

25

71

Step 8: Analyze and apply results. Once you have compiled your data and calculated the results, you can compare them to local or national benchmarks. There may be multiple benchmarks, depending on your topic and the performance measure you calculated. You should take into account the differences between your population and those you're comparing it with, as appropriate. If the measure is truly important to the group, you may wish to set a performance goal based on what the group feels is appropriate and reasonable and make it the focus of a quality improvement initiative.

Example: At 57 percent, your group's breast screening rate is less than the national benchmark of 68.9 percent. This benchmark is the mean for commercial HMO patients, according to the HEDIS 2007 Audit Means, Percentiles and Ratios, the NCQA's annual report of health plan performance data (view it at http://www.ncqa.org/tabid/334/default.aspx). Of the 35 patient charts that had no documentation of a mammogram, only 10 records showed that the physician had discussed the need for a mammogram with the patient. The challenge is now to drill down to figure out whether the issue was discussed but not documented in those other charts or whether it was simply overlooked. Telephone contact with the 25 identified patients might help you begin to clarify this so that an appropriate intervention can be designed.

Make it count

Chart audits can be useful tools in improvement and safety efforts. It is essential to define precisely what you want to measure and the criteria by which you will measure it. (If you're floundering, you probably haven't defined this well enough.) Sample sizes can be chosen informally or determined in a statistically valid fashion. Summarize your data in a way that makes sense for the problem you're addressing. Make sure to act on problems you find, and remeasure later to see that your changes made a difference. You and your patients will be glad you did.

About the Authors

Barbara Gregory is coordinator for the Duke Center for Community Research Virtual Library in Durham, N.C.

Cheryl Van Horn is quality analyst for the Department of Community and Family Medicine at Duke University.

Dr. Kaprielian is professor and vice-chair for education in the Department of Community and Family Medicine at Duke University, where she also serves as director for quality improvement and continuing medical education.

Author disclosure: nothing to disclose.

Send comments to fpmedit@aafp.org.

 

Editor's note: An expanded and interactive version of this content is available at http://patientsafetyed.duhs.duke.edu.

 

Copyright © 2008 by the American Academy of Family Physicians.
This content is owned by the AAFP. A person viewing it online may make one printout of the material and may use that printout only for his or her personal, non-commercial reference. This material may not otherwise be downloaded, copied, printed, stored, transmitted or reproduced in any medium, whether now known or later invented, except as authorized in writing by the AAFP. Contact fpmserv@aafp.org for copyright questions and/or permission requests.

Want to use this article elsewhere? Get Permissions


Article Tools

  • Download PDF
  • Print page
  • Share this page
  • FPM CME Quiz

Information From Industry