Return to Web Version
American Academy of Family Physicians

Radiology (Position Paper)

See also:
Imaging Personnel

Family Physician Interpretation of Outpatient Radiographs

Overview and Justification

Most family physicians provide the majority of patient care in the outpatient setting. Diagnostic radiographs are an integral part of the evaluation and management of acute and chronic illnesses for patients seen in the office. Those practices that have radiology services on-site usually do not have a radiologist on staff, so the family physician orders and interprets the radiographs, and then renders patient care based on the initial interpretation. The radiograph is often reread by a radiologist, and care may be modified if there is a clinically significant discrepancy between the readings. As a result of these circumstances, the family physician integrates the patient’s history, physical examination, laboratory testing and his or her interpretation of diagnostic radiographs to develop a working diagnosis and treatment plan, with only occasional changes to those plans after the radiologist’s rereading.

The United States spent over 1.4 trillion dollars for personal healthcare in 2003.1 Most published data about use of imaging services comes from Medicare billing data. In 2003, 17 percent of Medicare’s $281 billion expenditures were for physician services and about 2 percent of physician services were for plain film radiography (musculoskeletal and chest).2 Maitino recently reported that Medicare utilization rates for noninvasive diagnostic imaging have been increasing although utilization for general radiology (plain films) has decreased slightly.3 Based on 1999 Medicare claims data, 30.8 percent of all non-invasive diagnostic imaging is performed by non-radiologists, but only 23.9 percent of general radiology.4 Assuming similar proportions for non-radiologists’ Medicare claims in 2003 and similar proportions for total personal healthcare expenditures, $1.17 billion was paid to non-radiologists for musculoskeletal and chest radiography in 2003.

Recent data specifically about office radiology use has not been published. Sunshine estimated that in 1989 1.8 percent of total radiology work was performed by family physicians and general practitioners.5 Specifically looking at general radiology technical and professional interpretation costs from 1989 Medicare data, Sunshine and colleagues estimated that non-radiologists performed 57 percent of the general radiology in the office or free-standing clinic setting.5 Family physicians and general practitioners performed 13 percent, with cardiologists, orthopedists, general internists, ophthalmologists and other specialists performing the other 44 percent.5 Using 1993 Medicare data, Spettell and associates estimated that 73.2 percent of outpatient radiographs of the chest, spine, pelvis, hip, and upper and lower extremities were performed and interpreted by nonradiologists.6 While using 1991 claims data from a private insurer, Levin found that 70 percent of office studies were performed by nonradiologists.7 These data suggest that a substantial portion of office radiography in the United States is performed by nonradiologists, although a relatively small portion of that is performed by family physicians. It is not clear if the amount of office radiography performed by family physicians is increasing or decreasing in recent years.

Physicians billing for office radiography may bill for the technical component (taking the pictures) or the professional component (reading the films) or both. A family physician with on-site radiography equipment will typically bill for the technical component of the imaging service. If the family physician also reads the radiographs and generates a separate written report, then the professional component would be billed also. If the radiographs are initially read by the family physician and the radiographs are read a second time by a radiologists who generates the written report, the radiologist bills for the professional component.

There has been concern that on-site radiography leads to more frequent use and results in increased health care costs.8-12 However, one study reported that a decision by an insurer to deny claims for professional charges for radiologic services performed by nonradiologists resulted in a 12 percent increase in expenditures.13 Another study of general practitioners in New Zealand reported an increase in emergency department referrals and hospital admissions in areas of decreased patient access to imaging services.14 More factors than simply having on-site radiography equipment may influence the frequency of use. Wilson reported an a comparison of primary care physicians with and without on-site radiography and found an increased frequency of chest x-rays among physicians with on-site radiography, but an equal frequency of lumbar spine x-rays.15 It is clear that on-site radiography influences the frequency of imaging procedures. However, the appropriate frequency of radiography use is not known, nor what negative effect increasing distance to imaging facilities has on appropriate use.

Because of its importance in initiating immediate therapy, office radiology is a significant part of the practice of many family physicians. The amount of radiologic services they provide is relatively small compared to the total and the exact costs of plain film radiography attributable to family physicians is not known.

Section I: Scope of Practice for Family Physicians

It is the position of the American Academy of Family Physicians that clinical privileges should be based on each individual physician’s training and experience, demonstrated abilities and current competence, not on specialty designation alone. This general policy should certainly apply to ordering and interpreting office diagnostic imaging.

The number of U.S. family physicians with radiology equipment in their offices has not been reported. A 1988 survey of Minnesota family physicians found that 87.3 percent had on-site radiographic equipment.16 An unpublished study in 1996 found that 76 percent of Wisconsin family physicians have radiographic equipment in their office and 87 percent have this equipment in the same building.17 Based on the distribution of nonradiologists’ share of office radiology work broken down by state, Minnesota (43 percent) and Wisconsin (55 percent) were close to the average for the entire United States (46 percent), although the amount of office radiology work done by nonradiologists did vary widely from 21 percent for Wyoming to 84 percent for North Dakota.5 The limited data suggest that a majority of family physicians use radiographic equipment in their offices.

There is no doubt that patient convenience and satisfaction are improved by the presence of on-site radiography. Traveling to another facility, especially for the elderly and the disabled, places an additional burden on patients and caretakers. There is a risk of delaying appropriate diagnosis and treatment, especially with late afternoon or Friday appointments. A study of general practitioners in New Zealand reported an increase in emergency department referrals and hospital admissions in areas of decreased patient access to imaging services.14 Patient care will be compromised if radiographs are required for appropriately evaluation and treatment, but circumstances prohibit travel or the patient refuses to go to another facility.

The Centers for Medicare and Medicaid Services’ (CMS) policy for Medicare Part B payment for the professional component of diagnostic radiography does not discriminate approval for payment based on specialty. Chapter 13, Section 20.1 of the Medicare Claims Manual states: “Carriers must pay for the PC [professional component] of radiology services furnished by a physician to an individual patient in all settings under the fee schedule for physician services regardless of the specialty of the physician who performs the service.”18 In a related policy regarding interpretations of x-rays in the emergency department, the CMS policy states that carriers will pay for radiology services if the following requirements are met “a. The services are personally furnished for an individual beneficiary by a physician. b. The services contribute directly to the diagnosis or treatment of an individual beneficiary. c. The services ordinarily require performance by a physician. d. Are identifiable, direct, and discrete diagnostic or therapeutic services furnished to an individual beneficiary, such as interpretation of x-ray plates, angiograms, myelgrams, pyelograms or ultrasound procedures. Payment for a professional component of a diagnostic procedure furnished to a beneficiary in a hospital includes an interpretation and written report for inclusion in the beneficiary’s medical record maintained by the hospital.”19 In addition, CMS policy also states “Generally, payment is made for only one interpretation of an EKG or x-ray procedure furnished to an emergency room patient.” When multiple claims for the same interpretation occur, “Payment is made for the interpretation and report that directly contributed to the diagnosis and treatment for the individual patient. We will consider the second interpretation to be a quality control service. The physician specialty is not the primary factor in deciding which interpretation and report to pay regardless when the service was performed.”19

The American Medical Association (AMA) approved policy D-385.974 Freedom of Practice in Medical Imaging in 2005 stating “Our AMA will: … (3) actively oppose efforts to require patients to receive imaging services at imaging centers that are mandated to require specific medical specialty supervision and support patients receiving imaging services at facilities where appropriately trained medical specialists can perform and interpret imaging services regardless of medical specialty; and (4) actively oppose any attempts by federal and state legislators, regulatory bodies, hospitals, private and government payers, and others to restrict reimbursement for imaging procedures based on physician specialty, and continue to support the reimbursement of imaging procedures being performed and interpreted by physicians based on the proper indications for the procedure and the qualifications and training of the imaging specialists in that specific imaging technique regardless of their medical specialty.”20

The American Academy of Orthopedic Surgeons (AAOS) approved a revised position statement in March 2000 regarding office radiograph performance and interpretation which concludes that “The AAOS believes that orthopaedists are entitled to adequate compensation for the cost and work involved in providing [musculoskeletal radiographic studies] in their offices. Any policy which prohibits orthopaedists from performing and interpreting diagnostic imaging studies in their offices interferes with the patient’s ability to receive optimal care.”21

The American College of Emergency Physicians reaffirmed a policy statement in October 2000 and endorsed the principle that “The emergency physician providing contemporaneous interpretation of a diagnostic study is entitled to reimbursement for such interpretation even if the study is reviewed subsequently as part of the quality control process of the institution in which the physician practices.”22

A majority of family physicians is estimated to have radiography equipment in their offices at this time. Patient care is improved when the family physician is able to fully integrate the patient’s history and physical examination with contemporaneous interpretation of diagnostic imaging and other diagnostic studies. Family physicians, like other physicians who use diagnostic radiography in their evaluation of patients, are entitled to appropriate compensation for their services.

Section II: Clinical Indications

Many clinical conditions that present in a family physician’s office, including acute trauma, acute illness, chronic musculoskeletal pain and chronic illnesses require diagnostic radiography as part of the evaluation.

Well-accepted criteria for diagnostic radiography are reported in the literature for acute knee and ankle injuries that are commonly evaluated and treated in the family physician’s office.23-27

Recommendations for indications for office radiographs from a family physician’s perspective have been published.28 Patient perception of necessity of imaging may influence the frequency of radiographs 15, 29 and non-clinical indications, such as patient reassurance or work-related injury, have also been reported in some studies.30, 31

Restrictions to limit types of studies that may be performed by non-radiologists through guidelines 32 or payment33 have been reported with limited initial impact on total costs. These studies did not include sufficient analysis to address the impact on different physician specialties.

The American College of Radiology (ACR) Committee on Appropriateness Criteria and its expert panels use literature review and a modified Delphi technique to develop clinical practice guidelines based on clinical indications for a large number of diagnostic imaging and therapeutic techniques.34,35 Representatives from fifteen non-radiology specialty organizations participate in the process, but there are no family physicians presently participating in the development of the ACR Appropriateness Criteria™.36 The criteria address a large variety of clinical conditions and each topic includes recommendations and a summary of the relevant literature. Many indications for plain radiographs are recommended for patients with acute and chronic conditions that frequently present in the family physician’s office.

Initial radiologic evaluation of a large variety of acute and chronic conditions is appropriately performed in the family physician’s office, with referral to another facility for more extensive imaging, if necessary.

Section III: Training Methodology

Training in diagnostic imaging interpretation begins during clinical training in medical school, although the amount and intensity of the training experience at various medical schools can vary widely.

The Residency Review Committee accredits family medicine residency programs and has developed a new set of requirements that will be effective July 1, 2006.37 The Diagnostic Imaging and Nuclear Medicine requirement has not changed from the previous requirement effective July 1, 20038 and states “The program must provide the residents with a structured opportunity to learn the appropriate application of techniques and specialty consultations in the diagnostic imaging and nuclear medicine therapy of organs and body systems. Instruction should include the limitations and risks attendant to these techniques. The format of the instruction should be adapted to the resources available, but must include radiographic film/diagnostic imaging interpretation and nuclear medicine therapy pertinent to family medicine.”

Extensive individualized training also occurs during acute and chronic patient care in the hospital, emergency department and continuity practice experience during residency training. This training occurs during consultation with family physician and emergency medicine preceptors, and during formal and informal consultation with interpreting radiologists. Additional training occurs when residents’ provisional readings that result in patient-care decisions are reinforced or adjusted based on the written radiologists’ reports.

For family physicians who include the use of diagnostic radiographs in their practices, training and feedback continues through consultation with practice colleagues and radiologists throughout the family physician’s career.

Section IV: Testing, Demonstrated Proficiency and Documentation

An appropriate level of competence for ordering and interpreting diagnostic radiographs, as judged by the supervising faculty, is expected for residents to advance through their training. Deficiencies would be addressed by more intense remedial training, as with any other educational category for family practice.

Testing of knowledge of indications for and interpretation of diagnostic radiographs is a part of the general testing for certification by the American Board of Family Medicine. Certification examinations include questions about diagnostic radiography and some radiographic images. Radiography is considered one of many general areas of medical knowledge tested, and there are no specific rules for numbers of interpretations of radiographs or questions about radiography on each primary certification or recertification examination.39

Section V: Credentialing and Privileges

The issue of hospital privileges is not relevant to outpatient radiograph interpretation. However, it is becoming more common for managed care and health insurance organizations to request that participating physicians go through a credentialing process, either to meet internal standards or as a part of an application for National Committee for Quality Assurance (NCQA) accreditation. The NCQA credentialing is based on licensure, Drug Enforcement Agency certification, education and training, malpractice claims history, medical board sanctions and Medicare/Medicaid sanctions, ongoing monitoring of sanctions and work history.40 Specific procedures or skills are not usually considered in the credentialing process.

The Joint Commission on Accreditation of Healthcare Organizations’ ambulatory standards includes general guidelines about equipment safety and maintenance, and clinician training and credentialing for services they provide, but no specific standards related to use of office radiography or interpretation of radiographs.41

Verrilli reported a program by BlueCross BlueShield of Massachusetts for technical and professional privileging.33 Technical privileging was based on radiography equipment and processes meeting appropriate standards of care. Professional privileging was based on a list of “appropriate” Physicians’ Current Procedural Terminology (CPT-4) codes for each specialty, specialty society recognition that imaging is an integral part their practice and physician training for conducting and interpreting the imaging test.

CMS may have an impact on credentialing as it relates to Medicare billing in the near future. In its’ March 2005 Report to Congress: Medicare Payment Policy, the Medicare Payment Advisory Commission (MedPAC) recommended that Congress should direct the Secretary of Health and Human Services to set standards for physicians that bill Medicare for performing and interpreting diagnostic imaging studies. MedPAC specifically recommended that “Medicare should not limit payment to specific specialties” and “standards should be based on some combination of physician training, experience and continuing education.”2

To date, there have been limited efforts to restrict performing and interpreting outpatient radiographs by family physicians through a credentialing process, but this may change if new standards are developed by CMS for Medicare billing.

Section VI: Miscellaneous Issues

A. Competence for interpretation of diagnostic radiographs

Several studies have evaluated the frequency of agreement between a primary care physician’s reading of office radiographs and the radiologist’s reading. Concordance between readings by family physicians and radiologists was found in 72.5 to 92.4 percent of all radiographs.42-46 In addition, concordance between readings by internists and radiologists was found in 92 percent of all radiographs.47 Concordance for extremity films was higher, from 79 to 96 percent.42,44,46 Concordance rates were lower for chest radiographs, ranging from 41.9 to 89.5 percent,42,44,46-50 probably reflecting a greater level of complexity. Different criteria for concordance were used, so results from different studies are not directly comparable.

Similar studies of primary care physicians’ and emergency physicians’ readings of radiographs obtained in emergency departments yielded concordance rates of 83.2 to 99.3 percent.51-57 As with the primary care office radiograph reading studies above, different criteria for concordance were used, so results are not directly comparable.

Studies of discordance between radiologists for readings of chest x-rays have shown disagreements for 18 to 57 percent of readings.58-65 Disagreements between radiologists for a variety of radiographs have been reported to occur between 3-4.4 percent of readings.66-68 Bergus and colleagues found that 35 percent of discordant readings were ultimately determined to have been correctly interpreted by the family physician.44

Primary care clinicians may have an advantage toward improved accuracy with more complete clinical history. A recent review by Loy reported the majority of studies showed higher accuracy of radiograph readings when clinical information was provided, a few with no effect and none with a decrease in accuracy.69

The literature suggests that although family physicians are not perfect in their interpretations of diagnostic radiographs, their error rates are similar to the rates experienced by radiologists.
B. Necessity of a second radiograph reading

A variety of studies have been published in recent years addressing the issue of whether a second reading by a radiologist impacts clinical care.

Smith reported a study of 1393 pairs of radiograph readings from 86 primary care clinicians in 9 ambulatory practices in Wisconsin. In a subgroup of 553 pairs when the clinician would not have hypothetically requested radiology consultation there was a 2.5% frequency of any change in clinical care and zero substantial changes in care.46 Similarly Halvorsen reported 4/508 radiographs with clinically significant discordance and zero substantial changes in care.42

Several studies of radiography readings in Emergency Departments have addressed the issue of changes in care that result from the radiologists’ second reading. Lufkin reported a change in care of 11/9599 (0.11%) when ED physicians were “confident” in their readings of plain radiographs.70 Two large studies reported 0.5 -1.1% episodes of changes or potential changes in patient care for all radiographs that had discordant readings between the radiologist and the ED physician.56, 57

Similarly, studies of orthopedists’ initial radiograph reading and clinical management with a second reading by radiologists uniformly found no change in clinical care resulting from the second reading.71-74

Addressing meaningful patient oriented outcomes, a seven year prospective study was conducted with 869 cancer patients randomized into a double reading group (oncologist and radiologist) and single reading group (oncologist with radiologist consultation by request) for plain radiographs taken during cancer monitoring visits. Järvenpää reported no difference in the time of detection of cancer recurrence or 5 year survival.75 Radiology second reading was not requested for 55.6% of the 1927 plain radiographs in the single reading group.76

These studies suggest that a second reading by a radiologist is not always necessary and selective request for radiology consultation is appropriate in some clinical circumstances.
C. Radiology consultation for office radiography

Family physicians refer patients for specialty consultation for numerous reasons and are usually able to determine when such consultation is needed. The Halvorsen and Smith surveys found that 73 percent16 and 82 percent17 of the respondents thought that family physicians should be able to select which radiographs should be reread by a radiologist. Review of the literature does not support mandatory rereading of all radiographs taken in family physicians’ offices.
D. Shortage of radiologists

Several recent articles have discussed an increasing shortage of radiologists in the United States.77-79 A shortage of radiologists in the United Kingdom has resulted in the training of radiographers, called Clinical Specialist Radiographers, to interpret and report results of radiographs done in Accident and Emergency Departments. High rates of agreement with radiologists’ interpretations have been reported and radiographers are interpreting and reporting in over 30 sites.80, 81 Saketkhoo reports 25% of community hospital emergency departments have radiology staffing shortages and 62% cannot consult a radiologist about plain radiographs at night.82 A 2003 national survey of US radiologists suggests that the supply and demand is fairly balanced, but there remains a shortage of radiologists in non-metropolitan areas.83

Family physicians selectively requesting which radiographs to send for radiologist consultation and second reading would free up radiologists’ time for interpretation of more complex radiographs and radiological interventions.
E. Formal relationships with other organizations

Cooperation in the development of quality improvement programs, radiography use guidelines and CMS standards for office imaging should be encouraged between the AAFP, the American College of Radiology, Intersocietal Accreditation Commission and other relevant organizations.
F. Broader dissemination of ACR Appropriateness Criteria

Broader use of the ACR Appropriateness Criteria™ may have some beneficial impact by encouraging appropriate outpatient radiography use and discouraging unnecessary or inappropriate use. Studies of radiograph guideline dissemination methods have had mixed results31,84,85 and studies reporting a decrease in inappropriate radiographs have only had modest improvements.29,86,87
G. Research agenda

The research agenda for interpretation of outpatient radiographs should focus on the following:
  1. Quantifying the present radiographic practices for family physicians in the United States and how a shift in billing for the professional component of radiology services or CMS standards for performing and interpreting outpatient radiographs would impact the financial model of the future of family medicine model practice.
  2. Developing effective quality improvement programs that ensure acceptable image quality, improve interpretation error rates, assure patient safety and define which radiographs should be referred for radiologic consultation.
  3. Defining effective methods to encourage appropriate outpatient radiography use and discourage unnecessary or inappropriate use.
  4. Defining the effect on patient oriented clinical outcomes of performance and interpretation of diagnostic radiographs by family physicians on-site versus off-site, especially in rural and other underserved areas.

Section VII: References

  1. Centers for Medicare and Medicaid Services. Highlights -- Table 2: National Health Expenditures Aggregate Amounts and Average Annual Percent Change, by Type of Expenditure: Selected Calendar Years 1980-2003. Retrieved October 2005 from: www.cms.hhs.gov/statistics/nhe/historical/t2.asp
  2. Medicare Payment Policy Advisory Commission. Medicare Payment Policy, Report to Congress. March 2005. Accessed November 29, 2005 http://www.medpac.gov/publications/congressional_reports/Mar05_Ch03.pdf
  3. Maitino A, Levin D, Parker L, Rao V, Sunshine J. Nationwide trends in rates of utilization of noninvasive diagnostic imaging among the Medicare population 1993-1999. Radiology 2003;227:113-117.
  4. Maitino A, Levin D, Parker L, Rao V, Sunshine J. Practice patterns of radiologists and non-radiologists in utilization of noninvasive diagnostic imaging among the Medicare population 1993-1999. Radiology 2003;228:795-801.
  5. Sunshine JH, Bansal S, Evens RG. Radiology performed by nonradiologists in the United States: who does what? Am J Roentgenol 1993;161:419-429.
  6. Spettell CM, Levin DC, Rao VM, Sunshine JH, Bansal S. Practice patterns of radiologists and nonradiologists: national Medicare data on the performance of chest and skeletal radiography and abdominal and pelvic sonography. Am J Roentgenol 1998,171:3-5.
  7. Levin DC, Merrill C. The practice of radiology by nonradiologists: cost, quality and utilization issues. Am J Roentgenol 1994;162:513-518.
  8. Hillman BJ, Joseph CA, Mabry MR, Sunshine JH, Kennedy SD, Noether M. Frequency and costs of diagnostic imaging in office practice — a comparison of self-referring and radiologist-referring physicians. N Engl J Med 1990;323:1604-1608.
  9. Levin DC, Edmiston RB, Ricci JA, Beam LM, Rosetti GF, Harford RJ. Self-referral in private offices for imaging studies performed in Pennsylvania Blue Shield subscribers during 1991. Radiology 1993;189:371-375.
  10. Radecki SE, Steele JP. Effect of on-site facilities on use of diagnostic radiology by nonradiologists. Invest Radiol 1990;25:190-193.
  11. Kouri B, Parsons G, Alpert H. Physician self-referral for diagnostic imaging: review of the empiric literature. AJR 2002;179:843-850.
  12. Litt A, Ryan D, Batista D, Perry K, Lewis R, Sunshine J. Relative procedure intensity with self-referral and radiologist referral: extremity radiography. Radiology 2005;235:142-147.
  13. Hillman B, Olson GT, Colbert RW, Bernhardt LB. Response to a payment policy denying professional charges for diagnostic imaging by nonradiologist physicians. JAMA 1995:274;885-887.
  14. Durham J, McLeod D. Use of diagnostic imaging services in the central region by general practitioners. N Z Med J 1999;112:233-236.
  15. Wilson I, Dukes K, Greenfield S, Kaplan S, Hillman B. Patients’ role in the use of radiology testing for common office practice complaints. Arch Intern Med 2001;161:256-263.
  16. Halvorsen JG, Kunian A. Radiology in family practice: a prospective study of 14 community practices. Fam Med 1990;22:112-117.
  17. Smith PD. Office use of x-rays by family physicians in Wisconsin. Presentation at 1997 Wisconsin Primary Care Research Forum. Retrieved December 2005 from: http://www.wisconsinmedicalsociety.org/uploads/wmj/beasley.pdf
  18. Medicare Claims Processing Manual. Chapter 13 - Radiology Services and other Diagnostic Procedures. Section 20.1 Retrieved November 2005. http://www.cms.hhs.gov/manuals/104_claims/clm104c13.pdf
  19. Ordering Diagnostic Tests and Interpretation of Test Results. Medicare Part B policy manual, April 1999: Policy no. PHYS-002, page 3. Retrieved November 2005 from: http://www.wpsic.com/medicare/policies/wisconsin/phys002.pdf
  20. Policies of House of Delegates. D-385-974 Freedom of Practice in Medical Imaging. Res.228, A-05. Retrieved November 2005 from: http://www.ama-assn.org/apps/pf_new/pf_online?f_n=browse&doc=policyfiles/DIR/D-385.974.HTM&&s_t=&st_p=&nth=1&prev_pol=policyfiles/DIR/D-383.999.HTM&nxt_pol=policyfiles/DIR/D-385.974.HTM&
  21. American Academy of Orthopedic Surgeons, In-office diagnostic imaging studies by orthopaedic surgeons 2000: document no. 1132; Retrieved November 2005 from: http://www.aaos.org/wordhtml/papers/position/1132.htm
  22. American College of Emergency Physicians. Interpretation of diagnostic studies. Retrieved November 2005 from: http://www.acep.org/webportal/PracticeResources/PolicyStatements/PracticeManagement/ InterpretationofDiagnosticStudies.htm
  23. Stiell IG, Greenberg GH, McKnight RD, Nair RC, McDowell I, Worthington JR. A study to develop clinical decision rules for the use of radiography in acute ankle injuries. Ann Emerg Med 1992;21:384-390.
  24. Pigman EC, Klug RK, Sanford S, Jolly BT. Evaluation of the Ottawa clinical decision rules for the use of radiography in acute ankle and midfoot injuries in the emergency department: an independent site assessment. Ann Emerg Med 1994;24:41-45.
  25. Stiell IG, Greenberg GH, Wells GA, McKnight RD, Cwinn AA, Cacciotti T. Derivation of a decision rule for the use of radiography in acute knee injuries. Ann Emerg Med 1995;26:405-413.
  26. Stiell IG, Wells GA, Hoag RH, Sivilotti ML, Cacciotti TF, Verbeek PR, et al. Implementation of the Ottawa Knee Rule for the use of radiography in acute knee injuries. JAMA 1997; 278:2075-2079.
  27. Nugent P. Ottawa ankle rules accurately assess injuries and reduce reliance on radiographs. J Fam Pract 2004;53:785-788.
  28. Halvorsen JG, Swanson D. Indications for office radiographs. J Fam Pract 1990;31:521-529.
  29. Kerry S, Oakeshott P, Dundas D, Williams J. Influence of postal distribution of The Royal College of Radiologists’ guidelines, together with feedback on radiological referral rates, on x-ray referrals from general practice: a randomized controlled trial. Family Practice 2000;17;46-52.
  30. Little P, Cantrell T, Roberts L, Chapman J, Langridge J, Pickering R. Why do GPs perform investigations? The medical and social agendas in arranging back x-rays. Fam Pract 1998; 15:264-265.
  31. Freeborn DK, Shye D, Mullody JP, Eraker S, Romeo J. Primary care physicians’ use of lumbar spine imaging tests: effects of guidelines and practice pattern feedback. J Gen Intern Med 1997;12:619-625.
  32. Moskowitz H, Sunshine J, Grossman D, Adams L, Gelinas L. The effect of imaging guidelines on the number and quality of outpatient radiographic examinations. AJR 2000;175:9-15.
  33. Verrilli D, Bloch S, Rousseau J, Crozier M, Yecies S. Design of a privileging program for diagnostic imaging: costs and implications for a large insurer in Massachusetts. Radiology 1998;208:385-392.
  34. Mendelson E. The development and meaning of appropriateness guidelines. Radiologic Clinics of North America 1995;33:1081-1084.
  35. American College of Radiology. Appropriateness Criteria ™, ACR Website edition, Copyright © 2004 by the American College of Radiology. Retrieved November 2005 from: http://www.acr.org/s_acr/sec.asp?CID=1845&DID=16050
  36. Waldrip, C. American College of Radiology, Department of Quality and Safety, Program Specialist. Personal communication November 2, 2005.
  37. Accreditation Council for Graduate Medical Education. Program requirements for graduate medical education in family medicine. July 1, 2006 requirements. Retrieved October 2005 from http://www.acgme.org/acWebsite/downloads/RRC_progReq/120pr706.pdf
  38. Accreditation Council for Graduate Medical Education. Program requirements for graduate medical education in family medicine. July 1, 2001 requirements. Retrieved October 2005 from http://www.acgme.org/acWebsite/downloads/RRC_progReq/120pr701.pdf
  39. Puffer, James C. President and Chief Executive Officer, American Board of Family Medicine, personal communication, November 2005.
  40. National Committee for Quality Assurance. 2004/2005 Standards. Retrieved November 2005 from: http://hprc.ncqa.org/cvoResult.asp
  41. Alcenius, Michael. Associate Director, Standards Interpretation/Accreditation Operations. Joint Commission on Accreditation of Healthcare Organizations, personal communication, November 2005.
  42. Halvorsen JG, Kunian A, Gjerdingen D, Connolly J, Koopmeiners M, Cesnik J. The interpretation of office radiographs by family physicians. J Fam Prac, 1989;28(4):426-432.
  43. Halvorsen JG, Kunian A. Radiology in family practice: a prospective study of 14 community practices. Fam Med 1990;22(2):112-117.
  44. Bergus GR, Franken EA Jr, Koch TJ, Smith WL, Evans ER, Berbaum KS. Radiologic interpretation by family physicians in an office practice setting. J Fam Pract 1995:41(14):352-356.
  45. Franken EA Jr, Bergus GR, Koch TJ, Berbaum KS, Smith WL. Added value of radiologist consultation to family practitioners in the outpatient setting. Radiology 1995;197:759-762.
  46. Smith P, Temte J, Beasley J, Mundt M. Radiographs in the Office: is a second reading always needed? J Am Board Fam Prac 2004;17:256-263.
  47. Knollmann BC, Corson AP, Twigg HL, Schulman KA. Assessment of joint review of radio-logic studies by a primary care physician and a radiologist. J Gen Intern Med 1996;11:608-612.
  48. Hopper KD, Rosetti GF, Edmiston RB, Madewell JE, Beam LM, Landis JR. Diagnositic radiology peer review: a method inclusive of all interpreters of radiographic examinations regardless of specialty. Radiology 1991;180:557-561.
  49. Strasser RP, Bass MJ, Brennan M. The effect of an on-site radiology facility on radiologic utilization in family practice. J Fam Pract 1987:24(6):619-623.
  50. Kruitzky L, Haddy RI, Curry RW Sr. Interpretation of chest roentgenograms by primary care physicians. South Med 1987;80(11):1347-1351.
  51. Mucci B. The selective reporting of x-ray films from the accident and emergency department. Injury 1983;14(4):343-344.
  52. McLain PL, Kirkwood CR. The quality of emergency room radiograph interpretations. J Fam Pract 1985;20(5):443-448.
  53. Warren JS, Lara K, Connor PD, Cantrell J, Hahn RG. Correlation of emergency department radiographs: results of a quality assurance review in an urban community hospital setting. J Am Board Fam Pract 1993;6(3): 255-259.
  54. Preston CA, Marr JJ 3d, Amaraneni KK, Suthar BS. Reduction of “callbacks” to the ED due to discrepancies in plain radiograph interpretation. Am J Emerg Med 1998;16:160-162.
  55. O’Leary MR, Smith MS, O’Leary DS, Olmsted WW, Curtis DJ, Groleau G. Application of clinical indicators in the emergency department. JAMA 1989;262:3444-3447.
  56. Brunswick J, Ilkhanipour K, Seaberg D, McGill L. Radiographic interpretation in the emergency department. American Journal of Emergency Medicine 1996;14:346-348.
  57. Klein E, Koenig M, Diekema D, Winters W. Discordant radiograph interpretation between emergency physicians and radiologists in a pediatric emergency department. Pediatr Emerg Care 1999;15:245-248.
  58. Herman PG, Gerson DE, Hessel SJ, Mayer BS, Watnick M, Blesser B. Disagreements in chest roentgen interpretation. Chest 1975;68:278-282.
  59. Tuddenham WJ. Visual search, imaging organization and reader error in roentgen diagnosis. Radiology 1962;78:694-704.
  60. Herman PG, Hessel SJ. Accuracy and its relationship to experience in interpretation of chest radiographs. Invest Radio 1975;10:62-67.
  61. Revesz G, Kundel HL. Psychophysical studies of detection errors in chest radiology. Radiology 1977;123:559-562.
  62. Hessel SJ, Herman PG, Swensson RG. Improving performance by multiple interpretations of chest radiographs: effectiveness and cost. Radiology 1978;127:589-594.
  63. Swensson RG, Hessel SJ, Herman PG. Omissions in radiology: faulty search or stringent reporting criteria? Radiology 1977;123:563-567.
  64. Yerushalmy J. The statistical assessment of the variability in observer perception and description of roentgenographic pulmonary shadows. Radio Clin North AM 1969;7(3):381-392.
  65. Potchen E, Cooper T, Sierra A, Aben G, Potchen M, Potter M, Siebert J. Measuring Performance in Chest Radiography. Radiology 2000;217:456-459.
  66. Hillman BJ, Swensson RG, Hessel SJ, Gerson DE, Herman PG. The value of consultation among radiologists. AM J Roentgenol 1976;127(5):807-809.
  67. Rhea JT, Potsaid MS, DeLuca SA. Errors of interpretation as elicited by a quality audit of an emergency radiology facility. Radiology 1979;132:277-280.
  68. Siegle R, Baram E, Reuter S, Clarke E, Lancaster J, McMahan C. Rates of disagreement in imaging interpretation in a group of community hospitals. Acad Radiol 1998;5:148-154.
  69. Loy C, Irwig L. Accuracy of diagnostic tests read with and without clinical information: a systematic review. JAMA 2004;292:1602-1609.
  70. Lufkin K, Smith S, Matticks C, Brunnette D. Radiologists’ review of radiographs interpreted confidently by emergency physicians infrequently leads to changes in patient management. Ann Emerg Med 1998;31:202-207.
  71. Bosse M, Brumback R, Hash C. Medical cost containment: analysis of dual orthopedic/radiology interpretation of x-rays in the trauma patient. The Journal of Trauma; Injury, Infection and Critical Care 1995;38:220-222.
  72. Clark R, Anderson m, Johnson B, Moore D, Herbert F. Clinicial value of radiologist’ interpretation of perioperative radiographs of orthopedic patients. Orthopedics 1996;19:1003-1007.
  73. Anglen J, Marberry K, Gehrke J. The clinical utility of duplicate readings for musculoskeletal radiographs. Orthopedics 1997;20:1015-1019.
  74. Parmar V, Stanitski D, Stanitski C. Interpretation of radiographs in a pediatric limb deformity practice; do radiologists contribute? J Pediatr Orthop 1999;19:732-736.
  75. Järvenpää R, Holli K, Hakama M. Double-reading of plain radiographs - no benefit with regard to earliness of diagnosis of cancer recurrence: a randomized follow-up study. Eur J Cancer 2004;40:1668-1673.
  76. Järvenpää R, Holli K, Hakama M. Resource savings in the single reading of plain radiographs by oncologist only in cancer patient follow-up: A randomized study. Acta Oncologica 2005;44:149-154.
  77. Sunshine JH, Cypel YS, Schepps B. Diagnostic radiologists in 2000: basic characteristics, practices, and issues related to the radiologist shortage. AJR 2002;178(2):291-301.
  78. Hawkins J. Addressing the shortage of radiologists. Radiology Management 2001;23(4):26-28.
  79. Bhargavan M, Sunshine J, Schepps B. Too few radiologists? AJR 2002;178:1075-1082.
  80. Brealey S, King D, Crowe M, Crawshaw I, Ford L, Warnock N, Mannion R, Ethell S. Accident and emergency and general practitioner plain radiograph reporting by radiographers and radiologists; a quasi-randomized controlled trial. The British Journal of Radiology 2003;76:57-61.
  81. Brealey S, King D, Hahn S, Crowe M, Williams P, Rutter P, Crane S. Radiographers and radiologists reporting plain radiograph requests from accident and emergency and general practice. Clinical Radiology 2005;60:710-717.
  82. Saketkhoo D, Bhargavan M, Sunshine J, Forman H. Emergency department image interpretation services at private community hospitals. Radiology 2004;231:190-197.
  83. Meghea C, Sunshine J. Who’s overworked and who’s underworked among radiologists? An update on the radiologist shortage. Radiology 2005;236:932-938.
  84. Matowe L, Ramsey C, Grimshaw M, Gilbert F, MacLeod M, Needham G. Effects of mailed dissemination of the royal college of radiologists’ guidelines on general practitioner referrals for radiography: a time series analysis. Clinical Radiology 2002;57:575-578.
  85. Cameron C, Naylor C. No impact from active dissemination of the Ottawa ankle rules: further evidence of the need for local implementation of practice guidelines. CMAJ 1999;160:1165-1168.
  86. Oakeshott P, Kerry S, Williams J. Randomized controlled trial of the effect of The Royal College of Radiologist’ guidelines on general practitioners’ referrals for radiographic examination. British Journal of General Practice 1994;44:197-200.
  87. Eccles M, Steen N, Grimshaw J, Thomas L, McNamee P, soutter J, Wilsdon J, Matowe L, Needham G, Gilbert F, Bond S. Effect of audit and feedback, and reminder messages on primary-care radiology referrals: a randomized trial. Lancet 2001;357:1406-1409.
  88. Kerry S, Oakeshott P, Dundas D, Williams J. Influence of postal distribution of The Royal College of Radiologists’ guidelines, together with feedback on radiological referral rates, on x-ray referrals from general practice: a randomized controlled trial. Family Practice 2000;17:46-52.
(B1999) (2006)