Letter sent to ACTPCMD re: draft report on Section 747 programs

Jerilyn Glass, MD, PhD
Deputy Executive Secretary
ACTPCMD
Health Resources and Services Administration
Room 9A-27
5600 Fishers Lane
Rockville, MD 20857

Re: Draft Report on Evaluating the Impact of the Title VII, Section 747 Programs

Dear Dr. Glass:

On behalf of the undersigned organizations, representing over 95,000 practicing family physicians and family medicine educators, we are writing to express our support for the draft 5th report of the Advisory Committee on Training in Primary Care Medicine and Dentistry. This report, titled Evaluating the Impact of the Title VII, Section 747 programs is well-crafted and provides a very substantive basis for a renewed look at the effectiveness of these programs by Congressional and Administration policymakers.

This report has many stellar qualities. First and foremost, it stresses the importance of a primary care workforce. It explains why evaluation of these programs has been a problem. It states clearly that evaluations have found a beneficial effect of these programs and it addresses the prominent poor evaluation of the programs conducted by the Office of Management and Budget (OMB), by showing why it is flawed.

The report emphasizes, and re-emphasizes as necessary, the statutory intent of Title VII, Section 747, both historically and in the present. The recommendations, seven key objectives, and outcome measures, all flow from that explicitly stated purpose of the program.

The report addresses the need for data collection, the cost of such collection and analysis, the need for additional funds to cover collection and analysis costs, and the requirement that such data collection should not be a burden to the grantees. This is the first time, to our knowledge, that a public policy making body has addressed the need for HRSA to take on this role, and the report explicitly states the case for doing so.

The Logic Model seems to provide a logically coherent starting point for necessary analyses of these programs. Instead of superimposing outcome measures by which the programs will be judged after the fact, it offers unbiased measures that will be known prior to the grant-making, that can be clearly identified, and of which there is consensus around both their appropriateness and meaningfulness. As the report states, (page 22),”The choice of outcome measures used must be scientifically
sound and programmatically relevant.” With the references included in the report to justify each proposed outcome measure, the report makes the likelihood of that occurring extremely high.

With the clarity of description of the programs, the evaluation framework of Diversity, Primary Care, Distribution, Quality and Infrastructure makes infinite sense, and really for the first time sets up a framework upon which consensus regarding evaluation can be achieved. The listing of performance measures by these categories makes everything logically consistent and rational.

All of these and many more make this an outstanding draft report. Below we have included some suggested edits to strengthen the report even more. The additions we suggest are consistent with the basis of the report and are more to add some consistency across areas, and they are supported by the report as currently constituted.

In the Executive Summary, (page 5 of the draft, second paragraph), a sentence states, “In successive reauthorizations, the focus of the Program shifted to education and training of primary care providers, …..” We recommend the addition of the word “production” in the sentence to read, “In successive reauthorizations, the focus of the Program shifted to production, education, and training of primary care providers, …..”

This is in keeping with the explicit purpose of the program as outlined in the report, and the historical changes in legislation. The reauthorization of 1992 (Health Professions Education Extension Amendments of 1992 (PL 102-408)), and the succeeding one in 1998 (Health Professions Education Partnerships Act of 1998, (PL 105-392)), each contain a preference for establishing an academic administrative unit for programs in family medicine or substantially expanding them. The 1998 statute expands this to include internal medicine and pediatrics. So we recommend that the broad issue of capacity that is in the statement of purpose and key objectives should also be included in the language of the executive summary, in the overview included under Background (1.1.),(second paragraph, page 12), and in the abstract (second paragraph, page 3).

In a similar vein, in paragraph two of the abstract (page 3), the word “and” should be added to the last sentence which states “…. to education and training of primary care providers and to serve medically and dentally underserved populations and high-risk groups.” Again, the issue of capacity as well as distribution should be included.

Another issue we suggest be included in the report, is the addition of the words “and maintained” in the paragraph beginning “Evaluation of the Title VII, Section 747 programs that have examined specific..” The second sentence describes the programs as having helped establish family medicine departments. The sentence should read “established and maintained family medicine departments…” A similar change should be made in the Executive Summary, (page 5) so it would read “establishment and maintenance of family medicine departments…”

The justification for the above changes is included in two General Accounting Office (GAO, now, Government Accountability Office) reports, which have shown that this money works. The GAO, in two reports in 1994, addressed the question of how do we know Title VII money is well spent? While the reports agreed with this committee’s belief that changing priorities and objectives have made analysis of the programs difficult, the reports were able to make some positive correlations with programs under Section 747. For example, a July 1994 report, states that "the programs were important for funding innovative projects and providing 'seed money' for starting new programs. For example, Title VII was considered important in the creation and maintenance [emphasis added] of family medicine departments and divisions in medical schools."

In another report, the GAO states, in October 1994, that "students who attended schools with family practice departments were 57 percent more likely to pursue primary care." In addition, the report goes on to say that "students attending medical schools with more highly funded family practice departments were 18 percent more likely to pursue primary care and students attending schools requiring a third-year family practice clerkship were [also] 18 percent more likely to pursue primary care." The money spent on Section 747 of Title VII is directly targeted in these areas.

We also recommend that these two GAO studies be included in the section entitled “History of Evaluation of Title VII, Section 747 Programs”,(Section 1.3.1, page 19).

Our last major recommendation relates to the table of Recommended Outcome Measures to be found on page 9 of the Executive Summary and page 34 of the actual report. It is also found in prose on page 6 of the Executive Summary and page 14 of the full report. Objective number 2 has a dependent clause at the end that states “with special emphasis on individuals from disadvantaged backgrounds and underrepresented minorities.” The first part of the sentence deals with improving capacity, and the near- and longer-term measures match that first part and do not address the dependent clause. Objective number 7 speaks directly to the issue of individuals with disadvantaged backgrounds and underrepresented minorities, and the measures contained there match that objective. We believe the dependent clause in key objective 2 should be dropped as redundant, and without accompanying outcome measures.

An additional recommendation related to this table is the need for clarity of the use of the term “program” or “programs.” While in many cases in the text of the full report it is clear the word programs relates to the Title VII, section 747 programs in toto, in the chart it becomes easy to misunderstand the use of the word programs to mean residency programs. Since section 747 funds departments as well as programs, it would help to clarify its use in the table.

On the first page of the full report ( Section 1.1, page 12, fifth bullet) the statement is made that “This act substantially shifted the focus for Title VII, section 747 to providing for MUCs and targeting primary care providers to fill this need.” One normally thinks of the term “shifts” as movement away from one area to another. In this case, we believe it is more accurate to use the wording “added an additional focus” rather than shifted focus. There are several foci for this program, and two statutory preferences – one for establishment, and one for underserved. We believe the sentence should read, “This act added an additional focus for Title VII, section 747 to providing for MUCs and targeting primary care providers to fill this need.”

We have one additional content area that we believe needs to be included to make explicit that education of the primary care workforce must include preparation for the future practice of primary care and not for the practice of yesterday. While the term “research” is contained in the report, specifically in the recommendations and measures for #4 (faculty development), it is absent from the non-academic areas. We recommend adding language in the report relating to the training of primary care physicians to manage the future of primary care practice. The report should make explicit that education in the methods of quality improvement could be the entry toward the development and deployment nation-wide of primary care clinicians capable of measuring their practices and testing innovations. This would open up curriculum development and related projects addressing areas such as patient safety, health behavior counseling, etc. to determine if they work. The idea would be to train primary care clinicians in research methods sufficient to measure their practices. Language regarding “training providers in the investigational ("research") methods to evaluate and improve their practices” could be added at the end of statement #9 under Objective 4 on page 50. The last sentence could then say "Examples of this measure include number of programs with grantees trained in investigational methods to evaluate and improve their practices to better address/respond to emerging health care needs."

We found one minor point in the abstract that is more grammatical than anything else. The third paragraph states that “Title VII, section 747 programs educate and train high quality primary care providers who go on to join the NHSC….” We believe it should read “Title VII, section 747 programs educate and train high quality primary care providers, many of whom go on to join the NHSC….”

We are very pleased to have the opportunity to comment on the draft report. The work the Committee has done in developing coherent descriptions, analyses, and recommendations regarding the current state of evaluation of Title VII, section 747 programs, and describing the needs of the future, is monumental. The document is extremely well written, clear, and without unnecessary text. We believe it will be a strong foundation for future work in evaluation of the Title VII, section 747 programs. We certainly hope Congress and OMB take notice.

Thank you again for the opportunity to be involved in the work of this important committee.

Sincerely,


_________________________________
William K. Mygdal, EdD, President
Society of Teachers of Family Medicine

_________________________________
Warren Newton, MD, President
Association of Departments of Family Medicine

_________________________________
Mary Frank, MD, President
American Academy of Family Physicians

_________________________________
Penny Tenzer, MD, President
Association of Family Medicine Residency Directors

_________________________________
Moira Stewart, PhD, President
North American Primary Care Research Group