The suggested eight-step sequence described here is intended to offer guidance to programs conducting their first self-study.
The self-study is an objective, comprehensive evaluation of the residency or fellowship program, with the aim of improving it. Underlying the self-study is a longitudinal evaluation of the program and its learning environment, facilitated through sequential annual program evaluations that focus on the required components, with an emphasis on program strengths and “self-identified” areas for improvement (“self-identified” is used to distinguish this dimension of the self-study from areas for improvement the Review Committee identifies during accreditation reviews).
To offer context for the self-study, there are two new concepts: 1) an exploration of program aims; and 2) an assessment of the program’s institutional, local and, as applicable, regional environment. Both are discussed in detail below. The focus on aims and the program’s environmental context is to enhance the relevance and usefulness of the program evaluation, and support improvement that goes beyond compliance with the requirements.
Assemble the self-study group
Membership: The members of the Program Evaluation Committee (PEC) are the ideal core foundation for the self-study group. They are familiar with APE Template, the process of completing the APE, and the resulting action plans and improvement efforts. Including the Program Coordinator is also recommended.
Added participants: The ACGME does not require additional participants in the self-study. However, it may be beneficial to have a few individuals to offer an external perspective. Examples include a departmental leader, a clerkship director, a chief resident, or an expert in education, curriculum design, or assessment. These individuals may be included if program leaders think their contributions would be beneficial. For institutional experts, the DIO may be able to provide suggestions.
CCC representative: It may be beneficial to include a member of the Clinical Competency Committee (CCC) on the self-study group, due to the focus on educational outcomes, which makes data on residents’/fellows’ Milestone performance an important component in self-study discussions.
Engage program leaders and constituents in a discussion of program aims
The first task of the self-study group is a discussion of program aims. Aims are program and institutional leaders’ views of key expectations for the program, and how it differentiates itself from other programs in the same specialty/subspecialty. Aims may focus on the types of trainees recruited into the program, training for particular careers (clinical practice, academics, research, primary/generalist care), and other objectives, such as care for underserved patients, health policy or advocacy, population health, or generating new knowledge. Aims may also include other objectives, such as care for underserved patients, health policy or advocacy, population health, or generating new knowledge.
Review this brief slide presentation on how to set and validate program aims.
Program aims should generally be vetted with program and institutional leadership, and in some institutions, setting aims will be an institution-level initiative. In setting aims, programs should generally take a longer-term strategic view. However, aims may change over time. Factors such as a shift in program focus initiated by institutional or department leadership, changes in local or national demand for a resident workforce with certain capabilities, or new opportunities to train residents and fellows in a different setting may prompt revision of program aims.
Examine opportunities and threats
The next step is to conduct an assessment of the program’s environment. The rationale for examining opportunities for and threats facing the program is to provide context for the self-study.
Opportunities: Opportunities are external factors that are not entirely under the control of the program, but if acted upon, will help the program flourish. Opportunities take many forms, such as access to expanded populations for ambulatory care at a local health center, partnering with an institution with a simulation center, other collaborations, or availability of new clinical or educational technology through agreements with external parties.
Threats: Threats also are largely beyond the control of the program, and like opportunities, come in many forms. They could result from a change in support for education at the national level, changing priorities at the institutional or state levels, or from local factors, such as erosion of a primary ambulatory system based on voluntary faculty. The benefit of assessing program threats is that plans can be developed to mitigate their effect.
Aggregate and analyze data to generate a longitudinal assessment of the program’s improvement
A key component to the self-study analysis is information from successive Annual Program Evaluations, beginning, at minimum, with the Annual Program Evaluation conducted the year the program transitioned into the Next Accreditation System. While there are no explicit requirements for how information from Annual Program Evaluations should be aggregated, the ACGME offers two suggested templates for use in this process:
- Template for reporting key information from a single annual program evaluation
- Template for aggregating action items and key information from successive annual program evaluations
For the data from the Annual Program Evaluation, the focus is on program strengths and self-identified areas for improvement; how improvements are prioritized, selected, and implemented; and follow-up to assess whether interventions are effective. Over time, this process should focus on improvement that goes beyond compliance with requirements, with particular attention to improvements relevant to the program’s aims and environmental context.
Additional data for the self-study should relate to ongoing improvement activities and the perspectives of program stakeholders, such as results of the annual ACGME Resident and Faculty Surveys, and other relevant departmental or institutional data.
Obtain stakeholder input
The data should be confirmed and augmented by information from program stakeholders (residents/fellows, faculty members, others as relevant). For some programs, important information may include the perceptions of representatives from other specialties who interact with the residents or fellows. To collect this information, the program may use surveys, meetings with residents/fellows, or a retreat. Feedback from recent graduates could also provide useful data on the program’s educational effectiveness.
Interpret the data and aggregate the self-study findings
The next step is to interpret the aggregated data from the self-study. Specific elements of the self-study findings will include:
- establishing a working set of program aims;
- listing key program strengths;
- prioritizing among the self-identified areas for improvement to select those for active follow-up, and define the specific improvement activities;
- discussing opportunities that may enhance the program, and develop plans to take advantage of them; and,
- discussing threats identified in the self-study, and develop plans to mitigate their impact.
Discuss the findings with stakeholders
The self-study findings from Step 6 (above) should be shared with faculty members and residents/fellows. This step should validate the findings and improvement priorities chosen by the self-study group with these key stakeholders.
Develop a succinct self-study document for use in further program improvement and as documentation for the program’s 10-year site visit
The final step is for the self-study group or an individual designated by the group to complete a succinct self-study document that describes the process and key findings in the areas of program aims, the environmental assessment, and program strengths and areas for improvement.
The ACGME’s self-study summary template (see below) does not include information on program strengths and areas for improvement. The rationale for this is to allow programs to conduct a frank assessment. Programs may opt to complete the summary template, and have a separate brief companion document that lists the strengths and areas of improvement that were identified through the self-study.
At the time of the 10-year site visit (12 to 18 months later), the program will be asked to provide a brief written update describing improvements resulting from the self-study. No information will be requested on areas identified during the self-study that have not yet resulted in improvements.
Conducting the self-study for a dependent sub-specialty program
The ACGME has placed added responsibility for oversight of sub-specialty programs on the core program and sponsoring institution.
The self-study group for the core program should try to coordinate activities with the self-study groups for any dependent sub-specialty programs, to take advantage of common dimensions, explore potential synergies, and reduce the burden that may be associated with conducting an independent self-assessment.
The 10-year site visits for sub-specialty programs will be coordinated with the visit of their respective core program.
A test of a voluntary self-study pilot visit
The ACGME initiated a pilot study for any program in the seven Phase I specialties (Diagnostic Radiology, Emergency Medicine, Internal Medicine, Pediatrics, Neurological Surgery, Orthopaedic Surgery, and Urology) with its initial 10-year site visit scheduled between April 2015 and January 2017 (note that the interval for the pilot was recently extended from the original end date of July 2016). The objective is to assess if an added site visit to review the self-study will accelerate program improvement. In this pilot, following the program’s self-study, a special (non-accreditation) visit for guided discussion and feedback would be conducted by a group of field representatives with added training in self-study review. Programs volunteer to be part of this pilot.
The self-study pilot visit will not be an accreditation visit, and data shared by programs will not be used to assess compliance with requirements. Twelve to 18 months after the self-study pilot visit, the program would have its scheduled 10-year site visit. Additional detailed information about the self-study pilot visit is provided in a memorandum by ACGME Chief Executive Officer Thomas J. Nasca, MD, MACP.
Click here to review the memorandum
Review Committee evaluation of programs with early 10-year site visits
For a program with an early 10-year site visit, the Review Committee’s evaluation of the self-study will not have any accreditation impact. The ACGME will conduct an extensive program evaluation of these early 10-year site visits, focusing in particular on collecting best practice information for conducting the self-study. This information will be disseminated via the ACGME website and the Journal of Graduate Medical Education.