V. Evaluation
- Chris Callac
V.A. Resident Evaluation
V.A.1. Feedback and Evaluation
Feedback is ongoing information provided regarding aspects of one's performance, knowledge, or understanding. The faculty empower residents to provide much of that feedback themselves in a spirit of continuous learning and self-reflection. Feedback from faculty members in the context of routine clinical care should be frequent, and need not always be formally documented.
Formative and summative evaluation have distinct definitions. Formative evaluation is monitoring resident learning and providing ongoing feedback that can be used by residents to improve their learning in the context of provision of patient care or other educational opportunities. More specifically, formative evaluations help:
- residents identify their strengths and weaknesses and target areas that need work
- program directors and faculty members recognize where residents are struggling and address problems immediately
Summative evaluation is evaluating a resident's learning by comparing the residents against the goals and objectives of the rotation and program, respectively. Summative evaluation is utilized to make decisions about promotion to the next level of training, or program completion.
End-of-rotation and end-of-year evaluations have both summative and formative components. Information from a summative evaluation can be used formatively when residents or faculty members use it to guide their efforts and activities in subsequent rotations and to successfully complete the residency program.
Feedback, formative evaluation, and summative evaluation compare intentions with accomplishments, enabling the transformation of a neophyte physician to one with growing expertise.
V.A.1.a) Faculty members must directly observe, evaluate, and frequently provide feedback on resident performance during each rotation or similar educational assignment. (Core)
V.A.1.b) Evaluation must be documented at the completion of the assignment. (Core)
V.A.1.b).(1) For block rotations of greater than three months in duration, evaluation must be documented at least every three months. (Core)
V.A.1.b).(2) Longitudinal experiences, such as continuity clinic in the context of other clinical responsibilities, must be evaluated at least every three months and at completion. (Core)
V.A.1.c) The program must provide an objective performance evaluation based on the Competencies and the specialty-specific Milestones, and must: (Core)
V.A.1.c).(1) use multiple evaluators (e.g., faculty members, peers, patients, self, and other professional staff members); and, (Core)
V.A.1.c).(2) provide that information to the Clinical Competency Committee for its synthesis of progressive resident performance and improvement toward unsupervised practice. (Core)
V.A.1.d) The program director or their designee, with input from the Clinical Competency Committee, must:
V.A.1.d).(1) meet with and review with each resident their documented semi-annual evaluation of performance, including progress along the specialty-specific Milestones; (Core)
V.A.1.d).(2) assist residents in developing individualized learning plans to capitalize on their strengths and identify areas for growth; and, (Core)
V.A.1.d).(3) develop plans for residents failing to progress, following institutional policies and procedures. (Core)
Learning is an active process that requires effort from the teacher and the learner. Faculty members evaluate a resident's performance at least at the end of each rotation. The program director or their designee will review those evaluations, including their progress on the Milestones, at a minimum of every six months. Residents should be encouraged to reflect upon the evaluation, using the information to reinforce well-performed tasks or knowledge or to modify deficiencies in knowledge or practice. Working together with the faculty members, residents should develop an individualized learning plan.
Residents who are experiencing difficulties with achieving progress along the Milestones may require intervention to address specific deficiencies. Such intervention, documented in an individual remediation plan developed by the program director or a faculty mentor and the resident, will take a variety of forms based on the specific learning needs of the resident. However, the ACGME recognizes that there are situations which require more significant intervention that may alter the time course of resident progression. To ensure due process, it is essential that the program director follow institutional policies and procedures.
V.A.1.e) At least annually, there must be a summative evaluation of each resident that includes their readiness to progress to the next year of the program, if applicable. (Core)
V.A.1.f) The evaluations of a resident's performance must be accessible for review by the resident. (Core)
V.A.2 Final Evaluation
V.A.2.a) The program director must provide a final evaluation for each resident upon completion of the program. (Core)
V.A.2.a).(1) The specialty-specific Milestones, and when applicable the specialty-specific Case Logs, must be used as tools to ensure residents are able to engage in autonomous practice upon completion of the program. (Core)
V.A.2.a).(2) The final evaluation must:
V.A.2.a).(2).(a) become part of the resident's permanent record maintained by the institution, and must be accessible for review by the resident in accordance with institutional policy; (Core)
V.A.2.a).(2).(b) verify that the resident has demonstrated the knowledge, skills, and behaviors necessary to enter autonomous practice; (Core)
V.A.2.a).(2).(c) consider recommendations from the Clinical Competency Committee; and, (Core)
V.A.2.a).(2).(d) be shared with the resident upon completion of the program. (Core)
V.A.3. A Clinical Competency Committee must be appointed by the program director. (Core)
V.A.3.a) At a minimum, the Clinical Competency Committee must include three members of the program faculty, at least one of whom is a core faculty member. (Core)
V.A.3.a).(1). Additional members must be faculty members from the same program or other programs, or other health professionals who have extensive contact and experience with the program's residents. (Core)
The requirements regarding the Clinical Competency Committee do not preclude or limit a program director's participation on the Clinical Competency Committee. The intent is to leave flexibility for each program to decide the best structure for its own circumstances, but a program should consider: its program director's presence on the other Clinical Competency Committee members' discussions and decisions; the size of the program faculty; and other program-relevant factors. The program director has final responsibility for resident evaluation and promotion decisions.
Program faculty may include more than the physician faculty members, such as other physicians and non-physicians who teach and evaluate the program's residents. There may be additional members of the Clinical Competency Committee. Chief residents who have completed core residency programs in their specialty may be members of the Clinical Competency Committee.
V.A.3.b) The Clinical Competency Committee must:
V.A.3.b).(1) review all resident evaluations at least semi-annually; (Core)
V.A.3.b).(2) determine each resident's progress on achievement of the specialty-specific Milestones; and, (Core)
V.A.3.b).(3) meet prior to the residents' semi-annual evaluations and advise the program director regarding each resident's progress. (Core)
V.B. Faculty Evaluation
V.B.1. The program must have a process to evaluate each faculty member's performance as it relates to the educational program at least annually. (Core)
V.B.1.a) This evaluation must include a review of the faculty member's (Core)
The program director is responsible for the education program and for whom delivers it. While the term "faculty" may be applied to physicians within a given institution for other reasons, it is applied to residency program faculty members only through approval by a program director. The development of the faculty members improves the education, clinical, and research aspects of a program. Faculty members have a strong commitment to the resident and desire to provide optimal education and work opportunities. Faculty members must be provided feedback on their contribution to the mission of the program. All faculty members who interact with residents desire feedback on their education, clinical care, and research. If a faculty member does not interact with residents, feedback is not required. With regard to the diverse operating environments and configurations, the residency program director may need to work with others to determine the effectiveness of the program's faculty performance with regard to their role in the educational program. All teaching faculty members should have their educational efforts evaluated by the residents in a confidential and anonymous manner. Other aspects for the feedback may include research or clinical productivity, review of patient outcomes, or peer review of scholarly activity. The process should reflect the local environment and identify the necessary information. The feedback from the various sources should be summarized and provided to the faculty on an annual basis by a member of the leadership team of the program.
V.B.1.b) This evaluation must include written, anonymous, and confidential evaluations by the residents. (Core)
V.B.2. Faculty members must receive feedback on their evaluations at least annually. (Core)
V.B.3. Results of the faculty educational evaluations should be incorporated into program-wide faculty development plans. (Core)
The quality of the faculty's teaching and clinical care is a determinant of the quality of the program and the quality of the residents' future clinical care. Therefore, the program has the responsibility to evaluate and improve the program faculty members' teaching, scholarship, professionalism, and quality care. This section mandates annual review of the program's faculty members for this purpose, and can be used as input into the Annual Program Evaluation.
V.C. Program Evaluation and Improvement
V.C.1. The program director must appoint the Program Evaluation Committee to conduct and document the Annual Program Evaluation as part of the program's continuous improvement process. (Core)
V.C.1.a) The Program Evaluation Committee must be composed of at least two program faculty members, at least one of whom is a core faculty member, and at least one resident. (Core)
V.C.1.b) Program Evaluation Committee responsibilities must include:
V.C.1.b).(1) acting as an advisor to the program director, through program oversight; (Core)
V.C.1.b).(2) review of the program's self-determined goals and progress toward meeting them; (Core)
V.C.1.b).(3) guiding ongoing program improvement, including development of new goals, based upon outcomes; and, (Core)
V.C.1.b).(4) review of the current operating environment to identify strengths, challenges, opportunities, and threats as related to the program's mission and aims. (Core)
V.C.1.c) The Program Evaluation Committee should consider the following elements in its assessment of the program:
V.C.1.c).(1) curriculum; (Core)
V.C.1.c).(2) outcomes from prior Annual Program Evaluation(s); (Core)
V.C.1.c).(3) ACGME letters of notification, including citations, Areas for Improvement, and comments; (Core)
V.C.1.c).(4) quality and safety of patient care; (Core)
In order to achieve its mission and train quality physicians, a program must evaluate its performance and plan for improvement in the Annual Program Evaluation. Performance of residents and faculty members is a reflection of program quality and can use metrics that reflect the goals that a program has set for itself. The Program Evaluation Committee utilizes outcome parameters and other data to assess the program's progress toward achievement of its goals that a program has set for itself. The Program Evaluation Committee utilizes outcome parameters and other data to assess the program's progress toward achievement of its goals and aims.
V.C.1.c).(5) aggregate resident and faculty:
V.C.1.c).(5).(a) well-being; (Core)
V.C.1.c).(5).(b) recruitment and retention; (Core)
V.C.1.c).(5).(c) workforce diversity; (Core)
V.C.1.c).(5).(d) engagement in quality improvement and patient safety; (Core)
V.C.1.c).(5).(e) scholarly activity; (Core)
V.C.1.c).(5).(f) ACGME Resident and Faculty Surveys; and, (Core)
V.C.1.c).(6) aggregate resident:
V.C.1.c).(6).(a) achievement of the Milestones; (Core)
V.C.1.c).(6).(b) in-training examinations (where applicable); (Core)
V.C.1.c).(6).(c) board pass and certification rates; and, (Core)
V.C.1.c).(6).(d) graduate performance. (Core)
V.C.1.c).(7) aggregate faculty:
V.C.1.c).(7).(a) evaluation; and, (Core)
V.C.1.c).(7).(b) professional development. (Core)
V.C.1.d) The Program Evaluation Committee must evaluate the program's mission and aims, strengths, areas for improvement, and threats. (Core)
V.C.1.e) The annual review, including the action plan, must:
V.C.1.e).(1) be distributed to and discussed with the members of the teaching faculty and the residents; and, (Core)
V.C.1.e).(2) be submitted to the DIO. (Core)
V.C.2. The program must complete a Self-Study prior to its 10-Year Accreditation Site Visit. (Core)
V.C.2.a) A summary of the Self-Study must be submitted to the DIO. (Core)
Outcomes of the documented Annual Program Evaluation can be integrated into the 10-year Self-Study process. The Self-Study is an objective, comprehensive evaluation of the residency program with the aim of improving it. Underlying the Self-Study is this longtudinal evaluation of the program and its learning environment, facilitated through sequential Annual Program Evaluations that focus on the required components, with an emphasis on program strengths and self-identified areas for improvement. Details regarding the timing and expectations for the Self-Study and the 10-Year Accreditation Site Visit are provided in the ACGME Manual of Policies and Procedures. Additionally, a description of the Self-Study Process, as well as information on how to prepare for the 10-Year Accreditation Site Visit, is available on the ACGME website.
V.C.3. One goal of ACGME-accredited education is to educate physicians who seek and achieve board certification. One measure of the effectiveness of the educational program is the ultimate pass rate.
The program director should encourage all eligible program graduates to take the certifying examination offered by the applicable American Board of Medical Specialties (ABMS) member board or American Osteopathic Association (AOA) certifying board.
V.C.3.a) For specialties in which the ABMS member board and/or AOA certifying board offer (s) an annual written exam, in the preceding three years, the program's aggregate pass rate of those taking the examination for the first time must be higher than the bottom fifth percentile of programs in that specialty. (Outcome)
V.C.3.b) For specialties in which the ABMS member board and/or AOA certifying board offer(s) a biennial written exam, in the preceding six years, the program's aggregate pass rate of those taking the examination for the first time must be higher than the bottom fifth percentile of programs in that specialty. (Outcome)
V.C.3.c) For specialties in which the ABMS member board and/or AOA certifying board offer(s) an annual oral exam, in the preceding three years, the program's aggregate pass rate of those taking the examination for the first time must be higher than the bottom fifth percentile of programs in that specialty. (Outcome)
V.C.3.d) For specialties in which the ABMS member board and/or AOA certifying board offer(s) a biennial oral exam, in the preceding six years, the program's aggregate pass rate of those taking the examination for the first time must be higher than the bottom fifth percentile of programs in that specialty. (Outcome)
V.C.3.e) For each of the exams references in V.C.3.a)-d), any program whose graduates over the time period specified in the requirement have achieved an 80 percent pass rate will have met this requirement, no matter the percentile rank of the program for pass rate in that specialty. (Outcome)
Background and Intent: Setting a single standard for pass rate that works across specialties is not supportable based on the heterogeneity of the psychometrics of different examinations. By using a percentile rank, the performance of the lower five percent (fifth percentile) of programs can be identified and set on a path to curricular and test preparation reform.
There are specialties where there is a very high board pass rate that could leave successful programs in the bottom five percent (fifth percentile) despite admirable performance. These high-performing programs should not be cited, and V.C.3.e) is designed to address this.
V.C.3.f) Programs must report, in ADS, board certification status annually for the cohort of board-eligible residents that graduated seven years earlier. (Core)
It is essential that residency programs demonstrate knowledge and skill transfer to their residents. One measure of that is the qualifying or initial certification exam pass rate. Another important parameter of the success of the program is the ultimate board certification rate of its graduates. Graduates are eligible for up to seven years from residency graduation for initial certification. The ACGME will calculate a rolling three-year average of the ultimate board certification rate at seven years post-graduation, and the Review Committees will monitor it.
The Review Committees will track the rolling seven-year certification rate as an indicator of program quality. Programs are encouraged to monitor their graduates' performance on board certification examinations.
In the future, the ACGME may establish parameters related to ultimate board certification rates.
LSU GME Knowledge Base
Copyright 2020 LSU School of Medicine unless otherwise specified.