NYCOM Learning Outcome Assessment Plan

Developed by Dr. Pelham Mead and Dr. John McCarthy, Directors at NYSCOM 2008-2009.

I. Mission of NYCOM

The New York College of Osteopathic Medicine of the New York Institute of Technology is committed to training osteopathic physicians for a lifetime of learning and practice, based upon the integration of evidence-based knowledge, critical thinking and the tenets of osteopathic principles and practice. The college is also committed to preparing osteopathic physicians for careers in primary care, including health care in the inner city and rural communities, as well as to the scholarly pursuit of new knowledge concerning health and disease. NYCOM provides a continuum of educational experiences to its students, extending through the clinical and post-graduate years of training. This continuum provides the future osteopathic physician with the foundation necessary to maintain competence and compassion, as well as the ability to better serve society through research, teaching, and leadership.

II. INSTITUTIONAL GOALS

1. The Osteopathic Philosophy: Upon graduation, a student must possess the ability to demonstrate the basic knowledge of Osteopathic philosophy and practice, as well as Osteopathic Manipulative Treatment.
2. Medical Knowledge: A student must possess the ability to demonstrate medical knowledge through passing of course tests, standardized tests of the NBOME, post-course rotation tests, research activities, presentations, and participation in directed reading programs and/or journal clubs, and/or other evidence-based medicine activities.
3. Practice-based learning and improvement: Students must demonstrate their ability to critically evaluate their methods of clinical practice, integrate evidence-based medicine into patient care, show an understanding of research methods, and improve patient care practices.
4. Professionalism: Students must demonstrate knowledge of professional, ethical, legal, practice management, and public health issues applicable to medical practice.
5. Systems-based practice: Students must demonstrate an understanding of health care delivery systems, provide effective patient care and practice cost-effective medicine within the system.
6. Patient Care: Students must demonstrate the ability to effectively treat patients and provide medical care which incorporates the osteopathic philosophy, empathy, preventive medicine education, and health promotion.
7. Communication skills: Students must demonstrate interpersonal and communication skills with patients and other healthcare professionals, which enable them to establish and maintain professional relationships with patients, families, and other healthcare providers.
8. Primary Care: NYCOM is committed to preparing osteopathic physicians for careers in primary care, including health care in the inner city, as well as rural communities.
9. Commitment to scholarly work: NYCOM is committed to preparing osteopathic physicians for the scholarly pursuit of new knowledge concerning health and disease. NYCOM is committed to preparing academic physicians in order to address this nation’s projected health care provider shortage and the resulting expansion of medical school training facilities.
10. Global Medicine and Health policy: NYCOM—as a global institution and leader in global health, education, and knowledge—is committed to creating opportunities for students to engage in global health practice, policy, and the development of solutions to the world’s vital problems.
11. Cultural Diversity: NYCOM is committed to attracting, retaining, and graduating students from diverse cultural and ethnic backgrounds and prepare them as osteopathic physicians to deliver the highest quality medical care, with the highest degree of compassion, understanding, and empathy toward cultural differences in our global society.

III. Introduction and Rationale

At NYCOM we believe it is our societal responsibility to monitor our students’ quality of education through continual assessment of educational outcomes. On-going program evaluation mandates the utilization of empirical data based on a scientific methodology; a longitudinal study. As defined, a longitudinal study is a correlational research study that involves repeated observations of the same items over long periods of time.

At Thomas Jefferson University, an innovative study was implemented circa 1970, which was ultimately titled “Jefferson Longitudinal Study of Medical Education”. As a result of implementation of this longitudinal study plan, Thomas Jefferson University was praised by the Accreditation Team for the Middle States Commission on Higher Education for “…..their academic interest in outcome data, responsiveness to faculty and department needs and the clear use of data to modify the curriculum and teaching environment….their use of this data has impacted many components of the curriculum, the learning environment, individual student development, and program planning…” (TJU, 2005).

The Jefferson Longitudinal Study of Medical Education has been the most productive longitudinal study of medical students and graduates of a single medical school. This study has resulted in 155 publications in peer review journals. Many of the studies were presented before national or international professional meetings prior to their publication (TJU, 2005).

According to Hernon and Dugan (2004), the pressure on higher education institutions to prove accountability has moved beyond the acceptance and reliance of self-reports and anecdotal evidence compiled during the self-regulatory accreditation process to an increasing demand from a variety of constituencies to demonstrate institutional effectiveness by focusing on quality measures, such as educational quality, and cost efficiencies.

Accountability focuses on results as institutions quantify or provide evidence that they are meeting their stated mission, goals, and objectives. Institutional effectiveness is concerned, in part, with measuring (Hernon and Dugan, 2004):

• Student outcomes: such as graduation rates, retention rates. Such outcomes are institution-based and may be used to compare internal year-to-year institutional performance and as comparative measures with other institutions.
• Student learning outcomes: oftentimes referred to as educational quality and concerned with attributes and abilities, both cognitive and affective, which reflect how student experiences at the institution supported their development as individuals. Students are asked to demonstrate acquisition of specific knowledge and skills.

This program evaluation approach provides for on-going data collection and analysis targeted at assessing outcomes of student achievement and program (curricular design) effectiveness. As a result of the NYCOM Learning Outcome Assessment Plan’s continual assessment cycle, the report is available (with scheduled updates) as a resource in the decision-making process through offering outcomes data, recommendations, and suggestions which inform key stakeholders of areas of growth and/or improvement, and proposed changes to policy that strengthen both overall assessment and data-driven efforts to improve student learning.

IV. Purpose and Design

The NYCOM Learning Outcome Assessment Plan incorporates data and outcomes from many different constituents, including pre-matriculation data, the 4-year pre-doctoral NYCOM curriculum and the 5-year pre-doctoral Scholars program, post-graduate data, and practice/career tracking of our graduates. The Assessment Plan Guide (Table 1) and the following ten principles guide the NYCOM Learning Outcome Assessment Plan:

1. The assessment plan provides outcomes of formative and summative assessment experiences of our students’ learning. Examples of formative assessment activities include post-course roundtable discussions, Institute for Clinical Competence (ICC) seminars, and the Course/Faculty Assessment program. Examples of summative assessments include the Graduation Questionnaire submitted to AACOM, COMLEX scores, NBOME subject exam scores, and Clerkship evaluations.
2. NYCOM’s primary reason for assessment is to improve student learning and development.
3. The development of an effective, valid assessment program is a long-term dynamic process.
4. The assessment plan utilizes the most reliable, valid methods and instruments of assessment.
5. The priorities of this assessment program are grounded in the core goals and competencies of the NYCOM Mission Statement.
6. The assessment involves a multi-method approach.
7. The assessment of student learning and development is a process that is separate from faculty evaluation.
8. The assessment plan is of greatest benefit to the institution through providing an evidence-based analysis for decision-making around program revision and program improvement.
9. Provides a substantive and sustainable mechanism which enables NYCOM to fulfill its’ responsibility of evaluating our graduates in order to ensure the quality, rigor, and overall effectiveness of our educational program as it relates to educating competent and compassionate physicians;
10. Ensures that the Assessment Plan and reporting process measures what it is intended to measure (student achievement and program effectiveness) and provides stakeholders with relevant and timely data to make informed decisions on changes in curricular design implementation, program planning, and the overall learning environment.

The specifics of the NYCOM Learning Outcome Assessment Plan include:

• A detailed description of planned assessment procedures;
• Detailed linkage of assessment outcomes to NYCOM’s Mission and the osteopathic core competencies (as identified in Institutional Goals);
• Identification of specific data sources / outcome indicators (see figure 1);
• Description of assessment reporting cycle;
• A process for linkage to strategic planning, if appropriate;
• Detailed outcomes baseline and established benchmarks;
• Dissemination to appropriate committee level and/or departmental level personnel for development of appropriate action plans to ensure continuous program improvement.
• Description of responsible department/staff to organize and coordinate data collection, report generation and presentation, as well as directing dissemination of information to appropriate staff
Statement(s) of Expected Outcomes Identification of where expected outcomes are addressed Methods/criteria to assess outcomes Development of Baseline Development of benchmarks
Demonstrate basic knowledge of Osteopathic Competencies
• Academic Coursework

• ICC Seminars

• Descriptors of changes in academic status (attrition/remediation)

• C/FA program

• COMLEX 1 & II

• DPC Curriculum

• LDB Curriculum

• Clinical Clerkships

• Residency Match Rate

• NBOME Shelf Exams

• 5-Year Scholars Program

• Alumni feedback
• Course Exams

• Simulation

• Formative / Summative Experiences

• Surveys

• Standardized Tests (NBOME)

• Faculty feedback (clerkships)

• Class ratings Vis a Vis:

• Standardized Tests

• Internal / External Surveys

• Analysis of Residency Trends Data

• End-of-year pass rates

• Analysis of Specialty Choice
Demonstrate medical knowledge
Demonstrate ability for critical reasoning
Demonstrate knowledge of professionalism, ethics
Demonstrate an understanding of health care delivery systems
Demonstrate the ability to effectively treat patients
Students must demonstrate interpersonal and communication skills
NYCOM is committed to preparing osteopathic physicians for careers in primary care
NYCOM is committed to preparing osteopathic physicians for the scholarly pursuit of new knowledge
NYCOM is committed to creating opportunities for students to engage in global health practice
NYCOM is committed to preparing students to effectively interact with people of diverse cultures

V. Outcomes

Figure 2 provides an illustration of the inter-face of constituents’ data which will comprise NYCOM’s Learning Outcome Assessment Plan.

Outcome Indicators—evaluating student learning outcomes and program effectiveness in conjunction with the medical education training continuum:

1. Pre-matriculation data, including pre-matriculation survey;
2. Academic course-work (scores on exams, etc.) “attrition rate”;
3. Student feedback (assessment) of courses;
4. COMLEX Scores (pass rates), including mean scores and comparison to national averages;
5. Residency match rate;
6. Feedback from 4th Year Exit Survey;
7. Feedback from (AACOM) Graduation Questionnaire;
8. Residency program director evaluations of residents;
9. Completion rates (Post-Doctoral programs);
10. Specialty certification and licensure;
11. Career choices (practice type–academic, research, etc.);
12. Geographic practice locations.
13. Alumni survey

VI. Outcomes Detail / Data Sources

1. Pre-matriculation data
   This data-set includes data captured prior to students entering NYCOM, which includes the following: AACOM pre-matriculation survey given to students; total MCAT scores, collegiate GPA (total GPA-including undergraduate/graduate); and interview score. As there are several cohorts of students in NYCOM (Traditional, BS/DO, Med-Prep, and Émigré) all data is inclusive.
2. Academic course-work
   This data-set includes data captured during 4-year pre-doctoral program and 5-year scholar program and includes the following: course grades (H/P/F); exam scores; end-of-year class rankings; ratings of clinical clerkship performance; descriptors of changes in academic status (attrition); percent of students remediating; and percent of repeat students.
3. Student feedback of courses
   This data-set includes student feedback from the new Course / Faculty Assessment (program) reports.
4. COMLEX Scores
   This will include scores on COMLEX I and II and will incorporate pass rates-both overall and first time, mean scores, and comparisons to national averages (NBOME).
5. Residency match rates and overall placement rate
   Data will be compiled as received from the AOA and the National Residency Match Program (NRMP).
6. Feedback from (AACOM) Graduation Questionnaire
   Annual survey report received from AACOM comparing NYCOM graduates responses to numerous categories (including demographics, specialty choice, overall perception of pre-doctoral training, indebtedness, and more) to nation-wide osteopathic medical school graduating class responses.
7. Completion rates (post-doctoral programs)
   Percent of NYCOM graduates completing internship/residency training programs.
8. Specialty certification and licensure
   Data compiled from state licensure boards and other specialty certification organization (board certification) on NYCOM graduates.
9. Career choices and geographic practice location
   Data includes practice type (academic, research, clinical, and so on) and practice location. Data obtained from licensure boards, as well as NYCOM Alumni survey.

VII. Compiling the Data

Discussions with departmental leaders and dean’s confirmed data gathering occurs at various levels throughout the institution. Development of a central repository (centralized database) facilitates data gathering, data mining and overall efficiency as it relates to data assessment, report generation, and report dissemination. This includes utilization of internal databases (internal to NYCOM) as well as interfacing with external organizations’ databases, including the AOA (American Osteopathic Association),
AMA (American Medical Association), and the ABMS (American Board of Medical Specialties).

VIII. Stakeholders

Information from the data collection serves to inform NYCOM administration, appropriate faculty, appropriate research endeavors, and academic/administrative committees, including the following:

• Curriculum Committee
• Student Progress Committee
• Admissions Committee
• Deans and Chairs Committee

Further, this data analysis serves the college for accreditation purposes and institutional self-study.

NYCOM Learning Outcome Assessment Plan

Stages of Implementation of NYCOM Learning Outcome Assessment Plan

I. STAGE 1 Responsible Department/Staff: Office of Educational Development & Assessment (EDA) is lead department responsible for directing all aspects/components of NYCOM Assessment Plan to fruition. Organizational reporting structure of EDA is to Associate Dean for Academic Affairs.

II. STAGE 2 Centralized database: Development of a (shared or central) repository (database) utilized by internal departments of NYCOM comprised of student data categorized as follows:

Pre-matriculation Data includes demographics, academic data (GPA), and other admissions data (MCAT’s, etc.).

Pre-doctoral Data includes responses to pre-matriculation surveys, course grades, end-of-year grade point averages, Course / Faculty assessment data, ratings of clinical clerkship performance, performance scores on COMLEX I and II, descriptors of changes in academic status (attrition), 4th year exit surveys, and AACOM Graduation questionnaires.

Post-graduate/Career Data includes residency choice, hospitals of residency, geographic location, chosen specialty, residency performance ratings, performance on COMLEX III, geographic and specialty area of practice following post-graduate graduation, licensure, board certification status, scholarly work, professional activities/societies, faculty appointments, type(s) of practice (academic, clinical, research).

This database supports and assimilates collaborative surveys utilized by internal departments in order to capture requested data (see item IV. Outcomes on page 4) essential for tracking students during and after post-graduate training.  Specific data (e.g., COMLEX III, board certification, and licensure) is provided by external databases, through periodic reporting means, or queries from NYCOM, therefore the database provides for assimilation of this external data, in order to incorporate into institutional reporting format.

III. STAGE 2 Benchmarks and Reporting: Compile/collate data in order to establish baseline metrics (see item V. Compiling the Data on page 3). Following development of these metrics, institutional benchmarks are established. Benchmarks align with Institutional Goals as written above.

Reporting of data analysis occurs on an annual basis.  An annual performance report is compiled from all survey data and external sources.  Timeframe for reporting is congruent with end of academic year.  Updates to report occur semi-annually, as additional (external) data is received.

Data reporting includes benchmarking against Institutional Goals (mission), in order to provide projections around effectiveness of learning environment, quality improvement indicators, long-range and strategic planning processes, and cost analysis/budgetary considerations.

Report dissemination to key stakeholders, as previously identified (in footnote 3, page 1) in addition to other staff, as deemed appropriate for inclusion in the reporting of assessment analysis.  

IV. STAGE 3 Process Evaluation: Annual evaluation of implementation of (longitudinal) Learning Outcome Assessment Plan for review and critique of assessment reports, as well as overall evaluation of report results (to include review of any programmatic changes, curricular modifications, etc.). Annual evaluation should focus on the question:
Is the institutional plan assessing what it intended to assess?