Use of Assessment System for Improvements
Plymouth State University is committed to ongoing assessment of its effectiveness in meeting the goals of the university plan and carrying out the institutional mission. An Assessment Advisory Group has been in place at the university level and has helped educate all PSU faculty on the need to assess students, programs, and the institutional mission. Additionally, since the last NCATE visit, the institution has reorganized the Office of Institutional Research. The new office is now the Office of Institutional Research and Effectiveness and is directed by an Associate Dean.
The new state approval process, which PSU completed in November 2009, has a strong focus on performance based assessment. All professional education programs were required by this process to demonstrate that their candidates met the competencies specified in the state standards for general (ED 610.02) and professional education (ED 612/614) for their area of certification. Most of the PSU professional education programs have implemented portfolio assessments that re aligned with state and national standards. Additionally, programs have been collecting data for the past three years as is required by NCATE on the following: content knowledge, planning, clinical field experience, and assessment of candidates impact on student learning.
All baccalaureate candidates are assessed in a variety of ways throughout the teacher education program. The four levels of assessment correspond to appropriate benchmarks within the teacher education programs. Unit faculty, the PSU Council of Teacher Education (CTE) and the Office of Teacher Certification (OTC) are responsible for coordination and oversight of the assessment system. Candidates are made aware of unit assessment policies through the college catalog, academic advising, and materials distributed by the OTC and the College of Graduate Studies (COGS), as well as throughout the Teacher Certification and COGS website. Post baccalaureate candidates are assessed using a similar gate system as the baccalaureate level. A full description of these systems is attached as an exhibit.
The unit has developed and revised outcomes based evaluation to assess the clinical field experience. This assessment was piloted in 2006 with Education, English, and Math Education candidates. After reviewing the data and feedback collected from the cooperating teachers and clinical faculty who used the evaluation during the pilot, the clinical field experiences assessment was approved by the Unit’s CTE. Each program developed supplemental items to assess the state and national content standards not included in the generic tool (Sections I – IV of the evaluation tool). Sections I – IV of the evaluation tool focuses on pedagogy; classroom atmosphere; reflective practice; and professionalism.
After the clinical field experience tool was approved by the CTE, a subcommittee of members was formed to align the assessment with the Unit’s Conceptual Framework, the NH general teacher education standards (ED 610s) and INTASC standards. The forms are now available as either PDF files or Word documents on the Teacher Certification and COGS website. Evaluations are collected by the OTC for data entry. Data is analyzed at the end of each semester and shared with programs during the bi-annual retreats.
In 2006, the unit established an electronic system for collecting and analyzing transition points of candidates. All faculty are now able to access up-to-date information on each candidate’s progress toward meeting the requirements for teacher certification. For example, faculty advisors can use the system to determine if a candidate has met the requirements to be accepted into teacher certification candidacy: minimum required GPA, passing scores on the Praxis I exam, and favorable recommendations from two faculty members. Candidates can also access their own information to track their progress. COGS has developed and approved a similar Gate system and the electronic system is being developed for implementation in early spring 2011.
The OTC is responsible for collecting and maintaining the records of undergraduate candidates once they are accepted as teacher certification candidates. The Graduate Coordinator of Certification and Conversion is responsible for collecting and maintaining the records of post baccalaureate candidates once they are accepted into a teacher certification/professional preparations program and complete the necessary forms. The OTC Coordinator of Assessment maintains all of the undergraduate files and is responsible for the data management as candidates move through the gate system.
Based on the data collected, unit faculty, the Coordinator of Assessment, Coordinator of Teacher Certification and Clinical Experiences, and Director of Teacher Education are able to counsel candidates who appear to be struggling. Advisors are reminded to utilize the system during advising which occurs during each semester. In situations where students are struggling with passing Praxis I, achieving the necessary minimum GPA, or departmental requirements, program coordinators in collaboration with the Coordinator of Assessment works with the student to provide services needed to be successful. Those who have difficulty within programs are identified early and encouraged to work with academic advisors, program faculty, and the Plymouth Academic Support Services (PASS) to develop strategies to succeed. If these strategies fail, candidates are provided with help and support in identifying other possible degree programs and career paths.
In situations when a candidate experiences serious difficulties during the student teaching experience, the candidate works with the Coordinator of Teacher Certification and Clinical Field Experiences, the clinical faculty, and the Director of Teacher Education to develop a focus plan for improvement. At the post baccalaureate level the only difference is the coordinator of Certification and Conversion replaces the Coordinator of Teacher Certification and Clinical Field Experiences. If the plan fails, the Coordinator, advisor, and clinical faculty along with the Director of Teacher Education devise a plan that will enable him/’her to graduate without teacher certification During the past year, programs have developed plans within their departments to make sure that a candidate has options in this situations. The infrequent occurrence of such situations suggests that the assessment used to determine continuation in and completion of professional education programs are predictors of candidate success.
Continuous Improvement
1. Faculty Evaluations The faculty evaluations are primarily used to predict candidate success within the teacher preparation programs. In order to analyze the reliability of the 19-item survey, a Cronbach’s alpha reliability analysis was computed. The purpose of this analysis was to determine internal consistency or average correlation of items to gauge reliability. Data from the fall 2009 and spring 2010 were used for this analysis. The reliability coefficient for fall was .69 and for spring .77. Based on the literature reliability coefficients above .70 is considered acceptable. After analyzing the data, a number of factors could have contributed to the moderate reliability. The faculty evaluations are completed early in a candidate’s professional schooling. Attrition rates have not been analyzed regarding which candidates drop teacher certification as a major. Additionally, some faculty do not complete all 19 items of the evaluation. The unit intends to continue to run these analyses annually and discuss the timing of when a candidate completes this stage of the Gate System.
2. The greatest changes have occurred at the post baccalaureate level and these include the following: hired a Coordinator of Certification and Conversion who is developing and implementing processes for field experiences; hired a Director of Assessment and Faculty Program Coordinators who is addressing the problems with advising and lack of consistency within programs; added a new Graduate Assistant who is strictly working with accreditation and data management; hired a contract faculty to coordinator the Elementary Education program; reallocated faculty members to be full time graduate faculty in order to facilitate program improvements; revising Graduate Internship Seminar in order to ensure that content and pedagogical content is exhibited for all program completers.
3. Moved the Office of Teacher Certification to the offices of Institutional Research and Assessment in order to collaborate on data projects and review of data.
4. The University purchased the assessment program TracDat to manage data not only for the Unit, but at the institutional level also. The teacher preparation programs intend be campus leaders in this effort to manage data using the new system. This process will begin in January 2011.
5. Developed, piloted, revised and aligned student teaching evaluation forms and ALL initial teacher certification programs are using these forms. In 2006 a committee of members from the CTE reviewed the literature relative to assessment of culminating field experiences. The initial evaluation form was piloted in the Spring 2007 by the Education and English Education departments. After reviewing and revising the first 19 items on general pedagogical knowledge, skills and dispositions, faculty reviewed and aligned with specific content state and SPA standards. Competencies not addressed within the first 19 items, were written by faculty and included as content specific items. Content validity was performed by sharing these forms with a committee that included public school teachers, administrators, clinical faculty, and unit faculty. The new evaluation form was utilized by undergraduate teacher preparation programs during the 2007-2008 academic year. The following year (2008-2009), a committee of CTE members aligned the first 19 items (generic pedagogical items) with the conceptual framework, INTASC, and state initial teacher certification standards (ED 610.02). Additionally each program reviewed the form for alignment with specific state and national content standards. The committee also reviewed the form for redundancy and language.
6. During the Spring of 2008, all post baccalaureate teacher certification candidates were assessed using the OTC Student Teaching Evaluation form. Data was collected however, analysis by level was not implemented until the Fall of 2009. Data from these evaluations is shared with unit faculty, cooperating teachers, and clinical faculty in order to assess program efficiency and whether standards are being met by the candidates. In reviewing candidate data from focus groups during the last student teaching seminar and the scores on the evaluation form, it is evident that candidates are approaching or meeting target on all items.
In an effort to analyze the consistency of the items on this form, a Cronbach alpha reliability analysis was computed using a random sample of 20 evaluation forms from the Fall 2009 candidates and Spring 2010 candidates. The alpha reliability coefficient for the Fall data was .93 and for the Spring .91, indicating a relatively high internal consistency.
The OTC and program coordinators will continue to mentor new clinical and cooperating teachers in order to maintain consistency across all candidates. This analysis will be computed annually. It should be noted that inter-rater reliability for clinical faculty and cooperating teachers has been consistently recorded above .90 indicating a strong relationship between the two supervisors.
7. Improved the Teacher Education website so that all forms are available electronically. In an effort to meet the mission of the university to limit the use of paper and for improved bookkeeping, the OTC is working with Instructional Technology Services (ITS) to make the student teaching evaluation forms an online submission. The intent is to go live with this by the Fall 2011.
8. In preparing for the state re-approval of all teacher certification, advanced preparation programs and endorsements, the unit has worked on reviewed program and unit assessments aligned with state and national standards for all programs and completed a self-assessment in order to identify weaknesses and strengths at the program and unit levels. A team of 27 reviewers spent three days on campus in November, 2009 to review our programs. Assessments, candidate work samples, interviews with candidates, faculty, administrators and staff were completed for each program. Five post baccalaureate programs were identified as needing improvement and received conditional approval until August 2012. The emphasis of these programs has been on reviewing current data, syllabi, and curriculum in order to address the unmet standards identified by the state team. These programs plan to submit a progress report in the Fall 2011 with new data, candidate work and assignments. Additionally two programs were reviewed, Chemistry Education and Curriculum Administrator for approval as new programs. These programs have received a provisional status until 2012 in order to produce candidate work for evidence to the state that the standards are being addressed by all candidates.
9. Compile and report data to programs on student surveys, overall unit data for student teaching, EBI results, and Praxis I and Praxis II data. This data was collected in the past but not shared with the programs within the unit. The data is shared during the May retreat for all teacher education (full and part time) faculty. The emphasis is on review of data and plans for improvement. Beginning this year, unit initiatives will be discussed, prioritized and shared with all faculty in an effort to continue overall Unit improvement.
10. Closed assessment loops within the Unit in order to continually work towards improvement. Since the hiring of the new Coordinator of Assessment the assessment systems have been reviewed and the process of revising goals and objectives based on data has begun. To date the following have been completed : shared results of the evaluation of clinical faculty with program coordinators, the Director of Teacher Certification and the Coordinator of Teacher Education and Clinical Field Experience; data has been aggregated and disaggregated by level and program and shared with unit faculty and program coordinators; utilizing data from the faculty evaluations as a predictor for success; reviewed and presented data correlating Praxis I scores with final student teaching evaluation scores for discussion on program and unit completion; focus group data collected from the student teachers during the last seminar has been compiled and shared with all stakeholders.
11. With the electronic sharing of results from the EBI survey this is the next assessment loop to be closed. Plans to review data from the Fall 2010 and Spring 2011 candidates will be analyzed and shared with faculty.
Overall the work to receive state approval and NCATE accreditation has drastically changed the way the unit collects, analyzes and reviews data. Faculty are excited to see results and discuss the strengths as well as the weaknesses of our candidates. Attendance at the bi annual retreat has increased dramatically since adopting an empirical evidence system.
In summary, the state and national review has prompted faculty to review their programs systematically and make necessary changes based on the data. This has been a positive paradigm shift within the Unit. Additionally, the baccalaureate and post baccalaureate programs have moved toward working as a Unit which was a problem in the past.