Comprehensive Standard 3.3.2: Quality Enhancement Plan
The institution has developed a Quality Enhancement Plan that (1) demonstrates institutional capability for the initiation, implementation, and completion of the QEP; (2) includes broad-based involvement of institutional constituencies in the development and proposed implementation of the QEP; and (3) identifies goals and a plan to assess their achievement. (Quality Enhancement Plan)
The University of South Florida Sarasota-Manatee (USF Sarasota-Manatee) developed its first Quality Enhancement Plan (QEP) , entitled the Incredi-Bull Critical Thinking QEP, to implement from 2016-2021. The definition of critical thinking chosen by USF Sarasota-Manatee is the following:
Critical thinking is the intellectually disciplined process of actively and skillfully conceptualizing, applying, analyzing, synthesizing, and/or evaluating information gathered from, or generated by, observation, experience, reflection, reasoning, or communication, as a guide to belief and action.
Below is evidence that USF Sarasota-Manatee meets each of the indicators regarding CS 3.3.2 from the SACSCOC Quality Enhancement Plan Guidelines Rubric.
(1) Institutional Capability
Indicator 3.A. Capability to initiate the plan
Successful implementation of the Incredi-Bull Critical Thinking QEP depends on the provision of adequate human and financial resources. Much of the QEP implementation will be embedded in already-existing activities, such as faculty teaching, professional development, and academic program assessment.
To meet the additional workload associated with the QEP, USF Sarasota-Manatee will hire a half-time Director and a half-time Administrator. The Director, as the leader of curricular change efforts, must be a member of the faculty. The QEP Director will report half-time to the Regional Vice Chancellor for Academic & Student Affairs and half-time to one of the college deans starting in May 2016. The QEP Director will also have a dotted-line report to the Assistant Vice President for Institutional Research & Effectiveness and SACSCOC Liaison to ensure support for functions related to accreditation and assessment.
The QEP Administrator will assist the Director and must be equipped to manage QEP-related social media and website content, and provide organizational and meeting support, among other tasks. The QEP Director’s Position Description and the QEP Administrator’s Position Description detail the responsibilities of each position.
table below presents a budget for the five years of plan implementation, plus the expenditures associated with professional development and marketing in Year 0. Work such as development of online professional development modules and development of LibGuides will be handled as part of the ongoing support efforts of the faculty/staff involved and are not included in the budget. USF Sarasota-Manatee, through the support of its Senior Leadership, has committed significant personnel and financial resources to successfully launch and implement the QEP over the course of the implementation period.
|QEP Budget for Year 0 – Year 5|
|Director .50 FTE||$0||$59,539||$64,898||$66,195||$67,519||$68,870||$327,021|
|Faculty Leader Grants||$0||$0||$2,000||$4,000||$4,000||$2,000||$12,000|
|ETS Proficiency Profile||$0||$3,045||$3,915||$4,785||$5,655||$6,525||$23,925|
*Includes salary and benefits.
Indicator 3.B. Capability to implement and complete the plan
USF Sarasota-Manatee’s QEP, Incredi-Bull Critical Thinking, seeks to use enhanced pedagogical practices that focus on critical thinking to improve students’ critical thinking skills. The implementation plan of Incredi-Bull Critical Thinking involves four major components:
- Incredi-Bull Faculty Development
- Incredi-Bull Curriculum Integration
- Incredi-Bull Campus Engagement
- Incredi-Bull Outcomes-Based Assessment
A diagram outlining the Implementation Plan Timeline shows the roll-out of the first three components above, including the semesters of development, piloting, and implementation.
Year 0 (2015-2016): Baseline and Awareness
The major goals for Year 0 (Planning Year) of USF Sarasota-Manatee’s Incredi-Bull Critical Thinking QEP include using existing and new assessments to build a robust bank of baseline data, pilot and refine the Incredi-Bull Critical Thinking Rubric, continue to build the faculty’s knowledge and skills related to critical thinking pedagogy, increase University-wide awareness of the QEP topic including the Critical Thinking Commitments and Critical Thinking Intellectual Standards, and appoint a QEP Director who will have a .50 FTE assignment for QEP administration. The Year 0 Implementation Table provides a description of each activity, including the semester of implementation and person(s) responsible, with sections for implementation elements related to administrative actions, faculty development, curricular implementation, and campus engagement.
Year 1 (2016-2017): Pilot Implementation
Year 1 of the implementation plan is marked by an expansion of faculty development and the piloting of many elements of curricular integration. The year begins with the transition of the QEP Steering Committee to the QEP Implementation Committee under the leadership of the newly-hired QEP Director. Continuing communication to the campus community about Incredi-Bull Critical Thinking will support widespread campus engagement. The Year 1 Implementation Table provides a description of each activity, including the semester of implementation and the person(s) responsible, with sections for implementation elements related to administrative actions, faculty development, curricular implementation, and campus engagement.
Years 2-5 (2017-2021): Implementation
The goals for Years 2-5 are to implement fully the elements piloted in Year 1, to continue to build faculty knowledge and skills for enhancement of pedagogical practices that focus on critical thinking, and to continue development and implementation of curricular integration and campus engagement elements to enhance students’ critical thinking skills. Years 2 and 3 will involve integration of critical thinking into an increasing number of courses and assignments in the upper level programming of all undergraduate programs. All elements introduced in early years will continue through the end of the QEP. The Years 2-5 Implementation Table provides a description of each activity, including the semester of implementation and the person(s) responsible, with sections for implementation elements related to faculty development, curricular implementation, and campus engagement. To support interpretation of this multi-year table, the years of implementation are noted for each activity.
The diagram above shows how the Professional Development workshops and online modules will be continuously assessed by the faculty, QEP Director, and QEP Administrator through participation in the activities and the Professional Development Feedback Survey. To provide annual formative assessment, the faculty in collaboration with the Institutional Research & Effectiveness (IR) Assessment Coordinator will review the teaching and learning practices through annual program assessment. In addition, the QEP Implementation Committee will review curricular and pedagogical summaries from the degree programs and implement any changes to the implementation plan. Finally, as summative assessment, the QEP Director will prepare the final QEP Impact Report.
The diagram above summarizes Goal 2’s student learning outcome assessment activities for critical thinking throughout the five-year implementation period. The first box shows collection of assessment data each semester. In SLS 2122 Foundations of Professional Success, selected upper-level courses, and capstone courses, students will produce work for faculty to assess using the Incredi-Bull Critical Thinking Rubric. At all orientations (freshman and transfer), students will take the Watson-Glaser. Then graduating seniors will take the Watson-Glaser in the capstone courses. Students will take the ETS Proficiency Profile as a pre-test in SLS 1107 Foundations of University Success and as a post-test in SLS 2122. At freshman orientations, students will take the California Institutional Research Program’s (CIRP) Freshman Survey, which contains items that measure students’ perceptions of critical thinking ability. Graduating seniors will take CIRP’s Graduating Senior Survey.
Annually as part of academic program assessment, faculty will review results of students’ critical thinking learning outcomes by degree program. Then the QEP Implementation Committee will review a random sample of student work using the Incredi-Bull Critical Thinking Rubric to score performance. These reviews will provide the formative assessment to make changes to the curriculum annually, as needed. In Year 5, the QEP Director will analyze the data across all five years and prepare the QEP Impact Report, as summative assessment of the entire project.
(2) Broad-based Involvement
Indicator 4.A. Development of the Plan
Below is a timeline of the activities undertaken in Year 0 (Planning Year) to select the QEP topic. Included are the dates and supporting documents of the Core Curriculum Committee that selected the Pillars of Intellectual Engagement that all USF Sarasota-Manatee undergraduates undertake. The list demonstrates documented consultation and input from a variety of constituencies.
|Timeline of QEP Development through Fall Semester, Year 0|
|Aug. 24, 2012||General Education Workshop||Meeting with broad faculty representation to gather input on the new Core Curriculum Learning Outcomes||Core Curriculum Committee|
|Oct. 9, 2012||Core Curriculum Committee Meeting||Committee agrees on USFSM’s Pillars of Intellectual Engagement||Core Curriculum Committee Minutes 10/9/2012|
|Aug. 5, 2014||QEP Steering Committee Formed||Regional Vice Chancellor for Academic & Student Affairs invites representatives from colleges and Student Services to join committee||QEP Committee Membership Invitation; QEP Steering Committee|
|Aug. 20, 2014||Student Government Meeting||Dr. Bonnie Jones’s presentation explaining the purpose and timeline of USFSM’s QEP; Student Government questionnaire regarding student success||Bright Ideas for USF Sarasota-Manatee’s QEP Prezi; QEP Student Questionnaire|
|Aug. 22, 2014||Faculty Welcome Seminar||Dr. Bonnie Jones’s presentation explaining the purpose and timeline of USFSM’s QEP;
Faculty questionnaire regarding student success
|Bright Ideas for USF Sarasota-Manatee’s QEP Prezi; Faculty Questionnaire & Results|
|Sept. 11, 2014||QEP Steering Committee Meeting||Committee charge; Timeline review; Decision to limit potential topic to the Pillars of Intellectual Engagement; Plan for gathering of data related to potential topics; Plan for gathering input from campus community about topic selection||QEP Agenda 09/11/2014; QEP Minutes 09/11/2014|
|Oct. 9, 2014||QEP Steering Committee Meeting||Discussion of Faculty Senate questionnaire results and Student Government questionnaire results; Presentation of existing data to identify student learning gaps; Plan for town hall meetings for faculty and staff||QEP Agenda 10/9/2014; QEP Minutes 10/9/2014; Faculty Questionnaire Results; Learning Gaps Data Presentation|
|Oct. 9, 2014||Community Leadership Council (CLC)||Presentation about the QEP purpose and topic selection process; Polling of CLC members||CLC Questionnaire Results|
|Oct. 22 & 23, 2014||QEP Town Hall Meetings||Intro to Quality Enhancement Planning; Intro to the USFSM Pillars of Intellectual Engagement; Voting of faculty and staff who have not voted in prior meetings||QEP Townhall Questionnaire & Results|
|Oct. 2014||Student QEP Topic Online Survey||Online survey of students’ QEP topic preferences||QEP Student Survey & Results|
|Nov. 6, 2014||QEP Steering Committee Meeting||Review of Town Hall Meeting results – Communication and Critical Thinking were top two choices; Discussion of SGA questionnaire results – Critical Thinking first choice (65%, n=20); Review of CLC poll results; Unanimous vote to select Critical Thinking as USFSM’s QEP topic; Plan for QEP Steering Committee members’ independent reading of texts from Found. for Critical Thinking;Need to build a critical thinking research literature collection – Dr. Hunsader to consult with Diane Fulkerson, USFSM Librarian||QEP Agenda 11/6/2014; QEP Minutes 11/6/2014|
|Dec. 2, 2014||QEP Steering Committee Meeting||Discussion of readings from Foundation for Critical Thinking; Selection of authors for QEP literature review – Drs Richie Reich, Tricia Hunsader, Diane Fulkerson; Drs. Curran and Hunsader to send university-wide email to invite faculty with Critical Thinking expertise or interest to become involved;Dr. Rose to invite Dr. Michael Gillespie to act as liaison between the Core Curriculum Committee and the QEP Steering Committee;
Schedule meeting to plan for SACSCOC Meeting Dec 6-9, 2014
|QEP Agenda 12/2/2014; QEP Minutes 12/2/2014|
|Dec. 3, 2014||SACSCOC Annual Meeting Planning||Planned SACSCOC session coverage for USFSM faculty and administrators attending the meeting|
|Dec. 11, 2014||QEP Steering Committee Meeting||Debrief from SACSCOC Annual Meeting from attendees;Plan to communicate with Critical Thinking QEP director from University of Louisville, Patty Payette who led the QEP workshop at SACSCOC; Drs. Curran, Hunsader, and Jones to meet with Dr. Mike Gillespie, critical thinking researcher and USFSM faculty member, about joining the QEP Steering Committee; Discussed student QEP-naming contest||QEP Agenda 12/11/2014; QEP Minutes 12/11/2014|
|Jan. 9, 2015||Conference Call with University of Louisville QEP leaders||Drs. Hunsader and Jones discussed Critical Thinking QEP with Drs. Edna Ross and Patty Payette of U of L; Recommendation of Dr. Gerald Nosich as faculty professional development leader;Discussion of Paul-Elder framework for Critical Thinking|
|Jan. 15, 2015||QEP Steering Committee Meeting||Unanimous vote to use the Paul-Elder Critical Thinking Framework as the basis of the USFSM QEP; Discussion and selection of draft Learning Outcomes;·Decision to plan spring faculty-staff professional development with nationally known speaker
SGA QEP Committee charged with recommending 3-5 potential names for the QEP by Jan. 27;
|QEP Agenda 01/15/2015; QEP Minutes 01/15/2015|
|Jan. 27, 2015||QEP Steering Committee Meeting||Dr. Gerald Nosich chosen and professional development speaker; Planned for meeting: QEP Committee and faculty teaching critical thinking capstone/pillar courses; Andrew Becht presented six student-suggested names for the QEP; committee blended choices; Incredi-Bull Critical Thinking unanimously voted as QEP name; Dr. Jones to order Critical Thinking booklet from the Foundation for all faculty; Drs. Gillespie and Jones, and Laura Hoffman: subcommittee to develop critical thinking measurement tools; Steering Committee members to share QEP SLOs with their college faculty at next faculty meetings and distribute Critical Thinking booklet||QEP Agenda 01/27/2015; QEP Minutes 01/27/2015|
|Feb. 10, 2015||QEP Steering Committee Meeting||Finalized plans for the QEP Logo Contest; Feb. 20, 2015 – contest announcement March 24, 2015 – semi-finalists selected by QEP Committee; Plans for full-day full-faculty professional development on Aug. 19, 2015 – approved by Core ASA||QEP Agenda 02/10/2015; QEP Minutes 02/10/2015|
|Feb. 24, 2015||QEP Steering Committee Meeting||Invited guests – instructors of all critical thinking capstone courses; Discussion of how critical thinking is taught and assessed in existing capstone courses and how current practices align with the QEP definition and learning outcomes;.QEP Steering Committee to continue to collaborate with these instructors through the process of developing a new critical thinking rubric; Draft of QEP Literature Review by Dr. Reich complete||QEP Agenda 02/24/2015; QEP Minutes 02/24/2015; CT Pillar Assignments;
CT Assessment Rubrics; QEP Literature Review
|Mar. 10, 2015||QEP Steering Committee Meeting||Finalized agenda for April 20 Critical Thinking workshops with Dr. Nosich; Plan for promoting workshop attendance among faculty and staff – series of invites; · Approval of revisions to critical thinking learning outcomes based on discussion with capstone instructors; Dr. Mike Gillespie to obtain feedback from capstone instructors regarding development of expectations for each learning outcome; Finalized Five Learning Outcomes; Discussed QEP Lead Evaluator||QEP Agenda 03/10/2015; QEP Minutes 03/10/2015|
|Mar. 24, 2015||QEP Steering Committee Meeting||Review of April 20 workshop agenda and meetings of key faculty and administrators with Dr. Nosich; Continued promotion of April 20 workshop via personal, face-to-face invitations from committee members; Review of all QEP Logo Contest submissions and selection of finalists; Planned for campus-wide (students, faculty, staff) vote for QEP Logo, April 8 – 15, winner to be announced at April 20 QEP workshop
Critical Thinking rubric out to capstone instructors for feedback on “Meets Expectations” language
|QEP Agenda 03/24/2015; QEP Minutes 03/24/2015|
|Apr. 7, 2015||QEP Steering Committee Meeting||Plan for presentation of QEP progress to Campus Board April 10; Discussion of April 20 workshop logistics and number of attendees (to date, 49 faculty and 30 staff are planning to attend; Several students to be invited this week); Discussion of feedback from capstone course faculty regarding rubric language for “Meets Expectations” – the 3rd set of expectations was most generalizable across courses/programs – starting point for final version; Dr. Gillespie charged with drafting the “Meets Expectations” language for committee approval; Rubric to be piloted fall 2015; Drs. Hunsader and Jones to discuss selection of lead evaluator – SACSCOC Summer Institute and Advisory Visit to inform selection; Campus-wide vote for QEP logo to go live tomorrow||QEP Agenda 04/7/2015; QEP Minutes 04/7/2015|
|Apr. 9, 2015||QEP Budget Meeting||Meeting of Drs. Curran, Hunsader and Jones with Dr. Terry Osborn, Regional Vice Chancellor for Academic and Student Affairs, and Ben Ellinor, Regional Vice Chancellor for Business and Financial Affairs; Plan an additional meeting when the budget is drafted|
|Apr. 10, 2015||Campus Board Meeting||Dr. Hunsader reported on the QEP progress, including: QEP name – Incredi-Bull Critical Thinking; April 20 workshop with national speaker Dr. Gerald Nosich; Selection of 5 measurable learning outcomes; Addition of Dr. Mike Gillespie, USFSM resident faculty member and critical thinking researcher, to the QEP Steering Committee||Campus Board Minutes 04/10/2015|
|Apr. 20, 2015||Critical Thinking Workshop with Dr. Gerald Nosich||Repeated half-day workshops on Critical Thinking attended by 75 USFSM faculty, staff and administrators|
|Apr. 21, 2015||QEP Steering Committee Meeting||Major ah-ha from the workshop: value of the Paul-Elder Critical Thinking Standards for providing feedback to students about their expression of CT; Committee agreed to form a subgroup on PD to internally plan August faculty workshop focused on training in the Paul-Elder CT Standards and the reliable use of the USFSM CT Rubric; A QEP subgroup will meet over the summer to discuss available nationally normed CT assessments||QEP Agenda 04/21/2015; QEP Minutes 04/21/2015|
|May 13, 2015||QEP Subgroup on Professional Development Meeting||Planned full-day full-faculty workshop on the Paul-Elder CT Standards and reliable use of the USFSM CT Rubric to be held Aug. 20, 2015|
|Jun-Dec, 2015||Drafting of QEP||Detailing of QEP Timeline with hyperlinks to key documents;|
|July 19-22, 2015||SACSCOC Summer Institute||Attendance by 6 USFSM administrators, faculty, staff|
|July 22-23, 2015||SACSCOC Advisory Visit with Vice President Dr. Mike Hoefer||Discussion of QEP progress, potential lead evaluators, formation of assessment plan, and QEP budget||QEP Minutes 07/22/2015; QEP Presentation – SACSCOC Advisory Visit|
|Aug. 11, 2015||QEP Steering Committee Meeting||Final planning for faculty workshop||QEP Minutes 08/11/2015|
|Aug. 17, 2015||Faculty Workshop on CT Rubric||Conducted full-day workshop with 40 faculty||QEP Workshop Presentation|
|Sept. 15, 2015||QEP Steering Committee Meeting||Discussed lead evaluator candidates; Discussed nationally normed assessment; formed task force to lead selection; Review/discussion of implementation plan||QEP Agenda 09/15/2015; QEP Minutes 09/15/2015|
|Sept. 22, 2015||Conference call with Connie Wolfe||Drs. Gillespie, Hunsader, and Jones: Discussion of Connie Wolfe’s experience and potential for selection as our lead evaluator|
|Sept. 29, 2015||QEP Steering Committee Meeting||Discussed marketing plan; Vote to invite Ms. Wolfe to serve as lead evaluator; Discussed faculty development plan||QEP Agenda 09/29/2015; QEP Minutes 09/29/2015|
|Oct. 13, 2015||QEP Steering Committee Meeting||Discussed marketing plan and budget; Task force preliminary recommendation: California CT Test – group will examine content and report back; Discussed/ amended Operational Outcomes; Planned for group edit of QEP doc||QEP Agenda 10/13/2015; QEP Minutes 10/13/2015|
|Oct. 22, 2015||QEP Steering Committee Meeting with Lead Program Faculty||Presented draft plans for implementation plan to lead faculty; Faculty provided feedback that will be integrated into the plan|
|Oct. 27, 2015||QEP Steering Committee Meeting||Working through budget approval; Summarized lead faculty input; Task force on national assessment eliminated Calif. CT Test due to dissatisfaction with content; Task force reviewing Watson-Glaser Test as top contender· Planning for ‘I’m Committed’ video as QEP launch at Faculty/Staff Holiday Party||QEP Agenda 10/27/2015; QEP Minutes 10/27/2015|
|Nov. 10, 2015||QEP Steering Committee Meeting||Preview of draft promotional items; Vote to approve Watson-Glaser test as nationally normed CT assessment; Discussed/edited Commitments Survey for students and Professional Development survey for faculty; Discussion of QEP Director and Administrator roles, budget and timeline||QEP Agenda 11/10/2015; QEP Minutes 11/10/2015|
|Nov. 24, 2015||QEP Steering Committee Meeting||Group reviewed/discussed Assessment Plan; Final edits on surveys; Updates on marketing items, Holiday Party, and QEP draft; Mapped plan and timeline for committee to review/edit QEP||QEP Agenda 11/24/2015; QEP Minutes 11/24/2015|
|Dec. 5-8, 2015||SACSCOC Annual Meeting||Drs. Hunsader and Jones presented the session: “Building Synergy for Critical Thinking: Harnessing the Power of We for an Effective QEP” at the annual meeting; Laura Hoffman presented a poster: “Psychometric Properties of Critical Thinking Instruments”; Dr. Sandra Stone, Dr. Mike Gillespie, and Sarah Fayard also attended from USF Sarasota-Manatee||QEP Presentation
|Dec. 9, 2015||QEP Steering Committee Meeting||Discussion of QEP Director and Administrator position descriptions; Reviewed outstanding areas of QEP draft; Debriefed from SACS Conference; Prepped for Holiday Party – stuffed goody bags||QEP Agenda 12/9/2015; QEP Minutes 12/9/2015|
|Dec. 10, 2015||USF Sarasota-Manatee Holiday Party||Screened the “I’m Committed” QEP Video; Provided goody bags and polo shirts with the QEP logo to faculty, staff, and QEP-involved students||QEP Video|
Indicator 4.B. Implementation of the Plan
The success of the QEP depends upon broad-based faculty and staff participation. Below is a summary of the roles each will play in the implementation of Incredi-Bull Critical Thinking.
QEP Director. The QEP Director will provide leadership for enactment of the Incredi-Bull Critical Thinking Implementation and Assessment plans, Chair the QEP Implementation Committee, lead research efforts related to QEP data, supervise the QEP Administrator, manage QEP communication and marketing, and develop annual reports on QEP initiatives. The QEP Steering Committee felt it important to have a faculty member assume the role of the QEP Director with 50% of time devoted to the QEP and 50% of the time to other faculty activities in the College of Arts and Sciences. The QEP Director will begin duties on May 16, 2016 and continue for five years.
QEP Administrator. The QEP Administrator will provide administrative support for the QEP Director and QEP Implementation Committee. Duties will include managing QEP-related databases, website, and social media, purchasing and travel support, producing documents and reports, organizing and communicating across diverse stakeholders. The QEP Administrator will be a half-time position. As the QEP implementation process occurs and duties are better-defined, the QEP Implementation Committee will determine whether or not this position should be expanded. The QEP Administrator begins on July 1, 2016.
QEP Steering Committee. The QEP Implementation Committee will have representation from across campus. The QEP Director will chair the Committee and the QEP Administrator will provide it with support. The Core Curriculum Faculty Coordinator will serve on the Committee to facilitate integration with the Pillars curriculum and assessment. In addition, faculty who are early adopters of the critical thinking pedagogy and assessment methods will represent each of the four colleges. The IR Assessment Coordinator will serve on the committee to ensure assessment support. Two representatives from Student Affairs and two students will provide feedback from their perspectives.The Committee will use information from the QEP Annual Assessment Report to guide adjustments to the Implementation and Assessment Plans moving forward.
Deans. The deans will be responsible for supporting the faculty in their academic units to integrate Incredi-Bull Critical Thinking into their programs. Deans will work with their QEP Implementation Committee representatives to monitor progress toward curricular implementation. Progress on the QEP will be included in unit plans and reports.
Incredi-Bull Faculty. Faculty who aspire to receive the Incredi-Bull Faculty designation will meaningfully integrate critical thinking into one or more of their courses. This designation will be awarded via a peer-review process.
Incredi-Bull Faculty Leaders. Incredi-Bull Faculty leaders who have been selected via a peer review process will conduct a year-long project involving leadership or mentorship in critical thinking pedagogy or scholarship.
Faculty. Faculty involvement in professional development and curricular integration are key to the QEP’s success.
The organizational chart below shows positions and reporting relationships forming the backbone of the QEP.
(3) Goals and Assessment Plan
Indicator 5.A. Goals
The two goals of the QEP and their outcomes are outlined in the diagrams below. Goal 1 is the operational goal that must first be achieved before USF Sarasota-Manatee can achieve Goal 2, which is the student learning outcome goal. For an explanation of how the QEP Steering Committee arrived at these goals, please see Indicators 2.A. and 2.B. in Core Requirement 2.12 QEP (Quality Enhancement Plan).
Indicator 5.B. Plan to Assess
Comprehensive assessment of both operational outcomes and student learning outcomes over the next five years is critical to reaching the two goals of USF Sarasota-Manatee’s QEP, Incredi-Bull Critical Thinking. The process of ongoing annual assessment assisted USF Sarasota-Manatee in identifying critical thinking as the area in need of improvement. Assessment of the QEP will integrate into the already-established USF Sarasota-Manatee Assessment System, using similar processes, timelines, and forms as other academic programs.
The QEP Director is accountable for assessment of the expected outcomes in collaboration with the Assessment Coordinator in Institutional Research & Effectiveness (IR). In addition, the QEP Administrator will track faculty participation in and satisfaction with the QEP. The Core Curriculum Committee, chaired by the Core Curriculum Coordinator, and the Lead Faculty in each degree program already collect student learning outcome data and analyze the information as part of USF Sarasota-Manatee’s annual program assessment, which includes analysis of students’ critical thinking skills. These annual assessment reports will continue to provide outcome data to the QEP Director and the Assessment Coordinator for campus-wide compilation and analysis by the QEP Implementation Committee. As annual assessment proceeds and after discussion of faculty-agreed-upon changes to the QEP plan, the QEP Director will document the changes that resulted from assessment data on the QEP Annual Assessment Report. In collaboration with the Core Curriculum Committee the QEP Director will also document the revisions to any assessment procedures, if needed, to test continuously the impact of the changes on student learning outcomes. Ultimately, the QEP Director will compile the QEP Impact Report to demonstrate how successfully USF Sarasota-Manatee met the QEP Goals.
The assessment tools used to measure the two QEP goals will include quantitative and qualitative assessment, direct and indirect measures, and formative (annual) and summative (five-year) reviews. Through the USF Sarasota-Manatee Assessment System, faculty annually collect student learning outcome data on critical thinking in all undergraduate programs. In addition to these annual degree program reports that document student learning outcomes in critical thinking, the QEP Director with assistance from IR and the QEP Implementation Committee members will compile an assessment report for the QEP each year.
Assessment of Goal 1: Enhance Pedagogical Practices that Focus on Critical Thinking
USF Sarasota-Manatee will use the following means of assessing the operational outcomes:
- Faculty Participation. The QEP Administrator will track faculty attendance at each professional development activity through sign-in sheets and the QEP database. The QEP Director will work with the QEP Implementation Committee and the Faculty Senate’s Tenure & Promotion Committee to develop and implement Incredi-Bull Faculty recognition procedures. When the procedures are in place, the QEP Administrator will track faculty who apply for and receive Incredi-Bull Faculty status and Incredi-Bull Faculty Leader awards, reporting the totals for each annually and at the end of the five years to the QEP Implementation Committee. Faculty and Deans will also document the Incredi-Bull Faculty designation as part of the faculty Annual Review process in the Faculty Academic Information Reporting (FAIR) System.
- Professional Development Feedback Survey. Faculty will complete the Professional Development Feedback Survey at the end of each professional development activity on critical thinking. Faculty use a five-point scale with a range of “Strongly Disagree” to “Strongly Agree” to respond to statements about the professional development activity. The statements cover the goals, length, information, resources, session leaders, etc. Several open-ended questions on the survey provide faculty with the opportunity to comment on a variety of aspects of the session and professional development as a whole on campus. After compiling the results of the survey, the QEP Administrator will share them with the presenters/developers of the professional development activities and with the members of the QEP Implementation Committee after each activity to provide formative feedback. Summaries of the survey results will become a part of the annual QEP Assessment Report and the QEP Impact Report, providing summative feedback on the professional development overall. Please see Appendix G for a copy of the Professional Development Feedback Survey.
- Training Resource Development. The QEP Director will work closely with E-Learning staff and faculty to develop training materials and workshops involving critical thinking pedagogy. The QEP Administrator will ensure that the materials are readily available and maintain an index of the resources, updating it throughout each year of the QEP and summarizing the numbers and types of resources at the end of the five years for the QEP Impact Report. Through the QEP database, the QEP Administrator will also track and summarize any sharing of the resources internally with faculty and outside USF Sarasota-Manatee, such as at professional meetings or in publications.
- Academic Assessment Reports. The QEP Administrator under the guidance of the QEP Director and IR’s Assessment Coordinator will review the critical thinking outcome information in the annual assessment report of each undergraduate degree program. Compiling this program data on the QEP Annual Assessment Report, the QEP Administrator will document faculty’s teaching and learning practices that focus on curricular changes to enhance students’ critical thinking skills. The QEP Implementation Committee will review these results across programs through the annual assessment process. The curricular improvements documented on the QEP Annual Assessment Reports will then form the foundation for the QEP Impact Report in the fifth-year review.
- QEP Annual Assessment Report. A compilation of the reports from the degree programs, showing any curricular changes resulting from assessment of the critical thinking portion of the degree programs, will be a part of the annual QEP Assessment Report. The QEP Implementation Committee will review the annual QEP Assessment Report in the Fall Semesters of 2016-21. As a result of the review, the Committee will make any necessary changes to the QEP operations or curriculum. Assessment of the activities involved with professional development, teaching and learning practices, and review and changes will provide formative feedback at least annually for continuous improvement. The annual QEP Assessment Reports will also provide documented information to assist the QEP Director in the preparation of the QEP Impact Report in Year 5.
Assessment of Goal 2: Enhance Students’ Critical Thinking Skills
USF Sarasota-Manatee will use the following means of assessing the student learning outcomes:
- Incredi-Bull Critical Thinking Rubric. Faculty developed the Incredi-Bull Critical Thinking Rubric in parallel with the Critical Thinking Commitments (adapted from Paul & Elder, 2014). The QEP Steering Committee drafted the rubric in spring 2015 and solicited feedback from faculty teaching critical thinking capstone courses. These faculty reviewed the rubric in view of the capstone course projects currently used to assess critical thinking; they guided revisions to ensure content validity for assessment of the Critical Thinking Commitments and to support its use for capstone projects across programs. Capstone faculty also participated in the development of rubric language to capture what it means to “meet expectations” for each of the five commitments, balancing the competing demands of specificity and applicability to a range of disciplines. Dr. Gerald Nosich’s spring 2015 professional development workshop informed the rubric development and resulted in Paul & Elder’s (2014) nine Intellectual Standards being integrated into the “meets expectations” rubric descriptors. Specifically, each of the standards was aligned with one or more of the commitments; the bolding of the Intellectual Standards within the rubric language will maintain faculty and student focus on these important indicators of quality. The rubric was piloted with 41 faculty during the fall 2015 faculty development workshop, resulting in substantial faculty buy-in and additional minor edits.
Distillation of the Critical Thinking Commitments to simple phrases for promotional materials prompted further revisions. Four of the five Commitments started with a verb, which was then applied to one or more nouns. The fourth commitment was modified so that all five commitments follow a verb – noun structure, using the upper levels of Bloom’s Taxonomy (1956) for the leading verbs. The final result is a collaboratively-developed rubric, grounded in theory, with buy-in from faculty across the University. The Rubric is specific enough to assess the Critical Thinking Commitments, generalizable to diverse programs and courses, and concise with language that can be effectively communicated via University-wide marketing materials.
Faculty in the following programs will pilot the Incredi-Bull Critical Thinking Rubric in Spring Semester 2016: Core Curriculum, Criminology, General Business, Hospitality, and Psychology. Currently, a baseline mean score of 3.0 forms the starting point for targets in Year 1. Following the spring pilot, the QEP Steering Committee can adjust the baseline, as needed. Likewise, the QEP Implementation Committee will adjust targets annually, as needed, based on student performance.
Undergraduate degree programs across campus will phase in the implementation of the Incredi-Bull Critical Thinking Rubric over the five years. In Spring Semester 2016 students taking SLS 2122 Foundations of Professional Success will complete an assignment that faculty will assess using the Rubric to set a baseline. In Year 1, faculty in four degree programs will use the Rubric to assess student critical thinking skills in the capstone courses. That number will rise to eight in Year 2 and progressively increase until all eighteen (18) programs are using the rubric in the capstone courses by Year 5. During Year 3 faculty in at least twelve (12) degree programs will score student assignments in one upper-level course (3000 or 4000) per program in addition to the capstone, increasing to all eighteen (18) programs by Year 5. Through the QEP Annual Assessment Report, members of the QEP Implementation Committee will document and review the scores to determine if student learning is improving and to set new targets, if needed. Within individual courses, faculty will collect student work samples from critical thinking activities/assignments and score student performance using the Incredi-Bull Critical Thinking Rubric. The QEP Administrator in collaboration with IR’s Assessment Coordinator will calculate a mean score on each of the five rubric items and overall. These scores will serve as a baseline to set targets annually. In general, faculty will expect students to score a mean of 3.0 in the first year of the capstone course assessment (Year 1 of QEP) and the first year of upper-level course assessment (Year 3 of QEP). By the end of the five years, the QEP Steering Committee set a 3.5 as the target mean score on all sections of the Rubric and overall.
- Watson-Glaser Critical Thinking Test. In deciding which standardized critical thinking measures to use for assessment, the QEP Instrument Subcommittee of the QEP Steering Committee met several times and considered the following measures: ETS iSkills Assessment and Proficiency Profile, California Critical Thinking Skills Test, Collegiate Assessment of Academic Proficiency, Watson-Glaser, Halpern Critical Thinking Assessment, Cornell Critical Thinking Test, Problem Solving Inventory, Proficiency Profile, and the Critical Thinking Assessment Test.
The selection of the Watson-Glaser Critical Thinking Test occurred for several reasons:
- The questions are relatable to students;
- It is one of the most widely used and validated measures in the workplace;
- It provides targeted feedback and developmental reports;
- It yields normative scores relative to a student population;
- It can be administered offsite in an unproctored environment, if desired; and
- Of the available options, it provided the closest alignment with USF Sarasota-Manatee’s CT Commitments, most notably #s 3 and 4.
The Watson-Glaser Critical Thinking Test, a 40-item multiple choice test, assesses both critical thinking and decision-making. The Watson-Glaser has been shown to be reliable and valid (Watson & Glaser, 2009). Using data from The Watson-Glaser Short form, test-retest reliability has been shown to range from .81 to .89. Similarly, internal consistency reliability of the full scale Watson-Glaser II has been shown to range from .81 to .83. The Watson-Glaser is content valid with respect to Paul & Elder’s (2002) critical thinking model. Confirmatory factor analyses support the three-factor structure (recognize assumptions, evaluate arguments, and draw conclusions). Evidence of convergent validity is shown by strong correlations with established reasoning tests (e.g., .70 with Miller Analogies Test for Professional Selection). The Watson-Glaser correlates moderately with open mindedness (.34), and negatively with Intensity (tendency to express anger) as well as the MBTI Feeling scale (-.27). Criterion-related validity evidence for the Watson-Glaser II includes correlations with supervisory ratings of Core Critical Thinking Behaviors (.25) and Overall Potential (.25).
Students will take the test as part of Orientation when they enter USF Sarasota-Manatee as a new freshman or new transfer student (pretest). Then all students will take the test in their senior capstone courses (post-test). The QEP Steering Committee members envision following the students individually from pre- to post-test, as well as student groups from year to year. Faculty will pilot the Watson-Glaser Critical Thinking Test with students in SLS 2122 Foundations of Professional Success in Spring Semester 2016 when students will review their Profile Reports (Appendix E) as part of the course. Students who enter USF Sarasota-Manatee with 59 or fewer credits hours must take SLS 2122. For purposes of setting targets, the QEP Steering Committee chose the 51st percentile for the first year of Capstone Course post-testing because this score is a little above the national average for each of the three subscales and overall. Then targets go up two percentiles a year over the QEP’s five years. The QEP Steering Committee will adjust these targets, as needed, based on the pilot in spring 2016. In Years 1-5 the QEP Implementation Committee will adjust the targets annually based on student performance during the previous year.
2. Proficiency Profile. The Educational Testing Service’s (ETS) Proficiency Profile measures proficiency in critical thinking in the context of humanities, social sciences and natural sciences. The exam contains multiple-choice questions designed to measure academic skills developed, as opposed to subject knowledge taught, in general education courses. This test provides a norm-referenced measure of overall critical thinking of USF Sarasota-Manatee students as compared to students who take the test nationally. The reliability coefficient reported by ETS is .78, meaning that the test has acceptable internal consistency. For QEP purposes and as a component of general education assessment, students will take the Proficiency Profile in their freshman Core Curriculum course, SLS 1107. As a post-test, students will take it in their sophomore Core Curriculum course, SLS 2122 Foundations of Professional Success.
USF Sarasota-Manatee has used ETS-developed tests to measure students’ critical thinking skills starting in 2008-09 with the test that preceded the Proficiency Profile, the Measure of Academic Proficiency and Progress (MAPP). The scores provided by these previous administrations provided data to assist the QEP Steering Committee in setting targets. USF Sarasota-Manatee’s upper-level students who took the Proficiency Profile in 2011 and 2012 scored in the 40th and 65th percentiles of college seniors nationally, as provided by ETS. In fall 2013 and fall 2014, entering freshmen at USF Sarasota-Manatee scored in the 77th percentile of college freshmen nationally. For purposes of QEP assessment, the QEP Steering Committee decided to pre-test on the Proficiency Profile in the Core Curriculum course SLS 1107 Foundations of University Success. Students taking the Core Curriculum’s SLS 2122 Foundations of Professional Success after they have completed most, if not all, of their General Education curriculum will provide post-test results. In the QEP’s first year, the Committee decided to set the post-test target at the 80th percentile of graduating seniors nationally for students in SLS 2122 who started as a first-time-in-college student at USF Sarasota-Manatee. By the fifth year, these USF Sarasota-Manatee students will be expected to score at the 84th percentile. For transfer students in the SLS 2122 course, the post-test target starts at the 51st percentile in Year 1 and climbs to the 55th by Year 5. The QEP Implementation Committee will make adjustments to targets, as needed, when reviewing the scores annually.
3. Cooperative Institutional Research Program (CIRP) Surveys. The CIRP Freshman Survey and theCIRP Senior Survey provide indirect measures of critical thinking skills by asking students to describe how strong or weak they are in critical thinking and problem-solving skills. The Higher Education Research Institute at the University of California – Los Angeles has administered these surveys nationwide for 50 years. USF Sarasota-Manatee has data from its first two freshman classes who took the CIRP. Of students in the 2013 entering class, 72% described their critical thinking skills as strong compared to 63% who entered in 2014. Therefore, the QEP committee decided to set the QEP’s first-year target as 68%, midway between the two scores. For students’ assessment of their problem-solving skills, 73% of the 2013 students assessed themselves as strong, as did 72% of the 2014 students. Therefore, the QEP Steering Committee set this benchmark at 73% for the first year of the QEP. USF Sarasota-Manatee students will take the Senior Survey for the first time in Spring Semester 2016. In 2014, 85% of the seniors nationally taking the CIRP indicated critical thinking and problem-solving skills as strong. Therefore, the QEP Steering Committee members chose 85% as the target for the first year of the QEP with an increase to 89% in year five, unless the spring pilot suggests different targets based on student performance.
4.Academic Assessment Reports. The same reports described above under operational Goal 1 will document assessment data for Goal 2, as well. The IR Assessment Coordinator will compile Rubric scores across the degree programs and document a composite score on the annual QEP Assessment Reports for review by the QEP Implementation Committee.
5. QEP Annual Assessment Report. Using the same report as described under Goal 1 (e) above, the IR Assessment Coordinator will work with the QEP Director to randomly select student work from each program for QEP Implementation Committee members to review and score independently on the Rubric. As a result of the review, the Committee will make any necessary changes to the QEP curriculum. Again, the annual QEP Assessment Reports will also provide documentation to assist the QEP Director in preparing the QEP Impact Report in Year 5.
The table of Baseline Data and Targets for Operational Outcomes – Goal 1 aligns the five operational outcomes with the corresponding assessment measures, baseline data from Year 0 (2015-16), and targets for each of the five years of the project.
A second table of Baseline Data and Targets for Student Learning Outcomes – Goal 2 aligns the five student learning outcomes with the corresponding academic assessment measures, baseline data from Year 0 (2015-16), and targets for each of the five years of the project.
Finally, the QEP Director in consultation with the QEP Implementation Committee will analyze outcome data and document changes in curriculum and assessment on the Incredi-Bull Critical Thinking Quality Enhancement Plan Annual Assessment Report template starting in 2016-17.
The assessment of outcomes will be an iterative process. During the fifth year of the QEP, the QEP Director in consultation with the QEP Implementation Committee will review the assessment data as a whole and prepare the QEP Impact Report to describe its overall effect in improving student learning outcomes in critical thinking skills. The Committee in collaboration with the University community will determine at that time which elements of the QEP activities to sustain beyond the five years.
SUPPORTING DOCUMENTATION AND EVIDENCE