The institution identifies expected outcomes, assesses the extent to which it achieves these outcomes, and provides evidence of improvements based on analysis of the results in each of the following areas:
220.127.116.11 educational programs, to include student learning outcomes
The faculty at the University of South Florida Sarasota-Manatee (USF Sarasota-Manatee) are committed to a “culture of assessment” and annually gauge programmatic outcomes, analyze the outcome data, and use results to improve degree programs. The use of results establishes the next academic year’s plan and continually improves course content and delivery design, whether face-to-face, on-line, or blended. Faculty also evaluate each degree program through either specialized accreditation reviews (AACSB – every five years; NCATE/CAEP – every seven years) or seven-year program reviews, as required by the Florida Board of Governors (BOG).
The faculty in all four colleges – College of Arts & Sciences (CAS), the College of Business (COB), the College of Education (COE), and the College of Hospitality & Tourism Leadership (CHTL) – under the direction of their deans, conduct academic assessment of their educational programs. Although the exact procedures followed across the colleges differ somewhat, the coordination through Institutional Research & Effectiveness (IR) results in a full academic assessment system across campus that improves student learning.
Student Learning Outcomes
The Florida Board of Governors (BOG) requires learning outcomes assessment (termed Academic Learning Compacts or ALCs) at the undergraduate level as detailed in BOG Regulation 8.016. Internally, USF System Policy 10-060 supports the BOG mandate by requiring ALCs for all undergraduate academic programs (regardless of mode of delivery). USF Sarasota-Manatee’s faculty have also developed learning outcomes for all graduate programs, too. Together this regulation and policy requires each program to:
- outline expected student learning outcomes in the areas of content/discipline-specific knowledge and skills, communication skills, and critical thinking skills;
- develop methods for assessing student achievement of the defined outcomes;
- assess student achievement of the outcomes; and
- use the evaluation results to improve student learning and program effectiveness.
BOG Regulation 8.016, Section (2c) further mandates that the “articulation and assessment of expected core student learning outcomes, as well as program evaluation and improvement, shall occur on a continuous basis.” IR submits the Student Learning Outcomes Assessment Status Report to the BOG every January 31, covering the period from January 1 to December 31 of the previous year.
Annual Degree Program Planning, Assessment, & Accountability
USF Sarasota-Manatee successfully completed its first independent assessment plans in 2009-10 allowing faculty to “close the loop” by making programmatic decisions as a result of assessment. Each year since, faculty determine initiatives needed to change an element of the curriculum. The initiative is documented and subsequently evaluated at the following year’s assessment review. Realizing the value of program assessment, faculty work cohesively with each other to develop rubrics, gather data, and analyze results. With assistance from IR, USF Sarasota-Manatee faculty strive for ‘best practice’ assessment to successfully fulfill program learning outcomes.
Faculty from all four colleges use a similar design template, the Assessment Planning Record Template, when preparing the annual plan and subsequent review of results report each year. While each program is different, all assessment plans share the following common elements:
- Section I – Program mission statement
- Section II – Program/student learning outcomes
- Section III – Means/methods of assessment
- Section IV – Recap of previous cycle analysis results and use of results by learning outcome
- Section V – Current cycle assessment results by learning outcome
- Section VI – Current cycle analysis of results, planned changes, faculty comments, and recorded curricular and assessment process changes
- Section VII – Summary of program improvement initiatives to be pursued in the coming year
- Section VIII – Dean’s response to faculty’s analysis
Each academic program’s faculty established learning outcomes that the curriculum is designed to address that are reviewed annually along with the program’s mission. Faculty select the methods of assessment to measure each program learning outcome for the academic year. They gather assessment data from student performance that is analyzed by IR and put into informational tables, when appropriate. Faculty further analyze assessment results and make curriculum decisions based on assessment cycle results with input from their deans. Faculty evaluate outcome measures and review rubrics to continually improve the development of more refined assessment methods. Multiple assessment measures for learning outcomes, including national comparisons where feasible, provide triangulated data for richer results that aid in curriculum advancement and planning.
Annual Assessment Cycle Timeline
At the start of each assessment cycle (the beginning of the academic year), faculty establish the plan for each learning outcome: the methods of assessment and the measures or levels of expectation. At the close of each assessment cycle, faculty record the following by degree program: assessment results and use of results in program improvement.
|Fall/Spring||Submission of Data by Faculty: Faculty submit to IR, or students input into TaskStream for the College of Education (COE), student learning outcome data gathered as described in the Means/Methods of Assessment portion of their annual assessment plan.|
|Summer||Samples Selected/Scores Calculated: For all but COE, IR prepares clean samples of student work and renders them anonymous with the removal of any personal identifiers. Sample size is representative of the quantity of work submitted. IR selects samples randomly. IR forwards scoring rubrics and the samples to faculty raters for scoring. IR tracks data, such as embedded test scores, on spreadsheets and adds them to the assessment report for comparison to previous year results.|
|Late Summer||Preparation for Faculty Review: IR records rubric scores onto spreadsheets along with calculated statistics (e.g. means, standard deviations) for review by faculty. In COE the assessment coordinator runs reports from TaskStream and inputs results into the assessment reports in preparation for faculty review.|
|Early Fall||Assessment Review: Faculty meet to review the results of the assessment cycle reports. Faculty comment and record initiatives. The college’s dean reviews and comments on each degree program’s results/plans.|
USF Sarasota-Manatee Academic Program Assessment Summary
Below is the documentation of program assessment in each of the active degrees listed on the USF Sarasota-Manatee Degree Inventory 2014-15.
* The AY 2014-15 planning cycle will not be complete until November 2015.
**Applied Science is one degree with concentrations in two colleges.
***Program had first students in 2014-15.
****Program had first students in 2012-13, but did not have enough students or data to assess.
The three inactive programs have a plan annually but did not always have enough students (or any students) to analyze data with confidence.
Early Childhood Education, BS/BA 2014-15 Plan 2013-14 Report 2012-13 Plan
Elementary Education, MA 2014-15 Plan 2013-14 Plan 2012-13 Report
Reading Education, MA 2014-15 Plan 2013-14 Report 2012-13 Report
Please note that faculty assess each of the active graduate and undergraduate certificates at USF Sarasota-Manatee through the corresponding degree programs, because the certificates include courses that are part of the corresponding degree programs. For example, the Information Technology Management Certificate includes courses that faculty review within the BS in Information Technology. Four graduate certificates do not have any enrollment and are inactive. The Reading Education Certificate, when it was active, was a part of the MA in Reading Education assessment. No students have ever enrolled in the certificates in Business Analytics, Health Care Quality Management, or Lean Operations.
Methods of Assessment
A review of the above reports shows multiple methods of assessment used by faculty within the four colleges at USF Sarasota-Manatee. The faculty of the individual degree program areas use the methods of assessment that are most pertinent to measure and evaluate their program. IR posts examples of these assessments on the IR Assessment Tools Webpage to allow faculty to share methods. Most program assessment is completed by a minimum of two assessment scorers using a faculty-developed rubric to measure success of the student learning outcomes. Faculty work with IR to monitor inter-rater reliability.These methods of assessment include the following:
- Cases studies and analyses
- Client reports
- Clinical summative evaluation
- Communications projects
- Completer/Employer survey
- Comprehensive portfolios
- Course-based critical tasks
- Demographic study assignments
- Embedded questions
- ETS Major Field Tests
- ETS Proficiency Profile
- Fieldwork tracking and evaluations
- Final projects in Capstone courses
- Florida Teacher Certification Exams for K-12 (FTCE)
- Florida Value-Added Model (VAM) scores
- Impact on Pupil Performance (IP3) assignment
- Internship evaluations/Fieldwork evaluations
- Internship performance evaluations
- Lab reports
- Major papers
- Mock interviews
- Performance assignments
- Practicum project reports
- Pre- and Posttest embedded questions
- Presentations/Projects (individual and team-based)
- Research proposals/projects
- Senior project
- Surveys such as Alumni/Graduate, Completer/Employer, Exit, Faculty Assessment of Outcomes, Student Self-Assessment
- Tax return preparation
- Transition projects
- Treatment plans
- Wayfind technical assessment
Program Review & Specialized Accreditation
The Florida Board of Governor’s (BOG) considers student learning outcomes assessment (BOG Regulation 8.016) in the broader context of academic program quality. Outcomes assessment is viewed by the BOG as closely connected to Academic Program Review (BOG Regulation 8.015) and specialized accreditation (BOG Regulation 3.006).
The BOG requires the periodic review of all academic degree programs in state universities at least once every seven years from the date of the preceding review or from the implementation date of new academic programs. Program reviews must document how individual academic programs are achieving stated student learning and program objectives within the context of the university’s mission, as illustrated in the academic learning compacts for baccalaureate programs. The results of the program reviews are expected to inform strategic planning, program development, and budgeting decisions at the university level and, when appropriate, at the state level.
USF System Policy 10-062 on Academic Program Review and Specialized Accreditations requires each USF System institution to designate an office “to be responsible for the review of all academic degree programs” and “to establish guidelines for the program review process.” USF Sarasota-Manatee has adopted its own Program Review Guidelines and recommended Program Review Forms for the self-study, the external reviewer, and the dean’s review. The office responsible for program review at USF Sarasota-Manatee is Academic Affairs. USF Sarasota-Manatee had its first two programs undergo program review in 2013-14: Hospitality and Information Technology (IT). In addition to the self-study, the external review, and the dean’s review, these two programs also submitted a final report to the BOG (Hospitality Program Review Final Report; IT Program Review Final Report). The Dean and the faculty used the recommendations and proposed action plans from the program reviews to develop their subsequent academic and unit assessment plans.
Regarding specialized accreditation, the College of Business’ degree programs underwent a sixth-year review by the Association to Advance Collegiate Schools of Business (AACSB), which extended accreditation in spring 2014 (AACSB Accreditation Letter). Likewise, the National Council to Accredit Teacher Education (NCATE) reviewed the College of Education programs and extended accreditation in spring 2015 (NCATE Accreditation Letter). Like Hospitality and IT, the Colleges of Business and Education use the recommendations from their specialized accreditors to inform their annual academic and unit assessment.
USF Sarasota-Manatee’s Program Review Timetable 2014-2021 indicates scheduled program review and specialized accreditation review for each degree program at USF Sarasota-Manatee.
Use of Results Summary
The 2014-15 plans and assessment history for 2013-14 and 2012-13, posted on the IR Academic Assessment Webpage, document the faculty’s report findings each academic year and subsequent curricular and assessment changes implemented by faculty for the following academic year. The reports further illustrate how faculty evaluate changes and modify the curriculum based on student performance.
College of Arts & Sciences – Use of Results
Unsatisfied with the student effort on two key projects, a Client Report and a Treatment Report, faculty in the Communication Sciences and Disorders program revisited these two assignments. A change was implemented to improve the low scores. Faculty developed master files in the course management system, Canvas. As a change to the assessment process in the 2014-15 Assessment Report – Communication Sciences and Disorders academic year (AY), faculty aligned the outcomes and their descriptors to convey more specific assignment expectations. This action served to strengthen an element of the assessment process compared to the results of the 2012-13 Assessment Report – Communication Sciences and Disorders. On the 2013-14 Assessment Report – Communications Sciences and Disorders, the Client Report mean was 3.7 compared to the previous year mean of 3.0. The Treatment Plan showed an improved mean from 2.8 to 3.3, demonstrating improved student learning on this facet of the curriculum.
College of Business – Use of Results
The College of Business (COB) at USF Sarasota-Manatee achieved successful reaccreditation by the Association to Advance Collegiate Schools of Business (AACSB Accreditation Letter) in 2014. Faculty employ methods of assessment that align with both AACSB and SACSCOC to ensure students meet Assurance of Learning (AOL) goals.
Example 1: The course-embedded examination results in the Managerial Decision Analysis course, QMB 6305, revealed deficiencies in data mining and data analytics on the part of MBA students. Therefore, the MBA program offered a new Information Systems (ISM) graduate level course entitled Business Analytics and Big Data Management to cover the topic areas in quantitative analysis, decision-making, critical thinking and information systems in Spring Semester 2014. Faculty believed this change would improve student performance in the graduate AOL Goal V: Operational Business Decision-Making. In the 2012-13 Assessment Report – MBA the mean score on this goal was 3.4. On the 2013-14 Assessment Report – MBA, there was a modest improvement in the mean (3.6). Faculty will continue to monitor this goal.
Example 2: Faculty implemented an oral communication and community engagement opportunity in the Strategic Management and Decision Making capstone course, GEB 4890. This change was based on assessment results in oral communication and student feedback from the 2012 mock interview pilot, which showed students lacked skills in this area. The mock interview experience allowed students to demonstrate oral communication skills in an interview-like setting. To improve upon the pilot experience, faculty embedded a formal business interview with human resources professionals from the community into the capstone as a requirement for all students. The University’s Career Center provides assistance to faculty in the coordination efforts of the mock interviews. The length of interview time was extended from ten to 30 minutes to allow deeper analysis of student communication skills. The involvement of local human resource professionals established relationships with the business community. Human Resource professionals provide valuable feedback to students. Mean scores in Content, Organization, and Delivery greatly improved from 2011-12 to 2014-15 as the table below reflects.
|Capstone Mock Interviews
GEB 4890 Embedded-Assessment
College of Education – Use of Results
The College of Education (COE) at USF Sarasota-Manatee achieved continued accreditation by the National Council for Accreditation of Teacher Education (NCATE) in spring 2015. Faculty use methods of assessment that align with both NCATE and SACSCOC requirements. The NCATE Accreditation Letter identified the standard on assessment as an area of strength for the COE, demonstrating that the faculty go beyond typical expectations for academic assessment of student outcomes.
In spring 2015, COE and IR streamlined the reporting of the varied assessment methods. Data, now arrayed in tables, enables faculty to observe several years of data trends at a glance. The improved assessment reporting method currently includes 2011-12 through 2014-15, and will be continued going forward. Faculty consensus is that the improved reporting procedure enables faculty to better recognize data trends that identify areas where the curriculum can be enhanced to improve student performance.
College of Hospitality and Tourism Leadership – Use of Results
The degree programs of Hospitality and Information Technology (IT) in the College of Hospitality and Technology Leadership (CHTL) underwent program review in 2013-14. As a result, the faculty expanded their methods of assessment to include leadership, ethics, and diversity to better measure these learning outcomes. Note that IT moved to the College of Business effective 2015-16.
Example 1: In the 2012-13 assessment report for the BS in Hospitality Management, the Hospitality faculty developed a new program mission statement and learning outcomes. This allowed the curriculum to better align the program with the standards of the professional accreditor, the Accreditation Commission for Programs in Hospitality Administration (ACPHA). The focus on diversity, ethics, and leadership was added to better align to the College’s mission. Curricular changes included removal of three prerequisite courses that duplicated similar courses. To improve student learning in the Marketing course, HFT 3003, Intro to Hospitality and Tourism was designated as a prerequisite for HFT 3503, Hospitality Marketing and Sales. Faculty consensus is that HFT 3003 provides a firm understanding of the hospitality industry for students. Faculty revised multiple assessment tools and rubrics to align to the new student learning outcomes. Faculty revised and implemented the new Course Learning Outcomes Evaluation, an Internship Evaluation, and the capstone exam in 2013-14. Faculty used the capstone exam, internship evaluation, learning outcome evaluation, and alumni survey to assess the outcome in 2013-14. As a result of the 2014-15 assessment plan faculty are monitoring the scores of the capstone assessment exam. They are also reviewing the case study assignments for rubric alignment. aculty consensus is that the assignment will require students to support decisions with facts.
Example 2: The MS in Hospitality Management program had its first two graduates in 2012-13. Due to the low number of graduates, faculty delayed the assessment process until 2013-14 when more data on graduates became available. In 2013-14, faculty added case studies to HFT 6596, Marketing, and HFT 6477, Finance. During this period (2012-13 and 2013-14) Marketing and Finance assessment included embedded questions and student self-assessment. Faculty decided in 2013-14 to add case studies to the assessment process in lieu of embedded questions to strengthen the support for student learning outcomes by having faculty rubric score the case studies. The case studies are used to measure the student learning outcome of communication. Further, faculty revised the learning outcome of ‘Quantitative Business Analysis’ to ‘Research Methods: ability to apply the appropriate procedures and research methodologies within the hospitality industry’ for clarity. A change to the assessment process for 2014-15 is that faculty will assess this learning outcome using HMG 6586, Research Methods for Hospitality and Tourism, by assessing the final research proposal and student self-assessment developed for 2014-15.
USF Policy 10-065, Credit Hours, establishes criteria used by USF Sarasota-Manatee’s Academic Programs Committee (APC) for approving the online delivery of a credit-bearing course. APC is the Faculty Senate’s committee that reviews all curricular changes. These criteria for online courses include academic engagement and student learning outcomes for distance education courses that are comparable to the traditional delivery of the courses/programs, or identified, comparable courses/programs. Additionally, USF Policy 10-065, Sections II(F) 3 and 4, establish requirements for assessing and reporting on comparability of student learning outcomes and for having procedures in place for the ongoing design of online courses/programs to ensure comparability with the traditional delivery of the courses/programs.
Currently, the BS in Communication Sciences and Disorders is the only degree program fully online. It goes through the same assessment process as degrees that are classroom-based.
Initiatives by USF Sarasota-Manatee resulted in better support of faculty teaching online. In fall 2013 instructor content migrated from Blackboard to the Canvas learning management system. E-Learning developed and offered multiple training opportunities to assist faculty efforts. Areas of support includeinstructional support,faculty resources,student resources, ande-learning workshops (QM example). Examples of support includestaff support,guides,tutorials, and ablog.
After using the Sloan C Online Learning Consortium with mixed results, USF Sarasota-Manatee committed to the process of Quality Matters in 2014-15 to ensure distance education quality. Quality Matters is a recognized national standard, comprehensive, collaboratively developed and maintained, collegial with faculty support, connected, based in research literature, and a continuous improvement model. All USF Sarasota-Manatee major programs and certificates conduct student learning outcomes assessment regardless of mode of delivery of the courses or programs. Principles of sound practice in the assessment of student learning outcomes established for USF Sarasota-Manatee’s traditional education programs/courses ensure that the online delivery of a course/program is comparable to the delivery of instruction in a traditional format.
USF Sarasota-Manatee implemented a one-year pilot, in two phases, of Quality Matters as a result of faculty/E-Learning Services task force. Quality Matters is a faculty-centered, peer review process that is designed to certify the quality of online and blended courses. The Quality Matters standards assure that the online components of these courses promote learner engagement and provide students with all the tools and information needed to be successful learners. USF Sarasota-Manatee is working to incorporate Quality Matters into all phases of its course design processes and to develop the capacity to conduct unofficial and official course reviews. The Assistant Director of E-Learning coordinates USF Sarasota-Manatee’s Quality Matters endeavors. In addition to being the E-Learning specialist, she is also a certified Quality Matters Peer Reviewer. Certified reviewers have taught online for more than two academic semesters and have taught within the most recent 18 months in addition to completing two Quality Matter courses pertaining to the QM Rubric and Peer Review process. Quality Matters will aid all four colleges in the University’s obligation to ensure both the continuous improvement of online/hybrid course design and equality in program quality regardless of mode of delivery (face-to-face, online, and hybrid).
Phase 1, faculty development, consists of the QM Rubric and Peer Review Course. The QM rubric is a set of standards by which to evaluate the design of online and blended courses. The Peer Reviewer Course is designed to prepare experienced online faculty to become Quality Matters Certified Peer Reviewers. The Peer Reviewer Course includes a review of Quality Matters, practice critiquing and writing helpful recommendations, and a Practice Review in which the participants are asked to review Specific Standards in an online course using QM’s Course Review Management System.
In phase 2, course reviews, Internal Course Review Teams conduct internal reviews for qualified courses. Eligible courses are those that have been taught for at least two semesters online. Those which have successfully undergone internal review, are submitted to QM for Official Course Review.
In 2014-15, 18 faculty and staff received Quality Matters “Applying the Rubric to Online Courses” certification, the first course in the certification process. In 2015-16 E-Learning Services plans to continue the Quality Matters implementation, create internal capacity to conduct faculty-driven peer reviews of online course design, and achieve the 2015-16 benchmarks of the QM Implementation Plan. Further, E-Learning staff are creating a CANVAS Getting Started Module based on Quality Matters Rubric best practices for use in online courses. This will be available to all USFSM faculty in 2015-16.
In 2014-15, steps taken to improve distance learning include:
- The Quality Matters Implementation, development of E-Learning Services Guidelines, processes, and best practices.
- A SharePoint Document Library to maintain a digital library of resources.
- Joint professional development sessions (DLASC Minutes: 4/20/2015, 6/16/2015) for E-Learning Services and Student Disability Services staff on ADA compliance and closed captioning for online videos.
- A reorganization of E-Learning Services, creating Assistant Director and Instructional/Multimedia Developer positions.
Additional information on the planning and evaluation process is provided in Core Requirement 2.5 (Institutional Effectiveness). Information on college level competencies and student achievement is provided in Comprehensive Standard 3.5.1 (General Education Competencies) and Federal Requirement 4.1 (Student Achievement).
SUPPORTING DOCUMENTATION AND EVIDENCE