Annual Report on the Assessment of Student Learning

University Committee on Assessment

Assessment and Teaching Enhancement Center

Emporia State University

October 2002

ESU claims a rich tradition of excellence in the assessment of student learning.  As early as 1922, Kansas State Teachers College offered test development services to measure learning outcomes in the public schools of the state. More recently in 1978, the Faculty Senate adopted a Competency Examination Requirement to ensure that all ESU graduates demonstrated proficiency in basic reading, writing and mathematics skills. ESU's first formal "assessment plan" was developed in 1987 and highly praised by the Kansas Board of Regents in 1988. Ironically, the famous "Assessment Mandate" of the North Central Association Commission on Institutions of Higher Education to make  ". . . assessment of academic achievement . . mandatory" came several months after the ESU approach to assessment was implemented.

During the North Central Association's   1994 accreditation study, the Evaluation Team characterized ESU's approach to assessment as commendable,and stated: Athe assessment program at ESU . . . is working well, has the support of administration, faculty, and students; and has resulted in identifiable improvements of student learning. In 1998, The North Central Evaluation Team characterized ESUs approach to assessment as commendable, and stated: the assessment program at ESU . . . is working well, has the support of administration, faculty, and students; and has resulted in identifiable improvements of student learning. The American Productivity and Quality Center cited ESU as a a Best Practice organization in the assessment of student learning. Since then other recognition has come from such organizations as the American Association for Higher Education, the National Academic Advising Association, and the International Quality and Productivity Center.

ESU places good teaching first among its priorities as an institution of higher education, and we value and even celebrate demonstrated student learning, performance, and achievement. We have campus wide and departmental honor societies, competitions for best papers, best essays, best artwork, and many award celebrations to recognize the academic achievements of our students. Our assessment programs provide an infrastructure which ensures that our academic practices contribute measurably and significantly to these achievements.

We approach our next accreditation study in 2004 as a university with considerable experience and accomplishment in assessment. Assessment is an institutional priority at ESU. We believe that we are approaching what the Higher Learning Commission calls Level Three or the most "mature" implementation of assessment: "Student learning has become central to the culture of the institution and finding ways to improve it is ongoing."

This document updates the status of the assessment of student academic achievement at Emporia State University. Assessment at ESU is diverse and decentralized. The assessment of majors is the responsibility of the faculty in the various departments. Multiple measures of achievement are employed in all academic programs, but there is no mandated set of measures for all majors. The individual reports can be obtained from the ESU network at H:\COMMON\2002 Assessment.

Changes in Assessment

Table 1 updates the data from previous years about the assessment techniques used in the various academic programs.

Table 1

Summary of Assessment Techniques Employed 1988-2002

¹ Self-Reported Data Collection" in the 1988 report included both Student Surveys

This shows the continuance of some trends, but some new directions as well. Two-thirds (or more) of the programs employ capstone experiences, student surveys (or focus groups), and/or alumni surveys (or focus groups), the three most commonly used assessment techniques. In addition, most programs now employ capstone experiences and/or data concerning the placement of their graduates in their assessment of student learning. Also, an increasing number of programs now employ portfolio analysis (48%) and/or exit interviews (43%).

The Levels of Implementation tool distinguishes direct measures of student achievement from indirect measures, and Table 1 points up this distinction. Direct indicators of learning are immediately observable, e.g., pre-and post-testing, capstone courses, oral examinations, internships, portfolio assessments, evaluation of capstone projects, standardized national exams, performance on licensure, certification, or professional exams, and juried performances. Indirect indicators of learning are subsequently observable, and include surveys of alumni, employers, and students, success of students in subsequent institutional settings, and job placement data.

ESU programs employ more indirect indicators, than direct indicators, but most programs use both. This year, a previous decline in the use of direct measures seems to be reversed.

Appendixes A and B list the assessment techniques used in each undergraduate and graduate major.

Tables 2 and 3 show the extent of the use of multiple assessment measures in the various assessment programs. All of ESU's undergraduate and graduate majors are evaluated by multiple measures. More than half of the undergraduate majors employ six or more measurement techniques, a significant improvement since last year. More than sixty percent of the graduate majors employ six or more measurement techniques, also a significant increase since last year.

Table 2

Growth in Use of Multiple Measures in Undergraduate Programs: 1988-2002

Table 3

Growth in Use of Multiple Measures in Graduate Programs: 1994-2002

Campus-wide Activities

Several campus-wide assessment projects either are ongoing or were completed during 2000-2002. Each of these activities has a separate report available which is not included in this document. Assessments of undergraduate advising were conducted in the appropriate schools and colleges, and reports can be obtained from the deans' offices. The 2002 Student Perceptions Survey was completed during the spring semester and is available from the Office of the Vice President for Academic Affairs. The May 2002 Graduating Senior Survey is available from the Assessment and Teaching Enhancement Center. The annual Employment/Enrollment Report is available from the Office of Career Services.

Linkage Study

Recent changes in the University Competency Examination Requirement compelled us to conduct a special study this year. In 1988 (the initial year of the formal Assessment plan), all ESU students were required to demonstrate competency in reading, writing, and mathematics by achieving minimum scores on the Pre-Professional Skills Test (PPST). Over the years, alternative tests have been accepted (for students not seeking teacher certification), and qualified students have been able to fulfill these requirements with high grades in designated courses. Consequently, average scores on competency tests have declined because some of the most "academically talented" students do not take them.

An attempt was made during 2002 to evaluate changes in the students who actually do complete these tests. Most ESU students complete the ACT test before entering college. The ACT provides baseline data concerning performance in the basic skills of reading, writing, and mathematics. With financial assistance from the College of Liberal Arts and Sciences, we were able to obtain a "linkage report" from ACT which compared the performance of ESU students taking the Collegiate Assessment of Academic Proficiency (CAAP) to their earlier ACT scores. These comparisons were weighed against changes in a reference group of all public university students who have completed the two tests. Following are our findings.

ACT analyzed data on 434 ESU students who had completed the ACT and CAAP reading tests.

• 4% of Emporia State University students made lower than expected progress on the CAAP as compared to 9% of the reference group.
• 85% of Emporia State University students made expected progress on the CAAP as compared to 82% of the reference group.
• 11% of Emporia State University students made higher than expected progress on the CAAP as compared to 9% of the reference group.

ACT analyzed data on 551 ESU students who had completed the ACT and CAAP mathematics tests.

• 5% of Emporia State University students made lower than expected progress on the CAAP as compared to 7% of the reference group.
• 84% of Emporia State University students made expected progress on the CAAP as compared to 82% of the reference group.
• 11% of Emporia State University students made higher than expected progress on the CAAP as compared to 11% of the reference group.

ACT analyzed data on 518 ESU students who had completed the ACT and CAAP writing tests.

• 1% of Emporia Sate University students made lower than expected progress on the CAAP as compared to 9% of the reference group.
• 82% of Emporia State University students made expected progress on the CAAP as compared to 80% of the reference group.
• 17% of Emporia State University students made higher than expected progress on the CAAP as compared to 11% of the reference group.

  The evidence provided by this Linkage Study indicates that our general education program is effective in developing basic skills among our students.
 

Levels of Implementation

According to NCA-HLC guidelines, colleges and universities in the maturing stage or Level Three in the implementation of Assessment Plans have reached the point where ". . . assessment has become an institutional priority, a way of life. . . .A 'culture of evidence' has emerged, sustained by a faculty and administrative commitment to excellent teaching and effective learning. Student learning has become central to the culture of the institution and finding ways to improve it is ongoing."

At the department or program level, HLC guidelines require positive answers to the following questions.

• Does the office of Institutional Research provide support for assessment programs at the departmental level? (Level 2)
• Are faculty members becoming knowledgeable about the assessment program, its structures, components, and timetable? (Level 2)
• Are faculty members working together to determine appropriate measures for learning outcomes and to identify improvements based on those results? (Level 2)
•  Does the department chair have the responsibility for maintaining a successful assessment program as a part of her/his formal position descriptions? (Level 2)
•  Does the department chair develop strategies to ensure that the assessment program is implemented? (Level 2)
• Are graduate and undergraduate students represented on the committees that collect and use assessment data? (Level 2)
• Do students, faculty, and staff view assessment activities as a part of the university culture and as a resource and tool for their efforts to improve undergraduate and graduate student learning? (Level 3)
• Has a "culture of evidence" emerged with all decision-making based on or justified by assessment data? (Level 3)
• Do faculty speak publicly and informally to their peers and the institution's other constituents in support of the assessment program and educate others (e.g., newly hired faculty, adjuncts, and part-time faculty and students) about its value? (Level 3)
• Do students reflect upon the work they have produced over the course of their academic program, put their thoughts into writing, and include judgments about how their work does or does not demonstrate attainment of the faculty's expected outcomes? (Level 3)
• Are important decisions based on or justified by assessment data? (Level 3)
• Do faculty and student leaders educate their peers about outcomes assessment through conversations, public presentations, and articles in the student newspaper? (Level 3)

The director of Assessment and members of the University Committee on Assessment reviewed the individual departmental reports. Based on this review: the following programs all show acceptance and implementation of assessment principles, substantial faculty involvement, and clear evidence that assessment data provide the basis for decision-making and other actions. All employ multiple measures, including both direct and indirect indicators of performance. All suggest strong interest in both summative and formative assessments. A commitment to the improvement of student learning is evident from the reports about these majors.
 

Level 3 (or getting close)

Accounting and Computer Information Systems

Business Administration and Education

Counselor Education and Rehabilitation Programs

Music

Nursing

Physical Sciences

School Leadership/Middle and Secondary Teacher Education

Theatre

Level 2 (clear progress toward Level 3) 

Art

Early Childhood/Elementary Teacher Education

Library & Information Management

Psychology and Special Education

All the programs listed above are reaching "the mature stage" that Higher Learning Commission guidelines describe. Many of these majors have attained or are seeking accreditation from professional agencies. We offer one caution even for these excellent assessment plans: students need to be aware of the assessment plan and participate in formative activities where appropriate. In addition, these plans (as well as all others) need regular review and modification when appropriate.

Other departments have plans which can develop into effective assessment programs. We could not place them in the highest categories because they are not yet implemented, some important parts of the assessment activities not completely developed, doubts about faculty ownership of the assessment plan, lack of student involvement, evidence of a reduction in assessment activities, or there is a lack of information about decisions made or actions taken on the basis of assessment data.  We are confident that these programs can soon develop evidence that their assessment programs are working effectively. Level 2 (solid plan & some implementation)

Communication

English

Health, Physical Education and Recreation

University Libraries

Social Sciences

There are also a few departments in which assessment is in its infancy, is progressing at a slow pace, or has stalled. These departments listed below need to move quickly to develop and implement acceptable assessment programs.Below Level 2 

Biological Sciences

Mathematics & Computer Science

Foreign Languages

Instructional Design and Technology

Sociology and Anthropology

  We remain convinced that ESU is ahead of most institutions in the implementation of sound assessment practice. The commitment to good teaching and student learning is strong at all levels. Our experience and accomplishments in assessment are extensive and well recognized. Assessment at ESU is working well. Nevertheless, the annual reviewing of the individual program reports leaves us with some concerns and suggestions for improvement.

 Recommendations

 1. ESU's mission, purpose, and related statements should be revised to meet HLC guidelines. The Institutional Improvement Plan should be revised to include the assessment of student learning and some plan for outcomes assessment in the "performance indicator" currently labeled "Quality teaching and resultant learning."

 2. We need to move straightforwardly toward building the "assessment structures" required by the Higher Learning Commission. These structures are actually necessary for Level Two, not Level Three status. While I believe there is some flexibility in what is required, we have not yet addressed all the structural issues. The administrative leadership of the university should review the HLC guidelines and decide how ESU should implement an appropriate structure for assessment. These guidelines include the following:

 "The administration has enlarged the responsibility of the OIR to include instruction and support to the Assessment Committee, academic unit heads, and academic departmental or program faculty.

 Members of the Assessment Committee serve as coaches and facilitators to individuals and departments working to develop or improve their assessment programs and activities.

 The Assessment Committee is working with unit heads and with faculty and student government leaders to develop effective feedback loops so that information (about assessment results and the changes tried where those results suggest improvement is needed) can be shared with all institutional constituencies and used to improve student learning."

  3. Senior administrators (deans, vice presidents, president) should revisit the decision to reorganize and relocate the institutional research function as ESU.

4. At least some institutional decisions should be based on assessment data. These could include decisions related to hiring, budget allocations, curriculum changes, and approval of new programs.

5. The current policies on student evaluation of teaching should be revised to make the faculty member (not the chair or the FRC) the primary audience for these evaluations. The faculty member should be encouraged to use these and other data when building a case for promotion, tenure, and/or merit increases.

6. Formal position descriptions should be developed for department chairs that explicitly assign an expectation to maintain a successful assessment program.

7. Reports on assessment should become part of a department's comprehensive annual report. The guidelines for these reports should provide a framework that summarizes accomplishments, challenges, and programmatic changes. The impact of the assessment program and related matters should be discussed in the context of the other important activities in the department. It is also essential that these reports be reviewed by the faculty in the department before they are submitted to a higher administrative level.
 

8. Senior administrators (deans, vice presidents, president) should make a concerted effort to enlist the support of the Faculty Senate, the Associated Student Government, and other appropriate ESU groups for building high quality assessment programs. At the least, the faculty and the students need to see the relationship between accreditation and outcomes assessment.

9. At ESU, we recognize good and innovative teaching, technological development, research, creativity, service, and other worthwhile contributions to the academic enterprise. It is time that we take note officially of meritorious contributions in outcomes assessment. Such awards may challenge the view that assessment is something "extra" to do. NCA-HLC guidelines also encourage this recognition. The academic leadership of the university should establish an awards program to recognize publicly individuals, groups, and academic departments for excellence in assessment plans or projects.

 10. ESU is a leader in developing online courses and programs. Recently, the NCA-HLC joined seven other regional accrediting commissions to issue a Statement of Commitment by the Regional Accrediting Commissions for the Evaluation of Electronically Offered Degree and Certificate Programs.The Off-Site and Distance Education and staff should work with the appropriate deans and chairs and the Center for Educational Technology to develop assessment plans for courses and programs delivered via distance learning technologies.

Submitted by Ed McGlone

October 21, 2002

Appendixes A & B

Abbreviations Used

Appendix A

Assessment Techniques

LD AC Locally Developed Achievement Measures

EX EV External Expert Evaluation

NA AC Nationally Standardized Achievement Tests

PE Persistence Studies

PO Portfolio Analysis

CA Capstone Experience

WR Writing Skill Assessment

EX Exit Interviews

ST SU Student Surveys (or Focus Groups or Advisory Committees)

AL SU Alumni Surveys (or Focus Groups or Advisory Committee)

PL Placement of Graduates

EM Employer Satisfaction Surveys

 

Graduate Majors

ACC Accounting

ARTT Art Therapy

BA Business Administration

BE Business Education

BIO Biology

CE Counselor Education

CI Curriculum and Instruction

EA Educational Administration

EC Early Childhood

ENG English

IDT Instructional Design

LIM Library Science (MLS)

LID Information Management (PhD)

MA Mathematics

MUS Music

PE Physical Education

PS Physical Sciences

PSY Psychology

REH Rehabilitation Counseling

SE Special Education

SPY School Psychology

SS Social Sciences

Undergraduate Majors

ACC Accounting

ART Art

BE Business Education

BIO Biological Sciences

BUA Business Administration

CH Chemistry

CIS Computer Information Systems

CO Communication

CS Computer Science

ECON Economics

ES Earth Science

ELED Elementary Education

ENG English

FL Foreign Languages

HE Health Education

HP Health Promotion

IRS Information Resources Studies

MA Mathematics

MGT Management

MAR Marketing

MUS Music

NUR Nursing

PE Physical Education

PH Physics

PS Physical Sciences

PSY Psychology

REC Recreation

REH Rehabilitation Services

SE Secondary Education (Phase)

SS Social Sciences

SOC Sociology and Anthropology

TH Theatre

Assessment Techniques Used in Undergraduate Programs at ESU

Appendix B

Assessment Techniques Used in Graduate Programs at ESU
 

Last Updated November 20, 2008