Assessing Student Learning at ESU: A Report from the Assessment Task Force

Emporia State University has a long tradition of involvement in assessment activities dating back to its establishment of the State of Kansas Scholarship Test program in 1922. As increased attention to assessment issues has been paid by administrators, governing boards, and commentators on higher education, ESU has remained in the forefront of the institutions concerned about understanding and improving student learning.

ESU was the first post-secondary institution in Kansas to establish a formal university-wide Assessment Plan in 1989. In 1994, the Evaluation Team for the North Central Association re-accreditation of ESU characterized the university's approach to assessment as "commendable," and stated: "the assessment program at Emporia State University . . . is working well, has the support of administration, faculty, and students; and has resulted in identifiable improvements of student learning."

If the years immediately preceding the development of the first formal plan are characterized as "developmental," then the period from 1989 to 1994 was one of "implementation." The current phase is one of "refinement," as the university has evolved a more mature and less of a public relations view of assessment. The faculty, staff, and administration believe in the value of assessment. They seek to focus on issues related to student learning, rather than simply meeting governing board or accreditation association requirements.

ESU's assessment program as it has evolved to the present can be characterized as by the following principles:

1) There are institution-wide commitments to attempt to understand what takes place when students learn and to improve the learning experience for students at ESU.

2) It is recognized that different approaches to assessment are appropriate at different levels and in different settings. Thus, a division chair may personally interview graduating seniors about their academic experience while the central administration conducts a telephone survey of a sample of the entire student body to ask about instructional effectiveness. Even in the same academic unit, different assessment techniques may be used for different programs.

3) The success of the assessment program is indicated by changes which attempt to improve student learning as well as by changes which can demonstrate such improvement. Experimentation and special study do not always produce solutions for problems, but they do signify a willingness to seek solutions.

Thus, assessment at ESU is diverse and decentralized. The assessment of majors is the responsibility of the faculty in the various divisions. All divisions use multiple measures of achievement, but there is no mandated set of measures for all majors. (See Appendixes A & B).

ESU does require that all undergraduate students must take and pass competency examinations in reading, writing, and mathematics. The university also conducts surveys of recent graduates, of placement after graduation, and of employer satisfaction with ESU graduates periodically. A Student Perceptions of Instruction study is conducted every two years. Data from these studies are provided to the academic units. In addition, special studies are also undertaken to assess achievement in particular areas, e.g., the development of critical thinking, knowledge about and sensitivity to multi-cultural issues, and computer literacy. All programs which are eligible are encouraged to become accredited and to meet the assessment requirements of the appropriate accrediting agency.

Task Force on Assessment

A Task Force on Assessment was appointed by Vice President John Schwenn in November of 1997 to evaluate the current status of assessment at ESU and to make recommendations for any needed changes by the end of the Spring 1998 semester. The members of the Task Force are Herbert Achleitner, School of Library and Information Management; Rob Catlett, School of Business; Eileen Hogan, Teachers College; Connie Schrock, College of Liberal Arts and Sciences; Rod Sobieski, Director of General Education; Terri Summey, Division of Library Services; and Ed McGlone, Director of Assessment and Teaching Enhancement Center and Chair of the Task Force.

The Task Force developed and administered a questionnaire to all academic divisions. Appropriate administrative units also received the questionnaire. Some of the material in this report is based on data received in response to the questionnaire. Appendix A contains a copy of the questionnaire. The members of the Task Force also reviewed relevant documents and reports and interviewed faculty and administrators in regard to the assessment program.

Changes in Assessment since 1994

On May 17, 1994, the University Planning Committee approved a document entitled "Addendum to the Assessment Plan for Emporia State University" (a.k.a. "Revisions of the Assessment Plan") which attempted to codify what had become "customary practice on campus." This document reviewed the changes that had resulted from the actual implementation of the 1989 Assessment Plan. It placed assessment in the context of the university's 1992 mission statement, emphasizing the centrality of achieving general education goals for all students and summarizing the assessment techniques used in both undergraduate and graduate programs. Special attention was given to the assessment-based "refinements" of the general education curriculum that had occurred. The document emphasized the importance of data gathering and analysis in the individual academic divisions, at the college and school level, and in the General Education Council. Near the end of the document, the divisions were " . . . formally authorized to extend the variety of assessment options beyond those that they had proposed in previous sections."

Table 1 updates the data reported in the 1994 document about the assessment techniques used in the various divisions. It indicates the continuance of some trends, but some new directions as well. More and more academic divisions are developing their own measures of program achievement and using nationally standardized measures as well. Student and alumni surveys (or focus groups) continue to be used in the overwhelming majority of programs. There is a decline in the use of persistence studies and portfolio analysis. Most significantly, different techniques have been developed and are being used across the campus. A majority of programs now employ capstone experiences, writing skills assessment, and data concerning the placement of their graduates. Nearly a majority use exit interviews and employer surveys in their program assessment.

Table 1
Summary of Assessment Techniques Employed

* "Self-Reported Data Collection" in the 1994 document included both Student Surveys and Alumni Surveys.

**Technique not included in the 1994 document.

Tables 2 and 3 also update data reported in the 1994 document and show the growth in the use of multiple assessment measures in the various programs. All but one of ESU's undergraduate majors and all of the graduate majors are evaluated by multiple measures. A majority of the undergraduate majors and almost that proportion of graduate majors employ six or more measurement techniques.

Table 2
Growth in Use of Multiple Measures 1988-98
41 Undergraduate Programs

Table 3
Percent of 17 Graduate Programs Using Multiple Measures 1994-98

The 1989 Assessment Plan was developed with an emphasis on improving undergraduate education. By tradition graduate programs have had a strong evaluation and assessment history. Traditional practices have focused on rigorous entrance requirements, qualifying exams, master theses, dissertations as well as individual mentoring. More recently, graduate programs have added formative course corrections, self assessment, diagnostic exams, capstone courses, and multimedia portfolios. The new challenge is to devise assessments for web-based courses and programs and for other mediated programs that account for the unique electronic learning environment.

Attitudes Toward Assessment

Data from the questionnaire administered in January and February of 1998 provide evidence that the commitment to assessment at ESU remains strong. Table 4 summarizes the responses in the general evaluation section of the questionnaire.

The overwhelming majority of respondents believe that assessment activities are supported by the Board of Regents, administrators, and the faculty. Eighty-two percent either agree or strongly agree with the statement that the "our faculty are actively involved in assessment activities." Fifty-nine percent believe that assessment activities have affected student learning, and 77% believe that assessment has affected the quality of our academic programs. Eighty-seven percent view assessment as affecting the accreditation of our programs. Sixty-nine percent agree or strongly agree with the statement that "ESU's overall approach to assessment is generally effective."

The data also suggest that assessment is not seen as strongly related to such matters as student recruitment, student retention, funding for the institution, and public recognition of ESU. In each of these areas, a majority or significant number of respondents marked "don't know/no opinion" about the effect of assessment. The same response was the most popular in response to the statement, "our institution invests too many resources in assessment."

Table 4

Division Responses to General Evaluation

(22 Divisions and Offices Responding)

Another part of the questionnaire sought opinions about how the results of assessment data are used at ESU. A strong majority of the respondents said that assessment data are used in planning and are communicated widely at the division level. Similarly strong majorities indicated that assessment activities affect the emphasis on student learning and curriculum development and revision at the division level. Slight majorities indicated that assessment is used in budgeting at the division level and is communicated widely and is used in planning at the school/college level. For the most part, however, respondents did not know how assessment activities affect these matters at the university level.

Actions Taken Because of Assessment

Many actions that have been taken as a result of assessment affect the curriculum. Portfolio assessment combined with surveys of student computer literacy and multi-cultural sensitivity have resulted in the following initiatives on a university-wide basis.

1) The general education curriculum has been modified to require course work which emphasizes knowledge of multi-cultural issues.

2) The General Education Council has proposed that graduation requirements be modified to include course work which develops computer literacy.

3) The university's competency examination requirement has been modified to permit alternative ways in which high achieving students can demonstrate proficiency in reading, mathematics, and writing.

4) All academic divisions have been encouraged to develop writing-intensive courses.

5) The General Education Council, in consultation with faculty in Biology, Teacher Education, and Psychology have launched a longitudinal study of correlates of critical thinking among ESU students.

6) The university has received a major grant from the U.S. Department of Education to support the assessment of the effectiveness of multi-cultural/diversity education.

At the Division level, there is a long list of changes that have occurred because of the analysis and discussion of assessment data. Following are twenty selected from this long list.

1) Revised student evaluations to reflect program goals more directly.

2) Establishment of exhibition and recital programs for students and alumni in the performing arts areas.

3) Renewed emphasis on student writing in lower division courses; requirement that undergraduate papers follow APA or MLA style format.

4) Incorporation of computer technology in courses in a number of non-science and science majors.

5) Improved advising checklists and program information sheets.

6) Revision of course requirements to increase flexibility and opportunity for interdisciplinary study.

7) Development of new courses and curricula to meet student needs in such area as theatre management, public relations, and space science.

8) Offering tutors and tutoring assistance for "difficult" courses; reorganization of developmental math courses.

9) Formation and rejuvenation of student organizations with an academic focus.

10) Field trips to laboratories, industrial organizations, and other educational institutions.

11) Development of on-site teaching experiences before student teaching in cooperation with local schools.

12) Addition of undergraduate research requirements

13) Experimentation with various instructional styles

14) Increased admission requirements in some majors

15) Addition of writing assignments in nearly all general education classes

16) Revision of syllabuses and readings to give more emphasis to women and people of color.

17) Provision of additional computing facilities for students.

18) Addition of foreign language requirements and electives.

19) Deletion of unnecessary and outdated courses.

20) Offering additional courses via Internet and other distance learning systems to meet needs of place-bound students.

In additional to curricular changes, the experience with assessment at ESU has led to a number of changes in the way we approach the assessment activity itself. The following exemplify some current trends.

1) Eight of the twenty-one academic divisions now have formal Assessment Committees that meet and report on a regular basis.

2) While survey data are still collected and used, several divisions and the General Education Council are now using focus groups to obtain assessment data from students, alumni, and other constituency groups.

3) Capstone experiences have been designed in a number of undergraduate programs. Most of these emphasize undergraduate research or the synthesis of the entire academic experience.

4) Many chairs and assessment committees are now conducting exit interviews with graduating seniors to seek their advice about how the learning experience can be improved.

5) Some divisions have formed Advisory Boards consisting of potential employers, alumni, and others who also review assessment data.

6) The General Education Council has sponsored forums on such assessment issues as "enhancing student learning" and "measuring critical thinking."

Recommendations

While approaches and techniques still evolving, our view is that assessment at ESU is working well. The overwhelming majority of the academic divisions have adopted assessment as an important part of their efforts to improve the environment for learning. Central and school/college administrative support for assessment remains strong. Our recommendations are aimed to "fine tune" rather than significantly redirect ESU's approach to assessment.

1. Any modification of the university's Assessment Plan should retain the distinctive feature of decentralization of the responsibility for conducting assessment activities to the division level. While reporting requirements (to the Board and other external agencies) must be met, the focus on understanding and improving learning is best maintained when the faculty have a large degree of freedom to devise their own approaches to assessment.

2. While one price paid for decentralization is a certain amount of ignorance about what is happening in other parts of the university, the central administration should clarify and communicate widely the role of assessment in such matters as budgeting and institutional planning.

3. Academic units who are offering courses and programs via distance learning technologies should give special attention to assessment issues. If existing assessment methodologies are not adequate or appropriate for these programs, then new types of assessment should be developed.

4. The Assessment Office should issue an annual report on the state of assessment practice among the various divisions. The report should include data on what techniques are being used and give special emphasis to innovative practices and other significant changes in practice.

5. The Vice President for Academic Affairs should sponsor an Annual Forum on Assessment in which appropriate issues are discussed and improvements in practices are explained.

6. Consideration should be given to assigning the responsibility for ongoing evaluation of assessment activities to a standing committee or council (e.g., the General Education Council.) While decentralization of decision-making authority should be retained, the quality of assessment practice across campus should be monitored by a faculty body.

Submitted by the Assessment Task Force:

Herbert Achleitner, School of Library and Information Management

Rob Catlett, School of Business

Eileen Hogan, Teachers College

Connie Schrock, College of Liberal Arts and Sciences

Rod Sobieski, Director of General Education

Terri Summey, Division of Library Services

Ed McGlone, Chair and Director of the Assessment and Teaching Enhancement Center

 Appendix A:  Assessment Techniques Used in Undergraduate Programs at ESU

 Appendix B: Assessment Techniques Used in Graduate Programs at ESU

 Appendix C: Survey Instrument

Last Updated March 19, 2007