Indirect Measurement

In a previous post, I explained how student learning outcomes can be assessed with “direct measures.”  Direct measures are assessments of student-produced materials, such as essays, projects, exams, and clinical demonstrations.  In this post, I will describe another type of assessment data called “indirect measures.”

Indirect measures are assessments of student learning without the use of student-produced materials.  Usually, these are surveys, exit interviews, syllabi or assignment reviews, and so on.  Since the evidence for student learning in this type of assessment is “indirect,” it is difficult to really tell if the student achieved the learning outcome.  Instead, these assessment types provide related but not direct evidence of student learning.

Indirect assessment methods rely on self-reported information, surveys, and reviews of related content.  Because of this, indirect assessment data provide evidence of student learning that is less certain.  Nevertheless, indirect assessments can provide a broad view of student learning.

Examples of Indirect Measures

Surveys

School Surveys – Students.  Many schools and classrooms create their own surveys, and they often ask students about their own learning.  For example, an end-of-semester course evaluation may ask students “How much do you feel you have learned in this class?” and students may respond that they have learned “a lot” or “not very much,”  or they may rank their learning on a scale of 1-5.  This data can be useful because it indicates student perceptions of their own learning.  However, the data is limited because it is self-declared and not measured objectively.  In other words, how much a student “feels” or “thinks” he or she has learned may not be supported by more objective data such as the student’s grades.

A couple common student surveys within the classroom are the Minute Paper and the Muddiest Point.  Both of these are end-of-class reflection activities that ask students to identify the most important and the most confusing things they learned.  The instructor uses these responses to provide follow up information and to clarify confusing material either at the start of the next class or by email after class.  Both of these activities are very useful, but they rely on self-reported information, so they do not provide objective data on whether the student truly knows that information (the way a quiz or short essay about that content would indicate).

Another type of student survey that provides indirect assessment data is an Alumni Survey.  An alumni survey is completed by students who have graduated, and the survey often asks students to reflect on the usefulness of their studies.  Common alumni surveys are at six months, one year, three years, and five years after graduation.  A sample question may be “How prepared were you to use the Teamwork Skills (or Critical Thinking or Scientific Reasoning or Communication Skills) you developed at our college (or your academic program) at your current work or graduate program?”  Again, this information is indirect because it relies on a self-assessment rather than on an objective measurement.  Nevertheless, the graduate has hopefully had enough experience in the real world by the time of the survey to determine how prepared or unprepared he or she was as a result of the academic program.  Even if the self-reported information does not truly indicate how knowledgeable or prepared the graduate was, it can still demonstrate how prepared and confident the graduate feels about his or her abilities.

School Surveys – Employers.  Many schools also frequently create Employer Surveys that are completed by workplace managers.  These surveys are often paired with alumni surveys, and they ask managers the same questions they ask recent graduates.  The goal of this assessment is to gain another (hopefully somewhat objective) perspective of the graduate’s abilities.  For example, the employer survey may ask “How prepared do you believe the graduate was to use Teamwork Skills (or Communication or Scientific Reasoning, etc.)?”  When the results come in, it’s useful to compare the employer and alumni surveys.  When there is agreement, then the question probably presents an accurate assessment of the graduate’s abilities.  When there is disagreement, then there may be a difference in what the graduate learned and what the employer was expecting.

National Surveys:  NSSE, Noel-Levitz, Campus Climate.  Schools also use surveys created by professional research organizations – the National Survey of Student Engagement (by Indiana University) and the Noel-Levitz College Student Inventory (by Ruffalo Noel Levitz) are two of the most common.  These surveys also rely on self-reported information rather than on direct, objective measurement, but they are very useful because the surveys have a high degree of reliability and a large pool of data for comparison at the state or national level.  For example, according to the NSSE information page, over 1,600 institutions have used the NSSE, and “approximately 6 million students have completed the survey.”  This large pool of data can provide a lot of comparisons for your students and academic programs, and schools can use this information to determine program quality and student preparedness and to provide information for recruiting and marketing.

Interviews

Exit Interviews and Focus Groups.  Another type of indirect assessment is an interview, which is often completed individually through an Exit Interview or in a group setting through a Focus Group.  An exit interview is often conducted at the end of an academic program when a student is about to graduate.  College staff or faculty meet with a student individually in person (or they ask the student to complete a questionnaire with open-ended questions), and they ask the student to reflect on his/her learning, on the academic program, on the resources and student support services, and so on.

A focus group is a similar interview, but the staff or faculty member conducts open-ended questions with a small group or with a full class of students.  The focus group may asks students about an individual assignment, skill, course, or instructor, or it may ask about the academic program and student support services more broadly.  Both types of interviews ask students to self-report their opinion, experience, or perception, so there is no direct, objective assessment.  Nevertheless, these assessment methods can be useful because they can provide both specific, individualized and broad, generalized feedback directly from students on a number of topics.  After the interview sessions, the staff or faculty identify recurring themes or talking points, and they make conclusions about program quality and improvements on the basis of this aggregate feedback.

Review of Documentation and Materials

Transcript Review.  Surveys and interviews ask students, graduates, and employers for their opinion, reflection, experience, or perception pretty directly.  Meanwhile, a Transcript Review (and other forms of materials review described below) assess student learning through an analysis of a document.  A transcript review is a very common process with transfer students, and it is often conducted by the Registrar’s Office, Advising Office, Admissions Office, or an academic department.  In a transcript review, the college faculty or staff member makes conclusions about student learning by examining course titles and grades on a student transcript.  For example, an academic advisor may see that the student received an A in College Writing I at the University of Illinois and conclude that the student has effective writing skills and does not need to enroll in additional writing courses.

Materials Review.  A Materials Review is a general term for an indirect assessment of student learning that uses various forms of documentation, such as a syllabus, homework assignments, lesson plans, and reading material.  In a materials review, the faculty or staff member tries to develop a comprehensive understanding of what the student has studied to determine if the student meets the learning outcomes.  This form of review is often conducted together with a transcript review when the college is not familiar with the transfer school’s academic program.  It is also often conducted for a new articulation agreement with another school – college faculty or staff review course documents to verify that incoming students are receiving adequate levels of instruction and assessment.

The materials review can also be conducted by an accrediting agency during a “site visit,” or it can be conducted internally by faculty and administrators as part of an internal program review.  In all of these cases, the reviewers examine syllabi, assignments, activities, lesson plans, and any other material that is available to determine if the course or program adequately and effectively promotes student learning.

Conclusion

Indirect assessments do not provide specific, direct evidence of student learning.  Nevertheless, the information that is gathered from surveys, interviews, and reviews of transcripts, syllabi, and other course documents can provide a lot of information about student learning, and this information can be used as evidence of quality or as a justification for improvement.

Lirim Neziroski, Ph.D., MBA is an academic leader and an assessment and technology expert at a liberal arts university in the Chicago area. Contact Lirim directly for additional resources and speaking, consulting, and writing opportunities.