Measuring learning

Reliable and valid assessment methods

A reliable assessment ensures that;

• the same students will consistently achieve the same result
• where different teachers grade an assessment, the same mark would be given.

In other words, assessments must be written so that what is required to succeed is clear and marking schemes are well defined and unambigious. When we fulfil these criteria, we can be more certain that students have a clear idea of what is expected of them, and we remove the possiblity that the assessment itself might produce an inconsistent result.

In practice, you can improve reliability by:

• ensuring that instructions and questions are clear and unambiguous;
• ensuring the time allowed for the assessment is appropriate;
• ensuring that the number of assessment items is sufficient to give a reliable measure of a student's ability;
• developing a marking scheme with explicit criteria; and
• checking marks (moderating), double marking (two assessors), re-marking a sub-set (sampling) or carrying out multiple assessments over time - especially if more than one marker is involved and/or if the assessment methods entail a degree of subjectivity (e.g. an essay) or professional judgement (e.g. a complex clinical or workplace competence)

A valid assessment ensures that;

• it accurately measures what it is meant to measure.

For example, if we have a learning outcome such as, "students will be able to critically analyse the evidence on community immunisation programmes" then an assessment that measures factual recall is not appropriate.

In practice, you can increase validity by;

• including a representative sample of the course content (content validity);
• measuring the progress toward the intended outcome of the course, for example, attributes/knowledge that the student should be developing (construct validity, the construct being 'what it is to be a good nurse/ doctor' etc)
• having sufficient predictive ability to determine how well the student will transfer her/his knowledge into the workplace (criterion validity: the criteria will provide a suitable measure of how the student may be expected to perform outside the relative safety of the teaching environment).

Reliability and validity of direct observation of student performance can be improved by using a structured set of criteria, with clear descriptions for each of these, as well as a description of the expected standard for each performance.

Standard setting

When considering education achievement (and how to measure it), there are two types of standards: 

• Relative standards compare those that are assessed - for example, scholarships that are rewarded to the 10 highest grades, independant of what those grades are.  This is called norm-referenced assessment and is useful when a number or percentage of students are selected but less helpful for the assessment of the health professional as it can lead to unnecessary competition between learners.  
• Absolute standards compare the grade of the student against a set grade-based criterion; for example, a student may need to answer 50% of the examination questions correctly in order to pass (regardless of how many other students achieve 50% or more).  This is call this criterion-based assessment.  This is the most common assessment used in health professions' education, it assumes simply that if students meet the criteria then they will pass the assessment. 

The validity of standards depends on a number of factors.  The methods must, for example, be appropriate for the purpose, and easily explained and implemented.  Additionally, the standard setters must understand the purpose of the test and know course content. 

Rubrics

A rubric is set of rules or instructions that can be attached to an assessment task. A rubric lists the criteria associated with various grades for a piece of work. The criteria for each of the grades will provide the characteristics and qualities needed to successfully complete the particular task or assignment to the standard for that particular grade. For example, the criteria for achieving an A grade might make reference to "evidence of critical thinking" and "evidence of independent research that that led to insight". The criteria for achieving a C grade might make reference to "clear and logical presentation" and "use of the resources provided as part of the course". The rubric makes clear what is required to achieve an A.

A rubric makes the criteria for marking explicit. The educator - and anyone else involved in the assessment (second marker for example) - can clearly see the criteria associated with the different grades.  Students can be guided by the rubric as they complete the assignment. The rubric will help students to understand what is expected of them and it will help to communicate high expectations to the students.  

You may find it useful to look at this page by Carnegie Mellon University that provides you with a full explanation of rubrics together with examples that could be useful when creating your own rubrics.