An assessment is commonly conveyed through the use of pre-set criteria that provide precise explanations of exactly which characteristics or behaviors will be assessed (e.g. commitment) and the importance (value) of each in the assessment. These are called marking rubrics in academics literature. In the previous blog I wrote about the IJS protocol sheets skaters receive at competitions. The bullet points for achieving grade of execution (GOE) provided to judges could be considered a marking rubric because they provide qualities for them to assess skater performances. For example, spin grade of execution has 6 different qualities that can positively impact the judges score of the spin.
Each criterion on a marking rubric is designed to be assessed in isolation and this, according to Sadler (2009), is a failure of the act of assessment for several reasons. The first reason is that marking rubrics do not acknowledge details of quality that cannot be easily explained in words. Such details are called tacit knowledge (Polanyi 1958). For example, it can be extremely challenging to explain in words exactly why a choreographic moment is beautiful. Perhaps it was sharp, angular, and commanding. In the spin example, sure the GOE rubric provides valid qualities, yet how can one assess other qualities that cannot easily be explained. For example, you see a spin and it has a nice entry; its fast and centered. Yet, there is something else about it that makes you like it more than other skater’s spins which were equally fast and centered but you just cannot explain why. In another example, a skater has shown great improvements yet we cannot articulate all of the details that brought us to this conclusion.
A second reason marking rubrics fail is because valuating pre-set criteria ignores the fact that each criterion should have different weights under different circumstances. Consider “mental toughness” (a term we dismiss. Look for an upcoming blog series on the subject). One could argue that mental toughness is a vital characteristic of athletic success in that athletes must demonstrate mental toughness to persist and overcome great challenges throughout their careers (which is very true). However, how could mental toughness be scored on an assessment for an athlete who pushed through an injury they probably should have taken time to heal? In this specific case, was the athlete’s mental toughness still an important trait? Or take that choreographic moment discussed above. In that case it was beautiful because it was sharp, angular, and commanding. Does this mean that sharp, angular, and commanding will have equal weight as you assess another choreographic piece that is soft, fluid, and delicate? And how much more important was “sharpness” over “command” or vice versa in your assessment of the sequence?
A third reason marking rubrics fail is that each criterion is designed to be judged separately and then added up and often the assessor will arrive at a different score than if they judged the work as a whole (Sadler 2009). A stellar piece of work (as a whole) could end up with a mediocre score when that work is broken down into separate criterions. Or, a mediocre work, when judged as a whole, could achieve an outstanding score when judged through individual criterions. To circumvent the issue, Sadler (2009)states that assessors resort to judging the whole first, then they adapt the individual scores to match the overall assessment. This renders the actual assessment less honest; less useful because the assessor is simply adapting each individual score to match their overall judgment. Let’s say for example that a skater made huge improvements this past year. We coaches think it was a breakthrough year for the individual. Yet, when we score that athlete’s characteristics individually, the picture presented is that of mediocrity. Do we go with the marking rubric protocol because, apparently, we were totally wrong all this time in our belief in that skater’s progress? Or, do we adapt the scores to better reflect the positive transformation that we witnessed over the season? This issue has already occurred in our previous assessments so we are well aware it certainly exists!
The last issue regarding the effectiveness of marking rubrics is that the goal is to make assessments detailed and inclusive yet not overcomplicated and confusing. This opens up the risk, from efforts to create simple and easy to read assessments, of leaving out important characteristics that should otherwise be recognized (Washor and Mojkowski 2013)or misinterpretations of the content. We have read about higher education institutions running into all sorts of issues with this predicament when attempts are made to roll out new practices [see chapters 14-16 for several examples here: (Merry, Price et al. 2013)and (Macdonald and Joughin 2009)]. There tends to be great confusion when too few or too many details are provided and both teachers and students interpret the content in ways that benefit their own perspective but not necessarily as originally intended. The IJS system was developed to replace the 6.0 scoring system. The 6.0 system was a “judge the whole” protocol. There were no details for skaters and coaches to understand why a score was achieved. The IJS system, on the other hand, tends to be very information heavy and is certainly at risk of misinterpretations. In essence, it takes this beautiful whole (a skater’s performance) and mechanistically breaks it down into many little categories and details. Which system is better? It all depends on perspective.
Right: In the 6.0 system, skaters were awarded an overall score (6.0 represented a perfect score) from the judges. When the scores came up at an event, we smiled if they were ‘good’ scores, and cried if they weren’t. Other times we just stared!
Ultimately, we need to use a rubric of some sort to assess our athletes. However, it is vital that we consider both the quality of their development as a whole and the individual components separately to better understand the big picture. The assessments and the rubrics should be adapted to the unique circumstances for each athlete and the importance and value of each criterion should be adapted accordingly.
Questions for parents
Please take time to deeply think about the question(s) below and provide sincere answers. The questions do not have right or wrong answers but are meant to trigger your thinking about the assessment process.
Please send your responses to: Garrett@acskating.comand Sean@acskating.com
This is a mental exercise: Create a simple marking rubric that has 5 qualities to rate foods (such as spiciness or satiety). Try to determine your favorite food. Why is it your favorite type? Why did it beat out your second favorite type and does it always win out? Does your second favorite sometimes come out victorious? Now, where do get your favorite example of this type of food? For example, if it’s a burrito, where is the best burrito – do you make it at home or get it at a restaurant? What makes that burrito better than others?
- Was it easy to use the criteria as a guide to score your foods? Did some factors matter more in some cases but not others? Were you able to achieve the same conclusions if you judged the foods based on your gut feeling alone? Did using a standard scoring system make this food assessment easier and more fun to do? Or did it turn the process into something else?