How my use of the EMRN rubric has changed over time

As approaches to alternative grading evolve, so do the tools we use

Photo by Igor Kasalovic on Unsplash

This article is a remix of a couple of posts that originally appeared on my blog at rtalbert.org almost nine years ago, not too long after I had started using specifications grading. Whenever I talk about my use of specs grading, the rubric that is the subject of these articles always comes up, and people always seem very interested. So I thought I would condense the old articles, bring everything up to the present day, and give this rubric some attention.

---

Over ten years ago, I made a decisive break with traditional percentage-based grading systems and embraced specifications grading. I was motivated by experiences in my calculus classes where, after 20 years of using traditional grading, I was finally fed up with the way it gives false positives and false negatives, stresses students out, and disadvantages students who need more flexibility and more choices to show evidence of learning. My implementation of specs grading has undergone a lot of iteration since then, and some of the ways I use it today are totally unlike how I did it in 2014.

But some things have stayed consistent, and one of those is my insistence on radical simplicity, especially in the criteria I use to evaluate student work. I wanted today to get into the weeds on the use of a particular tool for keeping grading simple, which has been mentioned here before but has not (as far as I know) ever received its own dedicated article: The EMRN rubric.

Origins

In the early days of my use of specifications grading, I stumbled¹ across an article by Rodney Stutzman and Kimberly Race which featured a four-level rubric called the EMRF rubric. It had a compellingly simple visual presentation:

The EMRF rubric

The classic, “canonical” version of specs grading that Linda Nilson wrote about in her book grades student work on a two-level scale — Pass/Fail. When I started out using specs grading, I found this two-level approach to be slightly confining. In particular I wanted to distinguish between work that wasn’t Pass, but wasn’t a total Fail either but instead was “close” to passing. So I used a three-level rubric, especially for mathematical proofs or complex application problems: Pass/Progressing/Fail. I later changed “Fail” to “No Pass”. The middle layer of the rubric is for work that doesn't quite meet the specifications I set out, but it's close, and pragmatically the difference is that students had to spend a token to revise and resubmit "No Pass" work but they could revise "Progressing" work for free.

What I found was missing from this three-level rubric was a designation for really excellent work. The "Pass" level was synonymous with "good enough", and in my courses this is exactly what I was getting: "good enough". There wasn't much incentive for excellence. Yes, I could set the bar very high for "good enough" and call that "Passing", but it always felt to me that there needed to be something like "Pass+" in my system, and then the requirements to earn an A in the course would require a certain number of instances of really excellent work, not just meeting the minimum specs.

The EMRF rubric does this. It is basically a Pass/Fail rubric in which instead of one decision that needs to be made about a student’s work — does it meet the specifications (Pass) or not (Fail) — there are two. The first is whether the specs are met. The second one is:

• If the specifications are met, then is the work really excellent (“E”) or not excellent but just OK (“M”)?

• If the specifications are not met, then is the work a reasonable good-faith effort that shows some understanding of the concepts (“R”) or not (“F”)?

At the time, in my discrete structures course, students worked on three kinds of assignments -- Assessments (short, timed, in-class quizzes that measure proficiency on one of 20 different learning targets), Miniprojects (which apply basic knowledge to new problems), and Course Management tasks that include preparation activities, daily homework, and weekly quizzes over the syllabus. Assessments and Miniprojects were graded on the EMRF rubric while course management tasks are graded Pass/Fail, usually on the basis of just completeness and effort. To get an "A" in the class, students must:

• Earn "Pass" grades (E or M) on all 20 learning target assessments, at least five of which must be “E”.

• Earn "Pass" grades (E or M) on 8 Miniprojects (out of 10--15 in all) at least two of which must be “E”.

• Pass at least 90% of all the course management tasks.

For a grade of “B” in the class, students needed to earn E or M on all 20 learning targets, at least three of which were E; earn E or M on six Miniprojects, at least one of which is an E; and pass 80% of the course management tasks. For a course grade of “C”, there were lower quantities of Miniprojects required, but the key difference between a C and higher course grades was that no E grades were required at all. That is, to earn a C in the class — minimum baseline competency — you didn’t need to demonstrate “excellence”, you just needed to be “good enough”.

The difference between EMRF and straight Pass/Fail is the kind of feedback the letter communicates in EMRF. A grade of M means This meets expectations but it also honestly communicates that it could still be better. For many students who get an M on an assignment, the existence of an E will impel them to retry an assignment to raise their grade even though it was "good enough" the first time. Likewise, a grade of R or F means the same thing in the grading system -- you still have to revise and resubmit if you want the work to be counted -- but it communicates two different things, an "R" saying There is partial understanding here, but something important is missing and an "F" says There was too much missing to really know if you understand. In other words, it gives marks that indicate progress.

Use cases

Here are some practical situations where I used the EMRF rubric. The first is from a class I used to teach called Cryptography and Privacy that was a mix of math and computing topics, and contemporary issues about privacy and ethics. One of the learning targets was:

I can find the value of a mod n for any integer a and positive integer n and perform modular arithmetic using addition and multiplication.

Each learning target assessment is a new variant of the same problem: Students are given eight basic modular arithmetic computations to do involving adding, multiplying, and exponentiating. Unlike most work I give students to do, I don't especially care if they show their work on this problem. All I care about is whether they can compute the answer correctly or not. So the EMRF rubric is simply:

• E = All eight answers are correct.

• M = Either 6 or 7 out of 8 are correct.

• R = All parts are attempted but fewer than 6 answers are correct.

• F = Not all parts are attempted.

The idea is that if students can do modular arithmetic correctly 8 times in a row in a single sitting, that's pretty exemplary and I am convinced they have mastered the concept. If they can do it correctly about 3/4 of the time then I consider that "good enough". Otherwise they need to practice some more and try again later. Except if they deliberately omit parts of the problem, they still have to redo it but the message it sends is that the work is not complete enough to warrant feedback — and I don’t give feedback on it, other than to say that they omitted parts of the problem. The message I am sending is that it’s OK to get things wrong since there is a feedback loop in place; but it is not OK not to give a complete good-faith effort.

The EMRF rubric gets a little more interesting when applying it to more complex work, like mathematical proofs. I have an activity I like to do in proof based courses where students "grade" a piece of work that I present to them. Students read through a proposition and proposed proof and then use clickers to rate the work according to the specifications for student work document. The polling question is simply, using our EMRF rubric, what letter would you assign — E, M, R, or F?

In one of my proof-based classes, for example, I gave a proof to analyze that was a mathematical induction proof that was correct except the base case was omitted. I gave the proof as a pre-class activity for students to read and comment on. In the pre-class activity, 65% of students correctly identified that the proposition was true but that the proof had a significant flaw. Most of the remaining 35% said the proof had only minor errors to be corrected that had to do with style and phrasing. In class, I used the clickers and a polling tool to ask what grade they would give it, if they were me. About 1/3 of the students said either E or M, about 1/3 said R, and about 1/3 said F. This means that a non-trivial number of students saw the flaw in the proof, identified it as significant, but marked it as “passing” anyway.

We had a really interesting discussion then about what constitutes passing versus non-passing work, and what differentiates E from M. Once students saw the missing base case, every one of them who voted E or M the first time switched to F! Students are much harsher graders than I am — I would have graded that proof as “R”, which led to another important conversation: Even significant errors are not catastrophic. A lot of the time, “R” level work is five minutes and two sentences away from “E”.

Changes

I realized early on that the letter “F” can no longer be used in any meaningful sense in any grading system besides the traditional one. It is just too closely identified with “failure”. I tried to tell students that it stood for “Fragmentary”, which was true and not a bad way to think about it; but the branding in students’ minds is too strong. So I changed the letter to “N”, to stand for “Not Assessable”. Thus, I ended up with the EMRN rubric which we’ve written about here on this Substack before.

Here is an updated visualization of this rubric. You are free to use it and share it; you can also find it at a dedicated web page I have set up for this: https://rtalbert.org/emrn/.

The EMRN rubric

You are free to share this rubric with others, including use in your syllabi. Just please make sure to abide by the terms of the Creative Commons license that’s shown; click the link for details. The simplest way to do this, is to just use this very image, which contains all the info needed to properly attribute the diagram.

In later implementations, there were some courses at the entry level where the work students were doing didn’t really lend itself well to distinguishing “excellent” from “meets expectations”. In Calculus, for example, I might want students to demonstrate knowledge of basic derivatives by differentiating three simple functions. There isn’t really an “excellent” way to do this, unless I gave students, say, ten functions and “excellence” meant they did 9/10 right. So in the interests of simplicity, I collapsed the EMRN rubric down to just “MRN” — anything to the left of “R” is considered “M”. For more complex work, like application problems, we kept the full EMRN rubric.

Even later, I started encountering issues with the two non-passing marks of “R” and “N”. Students could still revise certain pieces of work even if they were marked “N”; but the existence of an “R” led some students to believe that student work marked “N” could not be revised. It’s a fair (if incorrect) conclusion to draw, and it caused a lot of confusion. Also, I began to have less and less use for tokens, as I moved away from traditional deadline structures. So eventually I dropped the “N”, and if students turned in work that would have been marked “N”, I’d take the wording that’s beneath the red box in the rubric and put it in their verbal feedback. This move also simplified my grading, and I think students got more out of the verbal feedback than they did out of the letter “N”.

The present

So in other words, what I currently use is… a two-level Pass/Fail rubric, just as Linda Nilson prophesied. But the EMRN rubric is at its core. And, I use the labels “Success” and “Retry” these days² and not “Pass” or “Fail” which I think are too emotionally charged to be of any use.

Grading work goes like this. Having looked at a piece of student work, I ask: Does the work demonstrate thorough understanding of the concepts, and does it meet the expectations outline in the assignment?

• If yes: Mark the work as Success. Then give feedback. Especially, if it was complete and well communicated (that is, it would have gotten an “E” in the EMRN rubric) then tell the student this and give them some kudos.

• If no: Mark the work as Retry. Then give feedback. Especially, if the work is fragmentary or systematically flawed (that is, it would have gotten an “N” in the EMRN rubric) then tell the student this, and offer no other feedback on the work³.

In this way, the EMRN rubric is not only good on its own, if you like four-level rubrics; it can also be repurposed into a three- or two-level rubric if that’s what you’re into. For me, I like the simplicity of two-level rubrics, and I think my students do too.

How this changes the narrative

In one of those classes several years ago, I handed back some work on assessments, and a student pulled me aside and asked if he could argue for a higher grade. He had done work on a problem where he made an initial algebra error that was serious, but then worked through the rest of the procedure correctly. I had given him an R; he was arguing for an M.

Sounds familiar, except this time the student was not grubbing for points -- what's a "point"? -- but rather presenting a coherent and well-considered explanation for why, in his opinion, his work meets the standards for M. (He did not argue for an E.) It was an exchange between two people on the same level. I did not agree with his argument in the end -- the standards for this problem clearly stated that a correct answer was necessary -- but after telling the student this, I could also say, "You definitely show evidence of understanding. You'll get this right next time." It wasn't about points, it was about quality and there's a world of difference here. And the student was OK with this, and did “E” level work the next time.

That's a narrative that I want to support.

1. I found it in a discussion in the Standards Based/Specifications Grading community on Google+. Does anybody remember Google+? Shout out to this platform, which was way ahead of its time in social media. This community or “circle” on that platform was the precursor to today’s Alternative Grading Slack workspace.

2. Yes, I am aware that “success” is a noun and “retry” is a verb. It annoys me; but I couldn’t think of anything better that wasn’t linguistically complicated.

3. I know we are all about feedback loops here. But I maintain that feedback is reserved for those who turn in complete, good-faith efforts at their work. Students are allowed, and encouraged, to get feedback on drafts and partial solutions by coming to my drop-in hours or asking email questions, and I am happy to answer those. But turning in work that is fragmentary gets nothing but a note about the fragmentary-ness of the work. Otherwise, what usually happens is that some will turn in deliberately fragmentary work and use the revision policy of the course as a black box — throw stuff at it, and see what the oracle (= professor) says about it, and keep repeating. I don’t like that and I don’t want to allow it.