Case study: Kay C Dee's specifications-graded biomedical engineering course
This week, we are bringing you another case study from our upcoming book, “Grading for Growth.” The core of the book will be case studies that feature instructors and how they use various alternative assessment systems in their classes.
Today’s case study focuses on Specifications Grading. You might find it helpful to review our short primer on “Specs” Grading, as well as the four pillars of alternative assessments.
We would love to hear your feedback on this case study. Do you find it helpful? Is there something more you’d like to know? Are there unnecessary extras we added in? Let us know in the comments!
Kay C Dee is a professor of Biomedical Engineering at Rose-Hulman Institute of Technology. RHIT is a teaching-intensive STEM-focused institution located in Terre Haute, Indiana. Here’s the description for Dee’s class “Regulatory Affairs - Medical Devices”: “Through this course, students will build a fundamental understanding of how the FDA [US Food and Drug Administration] regulates medical devices in the United States, with an emphasis on pathways to market.”
This course is interesting in many ways. It is a senior-level undergraduate class for biomedical engineers that was designed to be taught online from its very first offering. The class usually has around 20 students. Dee designed it to be extremely practical:
I believe it is important for the students to be navigating the FDA’s (voluminous and circuitous) websites and databases to find and use information, guidance, interpretations, templates and forms, etc., just as a professional does. They don’t need lectures on how to do that, or to be led step-by-step by a professor - they need a set of instructions, and then an interesting problem to solve as a motivation to explore and learn.
The course is asynchronous (no “live” class sessions), and interaction happens in many ways. Dee assigns regular discussion assignments using a learning management system, uses peer review assignments, holds frequent online office hours, and offers frequent email communication. Creating the course to be offered asynchronously online gave Dee room to focus on what matters: “... I could spend less time and energy performing ‘lectures’ in front of a classroom, and more time 1) working on compelling and informative projects for the course, 2) giving students feedback on their work, beyond just a simple ‘8/10 good’.”
The class is organized around 10 weekly modules. Content for each module is posted on the course’s LMS. To help understand the assessments that follow, here is an outline of the content in one particular module on “design controls.”
Watch a short instructor video describing the basic “design controls” required by the Code of Federal Regulations in medical device development.
Take an auto-scored comprehension quiz (this is only a self-check, not part of any grade).
Watch a short corporate YouTube video on common design control mistakes.
Read two online articles about design controls written by industry experts.
Read a trade news article about design control violations at a well-known medical device manufacturer.
Read the FDA’s FAQ on a form used to report design control violations.
Optionally, read about the legal troubles of the medical device manufacturer and its leadership after discovery of the design control failures.
There are two “activity packages” tied directly to each module’s content. These activity packages are collections of assessments that share a similar level in Bloom’s taxonomy.
Level I activity packages focus on basic proficiency and fundamental concepts in each module, including reading quizzes and traditional homework assignments.
Level II activity packages “...let students demonstrate that they can apply information and skills from that module to situations of varying levels of realism.”
There are also Level III activity packages, which are major projects that cut across modules. They are designed to let students showcase deep understanding of course material and “[the] ability to relate course material to current events, professional development, societal issues, scientific research, etc.”
Each activity package may include several separate assessments, such as a reading assignment, a quiz, and a homework assignment. Dee provides a list of specifications for the entire package that describe what it means for all of a student’s work on a package to be “satisfactory”. To complete a package, all specifications must be met at the specified levels.
We’ll begin by looking at an example of a Level I activity package, which is focused on a basic understanding of the “design controls” module described above. There are two assessments within this activity package. The first is a “Design Controls Assignment” that asks students to obtain the most recent FDA guidance document on design controls (from the FDA’s website). They use this guidance document to answer several questions. For example:
Page 28 of this guidance states that “In practice, design review, verification, and validation overlap one another, and the relationship among them may be confusing. As a general rule, the sequence is: verification, review, validation, review.” Please use your own words to write a detailed explanation of the differences between design review, verification, and validation, and give hypothetical examples of each.
The second assessment in the activity package is a “Level 1 quiz” with direct questions on design controls.
Here are the corresponding specifications for the entire package. The purpose of specifications isn’t to be instructions for an assignment, but rather a description of what success looks like.
Notice how the specifications for the Design Controls Assignment include both a completion requirement, and a description of the quality of the responses. For a direct assessment like the quiz, the specifications include a numerical cut-off. All of these are good options for different types of assessments.
Next, we’ll look at a Level II activity package for a different module. In the module, students have examined a case study about designing a specific insulin infusion pump. The “MAUDE [Manufacturer and User Facility Device Experience] Databases Assignment” in the Level II package begins with a basic task designed to get students used to working with the MAUDE database:
Use the Product Classification database […] to search for infusion pumps in general, and then to find information about insulin infusion pumps.
a) How many different product codes are there for infusion pumps in general?
b) What is the most likely product code of the specific device in our case study?
c) What is the citation for the regulation in the CFR that governs this specific product code?
d) What class of medical device is our case study pump most likely to be?
Then, it continues to an advanced task that involves a more complex use of the database. Here we give an example of the questions involved:
Use the MAUDE database to learn about problems that have been reported with currently-marketed insulin infusion pumps. [Details about how to search in various ways.]
a) From the reported adverse events, what are the likely top three problems that the currently-marketed devices are experiencing – and so are three problems that our case study team would want to pay extra attention to as they finalize their design?
b) Briefly describe the search(es) and analyses you performed to come to the conclusions above.
Below are the corresponding specifications for the MAUDE Databases Assignment, which describe the qualities of a successful submission of this package. These specs show how specifications can cover a range of qualities. Some of the specs include completion, following instructions, spelling and grammar, and so on, but they mostly focus on higher-level qualities that show whether a student has deeply engaged with the assignment or not. This is intentional: Level II packages are at a higher level of Bloom’s taxonomy than Level I packages, and are intended to demonstrate application of skills in a variety of contexts. Describing and assessing this sort of higher-level work is a strength of Specifications Grading.
Finally, we’ll take a look at an example of a Level III activity package. These packages involve major projects, and are only required for grades of B or higher. There are six Level III packages that students can choose from, with at most four required (for an A). As Dee says, “Choice is valued by students. We tend to work harder and enjoy learning more when we are tackling a topic that interests us.” Each Level III package can be completed during a specific 3-week time window. These windows are staggered throughout the course, to give students further options and keep them working regularly.
Here is Dee’s short description of one particular Level III package. Dee makes this description, and others, available at the start of the semester to help students decide on which packages to attempt.
Name: ISO14971 Project – Risk management in medical device design = safety by design.
Time available to work on package: Weeks 4 - 7
Creativity: Low
Immediate Career Practicality: High
Major activities: This project is simulation-intensive, requiring the ability to handle ambiguity, create detailed analyses, and apply broad principles to specific examples. Simulation/Ambiguity: Independently learning about and summarizing Negative Pressure Wound Therapy systems [NPWT]. Ambiguity/Detailed Analyses/Application: Using ISO14971 to create a detailed risk assessment and a comprehensive risk management plan for NPWT; self-assessment of work.
As always, detailed instructions are available on the LMS. The project includes filling in a risk management plan and a “risk assessment spreadsheet”--something practiced during the semester. Students have the opportunity to submit a draft which receives detailed feedback before the final submission. Here are the specifications for the project described above. Again, notice the range of specific requirements and descriptions of qualities of the submission:
Dee marks each item as either completed or not, with detailed feedback left on the assignment itself. Indeed, the main focus is on feedback, which Dee sees as a major benefit of using Specs.
If all specifications for an activity package are completed, then the student earns a “Pass” on the package. This is a key difference between Specifications and Standards-Based Grading: In Specs, the entire submission is graded as a whole, rather than using the work as evidence for discrete standards.
If the specifications aren’t satisfactorily met, students may use a token to revise. Dee’s token system scales with the complexity of the assignments. Students begin the semester with two “free” tokens and can earn up to three more by completing designated “extra” assignments. One token lets students re-do a Level I package, which is regraded using the same specifications. Two tokens give a 3-day extension on a Level II package, or give 3 days to redo a Level II package. Finally, three tokens give a 5-day extension or 5 days to redo a Level III package. In all cases, tokens are time-limited: Redos in particular must be requested within two days of receiving feedback. Tokens are automatic and require no excuse: Students just need to email Dee to let her know that they are using tokens. Dee says that they represent “Common, decent respect for other humans and the many demands on their time and energy.”
Passing activity packages is the key requirement for earning final grades. Dee presents this in her syllabus as a list of requirements for each grade:
A: Complete all ten Level I Activity Packages, all ten Level II Activity Packages, and at least four Level III Activity Packages.
B: Complete at least nine Level I Activity Packages, at least nine Level II Activity Packages, and at least three Level III Activity Packages.
C: Complete at least nine Level I Activity Packages, and at least eight Level II Activity Packages.
D: Complete at least nine Level I Activity Packages.
F: Fail to meet the requirements for earning a D.
Here is one way to visually represent the overall organization of modules and grade requirements:
We’ve visualized each activity package with a checkbox that a student could mark when completing that package. This is a common way to present grade requirements to students, and helps them both keep track of their grade progress, and get more invested in the assessment system. To earn a D, students must check off all red boxes. A C requires earning a D, plus all orange boxes, and so on. Each module has two packages (Level I and Level II), and the Level III packages cut across multiple modules. One detail: students could choose any nine Level I activity packages for a D, not just modules 1 through 9.
The key is that students have choice at every level: By selecting a desired grade, they open up a degree of choice as to which (and how many) activity packages to complete. The level of packages, and the number of packages, both increase for higher grades. This is what Linda Nilson describes as “more hurdles” and “higher hurdles” when designing a specifications grading system. As Dee writes:
Students who aren’t aiming to earn an A don’t need to waste their time (and my grading time) turning in every single course assignment performed at a poor level. I’d rather be confident that students who passed my course with … a grade of C showed mastery of the fundamentals, than say that they consistently did poorly on assignments.
How has this course structure worked for Dee? She cites four major benefits. First, grading low-level (Level I) activities on a Pass/Fail basis takes much less time than traditional grading. There’s no concern about whether an error should cost 1 point or 2.
Second, the structure gives Dee more time to dig into the advanced Level III projects, “giving richer feedback and responding to what students have written as if we were having a conversation - which, often, we do end up continuing in person.”
Third, Dee can set high standards. To pass any package, students must do what she describes as at least “B+/A- level work,” rather than getting by with consistently poor work and partial credit. And because she sets high standards, she sees that students meet them! Students “produce higher-quality work when the quality standards are clear and the consequence of being sloppy or procrastinating could be a ‘fail.’” Built-in opportunities for reassessment help students reach the high bar. Dee reports that, in general, she’s noticed a shift to slightly higher final grades since changing to Specifications grading. In particular, she sees fewer D or F grades, and small corresponding increases in C and A grades. She can be certain that, whatever a student’s grade, the work they completed was high-quality.
Finally, the process of writing specifications and organizing the activity packages forced Dee to prioritize learning goals. She has to hold herself to high standards: “I have to determine what goals are critical enough that an assignment should fail if they are not met, and I have to commit to following through with the specifications I set.”
Dee’s assessment system is an excellent example of Specifications Grading. The entire course is organized around “activity packages,” which are graded holistically: Either a student has met all specifications, or they haven’t (yet!) succeeded. This illustrates how Specs is an ideal assessment system for classes that involve putting together multiple ideas and demonstrating skills with process, application, and synthesis. Students who wish to earn a higher grade must complete both more of these packages, and packages at higher levels of Bloom’s taxonomy.
Thanks for reading this case study! Do you find it helpful? Was there something you especially liked? What would you add or remove? Let us know in the comments!
This post - because it describes the grading processes in enough detail that someone could use parts of those processes in their own courses - might make it seem as though specs grading is all about grading. But actually, my experience has been the exact opposite of Kenoc's comment. Instead of spending labor and energy on assigning grades to every activity every week, I spend more time interacting with students and helping them learn.
The problem with specifications grading is that it puts most of the emphasis on grades and very little emphasis on learning.