That makes a lot of sense for LOs based on "do". How does this work for LOs based on "explain"? In this case, I need non-isomorphic problems so that the student can't simply memorize definitions/processes without truly understanding the motivation/mental model behind the process.
I'd say it depends on exactly what sort of explanation you're hoping for. For my "explain" LOs, I'm often using pathways/mechanisms where one can start at one end or the other, where there's an inversion of the signal (i.e., an increase in A leads to a decrease in B), and/or where flux through the pathway can be either increased or decreased. With all of these possibilities, students can't usually bulk-memorize their way through everything without understanding it; they have to notice the specific prompt they're given and choose the right direction etc. For example, while blood flow is unidirectional, air flow is bidirectional (into the lungs and then out of the lungs), which leads to the possibility of LOs like this one....
LO: Given information about air flow (into or out of lungs), lung pressure, thoracic cavity/lung volume, or respiratory muscle state (contracting or relaxing), make predictions about these other parameters.
Example A: The diaphragm muscle is getting shorter. Is air currently flowing toward the lungs or toward the environment? Explain your reasoning.
Example B: Air is currently flowing from the lungs toward the outside environment. How does the current air pressure inside the lungs compare to the environmental air pressure? Explain your reasoning.
Both of these still fundamentally have a "do" component, where the "do" is to predict the resulting behavior. I am thinking of examples from computer science where my primary concern is whether the student has developed an accurate, useful mental model of something (e.g., a ripple-carry adder or a carry-lookahead adder). Predicting the output of these devices can be brute-forced without really understanding the "big picture" that motivates the design.
OK. This might be a disciplinary difference, or perhaps not. Can you post a fairly detailed example with the LO plus an assessment question matched to that LO?
Thanks for sharing this; it’s interesting to see the percentages. To go a step further in the direction of thinking of test questions as part of a feedback loop, if one is writing new questions for an exam anyway, the previous exams’ questions can be used to help students practice those concepts. They can either be assigned as low-stakes homework, or given to students as a sort of exam study guide. I decided some years ago that if I was worried that some students might be getting old exams from previous students, then I would make sure that *all* students had access to those materials to level the field, and provide additional practice / learning opportunities. So I regularly hand out the previous semester’s iteration of each exam a week or so in advance of the new one. (It’s optional for students to do it, and I don’t grade it, but will answer questions if students have them.) One caveat: I have found that it’s helpful to remind the students that the practice version may seem easier to them than the actual exam will, since they may be more stressed taking the exam and/or may not be doing the practice at the same pace or without notes etc.
I strongly agree with making previous exams available as practice exams; I do something similar for similar reasons. One point I try to make with students is that they can use these practice exams in multiple ways, e.g. to get a general feel for questions vs. to do a full-blown exam simulation and to use it as a diagnostic to identify areas of weakness. Any use is fine, but only the latter will give a clear sense of exam difficulty.
Brilliant. This focus on transparent feedback loops is so crucial, especially when you think about how iterative proceses are core to things like traning AI models. Do you think the resistance to emphasizing these loops comes from a traditional focus on just 'getting the answer'?
That makes a lot of sense for LOs based on "do". How does this work for LOs based on "explain"? In this case, I need non-isomorphic problems so that the student can't simply memorize definitions/processes without truly understanding the motivation/mental model behind the process.
I'd say it depends on exactly what sort of explanation you're hoping for. For my "explain" LOs, I'm often using pathways/mechanisms where one can start at one end or the other, where there's an inversion of the signal (i.e., an increase in A leads to a decrease in B), and/or where flux through the pathway can be either increased or decreased. With all of these possibilities, students can't usually bulk-memorize their way through everything without understanding it; they have to notice the specific prompt they're given and choose the right direction etc. For example, while blood flow is unidirectional, air flow is bidirectional (into the lungs and then out of the lungs), which leads to the possibility of LOs like this one....
LO: Given information about air flow (into or out of lungs), lung pressure, thoracic cavity/lung volume, or respiratory muscle state (contracting or relaxing), make predictions about these other parameters.
Example A: The diaphragm muscle is getting shorter. Is air currently flowing toward the lungs or toward the environment? Explain your reasoning.
Example B: Air is currently flowing from the lungs toward the outside environment. How does the current air pressure inside the lungs compare to the environmental air pressure? Explain your reasoning.
Both of these still fundamentally have a "do" component, where the "do" is to predict the resulting behavior. I am thinking of examples from computer science where my primary concern is whether the student has developed an accurate, useful mental model of something (e.g., a ripple-carry adder or a carry-lookahead adder). Predicting the output of these devices can be brute-forced without really understanding the "big picture" that motivates the design.
OK. This might be a disciplinary difference, or perhaps not. Can you post a fairly detailed example with the LO plus an assessment question matched to that LO?
We, as instructors, are required to keep student exams at my university. So, I only give them temporary access.
Fascinating! Good for you for doing the best you can under those circumstances.
Thanks for sharing this; it’s interesting to see the percentages. To go a step further in the direction of thinking of test questions as part of a feedback loop, if one is writing new questions for an exam anyway, the previous exams’ questions can be used to help students practice those concepts. They can either be assigned as low-stakes homework, or given to students as a sort of exam study guide. I decided some years ago that if I was worried that some students might be getting old exams from previous students, then I would make sure that *all* students had access to those materials to level the field, and provide additional practice / learning opportunities. So I regularly hand out the previous semester’s iteration of each exam a week or so in advance of the new one. (It’s optional for students to do it, and I don’t grade it, but will answer questions if students have them.) One caveat: I have found that it’s helpful to remind the students that the practice version may seem easier to them than the actual exam will, since they may be more stressed taking the exam and/or may not be doing the practice at the same pace or without notes etc.
I strongly agree with making previous exams available as practice exams; I do something similar for similar reasons. One point I try to make with students is that they can use these practice exams in multiple ways, e.g. to get a general feel for questions vs. to do a full-blown exam simulation and to use it as a diagnostic to identify areas of weakness. Any use is fine, but only the latter will give a clear sense of exam difficulty.
Brilliant. This focus on transparent feedback loops is so crucial, especially when you think about how iterative proceses are core to things like traning AI models. Do you think the resistance to emphasizing these loops comes from a traditional focus on just 'getting the answer'?