Executive Summary

Denise Nacu created a pair of multimodal midterm and final exams for her Human-Computer Interaction classes, but the time it took to grade them caused stress for her and her students.

Putting Denise’s exams online was difficult because parts of them required students to physically draw on the exam. We shifted the exams into two-part asynchronous, online-only formats with a D2L quiz for the multiple-choice and short-answer questions, and D2L dropbox with release conditions for submitting the design questions.

This solution saved Denise hours of grading and allowed her to return all final grades to her students within 48 hours of the last student completing their exam—a win for all involved.

Introduction

This post describes how Denise and I moved her midterm and final exams online using Desire2Learn. We’ll cover what the exams looked like at first, how we adjusted the format to a fully online format, and what we learned in the process.

The Classic Exam

Denise’s exams are well thought out and evaluate students on all levels of Bloom’s Taxonomy, from remembering through creating. Students complete a three-part exam made up of:

• 17 multiple-choice questions,
• 3 long-answer questions, and
• 2 open-ended design questions.

The exam rules are: no notes or books and students have a maximum of 120 minutes to complete the exam.

The exam itself worked perfectly, but grading took a long time and added responsibilities made grading in a timely manner increasingly difficult. To understand the challenge for an online-only setting, we need to understand a bit more about the types of questions Denise is asking the students to complete.

Multiple Choice (MC) and Long Answer (LA) Questions

Denise’s multiple choice and long answer questions did a great job of evaluating students on the lower part of Bloom’s and are pretty easy to put in an online format. Just add the questions to the UW D2L Quiz Question Import tool, batch upload the questions into D2L, and we’re virtually done.

Design Questions (DQ)

The design questions are also important for evaluating student mastery on the steeper slopes of Bloom’s—evaluation and creation—and herein lie the challenges we faced in bringing the exam online.

Design Question One

For the first design question, students are provided with an example interface and a set of assumptions under which to operate. Students were then asked to:

“Analyze this design and identify at least four problems you see. For each, you must 1) clearly identify the problematic element, 2) name a concept or design principle addressed in this class to justify your choice, and 3) use that concept or principle to explain why the element is problematic. Use the space below and the reverse of this page if needed to write your  response.”

The constraint here is that students need to draw on the example interface to indicate the areas which would cause problems, and support their written argument.

Design Question Two

For the second design question students are asked to:

“Using the space below, create a sketch for a revised interface that improves upon the design above.  Be sure to account for all of the tasks the user needs to accomplish listed in Step 1. Do not add extra tasks.  Explain how your design is an improvement and explicitly refer to concepts or principles addressed in this class.”

The obvious constraint here is that students need to create an example interface to improve upon the functionality of the first interface, and support it with a written argument.

The Conversion Challenges

So, problems with porting the exam to online are that students need to identify areas of a graphic to provide commentary, and create an improved example interface. Both of these activities require students to do something impossible with D2L’s existing quiz tool—draw.

While drawing on an electronic final is a hurdle to overcome, our additional challenges were that the exam should require students to:

1. complete the exam parts in order, MC/LA first and DQ second, and
2. complete the exam in a certain amount of time.

So, how did we do it and what did we sacrifice?

The Exam Online

To move the exam online we settled on using two components: D2L quizzes and D2L dropbox.

Before the exam date, Denise describes the exam format to her students and answers any questions to ensure there’s minimal confusion. Students submit other class assignments to the dropbox before the exam and most have experience using the quiz tool which helps ensure the exam is a success.

On the exam date, the quiz becomes available at 9:00 AM and students have 60 minutes to complete the quiz component. Upon quiz completion, the dropbox component becomes available and this must be submitted by 5:00 PM CST.

Let’s unpack the quiz and dropbox components in a bit more detail.

The Quiz

The quiz component delivers the multiple-choice and long-answer questions for the students to complete within 60 minutes. Advantages to quizzing are that:

• the multiple-choice questions are auto-graded, allowing Denise to focus on grading the more intensive portions of the exam, and
• this part is timed to keep students from relying too heavily on textbooks or notes.

On the last page of the quiz is a reminder for students to access the dropbox portion of the exam next. Once the quiz is submitted, the dropbox exam component becomes available for students.

The Dropbox

Upon accessing the dropbox, students download a DOCX file with the design questions and further instructions. To complete this portion of the exam, students can annotate the interface to describe the related design problems for the design question one.

For design question two, students can:

“…draw your design by hand or use your preferred wireframing tool (like myBalsamiq)… [and] paste a screenshot of your wireframe – or photo of your sketch – [in the DOCX] and explain how your design is an improvement.”

This open-endedness allows students to use the tools they prefer to create the required images. Once they’ve created their interface placing the photograph or screenshot in the DOCX is straight-forward. Upon completing the design questions, students submit the document to the dropbox by 5pm to complete their exam.

In addition, we added Denise’s design question rubric to D2L to further speed design question grading.

Truth and Sacrifices

We knew there would be sacrifices in moving the exam online, and there were.

Since we used two tools, there is no way in D2L to limit the whole time students take to complete the exam. The “9am-5pm” method provides the students incentive to begin the exam earlier to have more time on the design questions. The added benefit is that students who might encounter technical issues would have more time to overcome them or ask for help.

Additionally, since the exam is asynchronous, students can use their notes and supplementary materials. This makes the design question portion of the exam less demanding and stressful, especially if the students have many hours to complete it.

Conclusion

Overall, bringing the exam online in this format was a functional success. All students were able to complete the exam, the grading was completed far more quickly than the paper-based approach, and we didn’t sacrifice too much.

There are certainly other methods and tools we could have used to move this exam online and I hope you’ll share your thoughts and ideas in the comments.

Denise’s Last Word

“Preparing to move my exams into an online format required some additional time to assure that the instructions were clear to students and that the D2L set-up was correct.  However, Ian was fantastic in thinking through the implementation with me, helping me understand D2L’s capabilities, and keeping the project focused on the assessment goals. In the end, I was very satisfied with how the exams assessed student learning in the class. And the online format saved me many, many hours of grading time.  Now that we have this approach worked out, I’m confident that I can use it again in future classes.”