In my last post, I detailed a study in the summer of 2016 using the Knewton Adaptive Learning engine built into Pearson’s MyMathLab. This was a limited study with a trial of Knewton in 4 developmental math courses. The results of the trial were compared to sections of the same courses in which the adaptive engine was not used. In that limited study we found that students got better scores overall on the MyMathLab quizzes and that they spent less time on task.

The summer cohort of students isn’t reflective of regular semester classes (in DePaul’s First-Year Program we typically see entering freshmen, where this is the first university level course they have encountered), so we implemented the same trial in 4 courses with larger enrollments and traditional students during the winter 2016 quarter. Please see my previous blog post for information about the Knewton engine and the previous trial.

Winter 2016 Trial 

We implemented the trial with 4 instructors in three different courses:

• MAT 100 – Introduction to Quantitative Reasoning (2 sections—one fully online)
• MAT 101 – Intermediate Algebra (2 sections)
• MAT 130 – Precalculus (1 section)

We used the work we had done last summer for MAT 100 and 101 and created a new study plan for MAT 130 using the procedures outlined in my prior post. As the quarter progressed, I became a bit concerned that the results would not mirror what we saw last summer. This was largely due to technical issues that kept cropping up (more later). Instead, I was quite surprised by the results.

The courses were structured as follows:

• All use MyLabsPlus
• Traditional Course
  • Lectures (either online or F2F
  • MML Homework—60% score required to take corresponding quiz
  • MML Quizzes—three attempts possible—highest score
  • Midterm, Final
• Knewton Adaptive Courses
  • Lectures (either online or F2F
  • Adaptive Study plan—Mastery of x objectives required to take quiz
  • MML Quizzes—three attempts possible—highest score
  • Midterm, Final

Summary Charts of the Quizzes

We compared the adaptive sections to regular sections of the classes as follows (F2F = face-to-face):

• MAT 100
  • 2 sections Adaptive online
  • 1 section Adaptive F2F
  • 4 sections regular F2F
• MAT 101
  • 2 sections Adaptive F2F
  • 4 sections regular F2F
• MAT 130
  • 1 section Adaptive F2F
  • 3 sections regular F2F

The results point to better student success with the adaptive program.

MAT 100 (Introduction to Quantitative Reasoning)

In the case of MAT 100, the comparison of the two F2F offerings shows a clear advantage to the adaptive approach. The fully online course did not fare as well, but still showed an advantage. (Online courses tend to be a bimodal distribution of grades. Either students do really well or very poorly). It should be noted that the MAT 100 students are on a liberal studies track and will take only one additional liberal studies math course during their college career. 

MAT 101 (Intermediate Algebra)

These students are on the track to take additional math courses during their college career and were placed into this class based on their math placement exam. The results were even more striking, comparing two adaptive sections to 4 regular sections.

MAT 130 (Precalculus)

These students were also placed into this class based on their math placement exam. There is a gap in the data for the adaptive class as those quizzes were not administered. Again, we see similar results comparing the one section of adaptive to the three sections of regular classes.

Student Survey

A written survey was conducted in each class that used adaptive learning with 67 responses

The most common comments about the study plan were about the amount of time the study plan seemed to take. There was a lot of frustration with the answer formats (rounding, precision, etc). There was also some frustration with obtaining mastery points in order to take the quiz. Those that liked the study plan felt it helped their understanding.

We wanted to know how many had taken a MyLabs course before and if they used a study plan or regular homework. It was about evenly split between first use/and used a study plan to those that have taken a course and used homework:

We then asked if they thought the study plan helped them:

If we ask just the students that have taken a MML course before and used homework, they much prefer sticking with what they know:

Technical Issues surprised us a bit as we thought it would have been worse. Problems included login issues, disappearing mastery points and the answer formats noted earlier. Nevertheless, 34% with technical issues is certainly not optimal.

Comments and Instructor Feedback

Instructor 1

“The adaptive study plan is designed to remediate and assist understanding and presumably will help the poorer performing student. Anecdotal evidence suggests it helps the good students, but weaker students lack the motivation to put the study plan to good use.”

Instructor 2

“I think that adaptive learning has it merits since each student comes with a different background and prior knowledge. My issues were with the software itself.

• Mastery points given before the class started.
• The student would have a differ number of mastery points then what I would export.
• If a student would try to go back and redo a section the mastery points were erased causing much frustrations.
• If the software was a bit more user friendly it would have great merit.

With that said, the students that had prior experience with MLP in the traditional F2F seem to have liked the traditional method of homework, yet those with no prior experience with MLP seem to have liked the adaptive learning. Maybe this is due to the fact that they did not have anything to compare it to.

Personally, as an instructor I can go either way now that I have had some practice with it, but I think more collaboration is needed in setting up the class and being aware of the pitfalls of the software. I think that adaptive learning has its merits since each student comes with a different background and prior knowledge.”

Instructor 3

“The Good:  An adaptive system that students could move through and focus on areas of weakness.

The Bad: Students not being able to keep up, weird point distributions in the system, confusing process for completing study plan and the quizzes.

The Ugly: The technical end.  Difficult to use study plan to grade (needs to organize more like homework if possible).  Many students got so far behind because of the adaptive system that they couldn’t keep up.  This would be better if the students had an open window of time for course completion, but 10 weeks is too quick for adaptive learning to be of value to the majority of the students.

Things I would change in future: I would have allowed one attempt on the quiz (to populate the study plan) earlier so that students could move quicker.  I would have also gone through and tried to use the system more to see where the majority of students were behind; I just couldn’t seem to find the time to do that and cover new content.“

Summary

• The adaptive classes scored better overall on the quizzes than the traditional courses
• Technical issues plagued all participants
  • Need to meet with Pearson to go over the issues
  • One free quiz without study plan seemed to work the best—quiz attempts 2,3 require study plan mastery
• Further implementation
  • Recommend moving all developmental courses to adaptive
• However…
  • Technical issues need to be addressed
  • Master courses need to be developed with coherent grading strategies
  • Training for all faculty that will use the courses