We have been exploring the use of the Knewton Adaptive Learning engine built into Pearson’s MyMathLab. We began with a limited study during the summer of 2016 with a trial in 4 developmental math courses. The results from the trial courses using Knewton were compared to sections of the same courses in which the adaptive engine was not used. Before continuing, you may be wondering what is this adaptive learning?

According to “What Is Adaptive Learning,” a post from David Kunz in The Knewton Blog:

Adaptive learning makes content dynamic and interactive, placing the student at the center of his or her individual learning experience. The platform monitors how the student interacts with the system and learns, leveraging the enormous quantities of data generated by a student’s online interactions with ordinary (textbook-like) and extraordinary (game- and social-media-like) content, with teachers and peers, and with the system itself. It assesses not only what a student knows now, but also determines what activities and interactions, developed by which providers, delivered in what sequence and medium, most greatly increase the possibility of that student’s academic success.

Differentiated instruction lets students focus on the concepts that they are struggling with, rather than bogging them down with busywork on topics on which they have already demonstrated understanding. In addition to supporting students who are falling behind with targeted remediation, It also gives advanced students the opportunity to move forward at their own pace.

Knewton and MyMathLab

One of the most time-consuming aspects, aside from creating the engine itself, is mapping the pathway through the content, so that prerequisite knowledge is identified for each concept. Each objective in a MyMathLab section is mapped, through a rigorous process, to one or more prerequisite objectives. If a student masters a prerequisite objective but later forgets the concept when it is needed for mastery of a later objective, the Knewton engine, based on this mapping, will direct the student back for additional study on this previous objective. A simple mapping may look something like this:

Summer 2016 Trial

Four courses in the Summer First Year Academic Success program were implemented using the Knewton engine. These were all developmental math courses and were offered in a compressed 4 or 5 week program. These were compared against regular sections of the same courses. The courses were structured as follows:

• All use MyMathLab
• Traditional Course
  • Lectures (either online or F2F)
  • MML Homework—60% score required to take corresponding quiz
  • MML Quizzes—three attempts possible—highest score
  • Midterm, Final
• Knewton Adaptive Courses
  • Lectures (either online or F2F)
  • Adaptive Study plan—Mastery of x objectives required to take quiz
  • MML Quizzes—three attempts possible—highest score
  • Midterm, Final

Setting up the Study Plan

There was a fair amount of work on the part of the instructors to initially set up the study plan, and training was offered to each of them to assist in the development. Essentially, the process is as follows:

• Set coverage to objective level
• Enable Study Plan and set Mastery standards
• Set question coverage and scoring
• Set prerequisites
• Define study plan
• Review Settings

We had to first identify which sections and which objectives would be covered:

Next we defined the Mastery Standards. Note that the mastery standards default to 100%. This is not the recommended approach as students struggle to achieve 100% mastery. A lot of this is due to the unforgiving nature of the answer format in MyMathLab.

Note that we also define the number of questions and level of difficulty for the Quiz Me, which is used to determine if the student masters that particular objective.

Next, we select the questions from each section that will determine the pool from which both the Quiz Me and practice problems will be drawn. Since the courses were taught using the traditional MyMathLab homework and quizzes, we selected the problems that were already used in the homework, section quizzes, and tests.

Each section has an associated quiz (separate from the Quiz Me). In the traditional sections, the student has to get a 60% on the section homework in order to be able to take the quiz. With the adaptive study plan, the homework was replaced by the study plan, and the student is then required to demonstrate mastery of the number of objectives specified for that section.

---

This is indicated to the student when they attempt to take a particular quiz:

The student view

When a student views the study plan for a particular section, they are presented for each objective with an opportunity to practice problems for that objective, and/or take the Quiz Me to demonstrate mastery of the objective. In some cases, as shown below, publisher video is available for each objective.

As the student passes each Quiz Me, the objective moves below the line to indicate objectives mastered. In this case, 3 of the 4 objectives have been mastered and the student is ready to take the quiz for the section.

If a student fails to master the objective, the Quiz Me is grayed out and they must complete at least one practice problem to be able to take the Quiz Me again. Note that traditional MyMathLab help is available in the practice problems, but no learning aids are available in the Quiz Me.

Preliminary Results

The summer trial was extremely limited, both in terms of the number of courses and the number of students. It was further confounded by differences in session length as well as the type of student. In the First Year program, we typically see entering freshmen, where this is the first university level course they have encountered.

• Aligned to same instructor—regular vs adaptive
• Very small sample—for summer courses 12-20 students
• Difference in session length
  • 10 weeks regular sessions
  • 4 weeks Summer I
  • 5 weeks Summer II
• Difference in types of students
  • FYAS
  • Regular session

Overall, our results tracked what we have seen in other studies of a similar nature, that is, the student scores were better and they spent less time on task overall. The time on task is reflected in the fact that if the students already understood the material, they could prove mastery by taking the Quiz Me—without having to do homework problems just to be able to take the quiz.

This was a fully online course, in both the adaptive and regular course.

As noted, time on task was significantly less than the traditional course. In this particular course, one student finished the entire course in 8 days with a score of 98%!

Similar results were seen for the other courses:

 

One of the instructors commented on the student performance:

“Comparing results of the same class (MAT 100) from Autumn 2015 (with homework assignments) and Summer 2016 (with study plan) I came to conclusion that students perform better using the study plan, because they can work on their weakest part of the sections more in order to master them. Also at the same time they don’t need to spend a lot of time on parts of the sections where they are already familiar with the material.”

Student comments were likewise favorable:

• Hi and thanks! I was nervous at first to take an online math class, especially since it’s my weakest subject but the study plan and quiz me has been helping me a lot! I find it really simple to use and it’s very helpful with the subject being taught.
• I definitely do find them helpful, at first I tried to go right into taking some of the quizzes without practicing and I was unsuccessful, so yes I find them helpful to get the juices flowing.
• I have been working hard on MyMathLab and taken lots of notes. I find that watching the video, taking the practice and quiz me, in that sequence, really helps me learn the material and prepare me for the real quiz.

Next steps

The initial trial was limited in scope. During the Winter of 2016–2017, we will be implementing the same trial in 4 courses with larger enrollments and traditional students. We are also including Math 130, the pre-calculus course. Other disciplines, such as Environmental Science, Chemistry, and Psychology are investigating the use of adaptive technologies built into their publisher materials.

Watch this space for the results coming this Spring.