Abstract: AnimalWatch is a mathematics tutor with
enhanced adaptive feedback precisely tailored
to girls’ instructional needs. Three evaluation studies with fifth grade
students support our
hypothesis that adaptive feedback is beneficial to girls’ math confidence. We
have also had high
levels of teacher participation including classroom activities, after-school
presentations and
summer workshops . This paper describes features of the tutor, evaluation
studies, work with
classroom teachers and dissemination activities.
1. Increasing Girls’ Self-Confidence in Mathematics
A major factor contributing to women’s lower participation
in science, engineering and mathematics careers is
that, beginning in junior high school, many girls begin to doubt their ability
to learn mathematics (Beal, 1994; Beller
& Gafni, 1996). As a result, girls are typically more likely than boys to
progress no further in mathematics than
eighth grade algebra, and are subsequently under-prepared for many science and
math-intensive majors and
programs at the university and graduate school levels.
This project focuses on development of an intelligent
tutor to provide effective, confidence-enhancing
mathematics instruction for girls in late elementary school (Hart et al., 1999).
The guiding hypothesis is that
mathematics instruction in the United States can be completely transformed
through the use of instructional
techno logy so as to be much more appealing to girls, and in turn, enhance girls’
interest in and prepa ration for math
and science careers.
In contrast to most educational software which is designed
primarily with the male user in mind, AnimalWatch
provides supportive, adaptive and effective math instruction tailored to girls’
interests and needs. AnimalWatch
accomplishes this via its “student model,” a module that draws on cutting edge
techniques in artificial intelligence to
create a representation of each student’s math understanding. The student model
is continually updated as the
student works on math problems and is used to 1) generate appropriately
difficult problems and 2) to respond to the
student’s errors with customized help and feedback tailored to her or his needs.
AnimalWatch also engages girls’
interest in math by blending mathematics with environmental biology, the science
subject that is of highest interest to
female students.
AnimalWatch has been developed with the collaboration of
local schools to help students learn fractions,
decimals and percentages at a 5th-6th grade level. In contrast to common
drill -and-practice systems, intelligent
tutoring systems modify themselves to conform to the students’ learning styles.
Once the student demonstrates
mastery of whole numbers, the tutor presents simple fractional problems that
require increasingly challenging
application of the cognitive subtasks involved in solving the problems (e.g.
adding fractions with like denominators,
adding fractions with different denominators, etc.).
When a student has trouble solving a problem, AnimalWatch
initiates a tutoring interaction with customized
hints and guidance that helps the student work through the problem. Similar
problems involving the same subskills
are given until the student can successfully work the problems. AnimalWatch
maintains an accurate assessment of
the student’s strengths and weaknesses. Online
self-assessment surveys conducted as students work with
AnimalWatch have showed that the tutor generates a more accurate assessment of
each student’s abilities than the
students themselves (Beck et al., 1997).
Figure 1: Example of addition problem in AnimalWatch
AnimalWatch uses Artificial Intelligence techniques for
problem generation, hint selection and student
modeling (Stern et al., 1999). Multimedia is used judiciously to engage the
student by animating key concepts and
providing interactive manipulables based on those used by classroom teachers.
Figure 2: Example of a hint on a simple multiplication
problem
2. Features for Girls in AnimalWatch
When the student logs on, he or she enters an
environmental biology storyline in which math problems are
presented within distinct contexts that unfold as the narrative progresses.
Students can select an endangered species,
such as the Right Whale or the Giant Panda, which includes an initial story
context. The student is invited to join an
environmental monitoring team and engagesin activities to prepare for the trip.
For example, in the case of the Giant Panda, problems
involve research at the library about the Panda’s habitat,
reading about the birth of a new Panda at the San Diego Zoo, estimates of the
expenses associated with a trip to
China, and analyses of the rate of decline of the Panda population over time,
etc. The student model estimates when
the student is ready to move on to the next environment (e.g., whale-watching
vessel; mountain terrain trip, etc.).
Each math problem includes graphics tailored to the problem, e.g., a map of Cape
Cod bay showing the migration
route of the Right Whale for a problem in which students calculate the
fractional progress of a pod over the course of
a week’s travel. The third context involves a return to the research “base” and
preparation of a report about the
species’ status.
Hints and instruction screens appear when the student has
made a problem solving error (example shown in
Figure 2). Hints and adaptive feedback have been shown to be especially
important to girls, whereas boys retained
their confidence in math even when working with a drill and practice version of
the system that simply presented
problems and responded to student errors with the message, “Try again.” This
gender difference in response to
different types of feedback is consistent with the theoretical framework:
Detailed and immediate help provides a
critical role for lower-confidence students, many of whom are female, who are
quick to assume that they do not have
the ability to understand difficult concepts.
The intelligent student model is continually updated based
on the student’s ongoing performance. Math
problems in AnimalWatch are not “canned” or pre-stored. Rather, hundreds of
problem templates are used to
generate novel problems “on the fly.” AnimalWatch currently includes mathematics
operations that match those
included in most fifth grade classrooms: whole number operations (multi- digit
addition /subtraction,
multiplicaton/division); introduction to fractions; addition and subtraction of
like and unlike multi-digit fractions;
reduction /simplification; mixed numbers; introduction to proportions/ratios; and
interpretation of graphs, charts and
maps. AnimalWatch has been implemented in the Java computer programming language
inorder to be easily
disseminated via CD-ROM or the Internet and easily used by most elementary
schools.
3. Project Findings: Evaluation Studies with AnimalWatch
Results from three evaluation studies will be described.
The first study focused on collaborative learning, and,
consistent with students who worked individually, students who worked with a
partner showed significant increases
in math self concept (Berry, et al., submitted). The increase in math self
confidence was observed for girls working
with a male partner, as well as those working with another girl. In terms of
math confidence and objective problem
solving, these results indicate that girls do well with AnimalWatch even when
working with a male partner.
The effectiveness on girls’ and boys’ performance of
different types of adaptive feedback was measured in
June 1999 with three classes of fifth graders. Preliminary evidence showed that
highly adaptive feedback is
especially important to girls: when students worked with a version of
AnimalWatch with the adaptive feedback
“turned off,” girls were more negatively affected than boys. The goal was to
identify specific hints and help that are
most beneficial to girls, particularly those who are at different stages of
cognitive development.
In this study, students worked with AnimalWatch for three
sessions over the course of one week . AnimalWatch
was adjusted to compare responses to different types of hints, such as those
shown in Figure 4. When the student
made a mistake, the system first provided messages with relatively little
content, e.g., “Are you sure you are using
the correct operation?” Low interactive text hints were used at first because
nearly half of the errors entered are
corrected by students on the next try after a simple prompt. If the student kept
entering a wrong answer, the system
presented a hint to guide the student in the whole problem-solving process. At
this point, AnimalWatch selected
randomly either hints that were highly conceptual (see Figure 2) or hints that
were more procedural in nature (see
Figure 4). Hints also varied in the degree of interactivity required from the
student: hints that were highly interactive
were also highly structured and walked the student through the solution process
in incremental steps (see examples in
Figure 4). We hypothesized these would be more helpful to girls.
Figure 3: Change in Confidence of Girls and Boys
Data on math problem solving were automatically collected.
Hint effectiveness was assessed by comparing the
number of errors made on subsequent problems of the same type, the idea being
that if a particular hint is helpful
then the student should be able to to solve a similar problem with significantly
fewer errors.
Figure 4: Examples of hints differing in interactivity
To evaluate girls’ reactions to the different types of
hints, ratings from a survey were analyzed, in relation to
gender and cognitive development. In a pre-test session, students’ cognitive
developmental stage was assessed via a
computer presented battery of Piagetian reasoning problems (Arroyo et al.,
1999). Students also completed a survey
about their AnimalWatch experience, including questions in which they were asked
to rate the helpfulness of the
different types of hints that they saw. The results indicated that, not
surprisingly, both boys and girls of lower
cognitive development needed more hints to solve the problems. However, there
was a strong relation for girls
between their cognitive developmental stage and their views of how helpful the
different hints were. In addition,
hints that were highly interactive (i.e., structured) were rated by girls as
significantly more helpful than less
interactive hints, and were more effective (i.e., were followed by fewer errors
in subsequent problems), whereas
there was no relation for boys, see Figure 5. Overall, the results indicated
that not only is adaptive feedback
especially important for girls, certain specific types of feedback are preferred
by girls, whereas boys do not appear to
show such consistent preferences.
Several evaluation studies showed significant improvements
in attitudes towards math (confidence, value,
liking) after students worked with AnimalWatch (Beck et al., 1999). The goal was
to assess the effects of working
with AnimalWatch on girls’ math confidence. With regard to students’ confidence
in math, analyses of variance
comparing the pre- and post-test data from the Academic Attitudes Questionnaire
indicated that working with
AnimalWatch led to significant increases in students’ math self concept (Figure
3).
Figure 5: Gender differences in response to high and low
interactive hints
Analyses of the AnimalWatch survey revealed that students
generally rated their experience highly: Means
range from 3.78 to 4.85 (on a 5 point scale) for questions such as “Would you
like to use AnimalWatch again?,”
“When you made errors, did AnimalWatch give you enough help?,” “Do you think the
computer is a good way to
learn math?.” On the question, “Did you like working with AnimalWatch?,” girls
gave significantly higher ratings
than boys (mean 4.50 for girls, 4.05 for boys).
4. Dissemination
Classroom teachers and teachers-in-training were involved
in the design of AnimalWatch and in the evaluation
studies. Workshops for the participating fifth grade teachers focused on gender
equity in math and science, as did
guest lectures to psychology, education and teacher training programs and
three-hour workshops for student teachers.
AnimalWatch received high marks from the teachers, who
rated it highly on such issues as appropriateness of
math topics, sufficiency of help, ease of use, and fit to their curriculum. They
also felt that working with
AnimalWatch would help prepare girls for high stakes achievement tests such as
the PSAT or the MCAS (a new
state assessment test suite in Massachusetts), on which girls typically perform
less well in math than boys. All the
participating teachers volunteered their classrooms as sites for future
evaluation studies.
Teachers responded very positively to AnimalWatch ’s ease
of use and resilience (i.e., there is little that a
student can do to “mess up” the computer and thus require teacher intervention).
They were very pleased that it will
run on any platform.
5. Current Research Issues
Our primary goal in the project is to help girls tackle
increasingly challenging math learning while maintaining
their confidence in their math ability. The critical mechanism is AnimalWatch’s
ability to immediately provide
in dividualized adaptive feedback and to pace the difficulty of problems to avoid
discouraging the student with
repeated failures. In particular, we are now concerned with how rapidly
AnimalWatch should “push” girls through
the math curriculum. To address these issues, a mechanism that quickly and
automatically estimates the optimal
sequence and rate of problem presentation for each student is being evaluated.
Another goal is to add an additional
species, with background research and context construction. At the request of
many teachers, we will add a user
feature that would allow teachers to review the progress of individual students
after each session with AnimalWatch.
An artificial intelligence learning mechanism that quickly
and automatically estimates the optimal sequence and
rate of problem presentation for each student is being evaluated. However,
determining when a student is bogged
down in problem solving is actually quite difficult, technically. First, the
overall skill level of the student must be
accounted for. If two students are given identical problems, the more skilled
student will solve the problem more
quickly than the less skilled student. Second, there are considerable individual
differences in how quickly students
work. For example, some students navigate the keyboard more quickly than others,
some students have the
multiplication tables memorized (more often, boys) while others prefer to use
pencil and paper (more often, girls)
etc. Any of these factors can impact the time required to solve a problem.
Finally, the time required to solve a
problem is a noisy variable . That is, if a student is given similar problems, he
or she may take very different amounts
of time to solve them. A small mistake at the beginning of the problem solving
process can drastically impact the
time required to solve a problem. All these issues are being investigated.
