*********************************************************************************
I have been monitoring the reaction to the Reuters story
over the last two weeks since it was published (March 28, 2016). I think it is too early to say what the
ramifications of this story will be, but I will try to give some reactions by educators
and then some predictions of how things may or may not change on the test and
on the part of colleges and universities.
Before doing so, however, in the interest of full
disclosure, I need to mention I talked several times over the past year with
two of the authors of the Reuters articles about topics related to students and
education in China. In addition, I raised the issue of testing in China in an
article I wrote last summer for NACAC’s Journal of Admission.
The
Reaction From Educators
While there has not been universal consensus among the
educators who have posted remarks about these stories on closed groups across
the web, many have placed blame for the problems with the SAT on squarely on the
College Board and its policy of recycling old questions on subsequent exams.
They argue that in an electronic age all old questions will be recorded and
given as study aids by those companies that gather all such information within
minutes after each administration of the test.
The College Board has been aware for years that old questions were being
gathered by these companies in China (and other places too). Given that test security has been compromised
over many years, The College Board responded
The Reaction From The College
Board
Shortly after the Reuters’ stories were posted, The College
Board responded with a “Note”:
The College Board is not perfect; we did our
best with the information we had at the time. We can and should do more. In
fact, we have recently taken more aggressive actions to secure the exam
internationally, including canceling an administration in China. We continue to
balance widening access with protecting the exam. We recently refused entry to
the first administration of the redesigned SAT to a number of high-risk registrants
— many of whom make a living violating security protocols. In the future, we
will need to do even more.
We have been
working with Reuters on these stories since summer 2015. During that period,
we’ve made College Board leadership available for multiple interviews and
shared more details about test security than we’ve provided any other news
organization. We’ve been transparent with Reuters in an effort to make sure its
readers better understand all that we do to protect the integrity of the SAT internationally
and ensure the validity of the test scores we report. In the spirit of full
transparency, we are sharing here the detailed letter we sent to Reuters
with the facts of our decision-making.
Even as we recognize where we need to do more,
we are so proud of the new SAT and its impact on students. This is a time of
remarkable, positive change for our century-old organization. In early March,
more than 460,000 students took the redesigned SAT, which better reflects what
students are learning in the classroom and connects them with distinct
opportunities such as free practice tools and college application fee waivers.
The “Note”
admits that there have been problems and that their are efforts to address some
of the security issues related to test administration. What the Note does not
address, and what has many educators worried, is that the College Board does
not directly address the use of old questions on new versions of the test.
What
This Means For Future Test Takers And For Colleges And Universities
One
reason the College Board did not address this question may have to do with the
fact that the most recent testing, in March, was the roll out of the new and
updated SAT. The first test administration did have higher security and anyone
not of college age was prevented from taking it (whether than can continue to
do this may be something courts will decide). The effort to keep out
professionals who are part of test preparation companies will not, however,
keep these companies from gathering all the questions on the test. Some test
prep companies get students to memorize questions and once the test is over all
the questions are gathered and are then used to prepare the next set of test
takers.
What
is important about the Note is not something that is included; rather, it is
what is not addressed that is most telling. There is nothing in the Note that would
lead me to believe that old questions will no longer get recycled on subsequent
tests. As long as old questions are recycled there will be an unfair advantage
to those who have had prior access to these questions.
To
come up with questions that allow The College Board to distribute scores across
a wide range of student ability keeps many psychometricians employed by the
Educational Testing Service (ETS). For those not familiar with the test design,
the SAT yields responses such that average scores for students are approximately
500 out of 800 on each section of the test. For the test to produce these
results each year questions have to be designed so that there is a reliable bell
curve on responses by students. There have to be, to put it simply, some
questions that are easy and others that are hard. For example, there are some questions
that the vast majority of test takers will get right and there are some
questions that are designed for almost all students to get wrong. Test
designers create a test in which the range of responses always leads to a mean
of close to 500. This requires test questions to be vetted before they are used
as a part of the official test. The students taking the official SAT are not
told which of the sections of the test is experimental and therefore are not
scored for the students taking that particular SAT on that particular date. The
experimental sections provide the data needed on the test questions. Once the psychometricians
have measured the experimental questions in terms of how many will answer the
question correctly, these questions can then be used as part of a “real” test
given at some future date. Those companies and individuals who have gathered
all the questions from the SAT on a particular date including the experimental
section will then give students who have signed up for test prep access to
these questions when they sign up for their test prep services. The Reuters stories supports what I have just
written. It also raises the question of whether students’ SAT scores should be
used for admission purposes.
Colleges
and Universities need to trust the SAT. If there is strong evidence that the
test security has been so compromised that scores may not be accurate for many individual
students, then the whole reason for requiring standardized tests comes into
question. It is almost impossible, however, to determine which student has
gained an unfair advantage by having access to old questions prior to the test
date the student choose to sit.
If getting
access to questions that have appeared on previous tests this is “cheating”,
then it is a far different kind than bringing in answers hidden in a calculator
or some other device. The “cheating” that occurs when old questions are recycled
occurs before the actual test is administered and therefore cannot be observed
or determined by proctors or anyone else. It is, to sum up, almost impossible
to prove if a student has benefitted from having had prior access. And this is
when things get ethically murky. Some admission readers assume, at the outset,
that certain students’ scores (those from China mostly) are not accurate
because they have had access to questions. To assume this, however, is not the
way we in the US say things should be done to individuals. Students should be innocent
until proven guilty. In addition, a reader who assumes a student had an unfair
advantage should permit the student to mount a defense. Students do not even
know that their scores are being, if not dismissed, then at least looked upon
with suspicion by admission officers. I have had a student write about his getting discriminated against for just these reasons.
The SAT
test does not simply give a score of an individual student; in addition, it
demonstrates where the student stands in percentiles compared to all other test
takers. Without a reliable score distribution the test would not work well for
comparing students, not only on an individual test but over multiple tests a
student might take. The SAT is also used by ranking businesses such as the US
News to measure the selectivity and reputation of individual universities. The
test scores must bear comparison to the test scores over previous and
subsequent years for them to be useful. For example, if the average score of
students who enroll at a particular college increases by 20 points over 5 years
the school’s reputation will go up. This is only true, however, it the test scores
can be usefully compared. In addition secondary schools as well as colleges are
often measured by the average SAT scores of their students. The use of SAT
scores to measure the educational quality of secondary school student will
increase in the near future because the new version of the SAT is tied to the
Common Core. As a result, SAT scores are now mandatory for all students in some
States that use the Common Core. If, however, the scores are be unreliable then
parents and educators who are anti-testing may mount lawsuits to halt the use
of these tests.
I
have been outlining why it should be in the best interest of the College Board
to discontinue the use of old questions on the SAT because I wanted to build a
case that would prove, if not beyond a reasonable doubt, then at least one worthy
of vigorous discussion. Now however, I need to suggest why the use of old
questions will not change unless the leadership of the College Board feels they
must build all new tests for each future date due to pressure from colleges and
the public at large, (and, in a less likely scenario, by the courts.)
The
College Board is well aware of the security issues and acknowledges them in
their Note, but they are not in a position to create new tests for each
subsequent administration. The reason that old questions may (and, in my
opinion, almost certainly will) continue to appear is a matter of time and
cost. The process of creating new questions that fit with the need to create
score outcomes that match closely with previous tests, the need to use
experimental questions to assure the questions perform in the way they need to
in the aggregate, and the need to do this over multiple test dates each year
is, as it stands now, an overwhelming task. While the College Board can tighten
security in ways that will prevent conventional cheating the issue of using
prior questions still remains unanswered.
The
SAT was designed long before the Internet. Test questions could be given across
time zones and countries with little fear of questions being forwarded or
posted. Now, however, the questions are
sent around the globe in minutes. What should the College Board do? They have
already created a new test in which some of the things students are asked to do
are not easily subject to memorized answers. This is particularly true of the
new writing section. From what I can tell the new section is a huge improvement
over its predecessor. It ask student to write based on marshaling evidence--Critical
thinking rather than memorization. It seems paradoxical then that most colleges
and universities have made the writing section optional for students. The collages
and universities do not want to discourage students who do not want to write an
essay from applying to their schools. The section that is least susceptible to
information sharing is the one that will be used least. I would hope that
colleges and universities would strongly encourage students to take the writing
section at least in places where students are thought to be given unfair advantages
due to the sharing of old questions. I would go so far as to encourage schools
get faculty to read the essay from the SAT as they are academic in nature and
may help to predict academic success as well or better than the multiple choice
questions. At the very least studies should be funded to look at how well the
SAT essay predicts academic success. I do not expect schools will follow this
proposal.
For
the last several years the number of colleges and universities that have gone
test optional has increased significantly. There are schools all over the US,
many of them quite well known that no longer require applicants to submit
SAT/ACT scores. I have written about this phenomenon before so I will not
rehearse it all here. What I can predict, however, is that many more schools
will go test optional in the coming years. First of all it increases the number
of students who will apply to schools. This helps schools look more selective when
making offers of admission and this in turn could help the rankings of the
schools. More importantly, the schools may have less trust in the scores they
receive and will, therefore, not put as much weight on them. For groups like Fairest
and others who think the standardized test are pernicious this will comes as welcome
news. For those who think that standardized tests, especially at the ends of
the spectrum (very high testers and very low testers) are good predictors of academic
success will be disappointed. Those who want to quantify students will also not
be happy. Given that the new SAT is ties to the Common Core, those who support
the Core will want the tests to remain a necessary part of evaluating schools
ad students.
At
this point it may sound like I think the SAT is in big trouble and that its
future is in jeopardy. While this may be true for some of the reasons I‘ve
listed and for some others as well. I do not think the issue with old questions
being recycled will be the reason for its fall from being a common part of the
admission process. In order for students to gain an advantage on the SAT by memorizing
questions the students have to be incredibly focused and willing to devote many
hours to the task. The answers student have require knowing the full question
and the answer from many hundreds if not thousands of old questions. In other
words, only incredibly motivated students can hope to get a significant
advantage form memorizing old questions. While there is now a new SAT and the
questions to memorize may not be as large as in the past, this will change
within a year when at least 7 tests will be administered per year.
It
should come as no surprise that the places where the issue of test prep and access
to prior questions are in Asia. The rote memorization of materials has long has
been a part of the educations system. Students spend untold hours memorizing
materials for their own national exams. What many may not realize then is that
these students are in fact cramming for the tests by memorizing the tests. This
in and of itself requires a great deal of time and effort. The other piece of
information about many of these students is that they are, in many cases bright
students. For example, many students in China who take the SAT could score well
above 2000 on the old SAT and above 1400 on the new test. These score while
near the top percentages of all testers are low for China. In recent years
student who are accepted to top schools in the US typical have 800 math scores
and well above 700 on both the critical reading and writing sections of the
SAT. There are hundreds of student scoring very well on the Sat and it is not
only because they have access to old questions; rather, it is that they have
studies hard to for tests all of their lives and are prepared to do well on the
test anyway.
The competition for getting into top US schools –which admit only
small numbers of applicants from China and Korea—is so fierce that student are
willing to devote tremendous effort to improve their scores. So far as I have
seen there are not many students in the US who are ready to devote their
summers to attending SAT camps that require 12 or more hours of SAT prep per
day. Some students from Asia see that
they need nearly perfect scores to be admitted to top schools and therefore age
willing to do this. It may be that there are others around the world and in the
US who are willing to put in the effort to memorize everything on tests, but
the overall percentage will be small. Will this small percentage undermine the
credibility of the whole tests? Not likely. Some admission officers have
already said that they look at scores from student in some places with a jaded
eye. In other words a student with a great score in China will not be looked at
the same as a student with a great score from Iowa. It is unlikely however that
most schools will not expect the majority of their students who will be offered
admission to submit scores and that most will assume the scores are accurate.
If I
were a VC and I was looking for a new investment opportunity that has the
potential to solve the problems with test security, I would look around for
someone to create a test like the PISA exam or even the Gaokao which would be given
once a year. The test would, however, not have a mean score like the SAT/ACT. In
other words, the test results would not be compared to other version of the
test given in other years. The test would be used t show how students did on a
particular test. This test could be used as a substitute for the SAT or in
addition to the SAT. I realize I am opening myself to a deluge of critical
comments as many educators feel students are living in a standardized test culture
and adding one more would only make things worse. It may make things worse for
some but for those who need to prove that the results of their tests are
accurate it may help the students and the colleges too.
The
pragmatist in me believes that in the world of admission most students and schools will muddle on with the
tests that are out there and that the concern for test security will not
undermine the use of the tests so along as they are tied to secondary school
and college rankings. I have been wrong about a lot of things before so I will
see how this looks a few years from now.
No comments:
Post a Comment