Testing in a Global Environment

By Jake Vogt, ATP Writer

What will be the role of the test publishing industry in the emerging global community? With increased globalization, researchers and policy makers throughout the world are becoming more interested in employing testing methodologies to understand similarities and differences in human behavior, to assist in contributing to social goals, and to resolve societal issues. In November 2012, a panel of testing leaders met at a conference at the University of Massachusetts, Amherst to discuss a variety of topics.  We followed up on one of those topics, International and Cross-Cultural Assessment, with some of the ATP members who participated in that panel.

“Pearson’s a global company,” said Kimberly O’Malley, Vice President of Research at Pearson Education. “We have services from preschool to high school, early learning, and professional certification. So the concept of international and cross-cultural testing, AND learning, is really a critical concept for our company. “

“We started testing in international locations in 2011,” said Craig Mills, Vice President of Examinations at the American Institute of CPAs. “We’re expanding bit by bit, and as we get familiar with international testing, we’ll go to more challenging areas where we face the need to change our infrastructure.”

“In the 1990s, we were primarily a US test being administered to US programs,” said Larry Rudner, Vice President of Research and Development at the  Graduate Management Admissions Council, noting that globalization has had a transformative effect on his organization’s testing practices. “Today, [our test-takers] are 56% non-U.S. citizens, and a large number of test-takers are taking it to go school domestically outside of the U.S.”

As technological advances make expansion into international and cross-cultural assessment more feasible and even necessary, the industry is confronted with new practical concerns arising from differences in cultural attitudes and realities. One of the major areas affected is test security, which is currently conceived according to underlying values that are not universal.

“There are also different attitudes toward ‘cheating’ around the world,” said Rudner, pointing out that not every culture holds that assessments must be taken by individuals without help. “Some countries view cooperation as the right way of doing things.” He added that in poorer countries, overseeing testing integrity becomes more challenging. “A lot of people in test centers are making minimum wage in very poor countries, so it really doesn’t take that much money to bribe a test center operator.”

Solving these practical issues means understanding and addressing cultural gaps, making them part of a larger ethical problem: how can we construct tests, and test procedures, that are fair to test takers across multiple backgrounds?

 “The big issue we worry about is equity for everybody,” Rudner said, adding that tests designed in the US and administered in other countries can “inadvertently offend somebody, or demonstrate US ethnocentricity.” Even when a test is well-designed in terms of objectively testing the correct knowledge and skill sets, a cultural difference between the test authors and the test taker can result in an unfair disadvantage. “A US citizen just reads over something and somebody else starts wandering away, whether it’s a reference to the FBI or George Washington or something like that. You’ve created something unfair and not consistent across the test-taking experience. It may not show up in the bias statistics, but it shows up in people’s attitudes.”

There is no concrete solution to the question of a test’s cross-cultural fairness, but technology can help efforts to standardize. O’Malley is enthusiastic about a concept called the Reading Maturity Metric, which is used to quanity the complexity of a text. “The Text Complexity Measure is not only a measure of the text’s difficulty, but a student’s ability to comprehend that text,” she said . “This underlying technology can be adapted very rapidly, so that we can begin to understand students’ performance not just on  a specific test on a specific location but across the world.

 It’s not just complexity and cultural subtleties that create problems, though. Even the communication of test requirements across language barriers can be problematic.

 “There’s too much reliance on literal translation,” Mills says. “There are a lot of really good translation companies around the world, but they rarely have to translate things that require the careful attention to nuance that assessments do.” Often, a translation will be correct in the sense that it conveys the literal meaning of the words used, but will completely fail to communicate the real purpose of a sentence. “One of my favorite examples is somebody translating ‘Out of sight, out of mind,’ and that came back as ‘Invisible, insane.’” The translation is literal, but “the concept isn’t there.”

O’Malley agrees that strict literal translation is insufficient, stressing “the richness of the [English] language” as an intangible but vital concern. “We can simplify the testing language so that students can understand it, but the question is, have we lost the richness of the language, and in doing so short-changed our students in the quality of the reading material?” She looks again to technology for possible solutions. “Pearson, along with many other companies, have the opportunity with technology to offer linguistic accommodations. When we’re testing on paper, we don’t have those opportunities, so I really think the challenge is getting students internationally on the computer.”

These challenges do not appear to be dissipating any time soon. International exchanges of tests are becoming more common, and interest in cross-cultural research continues to grow.

In addressing the issues in these efforts, Rudner suggests that the testing industry might benefit from a coordinated approach.  “I think an organization such as ATP may need to provide some guidance,” he said. “We’ve got all of these different testing companies that are trying to be around internationally, and they’re all reinventing the same thing. That’s quite inefficient.”

Mills agrees. “A weakness that we have right now is that most testing programs are trying to solve their issues of how to do business internationally by themselves,” he said, expressing hope for the establishment of “a common approach to other countries as we try to figure out how to test fairly and effectively there.”

“We need to understand that we’re not gonna change the world,” Mills added. “We’re gonna have to change testing instead.”