IN CASE YOU MISSED IT: CEO Panel Video and Summary

Assessment in a Covid-19 World

Excerpts from the Experts: A summary of the CEO panel discussion on COVID-19^’s impact on the testing industry hosted by the Association of Test Publishers, April 17, 2020

“Education is sacred. It’s really what changes the world.”  Those were the words of Law School Admission Council’s (“LSAC”) CEO Kellye Testy in a recent online webinar offered by the Association of Test Publishers, which brought together a group of top industry leaders to make the case for testing and assessment in a challenged, COVID-19 world.  

Testy’s words led to an eloquent dialogue on issues impacting the industry by a panel of leaders representing a diverse collection of testing organizations.  Testy was joined by  ACT CEO Marten Roorda, Curriculum Associates CEO Rob Waldron, Questionmark Founder John Kleeman and Business Development Vice President Alex Tong from China-based ATA - all of whom agreed that, despite critics who may be using the current epidemic to disparage the industry for their own agendas, it is testing that upholds the standards that make education sacred. And they also agreed, unequivocally, that testing, even in these pandemic times, levels the playing field when it comes to underserved students of all ages and abilities

“For so many individuals it (testing) allows them to show their potential…and if they are disadvantaged or remote or have accessibility issues, it gives them a level playing field so that everyone can show their potential. And it also allows organizations to select candidates on merit, not just on who interviews well,  testing lets you genuinely measure skills and knowledge and you can reduce human error and bias,” Kleeman noted.

Kleeman went on to emphasize the importance of testing in a COVID-19 world in which testing, certification and licensure play a role that “ensures the people that have responsibility for life and limb are properly qualified.” 

ACT CEO Marten Roorda, feeling especially besieged by government edicts closing schools and cancelling standardized testing across the U.S., pointed to ACT’s active role in recent years in moving away from summative, high-stakes testing and more towards integrated learning.

“ACT has been looking at (integrated learning) solutions for the past four years,” Roorda said, noting that ACT has been lessening the emphasis on high stakes testing and increasing the potential of e-learning, with measurement throughout a school career taking the place of summative assessments. Roorda said he sees an opportunity in the current situation to push the testing industry in new directions with less emphasis, he said, “on physical locations and high stakes summative assessment.”

Testy weighed in from the law school admissions perspective, noting that the Council too has been integrating e-learning and assessment in its new LSAT prep product, LawHub . At the same time, she defended the continued need for high stakes testing in certain critical fields which she said, “is still right for our world and promotes the right values.”

Speaking from the Asian perspective, ATA’s Vice President Alex Tong expressed his dismay at the attacks on testing he has observed in the U.S. from such initiatives as opt-out, and the waivers now being initiated for many standardized tests. “In our culture that would just not happen,” he said, “Asians are very supportive when it comes to testing. Testing is something we embrace.”  And because of that, he said the current pandemic environment “has been painful…all tests have been cancelled – everyone is asking, what can we do?”

In response, Tong noted, ATA quickly rolled out its three pillar products which include solutions for online testing, online learning and online interviews – and made all three free of charge to the public. As for high stakes testing, the company is still hoping for a green light in July so that students can take the University entrance exam, which Alex noted, continues to be an un-contested pillar of education in China.

So, as all leaders defended the need for testing in a COVID-19 world, the conversation quickly turned to the topic of how to do it in a remote digital environment, with questions focusing on remote proctoring and cyber-security issues.

Waldron, from Curriculum Associates who moderated the panel, defined remote proctoring as two options: Record-and-review or live-on-line. Live-on-line, he noted, is most often used for high-stakes testing. “A remote proctor watches you through a webcam – they check that no one else is in the room, they check that you are not using a cell phone, and they observe just as they would in a physical test center.”

“Record-and-review doesn’t have a live proctor involved – the webcam records what you are doing taking the test and somebody can look at that (the recording) later on,” Waldron explained. Many record-and-review proctoring solutions, he said, offer a “system that flags issues, such as if they see two faces on the screen.”

Kleeman noted that “remote proctoring has been around for ten years and has been widely used in IT Certification…but a lot of other sectors haven’t been looking at it before, but obviously they’re now having to look at it.”   And in response to cybersecurity, Kleeman’s company, Questionmark, rolled out a cybersecurity test for home workers which they are offering free- of-charge during the current crisis.

Testy reported that LSAC had already experimented with a digital LSAT in 2019 which made the Concil particularly nimble when it came to a quick pivot and roll out of its LSAT-Flex in response to the current crisis. And she admitted that remote proctoring was one of their biggest hurdles. “It’s not easy to take a test in which you depend heavily on validity, reliability and fairness and take it to another mode.” She said.  And one of her top concerns, she noted, was legal defensibility. “Any high-stakes (test) where there is remote proctoring will be subject to lawyers being lawyers.”

Kleeman agreed, noting that in a remote proctoring environment it is really important to be up-front on privacy. “People are worried if their taking tests at home then maybe you are doing something with the data and you’re going to use it for marketing purposes or other purposes, so really clear privacy policies are important,” he said. “You need to make it very clear that any data you are collecting is just being used for the test purpose, you don’t want any ambiguity in your privacy policies.”

Kleeman, who is also Chair-elect of the Association of Test Publishers (ATP),  noted that the Association has recently offered a Privacy Bulletin focused on remote proctoring that is available free to charge on its website [www.testpublishers.org]. He urged organizations to download and follow the guidance in such resources. “I strongly encourage everybody to be up front and transparent…the testing industry needs to be absolutely clear that we are capturing data for the test’s purpose and not for any other purpose, and that it (the data) will be deleted after a period of time.”

Of course, any discussion on remote proctoring or e-learning quickly leads to the issue of  equity and accessibility.  Even the term “accessibility” has taken on new meaning in recent times. Where once it referred to test candidates with physical and cognitive challenges – it now encompasses access to hardware, software and the Internet. And it even drills down to bandwidth and firewall challenges.

“If you’re going to do remote (teaching or testing), how are you going to deal with the issue of equity? ” Waldron asked the panelists. He noted that his organization had done extensive research using zip codes and other demographic indicators. “We know, for example, that 33% of kids in poverty have no access to Internet at all, and another third have very limited access and don’t have quiet places where kids can test.”

Testy offered that LSAC has had a fee waiver program for several years that really tries to reach out to students from economically disadvantaged backgrounds.  She also noted that LSAC launched a digital program that provided tablets last year that enabled those without access to hardware. “In sum,” she said, “I’m glad we are focusing on equity and this crisis has made it apparent how important that is.”

Roorda noted that, “We (ACT) are well aware of the equity issues and trying to create alternatives…so if kids can’t find a space at home there may be space we can find for them nearby, like a local school. We offer testing in 25 states so we have locations we can use in libraries, or learning centers offered by Kaplan and others.”

Roorda also noted affordability is another dimension of accessibility. “Right now, we are offering the ACT free of charge to students who receive a free or reduced lunch.” He also reported that ACT is currently investigating options for free remote proctoring for disadvantaged students and also “at-home” accommodations for those who are physically and cognitively challenged.

On the topic of accommodation, Tong reported that in China an ADA – type law does not exist. Nevertheless, he said, ATA always strives to address the special needs of test takers, and he sees a silver lining in online testing as it relates to accommodation. “We have been using all the tools available such as text enlargement, speech recognition and many other accommodations. To us, as a vendor, we see that having it (testing) online is much easier to accommodate special needs than in a test center.”

On another positive note, Tong reported that “in Asia we have been trying to push the formative assessment instead of always pushing the students to sit for a very high stakes test which determines their life…but nobody listened…because the score was so important. But now we have an opportunity that is forcing people to see (the value) in online education with embedded assessment, badging, and everything else that goes along with it, and doesn’t require a high stakes exam at the end. So I see a light at the end of the tunnel that there is something positive that might come from the pandemic.”

That theme of positivity in a time of crisis resonated with all the panelists. Waldron paused the discussion briefly to give a charming anecdote about his personal experience years ago as a young bridegroom. “I was married on the rainiest day in the history of Rhode Island, “ Waldron recalled. “We were in a tent that seemed like it was about to fall down and the caterer came up to us and said to my wife, ‘here is what I have to say about rain and weddings – if you are worried and freaked out about it, all of your guests are going to be freaked out for you. And if you think it’s hysterical and funny, they’ll think it’s funny, and they are going to have the best time of their lives…everyone follows the bride.’ And when this crisis started,” said Waldron, “I was…all keyed up and my lovely wife said to me, ‘remember…you’re the bride.’ And I realized it would become self-fulfilling…your attitude about a problem becomes self-fulfilling.”

Waldron went on to talk about the silver lining of seeing technology being embraced by educators during the pandemic in ways he had never seen before. “Whenever we’d talk to teachers about technology it was a wrestling match, even though the technology was available. So one of the changes we are seeing is how much more comfortable teachers are becoming with remote learning.”

Tong added, (referring to the Association of Test Publishers’ decision to cancel its in-person conference program and turn it into a virtual event), that, “even our industry conference having to go online is good because of the inclusiveness. For Asians who have always attended the conference by traveling long distance and then suffering from jet lag, and also for people who would not otherwise have access to the conference…having it online will be more inclusive.”

As far as the challenges being presented to leaders, one of the overarching themes seemed to be how to keep testing from being devalued. One member of the audience, Chad Buckendahl, chimed in with a question directly related to that theme, noting that “testing in the certification space, and perhaps in education as well, is seeing an opportunity to call for waivers of eligibility for testing requirements. How do we keep from setting bad precedents?” he asked.

Testy responded, “We need to communicate the message that testing is so important and such an equalizer in so many ways…our industry needs to get that message out. There is a lot of pressure right now to go around standards. In the legal education context the American Bar Association seems to understand the need to hold firm. We need to make the case (for testing) so it is well understood.”

Roorda agreed and noted that, “(college) admissions testing assessment plays a role in offering opportunities to underserved students…so if we do away with it, then higher education will go only to privileged students and privileged families.”  He also pointed out that the need for testing will be even greater in the post-pandemic education environment that is going to have to deal with “serious gaps in education worldwide. Assessment will play an important role in identifying those gaps and informing educators.”

Waldron agreed, “When you have groups of students falling behind you need to assess their levels before you can personalize their learning on the right level. The biggest waste of time is giving instruction that they can’t handle just yet. Assessment will save time in getting them (students) back to grade level – it is a more efficient mechanism.”

All panelists agreed that though they mourned the loss of the in-person Innovations in Testing conference this year, they were looking forward to the Virtual event.

Roorda noted that he has been attending ATP conferences since 2004, and he looks forward to the Virtual Conference where he hopes to learn more about, “…innovation and the crossover between learning and testing…content generation, and especially navigation.” He added that some of the questions he hoped to have answered included management and efficacy of data.  “Online learning and measurement will create data, what do we do with that data? How do we use it to inform students and parents?”

Testy concluded with her thoughts about the upcoming Virtual Conference, “I always look forward to learning about emerging technologies, hopefully as we move online we will still be able to learn about some of the cutting edge work that is happening toward providing the tools we all need, to do what we all want to do, to further the assessment goals we’ve all talked about today.”

Click here for a recording of this discussion