This story was produced by The Nevada Independent , a nonprofit news organization covering politics and public policy in Nevada, and The Hechinger Report, a nonprofit, independent news organization focused on inequality and innovation in education. Sign up for the Hechinger newsletter and The Nevada Independent newsletter.

Sal Khan made a radical decision last summer. It was time, thought the ed tech pioneer, to bring his nonprofit online education empire to brick-and-mortar schools — lots of them.

To that end, his company built a new software tool designed for classroom use and piloted it in five school districts — including the massive Clark County School District serving Las Vegas and its suburbs — in the 2019-20 school year. Although Khan Academy was one of the first online learning organizations to promote the idea that kids could learn at home at their own pace, Khan denied the suggestion that working with traditional schools was a significant change in direction, instead calling it “a natural evolution of our work.”

If it went well, Khan Academy would prove, on a large scale, that teachers are critical to the successful use of technology in learning. And it would provide a key to ensuring more children become proficient in math, a stubborn problem in education nationally. But those goals will have to wait. The experiment has now been largely put on hold by the pandemic, but the questions it raises, both for technologists and for schools, illuminate some of the ongoing obstacles to technological innovation in schools.

Khan’s theory is that a big part of the reason kids fall behind in math is because they’ve missed a key skill at some point.

“Let’s say you’re learning negative numbers, but it has a decimal or fraction in it,” Khan said. If you don’t understand fractions, you’ll be too confused to grasp negative numbers, so you’ll tune out, he argued. “But,” he said, “if you give students opportunities to work at their true learning edge and fill in gaps without any shame, any stigma, that encourages them.”

The idea for the new program was to organize Khan Academy’s online library of learning videos and problem sets into a game-like software program to help kids practice skills they need to master. To ensure they’re fed the right practice problems, the tool pulls in their results from a test known as Measures of Academic Progress, or MAP, which measures discrete academic skills. (The developers of MAP, a nonprofit called Northwest Evaluation Association, or NWEA, worked with Khan to align their practice material with the skills measured by the test.) The version of the tool launched this fall focused on late elementary and middle school math, with new topics planned for the future.

In January, before any test results had come in, Khan was cautious about overstating the likely success of the new program.

"It’s not like you can just drop the tool in and the flowers will bloom and the birds will chirp," he said.

But based on initial data from a much smaller pilot in Long Beach, California, in 2017-18 and another in 33 tiny Idaho districts in 2013-14, he was optimistic. Studies of those efforts found that students who spent just 30 minutes per week on Khan Academy under the supervision of teachers significantly improved their performance on statewide standardized tests. Still, the nonprofit had never attempted anything on the scale of this school year’s five-district pilot.

Clark County, with more than 320,000 students, was the biggest district, with many times more students than Khan’s previous school-based efforts.

Khan provided the first year of teacher training, tech support and use of the new tool for grades 3 through 8 at no charge. The MAP tests, however, weren’t free: Clark County paid about $1.9 million to NWEA in 2019-20 for students in kindergarten through eighth grade to take the assessments. The original plan called for the district to pay Khan Academy for training and tech support next year. 

In January, Clark County Superintendent Jesús Jara said he saw the tool as a potential game changer for his sprawling school district, which battles large class sizes, a chronic teacher shortage, student transiency, a 70 percent poverty rate and dismal math achievement scores (just 24 percent of Clark County eighth graders were proficient in math in 2019, according to a national assessment).

Given all the hurdles, it seemed that if the tool could help students and teachers here, it could help them anywhere. Tara Powell, the principal of B. Mahlon Brown Academy of International Studies, a Clark County middle school, embraced the digital program from the beginning but knew its success hinged on teachers. 

“You could pop a kid on a computer and then get nothing out of it,” Powell said. Teachers are the key to the successful introduction of a new technology, she said. “It’s that relationship. It’s that prompting. It’s that coaching that the teacher has to put in and is willing to put in.”

Related: The overlooked power of Zuckerberg-backed learning program lies offline

In early February, things appeared to be going well. Across the five pilot districts, 70 percent of participating students had created an account and 30 percent were using the program, according to Khan’s data for that month.

Then the pandemic struck. On March 16, students went from attending schools with marked disparities to staying in homes with even more dramatic resource gaps. During the last week of April, more than 257,000 students had engaged in digital distance learning and had two-way communication with their teachers. Another 8,000 had a documented exemption, such as no internet access or an inability to leave home to pick up academic paper packets. By the time the school year ended, 4,370 students had not been reached at all during the coronavirus-related closure. District officials said they don’t know exactly how many children lack internet access. 

“This has amplified the inequities in our community for our children,” Jara said of the closures shortly after they began. But the district is trying to address the problem: It distributed nearly 111,500 Chromebooks to students for home use during the closure and deployed special Wi-Fi-equipped buses to various neighborhoods.

While using the Khan tool, which remains accessible online, could help some students keep up, it’s hardly assured that most will do so. Roughly 38,000 students logged on during the week before Nevada schools shut down. A month into the closure, the number of students logging into MAP Accelerator dropped to 21,423.

Students with teachers who used the program regularly in class could be more likely to use the tool at home. That has been true at Brown, one of the two middle schools The Nevada Independent and The Hechinger Report have followed since the pilot program started last fall.

Melinda Smith, a sixth-grade math teacher at Brown, had been using the software as a warm-up activity at the beginning of each of her 80-minute math classes. She especially appreciated the data the tool provided which showed how well students understood various math skills. She said it helped her realize when a concept needed to be taught again, either to the whole class or to small groups.

“At a quick glance, I can see that he’s better in number sense and number systems,” Smith said, browsing one student’s data set. She said she used the information to encourage kids to keep working on trickier skills.

Smith liked tracking student progress in real time. On her computer, she could watch students advance through skill sets as they worked during class time, and, if they were zoned out, she could see that, too. The software only counts active usage, not idle minutes.

One morning in early January, Brown sixth grader Gregg Whitney, 12, sat at a computer station in his school’s library steadily solving math problems with the help of Khan’s video tutorials. Correct answers yielded virtual confetti. As he accumulated points and “leveled up,” he also earned increasingly complex avatars. That morning, his screen featured an aquatic dragon avatar. Math has not always been Gregg’s best subject, but this was different.

“They’re putting the fun into math,” he said.

Ideally, the platform can motivate students in a way traditional letter grades cannot. Like runners vying for that personal record, students can aim to outperform their past selves. The hope is that as students master a new skill and go on to the next level, they will learn that hard work can lead to measurable success, knowledge that could both help lift struggling students and push high-achieving students.

But not all teachers were as enthusiastic about the program as those at Brown. And student interest has appeared to follow that of their teachers. Interviews with more than a dozen teachers revealed mixed levels of support. Some were excited to use the tool in their classrooms, while others expressed skepticism or mild interest.

The day after Gregg pointed out the detailed awesomeness of his aquatic dragon, Jahein Flanagan, 13, sat in a classroom at Ed Von Tobel Middle School with a school-issued laptop open before him.

His teacher, Leila Cryer, a long-term substitute, had asked him and his classmates, all of whom struggled to keep up in math, to practice fractions on the Khan tool while using the previous day’s class notes as a guide. (Teachers can assign specific topics if they prefer the whole class practice the same skill.)

Jahein considered the question Khan had fed him: “What does the 6 in 2/6 represent?” He picked an answer at random, which, he said, was how he usually approached the program.

He didn’t use the digital scratch pad to think through an answer and he said he hadn’t previously noticed the link to the video tutorial included with every question. When he clicked on it, the video on the parts of a fraction played without sound and he had no headphones. He closed it out, appearing unimpressed, and read the question once more. Again, he picked the wrong answer. His third pick was correct, but he said it was still a guess.

Related: How to reach students without internet access at home? Schools get creative

One reason for the big difference in how the two schools used the tool is that while some training efforts were district-wide, individual schools had to request much of the offered support. By early March, 1,000 school district staff members, including principals, teachers and math specialists, had signed up for training sessions offered by Khan. And 55 schools had requested some form of support, which could include anything from a webinar to a site visit. 

After the shutdown, Brown students still spent an average of 14 minutes per week using Khan, while Von Tobel students averaged one minute per week, according to district data from the last week of April.

Teachers’ backgrounds and professional training could also affect how the tool is used. Roughly 750 licensed positions remained unfilled when students entered classrooms in August and many schools in the district had to hire long-term substitutes who had less training (if any) in teaching math. Von Tobel was short two full-time math teachers in its four-person seventh grade math department in January, while Brown had a full slate of qualified math teachers.

Brown, located in the Las Vegas suburb of Henderson, also has fewer students living in poverty, though not by a wide margin. Three quarters of Brown’s students qualify for free- or reduced-price lunch, a federal measure of student poverty. At Von Tobel, in northeast Las Vegas, 100 percent of students qualify for meal assistance.

Related: Schools on a screen: One tech-savvy school district has lessons for others going online

But it’s not clear whether these issues make a difference. As of January, 56 percent of students at Brown demonstrated typical growth during the six months between their fall and winter MAP assessments. A very similar 53 percent of students at Von Tobel did the same.

For now, with the pandemic still keeping school closed and spring assessments cancelled, it is impossible to answer the original question of whether integrating Khan Academy into public schools will improve learning.

Initially, Khan Academy intended to add a language arts tool in fall 2020 and begin to charge for training and tech support. Khan said only that was still being worked out. And Jara said paying for the tutoring tool remains a priority, but with the normally bustling and neon-lit Las Vegas Strip a ghost town of padlocked casinos for more than two months, the state lost its main economic engine. And with all state agencies, including the Nevada Department of Education, put on notice for cuts, it’s unclear where the money for an experimental math improvement tool will come from.

But Jara wasn’t ready to quit.

“We have to look at ways to find the resources,” he said in mid-May when asked if he would pay for Khan Academy’s tool in the 2020-21 school year. “We have to do much better for our students.”

That’s going to be difficult, under the circumstances. But assuming the effort continues, The Nevada Independent and The Hechinger Report will continue to follow the story.