A network of small public high schools in New York City is exploring whether artificial intelligence can change the way teachers receive feedback about their classroom instruction.

Urban Assembly, a network of 21 schools, is working with the American Institutes of Research to develop an AI-powered tool that can help instructional coaches analyze videos of teachers delivering lessons and offer feedback, according to network leaders.

Artificial intelligence is already transforming the classroom experience for many New York City students, who say chatbots like ChatGPT can help them understand difficult topics and speed up their research. But the technology has also sparked fierce pushback from some educators and officials worried about its potential to encourage cheating and spread misinformation and bias.

After initially banning ChatGPT on school devices over concerns about academic dishonesty, New York City’s Education Department pledged to teach students to use the technology responsibly, and plans to open an institute to study its applications in schools.

The use of AI in teacher coaching brings up similar questions. Proponents say it could save lots of time for instructional coaches and expand access to feedback that improves the quality of teaching. But some teachers said they still had questions about how accurately the technology can capture subtle classroom interactions, how useful its data will be, and whether it will be skewed by biases.

Judy Cappuccio, a math teacher and instructional coach at Urban Assembly Institute of Math and Science for Young Women in Downtown Brooklyn, said she’s “open” to the idea of assistance from the AI tool, but has a “healthy amount of skepticism.”

“I would like to see it in action. It would take me some verification at first to trust it,” she said.

Several schools in Urban Assembly’s network are already part of a pilot where educators record themselves teaching and analyze the videos in detail with instructional coaches to improve their practice — a practice Urban Assembly CEO David Adams likened to athletes reviewing game tape.

The problem, Adams said, is that it can take the instructional coaches hours to review a single video, limiting the scale of the program. That means teachers aren’t getting enough feedback, and they’re getting it less often than they should be, he said.

That’s where the new AI-powered tool comes in. At the end of a two-year rollout, project leaders from the American Institutes of Research hope it will be able to measure things like how often students and the teacher are talking, laughing, and yelling, according to a proposal researchers submitted to Urban Assembly.

The tool will initially roll out to the 21 schools in the Urban Assembly network, though Adams hopes to eventually expand its use. It will cost around $500,000 to develop, test, and implement over two years, according to the network.

The tool will also be able to use “natural language processing,” a branch of AI that seeks to understand the meaning of language, to evaluate how “positive,” “respectful,” or “insulting” the teacher’s language is.

Some of the details captured by the AI tool might seem small, but they can offer clues about the climate of a classroom that teachers can learn from, Adams said.

When kids and teachers are laughing together, for example, it can be a sign that they’re “in the same emotional space” and students are better equipped to absorb the lesson, Adams said.

Capturing and documenting those moments on video can help teachers “replicate and grow” them, added Kiri Soares, the principal of Urban Assembly Institute of Math and Science for Young Women, one of the schools planning to pilot the new tool.

The tool won’t replace the instructional coaches, but will save them time by pointing them to relevant sections of the video, producing audio transcripts, and quickly gathering data that would take humans hours to compile, Adams said.

Ultimately, the tool could enlarge the program and allow more teachers to benefit, Adams argued.

The tool won’t be used in an evaluative capacity and won’t be tied to performance reviews conducted by the school principal, he added. The program is meant to be supportive and highlight what teachers do well, not just where they need to improve, Adams said.

Project leaders propose using the tool to help schools expand an existing teacher feedback program called CLASS, which taps instructional coaches to evaluate educators on metrics ranging from academic content to their relationships with students, based on video recordings of their classroom lessons.

Using videos rather than live observations can give a more honest glimpse of the classroom, and gives teachers the chance to see themselves in action, proponents said.

Teachers interested in AI proposal but have questions

Liza Backman, a science teacher and instructional coach at Urban Assembly Institute of Math and Science for Young Women in downtown Brooklyn, said she could see the benefits of having the type of data the AI tool can quickly gather at her fingertips.

“I think it’s a tally that would be interesting,” she said.

Still, she cautioned, those data points are only useful if there’s an instructional coach to help make sense of what they mean within the context of the class.

“Some of the lessons, there would be no laughter because we were talking about a very serious topic,” she noted.

Backman also raised questions about what kinds of school environments would be featured in the videos used to train the AI, and whether any biases could be baked in as a result.

“If you feed it videos from primarily white schools, versus primarily Black and brown schools, how will it navigate names?” she asked.

Adams said the tool in development for Urban Assembly schools would be trained at other Urban Assembly schools with similar demographics.

There are other potential downsides.

The AI-powered tool may miss out on meaningful moments from a classroom video that don’t fit cleanly into one of the categories it’s meant to track — moments an instructional coach would’ve caught if they’d been watching, said Soares, the principal of Urban Assembly Institute.

But that’s a worthwhile tradeoff if she can expand the number of teachers participating in the program, she said.

“Yes, we might miss out on some of those moments,” she said. “But more people will get more things.”

Michael Elsen-Rooney is a reporter for Chalkbeat New York, covering NYC public schools. Contact Michael at melsen-rooney@chalkbeat.org.