Artificial intelligence systems may be good at generating text, recognizing images, and even solving basic math problems—but when it comes to advanced mathematical reasoning, they are hitting a wall.

For all of the recent strides we’ve made in the math world—like a supercomputer finally solving the Sum of Three Cubes problem that puzzled mathematicians for 65 years—we’re forever crunching ...